Title: A graphon-signal analysis of graph neural networks
Venue: NeurIPS 2023
Accepted: yes
Keywords: graph neural network

graphon

generalization

stability

sampling

Szemerédi regularity lemma
Abstract: We present an approach for analyzing message passing graph neural networks (MPNNs) based on an extension of graphon analysis to a so called graphon-signal analysis. A MPNN is a function that takes a graph and a signal on the graph (a graph-signal) and returns some value. Since the input space of MPNNs is non-Euclidean, i.e., graphs can be of any size and topology, properties such as generalization are less well understood for MPNNs than for Euclidean neural networks. We claim that one important missing ingredient in past work is a meaningful notion of graph-signal similarity measure, that endows the space of inputs to MPNNs with a regular structure. We present such a similarity measure, called the graphon-signal cut distance, which makes the space of all graph-signals a dense subset of a compact metric space -- the graphon-signal space. Informally, two deterministic graph-signals are close in cut-distance if they ``look like'' they were sampled from the same random graph-signal model. Hence, our cut distance is a natural notion of graph-signal similarity, which allows comparing any pair of graph-signals of any size and topology. We prove that MPNNs are Lipschitz continuous functions over the graphon-signal metric space. We then give two applications of this result: 1) a generalization bound for MPNNs, and, 2) the stability of MPNNs to subsampling of graph-signals. Our results apply to any regular enough MPNN on any distribution of graph-signals, making the analysis rather universal.

A graphon-signal analysis of graph neural networks

Abstract

We present an approach for analyzing message passing graph neural networks (MPNNs) based on an extension of graphon analysis to a so called graphon-signal analysis. A MPNN is a function that takes a graph and a signal on the graph (a graph-signal) and returns some value. Since the input space of MPNNs is non-Euclidean, i.e., graphs can be of any size and topology, properties such as generalization are less well understood for MPNNs than for Euclidean neural networks. We claim that one important missing ingredient in past work is a meaningful notion of graph-signal similarity measure, that endows the space of inputs to MPNNs with a regular structure. We present such a similarity measure, called the graphon-signal cut distance, which makes the space of all graph-signals a dense subset of a compact metric space - the graphon-signal space. Informally, two deterministic graph-signals are close in cut distance if they 'look like' they were sampled from the same random graph-signal model. Hence, our cut distance is a natural notion of graph-signal similarity, which allows comparing any pair of graphsignals of any size and topology. We prove that MPNNs are Lipschitz continuous functions over the graphon-signal metric space. We then give two applications of this result: 1) a generalization bound for MPNNs, and, 2) the stability of MPNNs to subsampling of graph-signals. Our results apply to any regular enough MPNN on any distribution of graph-signals, making the analysis rather universal.

1 Introduction

In recent years, the need to accommodate non-regular structures in data science has brought a boom in machine learning methods on graphs. Graph deep learning (GDL) has already made a significant impact on the applied sciences and industry, with ground-breaking achievements in computational biology [2, 10, 17, 28], and a wide adoption as a general-purpose tool in social media, e-commerce, and online marketing platforms, among others. These achievements pose exciting theoretical challenges: can the success of GDL models be grounded in solid mathematical frameworks? Since the input space of a GDL model is non-Euclidean, i.e., graphs can be of any size and any topology, less is known about GDL than standard neural networks. We claim that contemporary theories of GDL are missing an important ingredient: meaningful notions of metric on the input space, namely, graph similarity measures that are defined for all graphs of any size , which respect and describe in some sense the behavior of GDL models. In this paper, we aim at providing an analysis of GDL by introducing such appropriate metrics, using graphon theory .

Agraphon is an extension of the notion of a graph, where the node set is parameterized by a probability space instead of a finite set. Graphons can be seen as limit objects of graphs, as the number of nodes increases to infinity, under an appropriate metric. One result from graphon theory (that reformulates Szemerédi's regularity lemma from discrete mathematics) states that any sufficiently large graph behaves as if it was randomly sampled from a stochastic block model with a fixed number of classes.

This result poses an 'upper bound' on the complexity of graphs: while deterministic large graphs may appear to be complex and intricate, they are actually approximately regular and behave random-like.

In this paper we extend this regularity result to an appropriate setting for message passing neural networks (MPNNs), a popular GDL model. Since MPNNs take as input a graph with a signal defined over the nodes (a graph-signal), we extend graphon theory from a theory of graphs to a theory of graph-signals. We define a metric, called the graph-signal cut distance (see Figure 1 for illustration), and formalize regularity statements for MPNNs of the following sort.

(1) Any deterministic graph-signal behaves as if it was randomly sampled from a stochastic block model, where the number of blocks only depends on how much we want the graph- signal to look random-like, and not on the graph-signal itself. (2) If two graph-signals behave as if they were sampled from the same stochastic block model,

then any (regular enough) MPNN attains approximately the same value on both.

Formally, (1) is proven by extending Szemerédi's weak regularity lemma to graphon-signals. As a result of this new version of the regularity lemma, we show that the space of graph-signals is a dense subset of the space of graphon-signals, which is shown to be compact. Point (2) is formalized by proving that MPNNs with Lipschitz continuous message functions are Lipschitz continuous mappings from the space of graph-signals to an output space, in the graphon-signal cut distance.

We argue that the above regularity result is a powerful property of MPNNs. To illustrate this, we use the new regularity result to prove two corollaries. First, a generalization bound of MPNNs, showing that if the learned MPNN performs well on the training graph-signals, it is guaranteed to also perform well on test graph-signals. This is shown by first bounding the covering number of the graphon-signal space, and then using the Lipschitzness of MPNNs. Second, we prove that MPNNs are stable to graph-signal subsampling. This is done by first showing that randomly subsampling a graphon-signal produces a graph-signal which is close in cut distance to the graphon-signal, and then using the Lipschitzness of MPNNs.

As opposed to past works that analyze MPNNs using graphon analysis, we do not assume any generative model on the data. Our results apply to any regular enough MPNN on any distribution of graph-signals, making the analysis rather universal. We note that past works about generalization in GNNs [14, 23, 26, 30] consider special assumptions on the data distribution, and often on the MPNN model. Our work provides upper bounds under no assumptions on the data distribution, and only mild Lipschitz continuity assumptions on the message passing functions. Hence, our theory bounds the generalization error when all special assumptions (that are often simplistic) from other papers are not met. We show that when all assumptions fail, MPNNs still have generalization and sampling guarantees, albeit much slower ones. See Table 1. This is also true for past sampling theorems, e.g., [18, 22, 27, 31, 32].

The problem with graph-signal domains. Since the input space of MPNNs is non-Euclidean, results like universal approximation theorems and generalization bounds are less well developed for MPNNs than Euclidean deep learning models. For example, analysis like in [6] is limited to graphs of fixed sizes, seen as adjacency matrices. The graph metric induced by the Euclidean metric on

adjacency matrices is called edit-distance . This reduction of the graph problem to the Euclidean case does not describe the full complexity of the problem. Indeed, the edit-distance is defined for weighted graphs, and non-isomorphic simple graphs are always far apart in this metric. This is an unnatural description of the reality of machine learning on graphs, where different large non-isomorphic simple graphs can describe the same large-scale phenomenon and have similar outputs for the same MPNN.

Other papers that consider graphs of arbitrary but bounded size are based on taking the union of the Euclidean edit-distance spaces up to a certain graph size [3]. If one omits the assumption that all graphs are limited by a predefined size, the edit-metric becomes non-compact - a topology too fine to explain the behavior of real MPNNs. For example, two graphs with different number of nodes are always far apart in edit-distance, while most MPNN architectures in practice are not sensitive to the addition of one node to a large graph. In [19], the expressivity of GNNs is analyzed on spaces of graphons. It is assumed that graphons are Lipschitz continuous kernels. The metric on the graphon space is taken as the L ∞ distance between graphons as functions. We claim that the Lipschitz continuity of the graphons in [19], the choice of the L ∞ metric, and the choice of an arbitrary compact subset therein, are not justified as natural models for graphs, and are not grounded in theory. Note that graphon analysis is measure theoretic, and results like the regularity lemma are no longer true when requiring Lipschitz continuity for the graphons. Lastly, in papers like [18, 26, 27, 31], the data is assumed to be generated by one, or a few graphons, which limits the data distribution significantly. We claim that this discrepancy between theory and practice is an artifact of the inappropriate choices of the metric on the space of graphs, and the choice of a limiting generative model for graphs.

2 Background

For n ∈ N , we denote [ n ] = { 1 , . . . , n } . We denote the Lebesgue p space over the measure space X by L p ( X ) , or, in short, L p . We denote by µ the standard Lebesgue measure on [0 , 1] . A partition is a sequence P k = { P 1 , . . . , P k } of disjoint measurable subsets of [0 , 1] such that ⋃ k j =1 P j = [0 , 1] . The partition is called equipartition if µ ( P i ) = µ ( P j ) for every i, j ∈ [ k ] . We denote the indicator function of a set S by 1 S . See Appendix A for more details. We summarize our notations in Appendix I.

2.1 Message passing neural networks

Most graph neural networks used in practice are special cases of MPNN (see [15] and [11] of a list of methods). MPNNs process graphs with node features, by repeatedly updating the feature at each node using the information from its neighbors. The information is sent between the different nodes along the edges of the graph, and hence, this process is called message passing . Each node merges all messages sent from its neighbors using an aggregation scheme , where typical choices is to sum, average or to take the coordinate-wise maximum of the messages. In this paper we focus on normalized sum aggregation (see Section 4.1). For more details on MPNNs we refer the reader to Appendix E.

2.2 Szemerédi weak regularity lemma

The following is taken from [13, 25]. Let G = { V, E } be a simple graph with nodes V and edges E . For any two subsets U, S ⊂ V , denote the number of edges with one end point at U and the other at S by e G ( U, S ) . Let P = { V 1 , . . . , V k } be a partition of V . The partition is called equipartition if || V i | - | V j || ≤ 1 for every i, j ∈ [ k ] . Given two node set U, S ⊂ V , if the edges between each pair of classes V i and V j were random, we would expect the number of edges of G connecting U and S to be close to the expected value e P ( U,S ) := ∑ k i =1 ∑ k j =1 e G ( V i ,V j ) | V i || V j | | V i ∩ U | | V j ∩ S | . Hence, the irregularity , that measures how non-random like the edges between { V j } k j =1 are, is defined to be

Theorem 2.1 (Weak Regularity Lemma [13]) . For every ϵ > 0 and every graph G = ( V, E ) , there is an equipartition P = { V 1 , . . . , V k } of V into k ≤ 2 c/ϵ 2 classes such that irreg G ( P ) ≤ ϵ . Here, c is a universal constant that does not depend on G and ϵ .

Theorem 2.1 asserts that we can represent any large graph G by a smaller, coarse-grained version of it: the weighted graph G ϵ with node set V ϵ = { V 1 , . . . , V k } , where the edge weight between the nodes V i and V j is e G ( V i ,V j ) | V i | , | V j | . The 'large-scale' structure of G is given by G ϵ , and the number of edges between any two subsets of nodes U i ⊂ V i and U j ⊂ V j is close to the 'expected value' e P ( U i ,U j ) . Hence, the deterministic graph G 'behaves' as if it was randomly sampled from G ϵ .

2.3 Graphon analysis

A graphon [4, 24] can be seen as a weighted graph with a 'continuous' node set, or more accurately, the nodes are parameterized by an atomless standard probability space called the graphon domain . Since all such graphon domains are equivalent to [0 , 1] with the standard Lebesgue measure (up to a measure preserving bijection), we take [0 , 1] as the node set. The space of graphons W 0 is defined to be the set of all measurable symmetric function W : [0 , 1] 2 → [0 , 1] , W ( x, y ) = W ( y, x ) . The edge weight W ( x, y ) of a graphon W ∈ W 0 can be seen as the probability of having an edge between the nodes x and y .

Graphs can be seen as special graphons. Let I m = { I 1 , . . . , I m } be an interval equipartition : a partition of [0 , 1] into intervals of equal length. The graph G = { V, E } with adjacency matrix A = { a i,j } m i,j =1 induces the graphon W G , defined by W G ( x, y ) = a ⌈ xm ⌉ , ⌈ ym ⌉ 1 . Note that W G is piecewise constant on the partition I m . We hence identify graphs with their induced graphons. A graphon can also be seen as a generative model of graphs. Given a graphon W , a corresponding random graph is generated by sampling i.i.d. nodes { X n } from he graphon domain, and connecting each pair X n , X m in probability W ( X n , X m ) to obtain the edges of the graph.

2.4 Regularity lemma for graphons

A simple way to formulate the regularity lemma in the graphon language is via stochastic block models (SBM). A SBM is a piecewise constant graphon, defined on a partition of the graphon domain [0 , 1] . The number of classes of the SBM is defined to be the number of sets in the partition. A SBM is seen as a generative model for graphs, where graphs are randomly sampled from the graphon underlying the SBM, as explained above. Szemerédi weak regularity lemma asserts that for any error tolerance ϵ , there is a number of classes k , such that any deterministic graph (of any size and topology) behaves as if it was randomly sampled from a SBM with k classes, up to error ϵ . Hence, in some sense, every graph is approximately quasi-random .

To write the weak regularity lemma in the graphon language, the notion of irregularity (1) is extended to graphons. For any measurable W : [0 , 1] 2 → R the cut norm is defined to be

where U, S ⊂ [0 , 1] are measurable. It can be verified that the irregularity (1) is equal to the cut norm of a difference between graphons induced by adequate graphs. The cut metric between two graphons W,V ∈ W 0 is defined to be d □ ( W,V ) = ∥ W -V ∥ □ . The cut distance is defined to be

where S [0 , 1] is the space of measure preserving bijections [0 , 1] → [0 , 1] , and V ϕ ( x, y ) = V ( ϕ ( x ) , ϕ ( y )) (see Section 3.1 and Appendix A.3 for more details). The cut distance is a pseudo metric on the space of graphons. By considering equivalence classes of graphons with zero cut distance, we can construct a metric space ˜ W 0 for which δ □ is a metric. The following version of the weak regularity lemma is from [25, Lemma 7].

Theorem 2.2. For every graphon W ∈ W 0 and ϵ > 0 there exists a step graphon W ′ ∈ W 0 with respect to a partition of at most ⌈ 2 c/ϵ 2 ⌉ sets such that δ □ ( W,W ′ ) ≤ ϵ , for some universal constant c .

The exact definition of a step graphon is given in Definition 3.3. It is possible to show, using Theorem 2.2, that ˜ W 0 is a compact metric space [25, Lemma 8]. Instead of recalling this construction here, we refer to Section 3.4 for the extension of this construction to graphon-signals.

3 Graphon-signal analysis

A graph-signal ( G, f ) is a graph G , that may be weighted or simple, with node set [ n ] , and a signal f ∈ R n × k that assigns the value f j ∈ R k for every node j ∈ [ n ] . A graphon-signal will be defined in Section 3.1 similarly to a graph-signal, but over the node set [0 , 1] . In this section, we show how to extend classical results in graphon theory to a so called graphon-signal theory. All proofs are given in the appendix.

3.1 The graphon signal space

For any r > 0 , define the signal space

We define the following 'norm' on L ∞ r [0 , 1] (which is not a vector space).

Definition 3.1 (Cut norm of a signal) . For a signal f : [0 , 1] → R , the cut norm ∥ f ∥ □ is defined as

where the supremum is taken over the measurable subsets S ⊂ [0 , 1] .

In Appendix A.2 we prove basic properties of signal cut norm. One important property is the equivalence of the signal cut norm to the L 1 norm

Given a bound r on the signals, we define the space of graphon-signals to be the set of pairs WL r := W 0 ×L ∞ r [0 , 1] . We define the graphon-signal cut norm , for measurable W,V : [0 , 1] 2 → R and f, g : [0 , 1] → R , by

We define the graphon-signal cut metric by d □ ( ( W,f ) , ( V, g ) ) = ∥ ( W,f ) -( V, g ) ∥ □ .

We next define a pseudo metric that makes the space of graphon-signals a compact space. Let S ′ [0 , 1] be the set of measurable measure preserving bijections between co-null sets of [0 , 1] , namely,

where ϕ is a measurable bijection and A,B,S are measurable. For ϕ ∈ S ′ [0 , 1] , we define W ϕ ( x, y ) := W ( ϕ ( x ) , ϕ ( y )) , and f ϕ ( z ) = f ( ϕ ( z )) . Note that W ϕ and f ϕ are only defined up to a null-set, and we arbitrarily set W,W ϕ , f and f ϕ to 0 in their respective null-sets, which does not affect our analysis. Define the cut distance between two graphon-signals ( W,f ) , ( V, g ) ∈ WL r by

Here, ( V, g ) ϕ := ( V ϕ , g ϕ ) . More details on this construction are given in Appendix A.3.

The graphon-signal cut distance δ □ is a pseudo-metric, and can be made into a metric by introducing the equivalence relation: ( W,f ) ∼ ( V, g ) if δ □ (( W,f ) , ( V, g )) = 0 . The quotient space ˜ WL r := WL r / ∼ of equivalence classes [( W,f )] of graphon-signals ( W,f ) is a metric space with the metric δ □ ([( W,f )] , [( V, g )]) = δ □ (( W,f ) , ( V, g )) . By abuse of terminology, we call elements of ˜ WL r also graphon-signals. A graphon-signal in ˜ WL r is defined irrespective of a specific 'indexing' of the nodes in [0 , 1] .

3.2 Induced graphon-signals

Any graph-signal can be identified with a corresponding graphon-signal as follows.

Definition 3.2. Let ( G, f ) be a graph-signal with node set [ n ] and adjacency matrix A = { a i,j } i,j ∈ [ n ] . Let { I k } n k =1 with I k = [( k -1) /n, k/n ) be the equipartition of [0 , 1] into n intervals. The graphonsignal ( W,f ) ( G, f ) = ( W G , f f ) induced by ( G, f ) is defined by

We denote ( W,f ) ( G, f ) = ( W G , f f ) . We identify any graph-signal with its induced graphon-signal. This way, we define the cut distance between a graph-signal and a graphon-signal. As before, the cut distance between a graph-signal ( G, f ) and a graphon-signal ( W,g ) can be interpreted as how much the deterministic graph-signal ( G, f ) 'looks like' it was randomly sampled from ( W,g ) .

3.3 Graphon-signal regularity lemma

(a)

(b)

(c)

To formulate our regularity lemma, we first define spaces of step functions.

Definition 3.3. Given a partition P k , and d ∈ N , we define the space S d P k of step functions of dimension d over the partition P k to be the space of functions F : [0 , 1] d → R of the form

for any choice of { c j ∈ R } j ∈ [ k ] d .

We call any element of W 0 ∩ S 2 P k a step graphon with respect to P k . A step graphon is also called a stochastic block model (SBM) . We call any element of L ∞ r [0 , 1] ∩ S 1 P k a step signal . We also call [ WL r ] P k := ( W 0 ∩ S 2 P k ) × ( L ∞ r [0 , 1] ∩ S 1 P k ) the space of SBMs with respect to P k .

In Appendix B.2 we give a number of versions of the graphon-signal regularity lemma. Here, we show one version in which the partition is fixed regardless of the graphon-signal.

Theorem 3.4 (Regularity lemma for graphon-signals - equipartition) . For any c > 1 , and any sufficiently small ϵ > 0 , for every n ≥ 2 ⌈ 9 c 4 ϵ 2 ⌉ and every ( W,f ) ∈ WL r , there exists a step graphon-signal ( W n , f n ) ∈ [ WL r ] I n such that

where I n is the equipartition of [0 , 1] into n intervals.

Figure 2 illustrates the graphon-signal regularity lemma. By identifying graph-signals with their induced graphon-signals, (6) shows that the space of graph-signals is dense in the space of graphonsignals with cut distance.

Similarly to the classical case, Theorem 3.4 is interpreted as follows. While deterministic graphsignals may seem intricate and complex, they are actually regular, and 'look like' random graphsignals that were sampled from SBMs, where the number of blocks of the SBM only depends on the desired approximation error between the SBM and the graph-signal, and not on the graph-signal itself.

Remark 3.5. The lower bound n ≥ 2 ⌈ 9 c 4 ϵ 2 ⌉ on the number of steps in the graphon-signal regularity lemma is essentially tight in the following sense. There is a universal constant C such that for every ϵ > 0 there exists a graphon-signal ( W,f ) such that no step graphon-signal ( W ′ , f ′ ) with less than 2 ⌈ C ϵ 2 ⌉ steps satisfies δ □ ( ( W,f ) , ( W ′ , f ′ ) ) ≤ ϵ . To see this, [8, Theorem 1.4, Theorem 7.1] shows that the bound in the standard weak regularity lemma (for graphs/graphons) is essentially tight in the above sense. For the graphon-signal case, we can take the graphon W ′ from [8, Theorem 7.1] which does not allow a regularity partition with less than 2 ⌈ C ϵ 2 ⌉ steps, and consider the graphon-signal ( W ′ , 1) , which then also does not allow such a regularity partition.

3.4 Compactness of the graphon-signal space and its covering number

We prove that ˜ WL r is compact using Theorem 3.4, similarly to [25, Lemma 8]. Moreover, we can bound the number of balls of radius ϵ required to cover ˜ WL r .

Theorem 3.6. The metric space ( ˜ WL r , δ □ ) is compact. Moreover, given r > 0 and c > 1 , for every sufficiently small ϵ > 0 , the space ˜ WL r can be covered by

balls of radius ϵ , where k = ⌈ 2 9 c 4 ϵ 2 ⌉ .

The Proof of Theorem 3.6 is given in Appendix C. This is a powerful result - the space of arbitrarily large graph-signals is dense in the 'small' space ˜ WL r . We will use this property in Section 4.3 to prove a generalization bound for MPNNs.

3.5 Graphon-signal sampling lemmas

In this section we prove that randomly sampling a graphon signal produces a graph-signal that is close in cut distance to the graphon signal. Let us first describe the sampling setting. More details on the construction are given in Appendix D.1. Let Λ = ( λ 1 , . . . λ k ) ∈ [0 , 1] k be k independent uniform random samples from [0 , 1] , and ( W,f ) ∈ WL r . We define the random weighted graph W (Λ) as the weighted graph with k nodes and edge weight w i,j = W ( λ i , λ j ) between node i and node j . We similarly define the random sampled signal f (Λ) with value f i = f ( λ i ) at each node i . Note that W (Λ) and f (Λ) share the sample points Λ . We then define a random simple graph as follows. We treat each w i,j = W ( λ i , λ j ) as the parameter of a Bernoulli variable e i,j , where P ( e i,j = 1) = w i,j and P ( e i,j = 0) = 1 -w i,j . We define the random simple graph G ( W, Λ) as the simple graph with an edge between each node i and node j if and only if e i,j = 1 .

We note that, given a graph signal ( G, f ) , sampling a graph-signal from ( W,f ) ( G, f ) is equivalent to subsampling the nodes of G independently and uniformly (with repetitions), and considering the resulting subgraph and subsignal. Hence, we can study the more general case of sampling a graphon-signal, where graph-signal sub-sampling is a special case. We now extend [24, Lemma 10.16], which bounds the cut distance between a graphon and its sampled graph, to the case of a sampled graphon-signal.

Theorem 3.7 (Sampling lemma for graphon-signals) . Let r > 1 . There exists a constant K 0 > 0 that depends on r , such that for every k ≥ K 0 , every ( W,f ) ∈ WL r , and for Λ = ( λ 1 , . . . λ k ) ∈ [0 , 1] k independent uniform random samples from [0 , 1] , we have

and

The proof of Theorem 3.7 is given in Appendix D.2

4 Graphon-signal analysis of MPNNs

In this section, we propose utilizing the compactness of the graphon-signal space under cut distance, and the sampling lemma, to prove regularity results for MPNNs, uniform generalization bounds, and stability to subsampling theorems.

4.1 MPNNs on graphon signals

Next, we define MPNNs on graphon-signals, in such a way that the application of a MPNN on an induced graphon-signal is equivalent to applying the MPNN on the graph-signal and then inducing it. A similar construction was presented in [26], for average aggregation, but we use normalized sum aggregation.

At each layer, we define the message function Φ( x, y ) as a linear combination of simple tensors as follows. Let K ∈ N . For every k ∈ [ K ] , let ξ k r , ξ k t : R d → R p be Lipschitz continuous functions that we call the receiver and transmitter message functions respectively. Define the message function Φ : R 2 d → R p by

where the multiplication is elementwise along the feature dimension. Given a signal f , define the message kernel Φ f : [0 , 1] 2 → R p by

We see the x variable of Φ f ( x, y ) as the receiver of the message, and y as the transmitter. Define the aggregation of a message kernel Q : [0 , 1] 2 → R p , with respect to the graphon W ∈ W 0 , to be the signal Agg( W,Q ) ∈ L ∞ r [0 , 1] , defined by

for an appropriate r > 0 . A message passing layer (MPL) takes the form f ( t ) ↦→ Agg( W, Φ ( t +1) f ( t ) ) , where f ( t ) is the signal at layer t . Each MPL is optionally followed by an update layer , which updates the signal pointwise via f ( t +1) = µ ( t +1) ( f ( t ) ( x ) , Agg( W, Φ ( t +1) f ( t ) )( x ) ) , where µ ( t +1) is a learnable mapping called the update function . A MPNN is defined by choosing the number of layers T , and defining message and update functions { µ t , ( t ξ k r ) , ( t ξ k t ) } k ∈ [ K ] ,t ∈ [ T ] . A MPNN only modifies the signal, and keeps the graph/graphon intact. We denote by Θ t ( W,f ) the output of the MPNN applied on ( W,f ) ∈ WL r at layer t ∈ [ T ] . More details on the construction are given in Appendix E.1.

The above construction is rather general. Indeed, it is well known that many classes of functions F : R d × R d → R C (e.g., L 2 functions) can be approximated by (finite) linear combinations of simple tensors F ( a, b ) ≈ ∑ K k =1 ξ k 1 ( a ) ξ k 2 ( b ) . Hence, message passing based on general message functions Φ : R 2 d → R p can be approximated by our construction. Moreover, many well-known MPNNs can be written using our formulation with a small K , e.g., [29, 36] and spectral convolutional networks [9, 20, 21], if we replace the aggregation in these method with normalized sum aggregation.

In Appendix E.1 we show that for any graph-signal ( G, f ) , we have Θ t ( W,f ) ( G, f ) = ( W,f ) Θ t ( G, f ) , where the MPNN on a graph-signal is defined with normalized sum aggregation

Here, n is the number of nodes, and { a i,j } i,j ∈ [ n ] is the adjacency matrix of G . Hence, we may identify graph-signals with their induced graphon-signals when analyzing MPNNs.

4.2 Lipschitz continuity of MPNNs

We now show that, under the above construction, MPNNs are Lipschitz continuous with respect to cut distance.

Theorem 4.1. Let Θ be a MPNN with T layers. Suppose that there exist constants L, B > 0 such that for every layer t ∈ [ T ] , every y ∈ { t , r } and every k ∈ [ K ] ,

where L µ t and L t ξ k y are the Lipschitz constants of µ t and t ξ k y . Then, there exists a constant L Θ (that depends on T, K, B and L ) such that for every ( W,f ) , ( V, g ) ∈ WL r ,

The constant L Θ depends exponentially on T , and polynomially on K,B and L . For formulas of L Θ , under different assumptions on the hypothesis class of the MPNN, we refer to Appendix F.

4.3 A generalization theorem for MPNN

In this section we prove a uniform generalization bound for MPNNs. For background on generalization analysis, we refer the reader to Appendix G.1. While uniform generalization bounds are considered a classical approach in standard neural networks, the approach is less developed in the case of MPNNs. For some works on generalization theorems of MPNNs, see [14, 23, 26, 30, 33].

When a MPNN is used for classification or regression, Θ T is followed by global pooling. Namely, for the output signal g : [0 , 1] → R p , we return ∫ g ( x ) dx ∈ R p . This is then typically followed by a learnable mapping R p → R C . In our analysis, we see this mapping as part of the loss, which can hence be learnable. The combined loss is assumed to be Lipschitz continuous 2 .

We model the ground truth classifier into C classes as a piecewise constant function C : ˜ WL r → { 0 , 1 } C , where the sets of different steps in ˜ WL r are Borel measurable sets, correspond to different classes. We consider an arbitrary probability Borel measure ν on ˜ WL r as the data distribution. More details on the construction are given in Appendix G.2.

Let Lip( ˜ WL r , L 1 ) be the space of Lipschitz continuous mappings Υ : ˜ WL r → R C with Lipschitz constant L 1 . By Theorem 4.1, we may assume that our hypothesis class of MPNNs is a subset of Lip( ˜ WL r , L 1 ) for some given L 1 . Let X = ( X 1 , . . . , X N ) be independent random samples from the data distribution ( ˜ WL r , ν ) . Let Υ X be a model that may depend on the sampled data, e.g., via training. Let E be a Lipschitz continuous loss function 3 with Lipschitz constant L 2 . For every function Υ in the hypothesis class Lip( ˜ WL r , L 1 ) (i.e. Υ X ), define the statistical risk

We define the empirical risk ˆ R (Υ X , X ) = 1 N ∑ N i =1 E ( Υ X ( X i ) , C ( X i ) ) .

Theorem 4.2 (MPNN generalization theorem) . Consider the above classification setting, and let L = L 1 L 2 . Let X 1 , . . . , X N be independent random samples from the data distribution ( ˜ WL r , ν ) . Then, for every p > 0 , there exists an event U p ⊂ ˜ WL r N , with probability

in which

where ξ ( ϵ ) = κ ( ϵ ) 2 log( κ ( ϵ )) ϵ 2 , κ is the covering number of ˜ WL r given in (7), and ξ -1 is the inverse function of ξ .

The theorem is proved in Appendix G.4. Note that the term ξ -1 ( N/ 2 C ) in (8) decreases to zero as the size of the training set N goes to infinity.

In Table 1 we compare the assumptions and dependency on the number of data points of different generalization theorems. All past works consider special assumptions. Our work provides upper bounds under no assumptions on the data distribution, and only mild assumptions on the MPNN (Lipschitz continuity of the message passing and update functions). In Table 2 in Appendix G.5 we present experiments that illustrate the generalization capabilities of MPNNs with normalized sum aggregation.

4.4 Stability of MPNNs to graph-signal subsampling

When working with very large graphs, it is often the practice to subsample the large graph, and apply a MPNN to the smaller subsampled graph [5, 7, 16]. Here, we show that such an approach is justified

theoretically. Namely, any (Lipschitz continuous) MPNN has approximately the same outcome on the large graph and its subsampled version.

Transferability and stability analysis [18, 22, 27, 31, 32] often studies a related setting. Namely, it is shown that a MPNN applied on a randomly sampled graph G approximates the MPNN on the graphon W from which the graph is sampled. However, previous analyses assumed that the generating graphon W has metric properties. Namely, it is assumed that there is some probability metric space M which is the graphon domain, and the graphon W : M×M→ [0 , 1] is Lipschitz continuous with respect to M , where the dimension of M affects the asymptotics. This is an unnatural setting, as general graphons are only assumed to be measurable, not continuous. Constraining the construction to Lipschitz continuous graphons with a uniformly bounded Lipschitz constant only accounts for a small subset of WL r , and, hence, limits the analysis significantly. In comparison, our analysis applies to any graphon-signal in WL r . When we only assume that the graphon is measurable, [0 , 1] is only treated as a standard (atomless) probability space, which is very general, and equivalent for example to [0 , 1] d for any d ∈ N , and to any Polish space. Note that graphon theory allows restricting the graphon domain to [0 , 1] since [0 , 1] , as a measure space, is very generic.

Theorem 4.3. Consider the setting of Theorem 4.2, and let Θ be a MPNN with Lipschitz constant L . Denote

Then

5 Discussion

We presented an extension of graphon theory to a graphon-signal theory. Especially, we extended well-known regularity, compactness, and sampling lemmas from graphons to graphon-signals. We then showed that the normalized sum aggregation of MPNNs is in some sense compatible with the graphon-signal cut distance, which leads to the Lipschitz continuity of MPNNs with respect to cut distance. This then allowed us to derive generalization and sampling theorems for MPNNs. The strength of our analysis is in its generality and simplicity- it is based on a natural notion of graph similarity, that allows studying the space of all graph-signals, it applies to any graph-signal data distribution, and does not impose any restriction on the number of parameters of the MPNNs, only to their regularity through the Lipschitzness of the message functions.

The main limitation of the theory is the very slow asymptotics of the generalization and subsampling theorems. This follows the fact that the upper bound on the covering number of the compact space ˜ WL r grows faster than the covering number of any finite-dimensional compact space. Yet, we believe that our work can serve as a point of departure for future works, that 1) will model subspaces of ˜ WL r of lower complexity, which approximate the support of the data-distribution in real-life settings of graph machine learning, and, 2) will lead to improved asymptotics. Another open problem is to find an essentially tight estimate of the covering number of ˜ WL r , which may be lower than the estimate presented in this paper.

Acknowledgments

We thank Ningyuan (Teresa) Huang for providing the experiments of Table 2.

Ron Levie is partially funded by ISF (Israel Science Foundation) grant #1937/23: Analysis of graph deep learning using graphon theory.

References

N. Alon, W. de la Vega, R. Kannan, and M. Karpinski. Random sampling and approximation of max-csps. Journal of Computer and System Sciences , 67(2):212-243, 2003. ISSN 0022-0000. doi: https://doi.org/10.1016/S0022-0000(03)00008-4. Special Issue on STOC 2002.

K. Atz, F. Grisoni, and G. Schneider. Geometric deep learning on molecular representations. Nature Machine Intelligence , 3:1023-1032, 2021.

W. Azizian and M. Lelarge. Expressive power of invariant and equivariant graph neural networks. In ICLR , 2021.

C. Borgs, J. T. Chayes, L. Lovász, V. T. Sós, and K. Vesztergombi. Convergent sequences of dense graphs i: Subgraph frequencies, metric properties and testing. Advances in Mathematics , 219(6):1801-1851, 2008.

J. Chen, T. Ma, and C. Xiao. FastGCN: Fast learning with graph convolutional networks via importance sampling. In International Conference on Learning Representations , 2018.

Z. Chen, S. Villar, L. Chen, and J. Bruna. On the equivalence between graph isomorphism testing and function approximation with gnns. In NeurIPS . Curran Associates, Inc., 2019.

W.-L. Chiang, X. Liu, S. Si, Y. Li, S. Bengio, and C.-J. Hsieh. Cluster-gcn: An efficient algorithm for training deep and large graph convolutional networks. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , KDD '19, page 257-266, New York, NY, USA, 2019. Association for Computing Machinery. ISBN 9781450362016. doi: 10.1145/3292500.3330925.

D. Conlon and J. Fox. Bounds for graph regularity and removal lemmas. Geometric and Functional Analysis , 22:1191-1256, 2012.

M. Defferrard, X. Bresson, and P. Vandergheynst. Convolutional neural networks on graphs with fast localized spectral filtering. In NeurIPS . Curran Associates Inc., 2016. ISBN 9781510838819.

J. M. S. et al. A deep learning approach to antibiotic discovery. Cell , 180(4):688 - 702.e13, 2020. ISSN 0092-8674.

M. Fey and J. E. Lenssen. Fast graph representation learning with PyTorch Geometric. In ICLR 2019 Workshop on Representation Learning on Graphs and Manifolds , 2019.

G. B. Folland. Real analysis: modern techniques and their applications , volume 40. John Wiley &Sons, 1999.

A. M. Frieze and R. Kannan. Quick approximation to matrices and applications. Combinatorica , 19:175-220, 1999.

V. Garg, S. Jegelka, and T. Jaakkola. Generalization and representational limits of graph neural networks. In H. D. III and A. Singh, editors, ICML , volume 119 of Proceedings of Machine Learning Research , pages 3419-3430. PMLR, 13-18 Jul 2020.

J. Gilmer, S. S. Schoenholz, P. F. Riley, O. Vinyals, and G. E. Dahl. Neural message passing for quantum chemistry. In International Conference on Machine Learning , pages 1263-1272, 2017.

W. L. Hamilton, R. Ying, and J. Leskovec. Inductive representation learning on large graphs. In Advances in Neural Information Processing Systems , page 1025-1035. Curran Associates Inc., 2017. ISBN 9781510860964.

J. M. Jumper, R. Evans, A. Pritzel, T. Green, M. Figurnov, O. Ronneberger, K. Tunyasuvunakool, R. Bates, A. Zídek, A. Potapenko, A. Bridgland, C. Meyer, S. A. A. Kohl, A. Ballard, A. Cowie, B. Romera-Paredes, S. Nikolov, R. Jain, J. Adler, T. Back, S. Petersen, D. A. Reiman, E. Clancy, M. Zielinski, M. Steinegger, M. Pacholska, T. Berghammer, S. Bodenstein, D. Silver, O. Vinyals,

A. W. Senior, K. Kavukcuoglu, P. Kohli, and D. Hassabis. Highly accurate protein structure prediction with alphafold. Nature , 596:583 - 589, 2021.

N. Keriven, A. Bietti, and S. Vaiter. Convergence and stability of graph convolutional networks on large random graphs. In Advances in Neural Information Processing Systems . Curran Associates, Inc., 2020.

N. Keriven, A. Bietti, and S. Vaiter. On the universality of graph neural networks on large random graphs. In NeurIPS . Curran Associates, Inc., 2021.

T. N. Kipf and M. Welling. Semi-supervised classification with graph convolutional networks. In ICLR , 2017.

R. Levie, F. Monti, X. Bresson, and M. M. Bronstein. Cayleynets: Graph convolutional neural networks with complex rational spectral filters. IEEE Transactions on Signal Processing , 67(1): 97-109, 2019. doi: 10.1109/TSP.2018.2879624.

R. Levie, W. Huang, L. Bucci, M. Bronstein, and G. Kutyniok. Transferability of spectral graph convolutional neural networks. Journal of Machine Learning Research , 22(272):1-59, 2021.

R. Liao, R. Urtasun, and R. Zemel. A PAC-bayesian approach to generalization bounds for graph neural networks. In ICLR , 2021.

L. M. Lovász. Large networks and graph limits. In volume 60 of Colloquium Publications , 2012. doi: 10.1090/coll/060.

L. M. Lovász and B. Szegedy. Szemerédi's lemma for the analyst. GAFA Geometric And Functional Analysis , 17:252-270, 2007.

S. Maskey, R. Levie, Y. Lee, and G. Kutyniok. Generalization analysis of message passing neural networks on large random graphs. In NeurIPS . Curran Associates, Inc., 2022.

S. Maskey, R. Levie, and G. Kutyniok. Transferability of graph neural networks: An extended graphon approach. Applied and Computational Harmonic Analysis , 63:48-83, 2023. ISSN 1063-5203. doi: https://doi.org/10.1016/j.acha.2022.11.008.

O. Méndez-Lucio, M. Ahmad, E. A. del Rio-Chanona, and J. K. Wegner. A geometric deep learning approach to predict binding conformations of bioactive molecules. Nature Machine Intelligence , 3:1033-1039, 2021.

C. Morris, M. Ritzert, M. Fey, W. L. Hamilton, J. E. Lenssen, G. Rattan, and M. Grohe. Weisfeiler and leman go neural: Higher-order graph neural networks. Proceedings of the AAAI Conference on Artificial Intelligence , 33(01):4602-4609, Jul. 2019. doi: 10.1609/aaai.v33i01. 33014602.

C. Morris, F. Geerts, J. Tönshoff, and M. Grohe. Wl meet vc. In ICML . PMLR, 2023.

L. Ruiz, L. F. O. Chamon, and A. Ribeiro. Graphon signal processing. IEEE Transactions on Signal Processing , 69:4961-4976, 2021.

L. Ruiz, Z. Wang, and A. Ribeiro. Graphon and graph neural network stability. In ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 5255-5259, 2021.

F. Scarselli, A. C. Tsoi, and M. Hagenbuchner. The vapnik-chervonenkis dimension of graph and recursive neural networks. Neural Networks , 108:248-259, 2018.

S. Shalev-Shwartz and S. Ben-David. Understanding Machine Learning: From Theory to Algorithms . Cambridge University Press, 2014. doi: 10.1017/CBO9781107298019.

D. Williams. Probability with Martingales . Cambridge University Press, 1991. doi: 10.1017/ CBO9780511813658.

K. Xu, W. Hu, J. Leskovec, and S. Jegelka. How powerful are graph neural networks? In International Conference on Learning Representations , 2019.

A Basic definitions and properties of graphon-signals

In this appendix, we give basic properties of graphon-signals, cut norm, and cut distance.

A.1 Lebesgue spaces and signal spaces

For 1 ≤ p < ∞ , the space L p [0 , 1] is the space of (equivalence classes up to null-set) of measurable functions f : [0 , 1] → R , with finite L 1 norm

The space L ∞ [0 , 1] is the space of (equivalence classes) of measurable functions with finite L ∞ norm

A.2 Properties of cut norm

Every f ∈ L ∞ r [0 , 1] can be written as f = f + -f -, where

and f -is defined similarly. It is easy to see that the supremum in (3) is attained for S which is either the support of f + or f -, and

As a result, the signal cut norm is equivalent to the L 1 norm

Moreover, for every r > 0 and measurable function W : [0 , 1] 2 → [ -r, r ] ,

The following lemma is from [24, Lemma 8.10].

Lemma A.1. For every measurable W : [0 , 1] 2 → R , the supremum

is attained for some S, T .

A.3 Properties of cut distance and measure preserving bijections

Recall that we denote the standard Lebesgue measure of [0 , 1] by µ . Let S [0 , 1] be the space of measurable bijections [0 , 1] → [0 , 1] with measurable inverse, that are measure preserving, namely, for every measurable A ⊂ [0 , 1] , µ ( A ) = µ ( ϕ ( A )) . Recall that S ′ [0 , 1] is the space of measurable bijections between co-null sets of [0 , 1] .

For ϕ ∈ S [0 , 1] or ϕ ∈ S ′ [0 , 1] , we define W ϕ ( x, y ) := W ( ϕ ( x ) , ϕ ( y )) . In case ϕ ∈ S ′ [0 , 1] , W ϕ is only define up to a null-set, and we arbitrarily set W to 0 in this null-set. This does not affect our analysis, as the cut norm is not affected by changes to the values of functions on a null sets. The cut-metric between graphons is then defined to be

Remark A.2. Note that δ □ can be defined equivalently with respect to ϕ ∈ S ′ [0 , 1] . Indeed, By [24, Equation (8.17) and Theorem 8.13], δ □ can be defined equivalently with respect to the measure preserving maps that are not necessarily invertible. These include the extensions of mappings from S ′ [0 , 1] by defining ϕ ( x ) = 0 for every x in the co-null set underlying ϕ .

Similarly to the graphon case, the graphon-signal distance δ □ is a pseudo-metric. By introducing an equivalence relation ( W,f ) ∼ ( V, g ) if δ □ (( W,f ) , ( V, g )) = 0 , and the quotient space ˜ WL r := WL r / ∼ , ˜ WL r is a metric space with a metric δ □ defined by δ □ ([( W,f )] , [ V, g )]) = d □ ( W,V ) where [( W,f )] , [( V, g )] , are the equivalence classes of ( W,f ) and ( V, g ) respectively. By abuse of terminology, we call elements of ˜ WL r also graphon-signals.

̸

Remark A.3. We note that ˜ WL r = ˜ W 0 × ˜ L ∞ r [0 , 1] (for the natural definition of ˜ L ∞ r [0 , 1] ), since in ˜ WL r we require that the measure preserving bijection is shared between the graphon W and the signal f . Sharing the measure preserving bijetion between W and f is an important modelling requirement, as ϕ is seen as a 're-indexing' of the node set [0 , 1] . When re-indexing a node x , both the neighborhood W ( x, · ) of x and the signal value f ( x ) at x should change together, otherwise, the graphon and the signal would fall out of alignment.

We identify graphs with their induced graphons and signal with their induced signals

B Graphon-signal regularity lemmas

In this appendix, we prove a number of versions of the graphon-signal regularity lemma, where Theorem 3.4 is one version.

B.1 Properties of partitions and step functions

Given a partition P k and d ∈ N , the next lemma shows that there is an equiparition E n such that the space S d E n uniformly approximates the space S d P k in L 1 [0 , 1] d norm (see Definition 3.3).

Lemma B.1 (Equitizing partitions) . Let P k be a partition of [0 , 1] into k sets (generally not of the same measure). Then, for any n > k there exists an equipartition E n of [0 , 1] into n sets such that any function F ∈ S d P k can be approximated in L 1 [0 , 1] d by a function from F ∈ S d E n up to small error. Namely, for every F ∈ S d P k there exists F ′ ∈ S d E n such that

Proof . Let P k = { P 1 , . . . , P k } be a partition of [0 , 1] . For each i , we divide P i into subsets P i = { P i, 1 , . . . , P i,m i } of measure 1 /n (up to the last set) with a residual, as follows. If µ ( P i ) < 1 /n , we choose P i = { P i, 1 = P i } . Otherwise, we take P i, 1 , . . . , P i,m i -1 of measure 1 /n , and µ ( P i,m i ) ≤ 1 /n . We call P i,m i the remainder.

We now define the sequence of sets of measure 1 /n

where, by abuse of notation, for any i such that m i = 1 , we set { P i, 1 , . . . , P i,m i -1 } = ∅ in the above formula. Note that in general ∪Q ̸ = [0 , 1] . We moreover define the union of residuals Π := P 1 ,m 1 ∪ P 2 ,m 2 ∪··· ∪ P k,m k . Note that µ (Π) = 1 -µ ( ∪Q ) = 1 -k 1 n = h/n , where k is the number of elements in Q , and h = n -k . Hence, we can partition Π into h parts { Π 1 , . . . Π h } of measure 1 /n with no residual. Thus we have obtain the equipartition of [0 , 1] to n sets of measure 1 /n

For convenience, we also denote E n = { Z 1 , . . . , Z n } .

Let

We can write F with respect to the equipartition E n as

for some { ˜ c j } with the same values as the values of { c j } . Here, E is supported in the set Π ( d ) ⊂ [0 , 1] d , defied by

Consider the step function

Since µ (Π) = k/n , we have µ (Π ( d ) ) = dk/n , and so

■

Lemma B.2. Let { Q 1 , Q 2 , . . . , Q m } partition of [0 , 1] . Let { I 1 , I 2 , . . . , I m } be a partition of [0 , 1] into intervals, such that for every j ∈ [ m ] , µ ( Q j ) = µ ( I j ) . Then, there exists a measure preserving bijection ϕ : [0 , 1] → [0 , 1] ∈ S ′ [0 , 1] such that 4

Proof . By the definition of a standard probability space, the measure space induced by [0 , 1] on a non-null subset Q j ⊆ [0 , 1] is a standard probability space. Moreover, each Q j is atomless, since [0 , 1] is atomless. Since there is a measure-preserving bijection (up to null-set) between any two atomless standard probability spaces, we obtain the result. ■

Lemma B.3. Let S = { S j ⊂ [0 , 1] } m -1 j =0 be a collection of measurable sets (that are not disjoint in general), and d ∈ N . Let C d S be the space of functions F : [0 , 1] d → R of the form

for some choice of { c j ∈ R } j ∈ [ m ] d . Then, there exists a partition P k = { P 1 , . . . , P k } into k = 2 m sets, that depends only on S , such that

Proof . The partition P k = { P 1 , . . . , P k } is defined as follows. Let

We must have | ˜ P| ≤ 2 m . Indeed, there are at most 2 m different subsets of S for the intersections. We endow an arbitrarily order to ˜ P and turn it into a sequence. If the size of ˜ P is strictly smaller than 2 m , we add enough copies of {∅} to ˜ P to make the size of the sequence 2 m , that we denote by P k , where k = 2 m . ■

The following simple lemma is proved similarly to Lemma B.3. We give it without proof.

Lemma B.4. Let P k = { P 1 , . . . , P k } , Q m = { Q 1 , . . . , Q k } be two partitions. Then, there exists a partition Z km into km sets such that for every d ,

B.2 List of graphon-signal regularity lemmas

The following lemma from [25, Lemma 4.1] is a tool in the proof of the weak regularity lemma.

Lemma B.5. Let K 1 , K 2 , . . . be arbitrary nonempty subsets (not necessarily subspaces) of a Hilbert space H . Then, for every ϵ > 0 and v ∈ H there is m ≤ ⌈ 1 /ϵ 2 ⌉ and v i ∈ K i and γ i ∈ R , i ∈ [ m ] , such that for every w ∈ K m +1

The following theorem is an extension of the graphon regularity lemma from [25] to the case of graphon-signals. Much of the proof follows the steps of [25].

Theorem B.6 (Weak regularity lemma for graphon-signals) . Let ϵ, ρ > 0 . For every ( W,f ) ∈ WL r there exists a partition P k of [0 , 1] into k = ⌈ r/ρ ⌉ ( 2 2 ⌈ 1 /ϵ 2 ⌉ ) sets, a step function graphon W k ∈ S 2 P k ∩ W 0 and a step function signal f k ∈ S 1 P k ∩ L ∞ r [0 , 1] , such that

Proof . We first analyze the graphon part. In Lemma B.5, set H = L 2 ([0 , 1] 2 ) and for all i ∈ N , set

Then, by Lemma B.5, there exists m ≤ ⌈ 1 /ϵ 2 ⌉ two sequences of sets S m = { S i } i m =1 , T m = { T i } i m =1 , a sequence of coefficients { γ i ∈ R } i m =1 , and

such that for any V ∈ K , given by V ( x, y ) = 1 S ( x ) 1 T ( y ) , we have

We may choose exactly m = ⌈ 1 /ϵ 2 ⌉ by adding copies of the empty set to S m and T m , if the constant m guaranteed by Lemma B.5 is strictly less than ⌈ 1 /ϵ 2 ⌉ . Let W ϵ ( x, y ) = ( W ′ ϵ ( x, y ) + W ′ ϵ ( y, x )) / 2 . By the symmetry of W , it is easy to see that (15) is also true when replacing W ′ ϵ by W ϵ . Indeed,

Consider the concatenation of the two sequences T m , S m given by Y 2 m = T m ∪ S m . Note that in the notation of Lemma B.3, W ϵ ∈ C 2 Y 2 m . Hence, by Lemma B.3, there exists a partition Q n into n = 2 2 m = 2 2 ⌈ 1 ϵ 2 ⌉ sets, such that W ϵ is a step graphon with respect to Q n .

To analyze the signal part, we partition the range of the signal [ -r, r ] into j = ⌈ r/ρ ⌉ intervals { J i } j i =1 of length less or equal to 2 ρ , where the left edge point of each J i is -r +( i -1) ρ r . Consider the partition of [0 , 1] based on the preimages Y j = { Y i = f -1 ( J i ) } j i =1 . It is easy to see that for the step signal

where a i the midpoint of the interval Y i , we have

Lastly, by Lemma B.4, there is a partition P k of [0 , 1] into k = ⌈ r/ρ ⌉ ( 2 2 ⌈ 1 /ϵ 2 ⌉ ) sets such that W ϵ ∈ S 2 P k and f ρ ∈ S 1 P k .

■

Corollary B.7 (Weak regularity lemma for graphon-signals - version 2) . Let r > 0 and c > 1 . For every sufficiently small ϵ > 0 (namely, ϵ that satisfies (17)), and for every ( W,f ) ∈ WL r there exists a partition P k of [0 , 1] into k = ( 2 ⌈ 2 c/ϵ 2 ⌉ ) sets, a step graphon W k ∈ S 2 P k ∩ W 0 and a step signal f k ∈ S 1 P k ∩ L ∞ r [0 , 1] , such that

Proof . First, evoke Theorem B.6, with errors ∥ W -W k ∥ □ ≤ ν and ∥ f -f k ∥ □ ≤ ρ = ϵ -ν . We now show that there is some ϵ 0 > 0 such that for every ϵ < ϵ 0 , there is a choice of ν such that the number of sets in the partition, guaranteed by Theorem B.6, satisfies

The reconcile these two conditions, we restrict to ϵ such that

There exists ϵ 0 that depends on c and r (and hence also on t ) such that for every ϵ < ϵ 0 (17) is satisfied. Indeed, for small enough ϵ ,

so

Moreover, for small enough ϵ ,

Hence, for every ϵ < ϵ 0 , there is a choice of ν such that

Lastly, we add as many copies of ∅ to P k ( ν ) as needed so that we get a sequence of k = 2 ⌈ 2 c/ϵ 2 ⌉ sets.

■

Denote c = 1 + t . In case we have

On the other hand, for we have

Theorem B.8 (Regularity lemma for graphon-signals - equipartition version) . Let c > 1 and r > 0 . For any sufficiently small ϵ > 0 , and every ( W,f ) ∈ WL r there exists ϕ ∈ S ′ [0 , 1] , a step function graphon [ W ϕ ] n ∈ S 2 I n ∩ W 0 and a step signal [ f ϕ ] n ∈ S 1 I n ∩ L ∞ r [0 , 1] , such that

where I n is the equipartition of [0 , 1] into n = 2 ⌈ 2 c/ϵ 2 ⌉ intervals.

Proof . Let c = 1 + t > 1 , ϵ > 0 and 0 < α, β < 1 . In Corollary B.7, consider the approximation error

with a partition P k into k = 2 ⌈ 2(1+ t/ 2) ( ϵα ) 2 ⌉ sets. We next equatize the partition P k up to error ϵβ . More accurately, in Lemma B.1, we choose

and note that

By Lemma B.1 and by the fact that the cut norm is bounded by L 1 norm, there exists an equipartition E n into n sets, and step functions W n and f n with respect to E n such that

Hence, by the triangle inequality,

In the following, we restrict to choices of α and β which satisfy α +(2 + r ) β = 1 . Consider the function n : (0 , 1) → N defined by

Using a similar technique as in the proof of Corollary B.7, there is ϵ 0 > 0 that depends on c and r (and hence also on t ) such that for every ϵ < ϵ 0 , we may choose α 0 (that depends on ϵ ) which satisfies

Moreover, there is a choice α 1 which satisfies

We note that the function n : (0 , 1) → N satisfies the following intermediate value property. For every 0 < α 1 < α 2 < 1 and every m ∈ N between n ( α 1 ) and n ( α 2 ) , there is a point α ∈ [ α 1 , α 2 ] such that n ( α ) = m . This follows the fact that α ↦→ (2 + r ) · 2 2(1+0 . 5 t ) ( ϵα ) 2 +1 / ( ϵ (1 -α )) is a continuous function. Hence, by (19) and (20), there is a point α (and β such that α +(2 + r ) β = 1 ) such that

■

By a slight modification of the above proof, we can replace n with the constant n = ⌈ 2 2 c ϵ 2 ⌉ . As a result, we can easily prove that for any n ′ ≥ 2 ⌈ 2 c ϵ 2 ⌉ we have the approximation property (18) with n ′ instead of n . This is done by choosing an appropriate c ′ > c and using Theorem B.8 on c ′ , giving a constant n ′ = ⌈ 2 2 c ′ ϵ 2 ⌉ ≥ 2 ⌈ 2 c ϵ 2 ⌉ = n . This leads to the following corollary.

Corollary B.9 (Regularity lemma for graphon-signals - equipartition version 2) . Let c > 1 and r > 0 . For any sufficiently small ϵ > 0 , for every n ≥ 2 ⌈ 2 c ϵ 2 ⌉ and every ( W,f ) ∈ WL r , there exists ϕ ∈ S ′ [0 , 1] , a step function graphon [ W ϕ ] n ∈ S 2 I n ∩ W 0 and a step function signal [ f ϕ ] n ∈ S 1 I n ∩ L ∞ r [0 , 1] , such that

where I n is the equipartition of [0 , 1] into n intervals.

Next, we prove that we can use the average of the graphon and the signal in each part for the approximating graphon-signal. For that we define the projection of a graphon signal upon a partition.

Definition B.10. Let P n = { P 1 , . . . , P n } be a partition of [0 , 1] , and ( W,f ) ∈ WL r . We define the projection of ( W,f ) upon ( S 2 P ×S 1 P ) ∩WL r to be the step graphon-signal ( W,f ) P n = ( W P n , f P n ) that attains the value

for every ( x, y ) ∈ P i × P j .

At the cost of replacing the error ϵ by 2 ϵ , we can replace W ′ with its projection. This was shown in [1]. Since this paper does not use the exact same setting as us, for completeness, we write a proof of the claim below.

Corollary B.11 (Regularity lemma for graphon-signals - projection version) . For any c > 1 , and any sufficiently small ϵ > 0 , for every n ≥ 2 ⌈ 8 c ϵ 2 ⌉ and every ( W,f ) ∈ WL r , there exists ϕ ∈ S ′ [0 , 1] , such that such that

where I n is the equipartition of [0 , 1] into n intervals.

We first prove a simple lemma.

Lemma B.12. Let P n = { P 1 , . . . , P n } be a partition of [0 , 1] , and Let V, R ∈ S 2 P n ∩W 0 . Then, the supremum of

is attained for S, T of the form is attained for S of the form

where s ⊂ [ n ] .

Proof . First, by Lemma A.1, the supremum of (21) is attained for some S, T ⊂ [0 , 1] . Given the maximizers S, T , without loss of generality, suppose that

we can improve T as follows. Consider the set t ⊂ [ n ] such that for every j ∈ t

By increasing the set T ∩ P j to P j , we can only increase the size of the above integral. Indeed,

where t, s ⊂ [ n ] . Similarly for any two signals f, g ∈ S 1 P n ∩ L ∞ r [0 , 1] , the supremum of

Hence, by increasing T to we get

We similarly replace each T ∩ P j such that

by the empty set. We now repeat this process for S , which concludes the proof for the graphon part. For the signal case, let f = f + -f -, and suppose without loss of generality that ∥ f ∥ □ = ∥ f ∥ 1 . It is easy to see that the supremum of (22) is attained for the support of f + , which has the required form. ■

Proof . Proof of Corollary B.11 Let W n ∈ S P n ∩W 0 be the step graphon guaranteed by Corollary B.9, with error ϵ/ 2 and measure preserving bijection ϕ ∈ S ′ [0 , 1] . Without loss of generality, we suppose that W ϕ = W . Otherwise, we just denote W ′ = W ϕ and replace the notation W with W ′ in the following. By Lemma B.12, the infimum underlying ∥ W P n -W n ∥ □ is attained for for some

We now have, by definition of the projected graphon,

Hence, by the triangle inequality,

A similar argument shows

Hence,

C Compactness and covering number of the graphon-signal space

In this appendix we prove Theorem 3.6.

Given a partition P k , recall that

is called the space of SBMs or step graphon-signals with respect to P k . Recall that ˜ WL r is the space of equivalence classes of graphon-signals with zero δ □ distance, with the δ □ metric (defined on arbitrary representatives). By abuse of terminology, we call elements of ˜ WL r also graphon-signals.

Theorem C.1. The metric space ( ˜ WL r , δ □ ) is compact.

̸

The proof is a simple extension of [25, Lemma 8] from the case of graphon to the case of graphonsignal. The proof relies on the notion of martingale. A martingale is a sequence of random variables for which, for each element in the sequence, the conditional expectation of the next value in the sequence is equal to the present value, regardless of all prior values. The Martingale convergence theorem states that for any bounded martingale { M n } n over the probability pace X , the sequence { M n ( x ) } n converges for almost every x ∈ X , and the limit function is bounded (see [12, 35]).

Proof . [Proof of Theorem C.1] Consider a sequence { [( W n , f n )] } n ∈ N ⊂ ˜ WL r , with ( W n , f n ) ∈ WL r . For each k , consider the equipartition into m k intervals I m k , where m k = 2 30 ⌈ ( r 2 +1) ⌉ k 2 . By Corollary B.11, there is a measure preserving bijection ϕ n,k (up to nullset) such that

where ( W n , f n ) ϕ n,k I mk is the projection of ( W n , f n ) ϕ n,k upon I m k (Definition B.10). For every fixed k , each pair of functions ( W n , f n ) ϕ n,k I mk is defined via m 2 k + m k values in [0 , 1] . Hence, since [0 , 1] m 2 k + m k is compact, there is a subsequence { n k j } j ∈ N , such that all of these values converge. Namely, for each k , the sequence

converges pointwise to some step graphon-signal ( U k , g k ) in [ WL r ] P k as j →∞ . Note that I m l is a refinement of I m k for every l > k . As as a result, by the definition of projection of graphon-signals to partitions, for every l > k , the value of ( W ϕ n,k n ) I mk at each partition set I i m k × I j m k can be obtained by averaging the values of ( W ϕ n,l n ) I ml at all partition sets I i ′ m l × I j ′ m l that are subsets of I i m k × I j m k . A similar property applies also to the signal. Moreover, by taking limits, it can be shown that the same property holds also for ( U k , g k ) and ( U l , g l ) . We now see { ( U k , g k ) } ∞ k =1 as a sequence of random variables over the standard probability space [0 , 1] 2 . The above discussion shows that { ( U k , g k ) } ∞ k =1 is a bounded martingale. By the martingale convergence theorem, the sequence { ( U k , g k ) } ∞ k =1 converges almost everywhere pointwise to a limit ( U, g ) , which must be in WL r .

Lastly, we show that there exist increasing sequences { k z ∈ N } ∞ z =1 and { t z = n k z j z } z ∈ N such that ( W t z , f t z ) ϕ t z ,k z converges to ( U, g ) in cut distance. By the dominant convergence theorem, for each z ∈ N there exists a k z such that

We choose such an increasing sequence { k z } z ∈ N with k z > 3 z . Similarly, for ever z ∈ N , there is a j z such that, with the notation t z = n k z j z ,

and we may choose the sequence { t z } z ∈ N increasing. Therefore, by the triangle inequality and by the fact that the L 1 norm bounds the cut norm,

■

The next theorem bounds the covering number of ˜ WL r .

Theorem C.2. Let r > 0 and c > 1 . For every sufficiently small ϵ > 0 , the space ˜ WL r can be covered by

balls of radius ϵ in cut distance, where k = ⌈ 2 2 c/ϵ ⌉ .

Proof . Let 1 < c < c ′ and 0 < α < 1 . Given an error tolerance αϵ > 0 , using Theorem B.8, we take the equipartition I n into n = 2 ⌈ 2 c α 2 ϵ 2 ⌉ intervals, for which any graphon-signal ( W,f ) ∈ ˜ WL r can be approximated by some ( W,f ) n in [ ˜ WL r ] I n , up to error αϵ . Consider the rectangle R n,r = [0 , 1] n 2 × [ -r, r ] n . We identify each element of [ ˜ WL r ] I n with an element of R n,r using the coefficients of (5). More accurately, the coefficients c i,j of the step graphon are identifies with the first n 2 entries of a point in R n,r , and the the coefficients b i of the step signals are identifies with the last n entries of a point in R n,r . Now, consider the quantized rectangle ˜ R n,r , defined as

Note that ˜ R n consists of

points. Now, every point x ∈ R n,r can be approximated by a quantized version x Q ∈ ˜ R n,r up to error in normalized ℓ 1 norm

where we re-index the entries of x and x Q in a 1D sequence. Let us denote by ( W,f ) Q the quantized version of ( W n , f n ) , given by the above equivalence mapping between ( W,f ) n and R n,r . We hence have

We now choose the parameter α . Note that for any c ′ > c , there exists ϵ 0 > 0 that depends on c ′ -c , such that for any ϵ < ϵ 0 there is a choice of α (close to 1 ) such that

where k = ⌈ 2 2 c ′ /ϵ 2 ⌉ . This is shown similarly to the proof of Corollary B.7 and Theorem B.8. We now replace the notation c ′ → c , which concludes the proof.

■

D Graphon-signal sampling lemmas

In this appendix, we prove Theorem 3.7. We denote by W 1 the space of measurable functions U : [0 , 1] → [ -1 , 1] , and call each U ∈ W 1 a kernel.

D.1 Formal construction of sampled graph-signals

Let W ∈ W 0 be a graphon, and Λ ′ = ( λ ′ 1 , . . . λ ′ k ) ∈ [0 , 1] k . We denote by W (Λ ′ ) the adjacency matrix

By abuse of notation, we also treat W (Λ ′ ) as a weighted graph with k nodes and the adjacency matrix W (Λ ′ ) . We denote by Λ = ( λ 1 , . . . , λ k ) : ( λ ′ 1 , . . . λ ′ k ) ↦→ ( λ ′ 1 , . . . λ ′ k ) the identity random variable in [0 , 1] k . We hence call ( λ 1 , . . . , λ k ) random independent samples from [0 , 1] . We call the random variable W (Λ) a random sampled weighted graph .

Given f ∈ L ∞ r [0 , 1] and Λ ′ = (Λ ′ 1 , . . . , Λ ′ k ) ∈ [0 , 1] k , we denote by f (Λ ′ ) the discrete signal with k nodes, and value f ( λ ′ i ) for each node i = 1 , . . . , k . We define the sampled signal as the random variable f (Λ) .

We then define the random sampled simple graph as follows. First, for a deterministic Λ ′ ∈ [0 , 1] k , we define a 2D array of Bernoulli random variables { e i,j (Λ ′ ) } i,j ∈ [ k ] where e i,j (Λ ′ ) = 1 in probability

W ( λ ′ i , λ ′ j ) , and zero otherwise, for i, j ∈ [ k ] . We define the probability space { 0 , 1 } k × k with normalized counting measure, defined for any S ⊂ { 0 , 1 } k × k by

where

We denote the identity random variable by G ( W, Λ ′ ) : z ↦→ z , and call it a random simple graph sampled from W (Λ ′ ) .

Next we also allow to 'plug' the random variable Λ into Λ ′ . For that, we define the joint probability space Ω = [0 , 1] k ×{ 0 , 1 } k × k with the product σ -algebra of the Lebesgue sets in [0 , 1] k with the power set σ -algebra of { 0 , 1 } k × k , with measure, for any measurable S ⊂ Ω ,

where

We call the random variable G ( W, Λ) : Λ ′ × z ↦→ z the random simple graph generated by W . We extend the domains of the random variables W (Λ) , f (Λ) and G ( W, Λ ′ ) to Ω trivially (e.g., f (Λ)(Λ ′ , z ) = f (Λ)(Λ ′ ) and G ( W, Λ ′ )(Λ ′ , z ) = G ( W, Λ ′ )( z ) ), so that all random variables are defined over the same space Ω . Note that the random sampled graphs and the random signal share the same sample points.

Given a kernel U ∈ W 1 , we define the random sampled kernel U (Λ) similarly.

Similarly to the above construction, given a weighted graph H with k nodes and edge weights h i,j , we define the simple graph sampled from H as the random variable simple graph G ( H ) with k nodes and independent Bernoulli variables e i,j ∈ { 0 , 1 } , with P ( e i,j = 1) = h i,j , as the edge weights. The following lemma is taken from [24, Equation (10.9)].

Lemma D.1. Let H be a weighted graph of k nodes. Then

The following is a simple corollary of Lemma D.1, using the law of total probability.

D.2 Sampling lemmas of graphon-signals

The following lemma, from [24, Lemma 10.6], shows that the cut norm of a kernel is approximated by the cut norm of its sample.

Lemma D.3 (First Sampling Lemma for kernels) . Let U ∈ W 1 , and Λ ∈ [0 , 1] k be uniform independent samples from [0 , 1] . Then, with probability at least 1 -4 e - √ k/ 10 ,

We derive a version of Lemma D.3 with expected value using the following lemma.

Lemma D.4. Let z : Ω → [0 , 1] be a random variable over the probability space Ω . Suppose that in an event E ⊂ Ω of probability 1 -ϵ we have z < α . Then

Proof .

and

■

As a result of this lemma, we have a simple corollary of Lemma D.3.

Corollary D.5 (First sampling lemma - expected value version) . Let U ∈ W 1 and Λ ∈ [0 , 1] k be chosen uniformly at random, where k ≥ 1 . Then

Proof . By Lemma D.4, and since 6 /k 1 / 4 > 4 e - √ k/ 10 ,

We note that a version of the first sampling lemma, Lemma D.3, for signals instead of kernels, is just a classical Monte Carlo approximation, when working with the L 1 [0 , 1] norm, which is equivalent to the signal cut norm.

Lemma D.6 (First sampling lemma for signals) . Let f ∈ L ∞ r [0 , 1] . Then

Proof . By standard Monte Carlo theory, since r 2 bounds the variance of f ( λ ) , where λ is a random uniform sample from [0 , 1] , we have

Here, V denotes variance, and we note that E ∥ f (Λ) ∥ 1 = 1 k ∑ k j =1 | f ( λ j ) | = ∥ f ∥ 1 . Hence, by Cauchy Schwarz inequality,

We now extend [24, Lemma 10.16], which bounds the cut distance between a graphon and its sampled graph, to the case of a sampled graphon-signal.

Theorem D.7 (Second sampling lemma for graphon signals) . Let r > 1 . Let k ≥ K 0 , where K 0 is a constant that depends on r , and let ( W,f ) ∈ WL r . Then,

The proof follows the steps of [24, Lemma 10.16] and [4]. We note that the main difference in our proof is that we explicitly write the measure preserving bijection that optimizes the cut distance. While this is not necessary in the classical case, where only a graphon is sampled, in our case we need to show that there is a measure preserving bijection that is shared by the graphon and the signal. We hence write the proof for completion.

Proof .

Denote a generic error bound, given by the regularity lemma Theorem B.8 by ϵ . If we take n intervals in the Theorem B.8 , then the error in the regularity lemma will be, for c such that 2 c = 3 ,

so

For small enough ϵ , we increase the error bound in the regularity lemma to satisfy

More accurately, for the equipartition to intervals I n , there is ϕ ′ ∈ S ′ [0 , 1] and a piecewise constant graphon signal ([ W ϕ ] n , [ f ϕ ] n ) such that

and for some 0 ≤ α ≤ 1 . If we choose n such that

then an error bound in the regularity lemma is

and

for some 0 ≤ α ≤ 1 . Without loss of generality, we suppose that ϕ ′ is the identity. This only means that we work with a different representative of [( W,f )] ∈ ˜ WL r throughout the proof. We hence have and

for some step graphon-signal ( W n , f n ) ∈ [ WL r ] I n .

Now, by the first sampling lemma (Corollary D.5),

Moreover, by the fact that f -f n ∈ L ∞ 2 r [0 , 1] , Lemma D.6 implies that

Therefore,

Similarly, we have

Now, let π Λ be a sorting permutation in [ k ] , such that

is a sequence in a non-decreasing order. Let { I i k = [ i -1 , i ) /k } k i =1 be the intervals of the equipartition I k . The sorting permutation π Λ induces a measure preserving bijection ϕ that sorts the intervals I i k . Namely, we define, for every x ∈ [0 , 1] ,

where J i,j : I i k → I j k are defined as x ↦→ x -i/k + j/k , for all x ∈ I i k .

By abuse of notation, we denote by W n (Λ) and f n (Λ) the induced graphon and signal from W n (Λ) and f n (Λ) respectively. Hence, W n (Λ) ϕ and f n (Λ) ϕ are well defined. Note that the graphons W n and W n (Λ) ϕ are stepfunctions, where the set of values of W n (Λ) ϕ is a subset of the set of values of W n . Intuitively, since k ≫ m , we expect the partition { [ λ ′ i , λ ′ i +1 ) } k i =1 to be 'close to a refinement' of I n in high probability. Also, we expect the two sets of values of W n (Λ) ϕ and W n to be identical in high probability. Moreover, since Λ ′ is sorted, when inducing a graphon from the graph W n (Λ) and 'sorting' it to W n (Λ) ϕ , we get a graphon that is roughly 'aligned' with W n . The same philosophy also applied to f n and f n (Λ) ϕ . We next formalize these observations.

For each i ∈ [ n ] , let λ ′ j i be the smaller point of Λ ′ that is in I i n , set j i = j i +1 if Λ ′ ∩ I i n = ∅ , and set j n +1 = k +1 . For every i = 1 , . . . , n , we call

the i -th step of W n (Λ) ϕ (which can be the empty set). Let a i = j i -1 k be the left edge point of J i . Note that a i = | Λ ∩ [0 , i/n ) | /k is distributed binomially (up to the normalization k ) with k trials and success in probability i/n .

̸

̸

Hence,

By properties of the binomial distribution, we have E ( ka i ) = ik/n , so

As a result

̸

̸

and for n > 10 ,

Now, by n = ⌈ √ k r log( k ) ⌉ ≤ √ k r log( k ) +1 , for large enough k ,

Similarly,

Note that in the proof of Corollary B.7, in (16), α is chosen close to 1 , and especially, for small enough ϵ , α > 1 / 2 . Hence, for large enough k ,

Similarly, for each k , if 1 -α < 1 √ log( k ) , then

Moreover, for each k such that 1 -α > 1 √ log( k ) , if k is large enough (where the lower bound of k depends on r ), we have

so, by 6 √ 2 < 9 ,

As a result, for large enough k ,

E Graphon-signal MPNNs

In this appendix we give properties and examples of MPNNs.

E.1 Properties of graphon-signal MPNNs

Consider the construction of MPNN from Section 4.1. We first explain how a MPNN on a grpah is equivalent to a MPNN on the induced graphon.

Let G be a graph of n nodes, with adjacency matrix A = { a i,j } i,j ∈ [ n ] and signal f ∈ R n × d . Consider a MPL θ , with receiver and transmitter message functions ξ k r , ξ k t : R d → R p , for k ∈ [ K ] , where K ∈ N , and update function µ : R d + p → R s . The application of the MPL on ( G, f ) is defined as follows. We first define the message kernel Φ f : [ n ] 2 → R p , with entries

We then aggregate the message kernel with normalized sum aggregation

Lastly, we apply the update function, to obtain the output θ ( G, f ) of the MPL with value at each node i

Lemma E.1. Consider a MPL θ as in the above setting. Then, for every graph signal ( G,A, f ) ,

Proof . Let { I i , . . . , I n } be the equipartition to intervals. For each j ∈ [ n ] , let y j ∈ I j be an arbitrary point. Let i ∈ [ n ] and x ∈ I i . We have

Therefore, for every i ∈ [ n ] and every x ∈ I i ,

and

■

E.2 Examples of MPNNs

The GIN convolutional layer [36] is defined as follows. First, the message function is

where M is a multi-layer perceptron (MLP) and ϵ a constant. Each layer may have a different MLP and different constant ϵ . The standard GIN is defined with sum aggregation, but we use normalized sum aggregation.

Given a graph-signal ( G, f ) , with f ∈ R n × d with adjacency matrix A ∈ R n × n , a spectral convolutional layer based on a polynomial filter p ( λ ) = ∑ J j =0 λ j C j , where C j ∈ R d × p , is defined to be

followed by a pointwise non-linearity like ReLU. Such a convolutional layer can be seen as J +1 MPLs. We first apply J MPLs, where each MPL is of the form

for some C ∈ R ( J +1) d × p , followed by the pointwise non-linearity. The message part of θ can be written in our formulation with Φ( a, b ) = b , and the update part of θ with η ( c, d ) = ( c, d ) . The last update layer U is linear followed by the pointwise non-linearity.

F Lipschitz continuity of MPNNs

In this appendix we prove Theorem 4.1. For v ∈ R d , we often denote by | v | = ∥ v ∥ ∞ . We define the L 1 norm of a measurable function h : [0 , 1] → R d by

We define Lipschitz continuity with respect to the infinity norm. Namely, Z : R d → R c is called Lipschitz continuous with Lipschitz constant L if

We denote the minimal Lipschitz bound of the function Z by L Z .

We extend L ∞ r [0 , 1] to the space of functions f : [0 , 1] → R d with the above L 1 norm.

Define the space K q of kernels bounded by q > 0 to be the space of measurable functions

The cut norm, cut metric, and cut distance are defined as usual for kernels in K q .

F.1 Lipschitz continuity of message passing and update layers

In this subsection we prove that message passing layers and update layers are Lipschitz continuous with respect to he graphon-signal cut metric.

and the update function is

We then apply an update layer

Similarly,

Lemma F.1 (Product rule for message kernels) . Let Φ f , Φ g be the message kernels corresponding to the signals f, g . Then

Proof . Suppose p = 1 For every x, y ∈ [0 , 1] 2

Hence,

Lemma F.2. Let Q,V be two message kernels, and W ∈ W 0 . Then

Proof .

So

■

■

As a result of Lemma F.2 and the product rule Lemma F.1, we have the following corollary, that computes the error in aggregating two message kernels with the same graphon.

Corollary F.3.

Next we fix the message kernel, and bound the difference between the aggregation of the message kernel with respect to two different graphons. Let L + [0 , 1] be the space of measurable function f : [0 , 1] → [0 , 1] . The following lemma is a trivial extension of [24, Lemma 8.10] from K 1 to K r .

Lemma F.4. For any kernel Q ∈ K r

where the supremum is attained for some f, g ∈ L + [0 , 1] .

The following Lemma is proven as part of the proof of [24, Lemma 8.11].

Lemma F.5. For any kernel Q ∈ K

r

For completeness, we give here a self-contained proof.

Proof . Any function f ∈ L ∞ 1 [0 , 1] can be written as f = f + -f -, where f + , f -∈ L + [0 , 1] . Hence, by Lemma F.4,

Next we state a simple lemma.

Lemma F.6. Let f = f + -f -be a signal, where f + , f -: [0 , 1] → (0 , ∞ ) are measurable. Then the supremum in the cut norm ∥ f ∥ □ = sup S ⊂ [0 , 1] ∣ ∣ ∫ S f ( x ) dx ∣ ∣ is attained as the support of either f + or f -.

Lemma F.7. Let f ∈ L ∞ r [0 , 1] , W,V ∈ W 0 , and suppose that ∣ ∣ ξ k r ( f ( x )) ∣ ∣ , ∣ ∣ ξ k t ( f ( x )) ∣ ∣ ≤ ρ for every x ∈ [0 , 1] and k = 1 , . . . , K . Then

Moreover, if ξ k r and ξ k t are non-negatively valued for every k = 1 , . . . , K , then

Proof . Let T = W -V . Let S be the maximizer of the supremum underlying the cut norm of Agg( T, Φ f ) . Suppose without loss of generality that ∫ S Agg( T, Φ f )( x ) dx > 0 . Denote q k r ( x ) = ξ k r ( f ( x )) and q k t ( x ) = ξ k t ( f ( x )) . We have

Let

Moreover, define v k t = q k t /ρ , and note that v k r , v k t ∈ L ∞ 1 [0 , 1] . We hence have, by Lemma F.5,

Hence,

Lastly, in case ξ k r , ξ k t are nonnegatively valued, so are q k r , q k t , and hence by Lemma F.4,

■

Lastly, we show that update layers are Lipschitz continuous. Since the update function takes two functions f : [0 , 1] → R d i (for generally two different output dimensions d 1 , d 2 ), we 'concatenate' these two inputs and treat it as one input f : [0 , 1] → R d 1 + d 2 .

Lemma F.9. Let η : R d + p → R s be Lipschitz with Lipschitz constant L η , and let f, g ∈ L ∞ r [0 , 1] with values in R d + p for some d, p ∈ N .

Then

Proof .

F.2 Bounds of signals and MPLs with Lipschitz message and update functions

We will consider three settings for the MPNN Lipschitz bounds. In all settings, the transmitter, receiver, and update functions are Lipschitz. In the first setting all message and update functions are assumed to be bounded. In the second setting, there is no additional assumption over Lipschtzness of the transmitter, receiver, and update functions. In the third setting, we assume that the message function Φ is also Lipschitz with Lipschitz bound L Φ , and that all receiver and transmitter functions

■

Theorem F.8. Let ( W,f ) , ( V, g ) ∈ WL r , and suppose that ∣ ∣ ξ k r ( f ( x )) ∣ ∣ , ∣ ∣ ξ k t ( f ( x )) ∣ ∣ ≤ ρ and L ξ k t , L ξ k t < L for every x ∈ [0 , 1] and k = 1 , . . . , K . Then,

Proof . By Lemma F.1, Lemma F.2 and Lemma F.7,

are non-negatively bounded (e.g., via an application of ReLU or sigmoid in their implementation). Note that in case K = 1 and all functions are differentiable, by the product rule, Φ can be Lipschitz only in two cases: if both ξ r and ξ t are bounded and Lipschitz, or if either ξ r or ξ t is constant, and the other function is Lipschitz. When K > 1 , we can have combinations of these cases.

We next derive bounds for the different settings. A bound for setting 1 is given in Theorem F.8. Moreover, When the receiver and transmitter message functions and the update functions are bounded, so is the signal at each layer.

Bounds for setting 2.

Next we show boundedness when the receiver and transmitter message and update functions are only assumed to be Lipschitz.

Define the formal bias B ξ of a function ξ : R d 1 → R d 2 to be ξ (0) [26]. We note that the formal bias of an affine-linear operator is its classical bias.

Lemma F.10. Let ( W,f ) ∈ WL r , and suppose that for every y ∈ { r , t } and k = 1 , . . . , K

Then, and

Proof . Let y ∈ { r , t } . We have

so,

Next, we have a direct result of Theorem F.8.

Corollary F.11. Suppose that for every y ∈ { r , t } and k = 1 , . . . , K

Then, for every ( W,f ) , ( V, g ) ∈ WL r ,

Bound for setting 3.

Lemma F.12. Let ( W,f ) ∈ WL r , and suppose that

Then, and

Proof . We have

■

■

Additional bounds.

Lemma F.13. Let f be a signal, W,V ∈ W 0 , and suppose that ∥ Φ f ∥ ∞ ≤ ρ for every k = 1 , . . . , K , and that ξ k r and ξ k t are non-negatively valued. Then

Proof . The proof follows the steps of Lemma F.7 until (25), from where we proceed differently. Since all of the functions q k r and q k t , k ∈ [ K ] , and since ∥ Φ f ∥ ∞ ≤ ρ , the product of each q k r ( x ) q k t ( y ) must be also bounded by ρ for every x ∈ [0 , 1] and k ∈ [ K ] . Hence, we may replace the normalization in (25) with

where for every k ∈ [ K ] , ρ k r ρ k t = ρ . This guarantees that v k r , v k t ∈ L ∞ 1 [0 , 1] . Hence,

where, by convention,

■

Theorem F.14. Let ( W,f ) , ( V, g ) ∈ WL r , and suppose that ∥ Φ ∥ ∞ , ∥ ξ k r ∥ ∞ , ∥ ξ k t ∥ ∞ ≤ ρ , all message functions ξ are non-negative valued, and L ξ k t , L ξ k t < L , for every k = 1 , . . . , K . Then,

The proof follows the steps of Theorem F.8.

Corollary F.15. Suppose that for every y ∈ { r , t } and k = 1 , . . . , K

and ξ, Φ are all non-negatively valued. Then, for every ( W,f ) , ( V, g ) ∈ WL r ,

The proof follows the steps of Corollary F.11.

F.3 Lipschitz continuity theorems for MPNNs

The following recurrence sequence will govern the propagation of the Lipschitz constant of the MPNN and the bound of signal along the layers.

Lemma F.16. Let a = ( a 1 , a 2 , . . . ) and b = ( b 1 , b 2 , . . . ) . The solution to e t +1 = a t e t + b t , with initialization e 0 , is

In case there exist a, b ∈ R such that a i = a and b i = b for every i ,

Setting 1.

Theorem F.17. Let Θ be a MPNN with T layers. Suppose that for every layer and every y and k ,

Let ( W,f ) , ( V, g ) ∈ WL r . Then, for MPNN with no update function

and for MPNN with update function

Proof . We prove for MPNNs with update function, where the proof without update function is similar. We can write a recurrence sequence for a bound ∥ Θ t ( W,f ) -Θ t ( V, g ) ∥ □ ≤ e t , by Theorem F.8 and Lemma F.9, as

The proof now follows by applying Lemma F.16 with a = 4 KL 2 ρ and b = 4 Kρ 2 L .

Setting 2.

Lemma F.18. Let Θ be a MPNN with T layers. Suppose that for every layer t and every y ∈ { r , t } and k ∈ [ K ] ,

with L, B > 1 . Let ( W,f ) ∈ WL r . Then, for MPNN without update function, for every layer t ,

and for MPNN with update function, for every layer t ,

Proof . We first prove for MPNNs without update functions. Denote by C t a bound on ∥ t f ∥ ∞ , and let C 0 be a bound on ∥ f ∥ ∞ . By Lemma F.10, we may choose bounds such that

We can always choose C t , K, L > 1 , and therefore,

Denote a = 2 KL 2 B 2 . We have

Now, for MPNNs with update function, we have

and we proceed similarly.

■

■

Theorem F.19. Let Θ be a MPNN with T layers. Suppose that for every layer t and every y ∈ { r , t } and k ∈ [ K ] ,

with L, B > 1 . Let ( W,g ) , ( V, g ) ∈ WL r . Then, for MPNNs without update functions

where

and for MPNNs with update functions where

Proof . We prove for MPNNs without update functions. The proof for the other case is similar. By Corollary F.11, since the signals at layer t are bounded by

We hence derive a recurrence sequence for a bound ∥ Θ t ( W,f ) -Θ t ( V, g ) ∥ □ ≤ e t , as

Setting 3.

Lemma F.20. Suppose that for every layer t and every y ∈ { r , t } and k = 1 , . . . , K ,

and ξ, Φ are all non-negatively valued. Then, for MPNNs without update function

and for MPNNs with update function

Proof . We first prove for MPNNs without update functions. By Lemma F.10, there is a bound e t of ∥ Θ t ( W,f ) ∥ ∞ that satisfies

Solving this recurrence sequence via Lemma F.16 concludes the proof.

Lastly, for MPNN with update functions, we have a bound that satisfies

we have and we proceed as before.

■

■

Lemma F.21. Suppose that for every y ∈ { r , t } and k = 1 , . . . , K

and ξ, Φ are all non-negatively valued. Let ( W,g ) , ( V, g ) ∈ WL r . Then, for MPNNs without update functions

and for MPNNs with update functions

Proof . We start with MPNNs without update functions. By Corollary F.15 and Lemma F.20, there is a bound e t on the error ∥ Θ t ( W, Φ f ) -Θ t ( V, Φ g ) ∥ □ at step t that satisfies

Hence, by Lemma F.16, and Z defined by (26),

where in the notations of Lemma F.16,

and

Next, for MPNNs with update functions, there is a bound that satisfies

Hence, by Lemma F.16, and Z defined by (26),

G Generalization bound for MPNNs

In this appendix we prove Theorem 4.2.

G.1 Statistical learning and generalization analysis

In the statistical setting of learning, we suppose that the dataset comprises independent random samples from a probability space that describes all possible data P . We suppose that for each x ∈ P there is a ground truth value y x ∈ Y , e.g., the ground truth class or value of x , where Y is, in general, some measure space. The loss is a measurable function L : Y 2 → R + that defines

■

.

similarity in Y . Given a measurable function Θ : P → Y , that we call the model or network , its accuracy on all potential inputs is defined as the statistical risk R stat (Θ) = E x ∼P ( L (Θ( x ) , y x ) ) The goal in learning is to find a network Θ , from some hypothesis space T , that has a low statistical risk. In practice, the statistical risk cannot be computed analytically. Instead, we suppose that a dataset X = { x m } M m =1 ⊂ P of M ∈ N random independent samples with corresponding values { y m } M m =1 ⊂ Y is given. We estimate the statistical risk via a 'Monte Carlo approximation,' called the empirical risk R emp (Θ) = 1 M ∑ M m =1 L (Θ( x m ) , y m ) . The network Θ is chosen in practice by optimizing the empirical risk. The goal in generalization analysis is to show that if a learned Θ attains a low empirical risk, then it is also guaranteed to have a low statistical risk.

One technique for bounding the statistical risk in terms of the empirical risk is to use the bound R stat (Θ) ≤ R emp (Θ) + E , where E is the generalization error E = sup Θ ∈T | R stat (Θ) -R emp (Θ) | , and to find a bound for E . Since the trained network Θ = Θ X depends on the data X , the network is not a constant when varying the dataset, and hence the empirical risk is not really a Monte Carlo approximation of the statistical risk in the learning setting. If the network Θ was fixed, then Monte Carlo theory would have given us a bound of E 2 of order O ( κ ( p ) /M ) in an event of probability 1 -p , where, for example, in Hoeffding's inequality Theorem G.2, κ ( p ) = log(2 /p ) . Let us call such an event a good sampling event . Since the good sampling event depends on Θ , computing a naive bound to the generalization error would require intersecting all good sampling events for all Θ ∈ T . Uniform convergence bounds are approaches for intersecting adequate sampling events that allow bounding the generalization error more efficiently. This intersection of events leads to a term in the generalization bound, called the complexity/capacity , that describes the richness of the hypothesis space T . This is the philosophy behind approaches such as VC-dimension, Rademacher dimension, fat-shattering dimension, pseudo-dimension, and uniform covering number (see, e.g., [34]).

G.2 Classification setting

We define a ground truth classifier into C classes as follows. Let C : ˜ WL r → R C be a measurable piecewise constant function of the following form. There is a partition of WL r into disjoint measurable sets B 1 , . . . , B C ⊂ ˜ WL r such that ⋃ C i =1 B i = ˜ WL r , and for every i ∈ [ C ] and every x ∈ B i ,

where e i ∈ R C is the standard basis element with entries ( e i ) j = δ i,j , where δ i,j is the Kronecker delta.

We define an arbitrary data distribution as follows. Let B be the Borel σ -algebra of ˜ WL r , and ν be any probability measure on the measurable space ( ˜ WL r , B ) . We may assume that we complete B with respect to ν , obtaining the σ -algebra Σ . If we do not complete the measure, we just denote Σ = B . Defining ( ˜ WL r , Σ , ν ) as a complete measure space or not will not affect our construction.

Let S be a metric space. Let Lip( S , L ) be the space of Lipschitz continuous mappings Υ : S → R C with Lipschitz constant L . Note that by Theorem 4.1, for every i ∈ [ C ] , the space of MPNN with Lipschitz continuous input and output message functions and Lipschitz update functions, restricted to B i , is a subset of Lip( B i , L 1 ) which is the restriction of Lip( ˜ WL r , L 1 ) to B i ⊂ ˜ WL r , for some L 1 > 0 . Moreover, B i has finite covering κ ( ϵ ) given in (23). Let E be a Lipschitz continuous loss function with Lipschitz constant L 2 . Therefore, since C| B i is in Lip( B i , 0) , for any Υ ∈ Lip( ˜ WL r , L 1 ) , the function E (Υ | B i , C| B i ) is in Lip( B i , L ) with L = L 1 L 2 .

G.3 Uniform Monte Carlo approximation of Lipschitz continuous functions

The proof of Theorem 4.2 is based on the following Theorem G.3, which studies uniform Monte Carlo approximations of Lipschitz continuous functions over metric spaces with finite covering.

Definition G.1. A metric space M is said to have covering number κ : (0 , ∞ ) → N , if for every ϵ > 0 , the space M can be covered by κ ( ϵ ) ball of radius ϵ .

Theorem G.2 (Hoeffding's Inequality) . Let Y 1 , . . . , Y N be independent random variables such that a ≤ Y i ≤ b almost surely. Then, for every k > 0 ,

The following theorem is an extended version of [26, Lemma B.3], where the difference is that we use a general covering number κ ( ϵ ) , where in [26, Lemma B.3] the covering number is exponential in ϵ . For completion, we repeat here the proof, with the required modification.

Theorem G.3 (Uniform Monte Carlo approximation for Lipschitz continuous functions) . Let X be a probability metric space 5 , with probability measure µ , and covering number κ ( ϵ ) . Let X 1 , . . . , X N be drawn i.i.d. from X . Then, for every p > 0 , there exists an event E p Lip ⊂ X N (regarding the choice of ( X 1 , . . . , X N ) ), with probability

such that for every ( X 1 , . . . , X N ) ∈ E p Lip , for every bounded Lipschitz continuous function F : X → R d with Lipschitz constant L F , we have

where ξ ( r ) = κ ( r ) 2 log( κ ( r )) r 2 and ξ -1 is the inverse function of ξ .

Proof . Let r > 0 . There exists a covering of X by a set of balls { B j } j ∈ [ J ] of radius r , where J = κ ( r ) . For j = 2 , . . . , J , we define I j := B j \ ∪ i<j B i , and define I 1 = B 1 . Hence, { I j } j ∈ [ J ] is a family of measurable sets such that I j ∩ I i = ∅ for all i = j ∈ [ J ] , ⋃ j ∈ [ J ] I j = χ , and diam( I j ) ≤ 2 r for all j ∈ [ J ] , where by convention diam( ∅ ) = 0 . For each j ∈ [ J ] , let z j be the center of the ball B j .

̸

Next, we compute a concentration of error bound on the difference between the measure of I j and its Monte Carlo approximation, which is uniform in j ∈ [ J ] . Let j ∈ [ J ] and q ∈ (0 , 1) . By Hoeffding's inequality Theorem G.2, there is an event E q j with probability µ ( E q j ) ≥ 1 -q , in which

with probability µ N ( E Jq Lip ) ≥ 1 -Jq . In this event, (28) holds for all j ∈ J . We change the failure probability variable p = Jq , and denote E p Lip = E Jq Lip .

Next we bound uniformly the Monte Carlo approximation error of the integral of bounded Lipschitz continuous functions F : χ → R F . Let F : χ → R F be a bounded Lipschitz continuous function with Lipschitz constant L F . We define the step function

Consider the event

Then,

To bound (1), we define for each X i the unique index j i ∈ [ J ] s.t. X i ∈ I j i . We calculate,

We proceed by bounding (2). In the event of E p Lip , which holds with probability at least 1 -p , equation (28) holds for all j ∈ J . In this event, we get

Recall that J = κ ( r ) . Then, with probability at least 1 -p

To bound (3), we calculate

By plugging the bounds of (1) , (2) and (3) into (29), we get

Lastly, choosing r = ξ -1 ( N ) for ξ ( r ) = κ ( r ) 2 log( κ ( r )) r 2 , gives κ ( r ) log( κ ( r )) √ N = r , so

Since the event E p Lip is independent of the choice of F : χ → R F , the proof is finished. ■

G.4 A generalization theorem for MPNNs

The following generalization theorem of MPNN is now a direct result of Theorem G.3.

Let Lip( ˜ WL r , L 1 ) denote the space of Lipschitz continuous functions Θ : WL r → R C with Lipschitz bound bounded by L 1 and ∥ Θ ∥ ∞ ≤ L 1 . We note that the theorems of Appendix F.2 prove that MPNN with Lipschitz continuous message and update functions, and bounded formal biases, are in Lip( ˜ WL r , L 1 ) .

Theorem G.4 (MPNN generalization theorem) . Consider the classification setting of Appendix G.2. Let X 1 , . . . , X N be independent random samples from the data distribution ( ˜ WL r , Σ , ν ) . Then, for every p > 0 , there exists an event E p ⊂ ˜ WL r N regarding the choice of ( X 1 , . . . , X N ) , with probability

in which for every function Υ in the hypothesis class Lip( ˜ WL r , L 1 ) , with we have

where ξ ( r ) = κ ( r ) 2 log( κ ( r )) r 2 , κ is the covering number of ˜ WL r given in (23), and ξ -1 is the inverse function of ξ .

Proof . For each i ∈ [ C ] , let S i be the number of samples of X that falls within B i . The random variable ( S 1 , . . . , S C ) is multinomial, with expected value ( N/C,...,N/C ) and variance ( N ( C -1) C 2 , . . . , N ( C -1) C 2 ) ≤ ( N C , . . . , N C ) . We now use Chebyshev's inequality, which states that for any a > 0 ,

We choose a √ N C = N 2 C , so a = N 1 / 2 2 C 1 / 2 , and

Therefore,

We intersect these events of i ∈ [ C ] , and get an event E mult of probability more than 1 -2 C 2 N in which S i > N 2 C for every i ∈ [ C ] . In the following, given a set B i we consider a realization M = S i , and then use the law of total probability.

From Theorem G.3 we get the following. For every p > 0 , there exists an event E p i ⊂ B i M regarding the choice of ( X 1 , . . . , X M ) ⊂ B i , with probability

such that for every function Υ ′ in the hypothesis class Lip( ˜ WL r , L 1 ) , we have

where ξ ( r ) = κ ( r ) 2 log( κ ( r )) r 2 , κ is the covering number of ˜ WL r given in (23), and ξ -1 is the inverse function of ξ . In the last inequality, we use the bound, for every x ∈ ˜ WL r ,

Since (31) is true for any Υ ′ ∈ Lip( ˜ WL r , L 1 ) , it is also true for Υ X for any realization of X , so we also have

Lastly, we denote

G.5 Experiments

The nontrivial part in our construction of the MPNN architecture is the choice of normalized sum aggregation as the aggregation method of the MPNNs. We hence show the accuracy and generalization gap of this aggregation scheme in practice in Table 1.

Most MPNNs typically use sum, mean or max aggregation. Intuitively, normalized sum aggregation is close to average aggregation, due its 'normalized nature.' For example, normalized sum and mean aggregations are well behaved for dense graphs with number of nodes going to infinity, while sum aggregation diverges for such graphs. Moreover, sum aggregation cannot be extended to graphons, while normalized sum and mean aggregations can. In Table 2, we first show that MPNNs with normalized sum aggregation perform well and generalize well. We then compare the normalized sum aggregation MPNNs (in rows 1 and 3 of Table 2) to baseline MPNNs with mean aggregation (rows 2 and 4 in Table 2), and show that normalized sum aggregation is not worse than mean aggregation.

The source code, courtesy of Ningyuan (Teresa) Huang, is available as part of https://github. com/nhuang37/finegrain_expressivity_GNN .

H Stability of MPNNs to graph subsampling

Lastly, we prove Theorem 4.3.

Theorem H.1. Consider the setting of Theorem 4.2, and let Θ be a MPNN with Lipschitz constant L . Denote

Then

Proof . By Lipschitz continuity of Θ ,

Hence,

and the claim of the theorem follows from Theorem 3.7.

■

As explained in Section 3.5, the above theorem of stability of MPNNs to graphon-signal sampling also applies to subsampling graph-signals.

I Notations

L p ( X ) or L p : Lebesgue p space over the measure space X .

µ : standard Lebesgue measure on [0 , 1] .

G = { V, E } : simple graph with nodes V and edges E .

A = { a i,j } m i,j =1 : graph adjacency matrix (page 4).

e G ( U, S ) : the number of edges with one end point at U and the other at S , where U, S ⊂ V (page 3).

e P ( U,S ) : density of of edges between U and S (page 3).

irreg G ( P ) : irregularity (1).

W 0 : space of graphons (page 4).

W : graphon (page 4).

W G : induced graphon from the graph G (page 4).

∥ W ∥ □ : cut norm (page 4).

d □ ( W,V ) : cut metric (page 4).

δ □ ( W,V ) : cut distance (page 4).

S [0 , 1] : the space of measure preserving bijections [0 , 1] → [0 , 1] (page 4).

S ′ [0 , 1] : the set of measurable measure preserving bijections between co-null sets of [0 , 1] (page 5).

˜ W 0 : space of graphons modulo zero cut distance (page 4).

L ∞ r [0 , 1] : signal space (2).

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors introduce an extension of the graphon analysis to a graphon-signal analysis, in order to measure generalization bounds of and stability of MPNNs. Specifically, they propose a variant of cut distance as a new graph-signal similarity measure that transforms the space of graph-signals to a dense subset of a compact metric space. \n\nGiven the definition of the cut distance, the authors are able to show Lipschitz continuity of MPNNs, that can yield results on generalization and stability properties of MPNNs.\n\nThe authors do not show any empirical result on the connection between the graphon-signal analysis and MPNNs.\n\nSTRENGTHS:\n1. The methodology of analysing MPNNs through the extension of graphon theory is quite interesting and fairly novel. \n2. The proposed approach can derive generalization and stability arguments for the MPNNs.\n3. In contrast to related work, the authors provide a rigorous analysis that is justified for modeling the graph space, even when the Lipschitz continuity of graphons is required.\n\nWEAKNESSES:\n1. There is no empirical evidence that can highlight the connection of the graphon-signal analysis with the MPNNs.\n\n2. There is no discussion about a related work [1], that also related stability arguments of MPNNs with respect to the graphon analysis. \n\n\n[1] Luana Ruiz, Zhiyang Wang and Alejandro Ribeiro. GRAPHON AND GRAPH NEURAL NETWORK STABILITY. 2021.\n\nQUESTIONS:\n1. Can the authors provide any empirical result that can show stability or generalization capabilities of MPNNs with respect to the graph-signals?\n\n2. Can the authors provide a further comparison of their analysis with the analysis in [1]? \n\nTypos:\nInconsistencies in title case format\n- '2.3 graphon analysis'\n- '3.3 graphon-signal regularity lemma'\n\n[1] Luana Ruiz, Zhiyang Wang and Alejandro Ribeiro. GRAPHON AND GRAPH NEURAL NETWORK STABILITY. 2021.\n\nLIMITATIONS:\n- The authors discuss the limitations of their theory with respect to the slow asymptotics and the subsampling theorems. \n- The authors do not discuss any potential negative impact (although I do not see any negative impact coming out of this theory).", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work focuses on analyzing message passing neural networks (MPNNs) by extending graphon analysis. It casts MPNNs as a function taking a graph-signal input (graph + signal on the graph) and proposing a similarity measure over graph signals, viewed as a dense subset of a compact metric space. This allows the authors to compare graph signals of different sizes and topological structures. More specifically, the authors formalize \"regularity statements\" for MPNNs by extending Szemerédi's weak regularity lemma to graphon signals. Furthermore, the authors demonstrate the power of their framework by using it to prove generalization bounds for MPNNs and establish stability of MPNNs under subsampling.\n\nThe work is a theoretical work that establishes a new framework for analyzing MPNNs and does not include any empirical results.\n\nSTRENGTHS:\nThe work is original and introduces a new framework for analyzing MPNNs via graphon signals. The approach is original and differs from prior graphon-based MPNN works in that it does not assume a generative model of the data.\n\nThe paper is generally well-written and clear.\n\nWEAKNESSES:\nIt is unclear what the practical implications of this result or analysis are. The paper could be strengthened by some experiments that show, for instance, that theoretically established generalization guarantees transfer over to empirical performance.\n\nThe exponential dependence in covering number, etc. render some of the theorems a bit impractical (though still of theoretical value).\n\nQUESTIONS:\nThe exponential growth of the covering number seems to be a limitation for practicality of the Generalization Theorem in Thm 4.2. Are there important settings for which the covering number is much smaller? Does the author believe there is any hope for improving the dependence on covering number (e.g., improving ξ(ε)) in Them 4.2, or is the bound somewhat tight in the general setting?\n\nLIMITATIONS:\nLimitations of this work pertaining to slow asymptotics are referred to in the Discussion section.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes a new metric to measure the distance between graphon signals (graphon parameters and signals defined over graphon jointly). The metric generalizes the previous metric for graphon only. The work also uses the metric to characterize the approximation error of graphon signals by performing equipartition of the space, and by sampling. The work also uses the metric to characterize the generalization bound of GNNs and stability of GNNs when they are employed to encode graphon signals.\n\nSTRENGTHS:\n1. The work reads solid and investigates a very fundamental problem. The defined metric may be useful for the future study. \n2. The study of the proposed metric is comprehensive, including approximation, generalization and stability.\n\nWEAKNESSES:\nThe extension of the metric from graphon to graphon signal seems incremental to me. The following results on equipartition and sampling approximation, GNN generalization, and stability results seem to be also just extensions. A better explanation of what the technical difficulty behind such an extension should be discussed. Even if there is no substantial technical challenge, what new insights can be collected based on the new extension should be provided and compared with previous observations in a better way. Currently, it is hard for readers to collect insights into these results. What I learned is that the approximation error converges to 0 very slowly ($O(\\sqrt{log k})$). But why do we have such a slow rate? Is it the best we can have? For the stability results, the issue is similar: does the result mean GNNs if used here are super unstable? In previous literature, the rate is about $O(1/k)$ or $O(1/\\sqrt{k})$. Also, the implication behind the generalization results is almost not discussed at all. \n\nSome minors: \n\n1. the notations in this work are complicated and are used in a kind of arbitrary way. It is hard for readers to track the used notations. I suggest to provide a table list of the key notations and think about how to unify them. \n2. Why the output of $\\Phi$ is in $R^p$ in Sec 4.1 if the output of $\\xi^k$ is $R^p$.\n\nQUESTIONS:\nPlease check the above weaknesses. \n\nI think the biggest issue behind this work is that discussion and comparison between the new results and previous ones are missing, which loses the chance to collect insights from the paper. Notations should be also improved.\n\nLIMITATIONS:\nPure theoretical work, not see negative societal impact", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-22"}

comments: SUMMARY:
This paper extends the graphon theory to the graphon-signal theory. Especially, the authors introduce a distance metric "cut distance" on graphon-signal and analyze their regularity, and compactness. Furthermore, the authors use Lipschitez properties of MPNN to bound the model generalization.

STRENGTHS:
I am not an expert in this field. But I carefully check the sketch of the points of the theoretical analysis is systemic and comprehensive.

WEAKNESSES:
I am not an expert in this field, but I think there are at least some points to improve the paper.

1. There are a lot of notations in this paper, it is better to summarize all the notations of this work and make a table in the appendix. It will help the review's comments

2. Some references are still missing for the training since adaptation. For example, the generalization gap in non-adversarial attacks should also be briefly discussed such as ( https://www.pnas.org/doi/10.1073/pnas.1907378117)

3. It would be better to have a figure to demonstrate the motivation of the distance metric.

QUESTIONS:
The questions are demonstrated in the weakness part.

LIMITATIONS:
The authors have adequately addressed the limitation of the public.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-25