Title: What functions can Graph Neural Networks compute on random graphs? The role of Positional Encoding
Venue: NeurIPS 2023
Accepted: yes
Keywords: graph neural network; random graph; positional encoding
Abstract: We aim to deepen the theoretical understanding of Graph Neural Networks (GNNs) on large graphs, with a focus on their expressive power.
Existing analyses relate this notion to the graph isomorphism problem, which is mostly relevant for graphs of small sizes, or studied graph classification or regression tasks, while prediction tasks on \emph{nodes} are far more relevant on large graphs. Recently, several works showed that, on very general random graphs models, GNNs converge to certains functions as the number of nodes grows.
In this paper, we provide a more complete and intuitive description of the function space generated by equivariant GNNs for node-tasks, through general notions of convergence that encompass several previous examples. We emphasize the role of input node features, and study the impact of \emph{node Positional Encodings} (PEs), a recent line of work that has been shown to yield state-of-the-art results in practice. Through the study of several examples of PEs on large random graphs, we extend previously known universality results to significantly more general models. Our theoretical results hint at some normalization tricks, which is shown numerically to have a positive impact on GNN generalization on synthetic and real data. Our proofs contain new concentration inequalities of independent interest.

What functions can Graph Neural Networks compute on random graphs? The role of Positional Encoding

Abstract

We aim to deepen the theoretical understanding of Graph Neural Networks (GNNs) on large graphs, with a focus on their expressive power. Existing analyses relate this notion to the graph isomorphism problem, which is mostly relevant for graphs of small sizes, or studied graph classification or regression tasks, while prediction tasks on nodes are far more relevant on large graphs. Recently, several works showed that, on very general random graphs models, GNNs converge to certains functions as the number of nodes grows. In this paper, we provide a more complete and intuitive description of the function space generated by equivariant GNNs for node-tasks, through general notions of convergence that encompass several previous examples. We emphasize the role of input node features, and study the impact of node Positional Encodings (PEs), a recent line of work that has been shown to yield state-of-the-art results in practice. Through the study of several examples of PEs on large random graphs, we extend previously known universality results to significantly more general models. Our theoretical results hint at some normalization tricks, which is shown numerically to have a positive impact on GNN generalization on synthetic and real data. Our proofs contain new concentration inequalities of independent interest.

1 Introduction

Machine learning on graphs with Graph Neural Networks (GNNs) [53, 5] is now a well-established domain, with application fields ranging from combinatorial optimization [6] to recommender systems [50, 11], physics [45, 1], chemistry [16], epidemiology [37], physical networks such as power grids [41], and many more. Despite this, there is still much that is not properly understood about GNNs, both empirically and theoretically, and their performances are not always consistent [52, 22], compared to simple baselines in some cases. It is generally admitted that a better theoretical understanding of GNNs, especially of their fundamental limitations, is necessary to design better models in the future.

Theoretical studies of GNNs have largely focused on their expressive power , kickstarted by a seminal study [54] that relates their ability to distinguish non-isomorphic graphs to the historical WeisfeilerLehman (WL) test [51]. Following this, many works have defined improved versions of GNNs to be 'more powerful than WL' [34, 35, 26, 49, 38], often by augmenting GNNs with various features, or by implementing 'higher-order' versions of the basic message-passing paradigm. Among the simplest and most effective idea to 'augment' GNNs is the use of Positional Encodings (PE) as input to the GNN, inspired by the vocabulary of Transformers [48]. The idea is to equip nodes with carefully crafted input features that would help break some of the indeterminancy in the subsequent message-passing framework. In early works, unique and/or random node identifiers have been used [32, 47], but they technically break the permutation-invariance/equivariance - consistency with a reordering of the nodes in the graph - of the GNN. Most PEs in the current literature are based on eigenvectors of the adjacency matrix or Laplacian of the graph [12, 13] (with recent variants to handle the sign/basis indeterminancy

Samuel Vaiter

[31]), random-walks [13], node metrics [56, 30], or subgraphs [4]. Some of these have been shown to have an expressive power beyond WL [30, 4, 31].

In some contexts however, WL-based analyses have limitations: they pertain to tasks on graphs (e.g. graph classification or regression) and have limited to no connections to tasks on nodes; and they are mostly relevant for small-scale graphs, as medium or large graphs have a negligible chance of being exactly isomorphic to one another , but exhibit different characteristics (e.g. community structures) that might be useful for learning. At the other end of the spectrum, the properties of GNNs on large graphs have been analysed in the context of latent positions Random Graphs (RGs) [24, 25, 43, 44, 29, 2, 36], a family of models slightly more general than graphons [33]. Such statistical models of large graphs are classically used in graph theory [18, 9] to model various data such as epidemiological [27, 39], biological [17], social [20], or protein-protein interaction [19] networks, and are still an active area of research [9]. For GNNs, the use of such models has shed light on their stability to deformations of the model [44, 29, 24], expressive power [25], generalisation [15, 36], or some phenomena such as oversmoothing [23, 3]. One basic idea is that, as the number of nodes in a random graph grows, GNNs converge to 'continuous' equivalents [24, 8], whose properties are somewhat easier to characterize than their discrete counterpart. As prediction tasks on nodes are far more common and relevant on large graphs modelled by random graphs, this paper will focus on permutation-equivariant GNNs, rather than permutation-invariant. In the limit, it has been shown that their output converge to functions over some latent space to label the nodes, but the descriptions of this space of functions and its properties are still very much incomplete. A partial answer was given in [25], in which some universality properties are given for specific models of GNNs, but for limited models of random graphs with no random edges , and specific models of GNNs that no not include node features or PEs. We will in particular extend some of their results to random edges, with the proper choice of PEs.

Contributions. In this paper, we significantly extend existing results by providing a complete description of the function space generated by permutation-equivariant GNNs (Theorem 1), in terms of simple stability rules, and show that it is equivalent to previous implicit definitions that were based on convergence bounds. We outline the role of the input node features , and particularly of Positional Encodings (PEs). We then study the several representative examples of PEs on large random graphs. In particular, we analyze SignNet [31] (eigenvector-based) PEs (Theorem 2), and distance-based PEs [30] (Theorem 3). We derive simple normalization rules that are necessary for convergence, and illustrate them on real data. Finally, our proofs contain new universality results for square-integrable functions and new concentration inequalities that are of independent interest. All technical proofs are available in the Appendix. The code to reproduce the figures can be found at https://github.com/nkeriven/random-graph-gnn .

2 Background on Random Graphs and Graph Neural Networks

Let us start with generic notations and definitions. The norm ∥·∥ is the Euclidean norm for vectors and the operator norm for matrices and compact operators between Hilbert spaces. The latent space X is a compact metric set with a probability distribution P over it. Square-integrable functions from X to R q w.r.t. P are denoted by L 2 q , and are equipped with the Hilbertian norm ∥ f ∥ 2 L 2 def. = ∫ X ∥ f ( x ) ∥ 2 dP ( x ) .

The (disjoint) union of multidimensional functions L 2 ⊔ def. = ⊔ q ∈ N ∗ L 2 q is a metric space for a metric defined as ∥ f -g ∥ L 2 if f, g ∈ L 2 q for some q , and 1 otherwise. Continuous Lipschitz functions between metric spaces X → Y are denoted by C Lip ( X , Y ) . For X = { x 1 , . . . , x n } where x i ∈ X , we define the sampling of f : X → R d as ι X f = [ f ( x i )] n i =1 ∈ R n × d . Given Z ∈ R n × d , the Frobenius norm is ∥ Z ∥ F and we define the normalized Frobenius norm as ∥ Z ∥ MSE = n -1 2 ∥ Z ∥ F . The notation comes from the fact that ∥ ι X ( f -f ⋆ ) ∥ 2 MSE = n -1 ∑ i ∥ f ( x i ) -f ⋆ ( x i ) ∥ 2 which is akin to a Mean Square Error.

Latent position Random Graphs. In this paper, we consider latent position random graphs [20, 28, 33], a family of models that includes Stochastic Block Models (SBM), graphons, random geometric graphs, and many other examples. They are the primary models used for the study of GNNs in the literature [24, 25, 29, 43]. We generate a graph G = ( X,A,Z ) , where X ∈ R n × d are unobserved latent variables, A ∈ { 0 , 1 } n × n its symmetric adjacency matrix, and Z ∈ R n × p are (optional) observed node features. The latent variables and adjacency matrix are generated as such:

where w : X × X → [0 , 1] is a continuous connectivity kernel and α n is the sparsity-level of the graph, such that the expected degrees are in O ( n 2 α n ) . Non-dense graph can be obtain with α n = o (1) , here we will go down to the relatively sparse case α n ≳ (log n ) /n , a classical choice in the literature [28, 24]. Note that the continuity hypothesis of the kernel w is not really restrictive: neither X nor the support of the distribution P need be connected. For instance, SBMs can be obtained by taking X to be a finite set. We do not specify a model for the node features yet, see Sec. 4.

Graph shift matrix and operator. When the number of nodes n grows on random graphs, it is known that certain discrete operators associated to the graph converge to their continuous version, as well as the GNNs that employ them [24, 8]. Here, some of our results will be valid under quite general assumptions, hence we use generic notations for our graph representations. When the results are only valid for particular examples, this will be specifically expressed.

We consider a graph shift matrix [46] S = S ( G ) ∈ R n × n , which can be either directly the adjacency matrix of the graph or various notions of graph Laplacians. We define an associated graph shift operator S : L 2 ⊔ → L 2 ⊔ such that the restriction S | L 2 q is a compact linear operator of L 2 q onto itself. Note that we reserve 'matrix' and 'operator' respectively for the discrete and continuous versions. The results in Sec. 3 will be valid under generic convergence assumptions from S to S , while the results of Sec. 4 will focus on the following two representative examples.

Example 1 (Normalized adjacency matrix and kernel operator) . Here S = ˜ A = ( nα n ) -1 A and S f = A f = ∫ w ( · , x ) dP ( x ) . This choice requires to know, or estimate, the sparsity level α n . In this case, our results will hold whenever α n ≳ (log n ) /n with an arbitrary multiplicative constant.

Note that this choice requires to know (or estimate) the parameter α n , otherwise we will not have convergence between S and S , which can be limiting. This is not the case of the next example.

Example 2 (Normalized Laplacian matrix 1 and operator) . Here S = L = D -1 / 2 A AD -1 / 2 A where D A = diag( A 1 n ) is the degree matrix of G , and S f = L f = ∫ w ( · ,x ) √ d ( · ) d ( x ) dP ( x ) where d ( · ) = ∫ w ( · , x ) dP ( x )

is the degree function . Whenever we opt for this choice, we assume that d min def. = inf X d ( x ) > 0 , and our results will hold whenever α n ⩾ C (log n ) /n with a multiplicative constant C that depends (in a non-trivial way) on w , see Thm. 9 in App. D.

To sometimes unify notations, when we adopt these examples, we define w S such that w S ( x, y ) = w ( x, y ) in the adjacency case and w S ( x, y ) = w ( x,y ) √ d ( x ) d ( y ) in the normalized Laplacian case. Therefore for these two examples the continuous operator has a single expression S f = ∫ w S ( · , x ) dP ( x ) .

Graph Neural Network. As mentioned in the introduction, we focus on equivariant GNNs that can compute functions over nodes , as this makes the most sense on large graphs that RGs seek to model. Recall that we observe a graph shift operator S and node features Z ∈ R n × p , and we return a vector per nodes Φ( S, Z ) ∈ R n × d L . We adopt a traditional GNN that uses the graph shift matrix S : given input features Z (0) ∈ R n × d 0 ,

where ρ is the ReLU function applied element-wise, and θ ( ℓ ) i ∈ R d ℓ × d ℓ +1 , b ( ℓ ) ∈ R d ℓ are learnable parameters gathered in θ ∈ Θ . We denote by Θ the set of all possible parameters. For all classic choices of S , our definition of GNNs are a special case of message-passing NN (MPNN), which can be defined with a more general 'aggregation' function. For the two examples above (adjacency and Laplacian), the aggregation function used is a sum, or a normalized sum.

We note that here we employ the ReLU function as a non-linearity, as some of our results will use its specific properties. Multi-Layer Perceptrons (MLP, densely connected networks) using the ReLU activation, and with potentially more than one hidden layer, will be denoted by f MLP γ , where γ gathers their parameters.

Following recent literature [12, 13], we consider inputing Positional Encoding (PE) at each node. Such PE are generally computed using only the graph structure and concatenated to existing node features Z , here we simply introduce a generic notation:

with some parameter γ ∈ Γ . In our notations, the PE module uses the node features Z , generally by concatenating them to its output. For short, we may denote the whole architecture with PE and GNN as Φ θ,γ ( S, Z ) def. = Φ θ ( S, PE γ ( S, Z )) . It is not difficult to see that if the PE computation is equivariant, then the whole GNN is equivariant: denoting by σ a permutation matrix of { 1 , . . . , n } ,

All the examples of PEs examined in Sec. 4 are equivariant.

3 Function spaces of Graph Neural Networks

In this section, we provide a complete and intuitive description of the function space approximated by equivariant GNNs applied on RGs. All technical proofs are provided in App. A. It has been shown [24, 25, 8, 29] that GNNs converge to functions over the latent space: when the node features are a sampling of a certain function ι X f (0) , then the output of the GNN is close to being a sampling of another function ι X f ( L ) . Assuming the node features or PEs approximate some function set B ⊂ L 2 ⊔ , we define the space of functions that a GNN can approximate as follows.

Definition 1. Given a base set B ⊂ L 2 ⊔ , the set of functions approximated by GNNs F GNN ( B ) is formed by all the functions f ∈ L 2 ⊔ such that: for all ε > 0 , there are θ ∈ Θ , f (0) ∈ B such that

In other words, F GNN ( B ) are the functions whose sampling can be ε -approximated by the output of a GNN, with probability going to 1 as n grows. Note that if the quantifiers of θ, f (0) and ε were reversed, the MSE would converge to 0 in probability. Here this is not the case: θ, f (0) may depend on ε , which is akin to an approximation level. Similar to the permutation equivariance of GNNs, there is a notion of continuous equivariance for functions well-approximated by GNNs [24, 25, 8], where the permutations are replaced by bijections over the latent space X . We adopt the notations F GNN ( B ) = F GNN ( B , w, P ) . For all continuous bijections ϕ over X , we define w ϕ ( x, y ) = w ( ϕ ( x ) , ϕ ( y )) , P ϕ = ϕ -1 ♯P where ♯ is the push-forward operation, and B ϕ = { f ◦ ϕ | f ∈ B} . Then, we have the following result.

Proposition 1. Let S = S ( A ) be a graph shift operator that only depends on the adjacency matrix of the graph in a permutation-equivariant manner. Then, for all continuous bijections ϕ : X → X ,

That is, if one 'permutes' the kernel w , the distribution P and the base set B , then the function space F GNN contains exactly the permuted version of the original space.

The goal of this section is to provide a more intuitive description of the space F GNN, which we will do under some basic convergence assumption from S to S . GNNs (2) basically include two components: dense connections and MLPs that can approximate any continuous function by the universality theorem [40], and applications of S . Hence, we define the following function space.

Definition 2. We define F S ( B ) ⊂ L 2 ⊔ the (minimal) S -extension of a base set B ⊂ L 2 ⊔ by the following rules:

(i) Base space: B ⊂ F S ( B ) ;

(ii) Stability by composition with continuous functions: for all f ∈ F S ( B ) with a p -dimensional output and g ∈ C Lip ( R p , R q ) , it holds 2 that g ◦ f ∈ F S ( B ) ;

(iii) Stability by graph operator: for all f ∈ F S ( B ) , it holds that S f ∈ F S ( B ) ;

(iv) Linear span: for all q , F S ( B ) ∩ L 2 q is a vector space;

(v) Closure: F S ( B ) is closed in L 2 ⊔ ;

(vi) Minimality: for all G ⊂ L 2 ⊔ satisfying all the properties above, F S ⊂ G .

In words, F S take a base set B , and extend it to be stable by composition with Lipschitz functions, application of the graph operator, and linear combination (of its elements with the same dimensionality). Our result will use the following assumption, which is naturally true for our running examples.

Assumption 1. With probability going to 1 , ∥ S ∥ is bounded. Moreover, for all f ∈ L 2 ⊔ ,

where P - → indicates convergence in probability.

Proposition 2. Assumption 1 is true for the adjacency matrix (ex. 1) and normalized Laplacian (ex. 2).

Under this assumption, the main result of this section states that the functions well-approximated by GNNs are exactly the S -extension of the base input features B .

Theorem 1. Under Assumption 1, for all B ⊂ L 2 ⊔ , we have:

Given the definition of GNNs (2) and construction of F S , Theorem 1 appears quite natural. Its proof, provided in App. A.3, is however far from trivial. The inclusion F S ( B ) ⊂ F GNN ( B ) is similar in spirit to previous convergence results [24], since one has to construct a GNN that approximates a particular function. It involves however a new extended universality theorem for MLPs for square-integrable functions (Lemma 3 in App. A.3), which uses the special properties of ReLU . The reverse inclusion F GNN ( B ) ⊂ F S ( B ) is quite different from previous work on GNN convergence: given f ∈ F GNN ( B ) whose only property is to be well-approximated by GNNs, one must construct a sequence of functions in F S ( B ) that converge to f , and uses the closure of F S ( B ) . The need to work within square-integrable function is here obvious, as we only have convergence of the MSE, an approximation of the L 2 -norm. For instance, this inclusion would not be true in the space of continuous functions.

Using composition with continuous functions, if F S ( B ) contains a continuous bijection ϕ : X → Im ( ϕ ) , then F S ( B ) contains all continuous functions, and by density all square integrable functions. That is, the equivariant GNNs are then universal over X : they can generate any function to label the nodes. Another criterion using the Stone-Weierstrass theorem (e.g. [21]), similar to the proofs in [25], is the following.

̸

Proposition 3. Assume that for all x = x ′ in X , there is a continuous function f ∈ F S ( B ) ∩ C Lip ( X , R ) such that f ( x ) = f ( x ′ ) . Then, F S ( B ) = L 2 ⊔ .

̸

In the rest of the paper, we study several examples of PEs and corresponding set B , that will generalize the results of [25]. We expect many other interesting characteristics of F S to be derived in the future.

4 Node features and Positional encodings

In the previous section, we have provided a complete description of the function space generated by equivariant GNNs when fed samplings of functions as node features, and the set of B is thus crucial for the properties of F S ( B ) . For instance, in the absence of node features and PEs, it is classical to input constant features to GNNs [25], such that the space of interest is F S (1) . However, similar to the failure of the WL test on regular graphs, if S 1 ∝ 1 (e.g. constant degree function), then F S (1) contains only constant functions ! The role of PEs is often to mitigate such situations.

Definition 3. The set of functions approximated by PEs F PE is formed by all the functions f ∈ L 2 ⊔ such that: for all ε > 0 , there is γ ∈ Γ such that

Note that, as before, γ may depend on ε . When passing PEs as input to GNNs, F PE serves as the base space B , and the space of interest to characterize the functions well approximated by the whole

architecture Φ θ,γ is therefore F S ( F PE ) . In fact, by simple Lipschitz property: for any f ∈ F S ( F PE ) and ε > 0 , there are θ ∈ Θ , γ ∈ Γ such that

In the rest of the section, we therefore aim to characterize F PE for several representative examples. We first briefly comment on observed node features, then move on to PEs. Proofs are in App. B.

4.1 Node features

A first, simple example, is when observed node features are actually a sampling of some function Z = ι X f (0) . This is a convenient choice that is often adopted in the literature [24, 25, 23, 8, 29]. In this case, by adopting the identity PE γ ( S, Z ) = Z , it is immediate that F PE = { f (0) } . A more realistic example is the presence of centered noise:

where ν = [ ν 1 , . . . , ν n ] and the ν i are i.i.d. noise vectors with E ν i = 0 and Cov ( ν i ) = C ν . This time, F PE cannot contain directly f (0) , as the Law of Large Numbers (LLN) gives

However, when applying the graph shift matrix at least once, one obtains convergent PEs.

Proposition 4. Consider the adjacency matrix (ex. 1) or normalized Laplacian (ex. 2). If the node features are a noisy sampling (6) and the PE are defined PE γ ( S, Z ) = SZ , then, F PE = { S f (0) } .

Of course this may not be the only possibility for removing noise from node features, and moreover it is not clear how realistic the node features model (6) actually is. The study of more refined models linking graph structure and node features is a major path for future work.

4.2 Positional Encodings

In this section, we consider classical PEs computed solely from the graph structure and show how they articulate with our framework. We consider two examples that are the most-often used in the literature: PEs as eigenvectors of the graph shift matrix [12, 13] (actually a recent variant that account for sign indeterminancy [31]), and PE based on distance-encoding [30] (again a variant that, as we will see, generalize other architectures [49]). For most of the results below, we will focus on two representative cases of kernels, that include many practical examples. We remark that these yield sufficient conditions to establish our results, but by no mean necessary. Other cases could be examined in future work.

Example a (Stochastic Block Models) . In this case, the space of latent variables X = { 1 , . . . , K } is finite, each element correspond to a community label. The kernel w is represented by a matrix C def. = [ w ( ℓ, k )] ∈ R K × K + that gives the probability of connection between communities ℓ and k , and P ∈ R K + is a probability vector of size K that sum to 1 .

Example b (P.s.d. kernel) . Here we assume that w is positive semi-definite (p.s.d.). This includes for instance the Gaussian kernel.

For any symmetric matrix (resp. self-adjoint compact operator) M , we denote by λ i M its eigenvalues and u i M its eigenvectors (resp. eigenfunctions), with any arbitrary choice of sign or basis here. Since in all our examples operators are either p.s.d. or finite-rank, the eigenvalues are ordered as such: first the non-zero eigenvalues by decreasing order (from positive to negative), then all zero eigenvalues.

4.2.1 Eigenvectors and SignNet

It has been proposed [12, 13] to feed the first q eigenvectors of the graph into the GNN, for a fixed q . A potential problem with this approach is the sign ambiguity of the eigenvectors, or even the basis ambiguity in case of eigenvalues with multiplicities. Here we consider only the sign ambiguity for simplicity: we will assume that the first eigenvalue of S are distinct. The sign ambiguity was alleviated in [31] by taking a sign-invariant function: considering an eigenvector u S i of S ,

1.00

0.75

0.50

0.25 0.00

0.25

0.50

0.75

1.00

1.5

1.0

0.5

0.0

0.5

1.0

1.5

Train

Test

1.00

0.75

0.50

0.25 0.00

0.25

0.50

0.75

1.00

0.10

0.05

0.00

0.05

0.10

Train

Test

1.00

0.75

0.50

0.25 0.00

0.25

0.50

0.75

1.00

2.5

2.0

1.5

1.0

0.5

0.0

0.5

test error=0.018

1.00

0.75

0.50

0.25 0.00

0.25

0.50

0.75

1.00

0.4

0.2

0.0

0.2

0.4

0.6

0.8

test error=0.565

where f : R → R p is a function applied to each coordinate of u S i to preserve permutation-equivariance. The resulting function is sign-invariant, and one can parameterized f . Given the first q eigenvectors u S i and a collection of MLPs f MLP γ i : R → R p i for some output dimensions p i , the PE considered in this subsection concatenates the outputs:

where p = ∑ q i =1 p i and the MLP are applied element-wise. The parameter γ gathers the γ i . The equation (8) involves a renormalization of the eigenvectors u S by the square root of the size of the graph √ n : indeed, as u S i is normalized in R n , this is necessary for consistency across different graph sizes. See Sec. 4.2.3 for a discussion and some numerical illustrations.

As can be expected, the eigenvectors of S generally converge to the eigenfunctions of S , under a spectral gap assumption. We provide the theorem below which handles all of our running examples. We suppose that the relevant eigenvalues have single multiplicities, to only have sign ambiguity.

Theorem 2. Consider either SBM (ex. a) or p.s.d. kernel (ex. b), and either adjacency matrix (ex. 1) or normalized Laplacian (ex. 2). Fix q , assume the first q +1 eigenvalues λ S 1 , . . . , λ S q +1 of S are two-by-two distinct. We define

Hence F PE contains the eigenfunctions of S , modified by the SignNet architecture to account for the sign indeterminancy. We further discuss this space in Sec. 4.2.3. An illustration is provided in Fig. 1.

4.2.2 Distance-encoding PEs

In [30], the authors propose to define PEs through the aggregation of a set of 'distances' ξ ( i, j ) from each node i to a set j ∈ V T of target nodes (typically, labelled nodes in semi-supervised learning, or anchor nodes selected randomly [56]):

where AGG is an aggregation function that acts on (multi-)sets, and ξ ( i, j ) is selected in [30] as randomwalk based distances ξ ( i, j ) = [( AD -1 A ) ij , . . . , (( AD -1 A ) q ) ij ] ∈ R q . For simplicity, since here we do not consider any particular set of target nodes, we just consider V T = V the set of all nodes. Moreover, to use our convergence results, we replace the random walk matrix with our graph shift matrix S . As aggregation, we opt for the deep-set architecture [58], which applies an MLP on each ξ ( i, j ) then a sum. Deep sets can approximate any permutation-invariant function. As we will see below, with the proper normalization to ensure convergence, we obtain:

where f MLP γ : R q → R p is applied row-wise and e j ∈ R n are one-hot basis vectors. We note that a similar architecture was proposed in a different line of work: it was called Structured Message Passing by [49], or Structured GNN by [25]. In these works, the inspiration is to give nodes unique identifiers,

e.g. , one-hot encodings e i . However, this process is not equivariant. To restore equivariance, [49] propose a deep-set pooling in the 'node-id' dimension PE γ ( S ) = ∑ j Φ γ ( S, e j ) , where Φ γ is itself a permutation-equivariant GNN, and the equivariance of PE γ is restored. By choosing Φ γ ( S , e j ) = n -1 f MLP γ ( n · [ Se j , . . . , S q e j ]) (which is a valid choice for a message-passing GNN), we obtain exactly distance-encoding PEs above.

In [25], powerful universality results were shown for this choice of architecture in the case of non-random edges a ij = w ( x i , x j ) and q = 1 . With our notations, they implicitely studied PE functions of the following form: ∫ f ( w ( · , x )) dP ( x ) . This allows to modify the values of the kernel before computing the degree function, and can therefore break potential indeterminancy such as constant degrees. Unfortunately, their proof technique and the concentration inequalities they use are not true anymore for Bernoulli random edges , which are far more realistic than deterministic weighted edges. Here we show that for a large class of kernels, concentration can be restored when we add an MLP filter on the eigenvalues of S with ReLU. Our definition of distance-encoding PEs is therefore:

where S γ 2 def. = h f MLP γ 2 ( S ) is a filter that applies an MLP f MLP γ 2 on the eigenvalues of S .

Theorem 3. Consider either SBM (ex. a) or p.s.d. kernel (ex. b), and either adjacency matrix (ex. 1) or normalized Laplacian (ex. 2). Consider the PE (10) . We define

where S δ x def. = { z ↦→ w S ( z, x ) } by abuse of notation. Then F Dist ⊂ F PE .

Note that here we only have an inclusion F Dist ⊂ F PE instead of an equality as in Thm. 2: indeed, we show that the PE (10) can approximate functions in F Dist, but they may converge to other functions. Nevertheless, as a consequence of our analysis, all the universality results of [25, Sec. 5.3] are valid with the choice of PE (10), see Appendix C for a reminder using our notations. This is a strict, and non-trivial improvement over [25], as their results were only derived for non-random edges. For this, Theorem 3 relies mostly on a new concentration inequality for Bernoulli matrices with ReLU filters in Frobenius norm, that we give below since it is of independent interest.

Theorem 4. Consider either SBM (ex. a) or p.s.d. kernel (ex. b), and either adjacency matrix (ex. 1) or normalized Laplacian (ex. 2). Define the Gram matrix W = [ w S ( x i , x j ) /n ] ij . For all ε > 0 , there is an MLP filter S γ = h f MLP γ ( S ) such that

The proof of this theorem, given in appendix B.3, is inspired by the so-called USVT estimator [7]. One notes that the use of an MLP graph filter is quite unconventional. A more classical choice is polynomial filters: this avoids the diagonalization of S by computing ∑ k a k S k , it is for instance the basis for the ChebNet architecture [10]. For the purpose of Theorems 3 and 4, polynomial filters do not work, and ReLU is of crucial importance : indeed, we need the filter to zero-out O ( n ) eigenvalues uniformly in some interval [ -τ, τ ] . This cannot be done with polynomials with a fixed number of parameters and growing n →∞ . On the other hand, when choosing f as an MLP with ReLU, due to the shape of this non-linearity, f MLP γ 2 can be uniformly 0 on a whole domain . Of course, polynomial filters offer great computational advantages, and perform well in practice, despite their flaw in our asymptotic analysis. Moreover, ReLU is technically non-differentiable. Designing filters that offer both computational advantages and exact approximation is still an open question. In practice, we observe that the ReLU-filter does learn to approximate its expected shape, when we

0.05

0.00

0.05

0.10

0.15

0.05

0.00

0.05

0.10

0.15

No filter

Learned filter

Ideal filter

minimize the reconstruction error ∥ S γ -W ∥ F on synthetic data where W is known, see Fig. 2.

4.2.3 Discussion

Approximation power. As mentioned above, in the absence of node features, one may opt for constant input, but this may lead to degenerate situations. PEs aim to counteract that, by increasing GNNs' approximation power. We quickly verify that this is indeed the case for our two examples.

Proposition 5. There are cases where F S (1) ⊂F S ( F Eig ) or F S (1) ⊂F S ( F Dist ) with strict inclusions.

Moreover, as mentioned in the previous section, existing universality results [25] can be generalized in our case, see App. C. Another interesting question is somewhat the opposite: given the already rich class of functions generated by PEs, are GNNs really more powerful?

Proposition 6. There are cases where F Eig ⊂ F S ( F Eig ) or F Dist ⊂ F S ( F Dist ) , with strict inclusions.

The proof, which is not so trivial, invokes functions with at least one round of message-passing after the computation of PEs, so the additional approximation power does not come only from MLPs. Intuitively, it seems natural that message-passing rounds are useful for other reasons, e.g. noise reduction or smoothing [23]. We leave these complementary lines of investigation for future work.

Renormalization. A striking point in our variants of PEs is the presence of various normalization factors by the graph size n to ensure convergence: the equation (8) involves a renormalization of the eigenvectors u S by the square root of the size of the graph √ n , while (10) involves a multiplicative factor n inside the MLP f MLP γ 1 (the 1 /n outside of the sum is more classical). Our analysis shows that these normalization factors are necessary for convergence when n →∞ , and more generally for consistency across different graph sizes.

In practice, this is generally not used. Indeed, if the training and testing graphs have roughly the same 'range' of sizes n ∈ [ n min , n max ] , then a GNN model can learn the proper normalization to perform, which is not the point of view of our analysis n →∞ . While in-depth benchmarking of PEs has been done in the literature [13] and is out-of-scope of this paper, we give a small numerical illustration of the effect of normalization in Table 1. We consider a synthetic dataset with a classic classification problem on (unobserved) latent variables x i and a Gaussian kernel W . To emphasize the effect of the normalization, we also examine a situation where the test graphs are much larger than the training graphs while following the same model, which we denote by out-of-dist . Concerning real data, since there are practically no datasets for node-classification with several graphs of sufficiently different size to test the renormalization, we artificially extract many subgraphs from a single large graph ( Citeseer ) with labelled nodes to create such a dataset, denoted by Citeseer-subgraphs . We also directly look at two graph-classification datasets with many graphs of different sizes. Note that, to emphasize the effect of PEs, we discard eventual node features and use only the graph structure.

On synthetic data exactly formed of random graphs of vastly different sizes, the renormalization is of course necessary to obtain good performance, as predicted by our theory: without it, the PEs do not converge when n grows. On real data, we see that renormalization generally improve performance, and this is more true for IMDB-BINARY, which contains a larger range of graph sizes, and distance-based PEs. Note that here we use relatively small GNNs that are not state-of-the-art (in particular since we do not use node features), as well as a different train/test split than most papers ( K = 5 CV-folds instead of K = 10 ): indeed, we do not want our models to learn the proper normalization on the limited range of sizes n in the dataset, so we limit their number of parameters and use a smaller training set. We do not expect our simple renormalization process to make a significant difference on large-scale benchmarks with state-of-the-art models [13], but this is a pointer in an interesting direction to be explored in the future. In particular, this type of normalization may be useful in real-world scenarii where the test graphs are far larger than the labelled training graphs.

5 Conclusion

On large random graphs, the manner in which GNNs label nodes can be modelled by functions. The analysis of the resulting function spaces is still in its infancy, and of a very different nature to the studies of graph-tasks , both discrete [54] or in the limit [36]. In this paper, we clarified significantly the nature of the space of functions well-approximated by GNNs on large-graphs, showing that it can be defined by a few extension rules within the space of square-integrable functions. We then showed the usefulness of Positional Encodings by analyzing two popular examples, established new universality results, as

well as some concentration inequalities of independent interest. Our theory hinted at some process for consistency across graphs of different sizes that can help generalization in practice.

This paper, which in large part consisted in properly defining the objects of interest, is without doubt only a first step in their analysis. Future studies might look at specific settings and derive more useful properties of the space F S , more powerful PEs, a better understanding of their limitations, or more realistic models for node features. In particular, a better connection with the existing WL-based theory for finite small graphs, and associated 'powerful' architectures, is a major path for future work.

Acknowledgments

The authors acknowledge the fundings of ANR GraVa ANR-18-CE40-0005 and ANR GrandMa ANR21-CE23-0006.

References

J. Arjona Martínez, O. Cerri, M. Spiropulu, J. R. Vlimant, and M. Pierini. Pileup mitigation at the Large Hadron Collider with graph neural networks. European Physical Journal Plus , 134(7):1-12, 2019.

Aseem Baranwal, Kimon Fountoulakis, and Aukosh Jagannath. Graph Convolution for SemiSupervised Classification: Improved Linear Separability and Out-of-Distribution Generalization. pages 1-30, 2021.

Aseem Baranwal, Kimon Fountoulakis, and Aukosh Jagannath. Effects of Graph Convolutions in Multi-layer Networks. pages 1-31, 2022.

Giorgos Bouritsas, Stefanos Zafeiriou, Fabrizio Frasca, and Michael M. Bronstein. Improving Graph Neural Network Expressivity via Subgraph Isomorphism Counting. arXiv:2006.09252 , 2020.

Michael M. Bronstein, Joan Bruna, Taco Cohen, and Petar Veliˇ ckovi´ c. Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges. 2021.

Quentin Cappart, Didier Chételat, Elias Khalil, Andrea Lodi, Christopher Morris, and Petar Veliˇ ckovi´ c. Combinatorial optimization and reasoning with graph neural networks. pages 1-43, 2021.

Sourav Chatterjee. Matrix estimation by Universal Singular Value Thresholding. Annals of Statistics , 43(1):177-214, 2015.

Matthieu Cordonnier, Nicolas Keriven, Nicolas Tremblay, and Samuel Vaiter. Convergence of Message Passing Graph Neural Networks with Generic Aggregation On Large Random Graphs. pages 1-39, 2023.

Harry Crane. Probabilistic Foundations of Statistical Network Analysis . Chapman & Hall, 2018.

Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst. Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering. In Advances in Neural Information and Processing Systems (NIPS) , 2016.

Xin Luna Dong, Xiang He, Andrey Kan, Xian Li, Yan Liang, Jun Ma, Yifan Ethan Xu, Chenwei Zhang, Tong Zhao, Gabriel Blanco Saldana, Saurabh Deshpande, Alexandre Michetti Manduca, Jay Ren, Surender Pal Singh, Fan Xiao, Haw Shiuan Chang, Giannis Karamanolakis, Yuning Mao, Yaqing Wang, Christos Faloutsos, Andrew McCallum, and Jiawei Han. AutoKnow: Self-Driving Knowledge Collection for Products of Thousands of Types. Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , pages 2724-2734, 2020.

Vijay Prakash Dwivedi and Xavier Bresson. A Generalization of Transformer Networks to Graphs. 2020.

Vijay Prakash Dwivedi, Anh Tuan Luu, Thomas Laurent, Yoshua Bengio, and Xavier Bresson. Graph Neural Networks with Learnable Structural and Positional Representations. In ICLR , pages 1-25, 2022.

Gerald B. Folland. Real Analysis . Wiley-Interscience, 1999.

Kimon Fountoulakis, Dake He, Silvio Lattanzi, Bryan Perozzi, Anton Tsitsulin, and Shenghao Yang. On Classification Thresholds for Graph Attention with Edge Features. pages 1-37, 2022.

Alex Fout, Basir Shariat, Jonathon Byrd, and Asa Ben-Hur. Protein Interface Prediction using Graph Convolutional Networks. In Neural Information Processing Systems (NeurIPS) , pages 6512-6521, 2017.

Michelle Girvan and M. E. J. Newman. Community structure in social and biological networks. Proceedings of the National Academy of Sciences of the United States of America , 99(12):7821 -7826, 2002.

Anna Goldenberg, Alice X. Zheng, Stephen E. Fienberg, and Edoardo M. Airoldi. A survey of statistical network models. Foundations and Trends in Machine Learning , 2(2):129-233, 2009.

Desmond J Higham, Marija Rašajski, and Nataša Pržulj. Fitting a geometric graph to a proteinprotein interaction network. Bioinformatics , 24(8):1093-1099, 2008.

Peter D. Hoff, Adrian E. Raftery, and Mark S. Handcock. Latent space approaches to social network analysis. Journal of the American Statistical Association , 97(460):1090-1098, 2002.

Kurt Hornik, Maxwell Stinchcombe, and Halbert White. Multilayer Feedforward Networks are Universal Approximators. Neural Networks , 2:359-366, 1989.

Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open Graph Benchmark: Datasets for Machine Learning on Graphs. Neural Information Processing Systems (NeurIPS) , (NeurIPS):1-34, 2020.

Nicolas Keriven. Not too little, not too much: a theoretical analysis of graph (over)smoothing. Advances in Neural Information Processing Systems (NeurIPS) , 2022.

Nicolas Keriven, Alberto Bietti, and Samuel Vaiter. Convergence and Stability of Graph Convolutional Networks on Large Random Graphs. In Advances in Neural Information and Processing Systems (NeurIPS) , pages 1-26, 2020.

Nicolas Keriven, Alberto Bietti, and Samuel Vaiter. On the Universality of Graph Neural Networks on Large Random Graphs. In Advances in Neural Information and Processing Systems (NeurIPS) , 2021.

Nicolas Keriven and Gabriel Peyré. Universal Invariant and Equivariant Graph Neural Networks. In Advances in Neural Information Processing Systems (NeurIPS) , pages 1-19, 2019.

M. Kuperman and G. Abramson. Small world effect in an epidemiological model. Physical Review Letters , 86(13):2909-2912, 2001.

Jing Lei and Alessandro Rinaldo. Consistency of spectral clustering in stochastic block models. Annals of Statistics , 43(1):215-237, 2015.

Ron Levie, Wei Huang, Lorenzo Bucci, Michael Bronstein, and Gitta Kutyniok. Transferability of spectral graph convolutional neural networks. Journal of Machine Learning Research , 22:1-41, 2021.

Pan Li, Yanbang Wang, Hongwei Wang, and Jure Leskovec. Distance encoding: Design provably more powerful neural networks for graph representation learning. Advances in Neural Information Processing Systems , 2020-Decem(1):1-29, 2020.

Derek Lim, Joshua Robinson, Lingxiao Zhao, Tess Smidt, Suvrit Sra, Haggai Maron, and Stefanie Jegelka. Sign and Basis Invariant Networks for Spectral Graph Representation Learning. pages 1-42, 2022.

Andreas Loukas. What graph neural networks cannot learn: depth vs width. In ICLR , 2020.

László Lovász. Large networks and graph limits. Colloquium Publications , 60:487, 2012.

Haggai Maron, Heli Ben-Hamu, Hadar Serviansky, and Yaron Lipman. Provably Powerful Graph Networks. In Advances in Neural Information Processing Systems (NeurIPS) , pages 1-12, 2019.

Haggai Maron, Ethan Fetaya, Nimrod Segol, and Yaron Lipman. On the Universality of Invariant Networks. In International Conference on Machine Learning (ICML) , 2019.

Sohir Maskey, Yunseok Lee, Ron Levie, and Gitta Kutyniok. Generalization Analysis of Message Passing Neural Networks on Large Random Graphs. pages 1-63, 2022.

Eli A Meirom, Haggai Maron, Shie Mannor, and Gal Chechik. How to Stop Epidemics: Controlling Graph Dynamics with Reinforcement Learning and Graph Neural Networks. pages 1-23, 2020.

Pál András Papp and Roger Wattenhofer. A Theoretical Comparison of Graph Neural Network Extensions. 2022.

Lorenzo Pellis, Frank Ball, Shweta Bansal, Ken Eames, Thomas House, Valerie Isham, and Pieter Trapman. Eight challenges for network epidemic models. Epidemics , 10:58-62, 2015.

Allan Pinkus. Approximation theory of the MLP model in neural networks. Acta Numerica , 8(May):143-195, 1999.

Martin Ringsquandl, Houssem Sellami, Marcel Hildebrandt, Dagmar Beyer, Sylwia Henselmeyer, Sebastian Weber, and Mitchell Joblin. Power to the Relational Inductive Bias: Graph Neural Networks in Electrical Power Grids , volume 1. Association for Computing Machinery, 2021.

Lorenzo Rosasco, Mikhail Belkin, and Ernesto De Vito. On learning with integral operators. Journal of Machine Learning Research , 11:905-934, 2010.

Luana Ruiz, Luiz F. O. Chamon, and Alejandro Ribeiro. Graphon Signal Processing. arXiv:2003.05030 , pages 1-13, 2020.

Luana Ruiz, Luiz F.O. Chamon, and Alejandro Ribeiro. Graphon neural networks and the transferability of graph neural networks. In Advances in Neural Information and Processing Systems (NeurIPS) , 2020.

Alvaro Sanchez-Gonzalez, Nicolas Heess, Jost Tobias Springenberg, Josh Merel, Martin Riedmiller, Raia Hadsell, and Peter Battaglia. Graph networks as learnable physics engines for inference and control. arxiv:1806.01242 , 2018.

Aliaksei Sandryhaila and José MF Moura. Discrete signal processing on graphs. IEEE Trans. Sig. Proc. , 61(7):1644-1656, 2013.

Ryoma Sato, Makoto Yamada, and Hisashi Kashima. Random features strengthen graph neural networks. SIAM International Conference on Data Mining, SDM 2021 , pages 333-341, 2021.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 30. Curran Associates, Inc., 2017.

Clément Vignac, Andreas Loukas, and Pascal Frossard. Building powerful and equivariant graph neural networks with message-passing. In Advances in Neural Information and Processing Systems (NeurIPS) , 2020.

Jizhe Wang, Pipei Huang, Huan Zhao, Zhibo Zhang, Binqiang Zhao, and Dik Lun Lee. Billion-scale commodity embedding for E-commerce recommendation in alibaba. Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , pages 839-848, 2018.

B Yu Weisfeiler and A A Leman. The Reduction of a Graph to Canonical Form and the Algebra Which Appears Therein. Nti , 2(9):12-16, 1968.

Felix Wu, Tianyi Zhang, Amauri Holanda de Souza, Christopher Fifty, Tao Yu, and Kilian Q. Weinberger. Simplifying graph convolutional networks. 36th International Conference on Machine Learning, ICML 2019 , 2019-June:11884-11894, 2019.

Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and Philip S. Yu. A Comprehensive Survey on Graph Neural Networks. IEEE Transactions on Neural Networks and Learning Systems , pages 1-21, 2020.

Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How Powerful are Graph Neural Networks? In ICLR , pages 1-15, 2019.

Pinar Yanardag and S.V.N. Vishwanathan. Deep Graph Kernels. pages 1365-1374, 2015.

Jiaxuan You, Rex Ying, and Jure Leskovec. Position-aware Graph Neural Networks. 2019.

Y. Yu, T. Wang, and R. J. Samworth. A useful variant of the Davis-Kahan theorem for statisticians. Biometrika , 102(2):315-323, 2015.

Manzil Zaheer, Satwik Kottur, Siamak Ravanbhakhsh, Barnabás Póczos, Ruslan Salakhutdinov, and Alexander J. Smola. Deep sets. Advances in Neural Information Processing Systems , 2017Decem(ii):3392-3402, 2017.

A Proofs of Sec. 3

A.1 Proof of Prop. 1

Denote G the distribution of ( X,A ) with ( w,P ) and G ϕ for ( w ϕ , P ϕ ) . Note that if X,A ∼ G ϕ , then ϕ ( X ) , A ∼ G , and recall that S = S ( A ) only depends on A .

Consider f ∈ F GNN ( B , w, P ) , ε > 0 , and θ, f (0) such that P G (∥ ∥ Φ θ ( S, ι X f (0) ) -ι X f ∥ ∥ MSE ⩾ ε ) → 0 . Then, denoting by ϕ ( X ) = { ϕ ( x 1 , . . . , ϕ ( x n ) } ,

which shows that f ◦ ϕ ∈ F GNN ( B ϕ , w ϕ , P ϕ ) and one inclusion. The other inclusion is immediate by doing the same reasoning for ϕ -1 .

A.2 Proof of Prop. 2

Define W = [ w S ( x i , x j ) /n ] the Gram matrix. Using Theorem 9, for both our examples we have

Since ∥ W ∥ ⩽ sup x,y | w S ( x, y ) | is bounded, it shows that ∥ S ∥ is bounded with probability going to 1 .

Let f ∈ L 2 q and any ε > 0 . Since continuous functions are dense in square-integrable functions on compact spaces (see e.g. [14, Sec. 8.2]), let g ∈ C Lip ( X , R q ) such that ∥ f -g ∥ L 2 ⩽ ε . We have

where we have used Lemma 8 and the fact that g is bounded.

Finally,

Using ∥ AB ∥ F ⩽ ∥ A ∥ ∥ B ∥ F and the LLN, and the fact that ∥ f ∥ L 2 , ∥ W ∥ , ∥ S ∥ are bounded, with probability going to 1 ,

which, since ε was chosen arbitrarily, concludes the proof.

A.3 Proof of Theorem 1

The proof uses intermediate results. Recall the definition of GNNs: given input node features Z (0) ∈ R n × d 0 ,

We can define a continuous equivalent of GNNs, called c-GNNs in the literature [24], using the operator S . Given f (0) ∈ L 2 d 0 ,

Then, under our assumption on the operators ( S, S ) , discrete GNNs converge to continuous GNNs.

Lemma 1. Suppose Assumption 1 holds. For all f ∈ L 2 ⊔ and θ ,

Proof. Writing Z (0) = ι X f and f (0) = f , we show by recursion on the layers that ∥ ∥ Z ( ℓ ) -ι X f ( ℓ ) ∥ ∥ MSE P - - - - → n →∞ 0 .

For ℓ = 0 , we have exactly ∥ ∥ Z (0) -ι X f (0) ∥ ∥ MSE = 0 . Assuming the convergence holds for ℓ -1 , we have,

using the Lipschitz property of ρ in the first line, and ∥ AB ∥ F ⩽ ∥ A ∥ ∥ B ∥ F after. The first term converges to 0 by recursion hypothesis since ∥ S ∥ is bounded with probability going to 1 , and the second converges to 0 by Assumption 1. This concludes the proof.

Lemma 2. Given a base space B ⊂ L 2 ⊔ , denote by F c ( B ) ⊂ L 2 ⊔ the following space of all functions f of the form:

for all k, L, d i . Then F c ( B ) is dense in F S ( B ) .

Proof. By definition, F c ⊂ F S since its contruction uses only rules that leave F S stable. Conversely, F c satisfies all the rules of stability of F S so by minimality F S ⊂ F c .

Lemma 3 (Universality in L 2 ) . Let f ∈ L 2 q and g ∈ C Lip ( R q , R p ) , for all ε > 0 , there exists an MLP f MLP γ that uses ReLU, with two hidden layers, such that

Proof. Denote by L g the Lipschitz constant of g . Let C k = [ -k, k ] q , and X k = { x ∈ X | f ( x ) ∈ C k } , and ξ k = ∫ X k ∥ f ( x ) ∥ 2 dP ( x ) . We have ξ k positive and increasing, and lim k →∞ ξ k = ∥ f ∥ 2 L 2 . Define k ε such that ξ k ε ⩾ ∥ f ∥ 2 L 2 -ε 2 1+ L 2 g + ∥ g (0) ∥ 2 , such that ∫ X c kε ∥ f ( x ) ∥ 2 dP ( x ) ⩽ ε 2 1+ L 2 g + ∥ g (0) ∥ 2 .

Since C k ε is compact, by the universality theorem [21, 40], there is an MLP f MLP γ ′ such that sup y ∈ C kε ∣ ∣ g ( y ) -f MLP γ ′ ( y ) ∣ ∣ ⩽ ε . Moreover, using the property of ReLU, it is easy to see that the following function can be implemented by an MLP:

Then, we define f MLP γ = f MLP γ ′′ ◦ f MLP γ ′ , where f MLP γ ′′ is applied coordinate-wise. As a result, we have f MLP γ ( y ) = f MLP γ ′ ( y ) on C k ε , and ∥ ∥ f MLP γ ( y ) ∥ ∥ ∞ ⩽ k ε outside. Then, we have

For the first term, since on X k ε we have f ( x ) ∈ C k ε , we use the approximation property and we have

For the second term, since ∥ f ( x ) ∥ 2 ⩾ dk 2 ε ⩾ 1 on X c k ε , we have

And for the third term, given the property of the built MLP,

which concludes the proof.

Lemma 4. Let f ∈ F c ( B ) . For all ε > 0 , there exists θ and f (0) ∈ B such that

Proof. Let f ∈ F c ( B ) be constructed as (12). Denote by L g ( ℓ ) i the Lipschitz constant of g ( ℓ ) i ∈ C Lip ( R d ℓ , R d ℓ +1 ) . Let ε > 0 .

We build the following continuous GNN: ¯ f (0) = f (0) , and

for well-chosen MLPs. We design them by increasing layer indices: assuming the MLPs up to layer ℓ -1 are choosen (i.e. ¯ f ( ℓ ) is chosen), we use Lemma 3 and choose θ ( ℓ ) i (which depends on θ (0) , . . . , θ ( ℓ -1) then) such that

Then we get

Hence by a simple recursion and since f (0) = ¯ f (0) we have

by our choice of ε ( ℓ ) , which concludes the proof.

Proof of Theorem 1. We start with the inclusion F S ⊆ F GNN. Let f ∈ F S ( B ) and ε > 0 . By Lemma 2, there is f c ∈ F c ( B ) constructed as (12) such that ∥ f -f c ∥ L 2 ⩽ ε/ 3 , and we use the weak law of large numbers to obtain that P ( ∥ ι X ( f -f c ) ∥ MSE ⩾ ε/ 3) - - - - → n →∞ 0 . . By Lemma 4, there exists θ such that ∥ ∥ Φ θ ( S , f (0) ) -f c ∥ ∥ L 2 ⩽ ε/ 3 . Again by the LLN, we have that P ( ∥ ∥ ι X (Φ θ ( S , f (0) ) -f c ) ∥ ∥ MSE ⩾ ε/ 3) → 0 . Finally, by Lemma 1), we also have

Using a triangular inequality, we have

We conclude by a union bound, and f ∈ F GNN ( B ) .

For the reverse inclusion, let f ∈ F GNN ( B ) . By hypothesis, for all m ∈ N , there are θ ∈ Θ , f (0) ∈ B such that

By the LLN,

Hence, b a union bound and triangular inequality, we obtain the deterministic bound ∥ ∥ f -Φ θ ( S , f (0) ) ∥ ∥ L 2 ⩽ 3 /m . Since Φ θ ( S , f (0) ) ∈ F S ( B ) and F S ( S ) is closed, by taking m →∞ we have f ∈ F S ( B ) .

A.4 Proof of Prop. 3

Remark that F S ( B ) ∩ C Lip ( X , R ) is in fact a subalgebra of C Lip ( X , R ) . Indeed it is a vector space, and moreover it is stable by multiplication: for f, g ∈ F S ( B ) ∩ C Lip ( X , R ) , by stability of F S ( B ) by composition with continuous functions we have that x ↦→ [ f ( x ) , 0] , x ↦→ [0 , g ( x )] are in F S ( B ) , then ( x ↦→ [ f ( x ) , g ( x )]) ∈ F S ( B ) by linearity (that is, F S is stable by concatenation), and since ( x, y ) ↦→ xy is continuous, ( x ↦→ f ( x ) g ( x )) ∈ F S ( B ) ∩ C Lip ( X , R ) .

Hence F S ( B ) ∩ C Lip ( X , R ) is a subalgebra of C Lip ( X , R ) and since it separates points by hypothesis, by the Stone-Weierstrass theorem [21] it is dense in C Lip ( X , R ) for the uniform norm, and a fortiori in ⊔ d C Lip ( X , R d ) by concatenation. Since continuous functions are dense in square-integrable functions [14, Sec. 8.2] and the L 2 norm is dominated by the uniform norm, it results that F S ( B ) is dense in L 2 ⊔ , and even F S ( B ) = L 2 ⊔ because it is closed.

B Proof of Sec. 4

We introduce general notations that are valid all throughout this section of appendix. In this appendix, we will only assume:

Assumption 2. We have the following.

a bounded kernel w S , and either w S is p.s.d. or X is finite;

the operator S f = ∫ w S ( · , x ) dP ( x ) ;

the Gram matrix W = [ w S ( x i , x j ) /n ] ;

a graph matrix S such that

in probability.

These assumptions are verified for both adjacency and normalized Laplacian: in the first case, we take w S = w , and ∥ A/ ( α n n ) -W ∥ → 0 by Theorem 9, and in the second, we take w S ( x, y ) = w ( x,y ) √ d ( x ) d ( y ) , which is bounded by our assumptions on d , and p.s.d. when w is itself p.s.d., and we have indeed ∥ L -W ∥ → 0 by Theorem 9. We will also use the following property.

Lemma 5. F PE is closed.

Proof. Let f m be a sequence in F PE that converges to a f ∈ L 2 q for some q . Remark that f m ∈ L 2 q for all m big enough. Let ε > 0 , and m be such that ∥ f m -f ∥ L 2 ⩽ ε/ 2 . By the law of large numbers, for any fixed m , ∥ ι X ( f m -f ) ∥ MSE converges to ∥ f m -f ∥ L 2 ⩽ ε/ 2 almost surely and a fortiori in probability, such that P ( ∥ ι X ( f m -f ) ∥ MSE ⩾ ε/ 2) - - - - → n →∞ 0 . By definition, there is γ ∈ Γ such that

and therefore f ∈ F

B.1 Proof of Prop. 4

We have

The first term goes to 0 by Prop. 2.

For the second term, using ∥ AB ∥ F ⩽ ∥ A ∥ ∥ B ∥ F , we have

The first term goes to 0 in probability since ∥ ε ∥ MSE is bounded with probability going to 1 and ∥ S -W ∥ → 0 for our examples of graph operators. For the second term, we have

Using Lemma 8 with the iid variable y j = ( x j , ν jℓ ) and E w S ( · , x j ) ν jℓ = 0 since ν and X are independent, we obtain

which concludes the proof.

B.2 Eigenvectors positional encodings: SignNet

In this whole section, the number of eigenvectors q is fixed, and we assume that λ S 1 , . . . , λ S q +1 are pairwise distinct. We first start by generic results that allows to go from the graph matrix S to the Gram matrix.

Lemma 6 (Intermediate result for SignNet) . Suppose that Assumption 2 holds, that λ S 1 , . . . , λ S q +1 are distinct, and that

Then the result of Theorem 2 holds.

Proof. Let f ∈ F Eig, written as (9). By Assumption 2, ∥ S -W ∥ → 0 . By Kato's inequality, we have that sup i ∣ ∣ λ S i -λ i W ∣ ∣ → 0 , and by hypothesis, the eigenvalues of W converge to those of S . Given the hypotheses on the eigenvalues of S , with probability going to 1 the q +1 first eigenvalues of S have single multiplicities. When it is the case, according to Davis-Kahan theorem (Theorem 8), for all i = 1 , . . . , q there is s i ∈ {-1 , 1 } such that

which, combined with our hypotheses, yields

For i = 1 , . . . , q , let f i : R → R p i be continuous functions and ε > 0 . By Lemma 3, there is f MLP γ i such that ∥ ∥ ( f MLP γ i -f i ) ◦ u S i ∥ ∥ L 2 ⩽ ε/ (2 q ) . Then, call L i the (uniform) Lipschitz constant of f MLP γ i on R . We have

The first term goes to 0 in probability by what precedes, while for the second

which proves that F Eig ⊂ F PE , and thus F Eig ⊂ F PE since F PE is closed by Lemma 5.

For the reverse inclusion, let f ∈ F PE . By hypothesis, for all m ∈ N , there is γ ∈ Γ , such that

By what precedes, if we write PE γ = ( Q f MLP γ i )( √ nu S i )] q i =1 , we have

By the LLN,

Hence, by a union bound and triangular inequality, we obtain the deterministic bound ∥ ∥ f -[( Q f MLP γ i ) ◦ u S i ] q i =1 ∥ ∥ L 2 ⩽ 3 /m . Since [( Q f MLP γ i ) ◦ u S i ] q i =1 ∈ F Eig, we have f ∈ F Eig .

We must now prove the hypothesis of Lemma 6. This is done separately for p.s.d. kernel and SBM.

B.2.1 Positive semidefinite kernels

In this subsection, we assume that w S is p.s.d. The following result is adapted from [42].

Theorem 5 (Adapted from [42]) . Suppose that Assumption 2 holds, that λ S 1 , . . . , λ S q +1 are pairwise distinct, and that w S is p.s.d. (Ex. b). Then:

in probability.

Proof. Denote by H the RKHS associated with w S , and by T H : H → H the kernel integral operator. Then, it is known [42] that the spectrum of S and T H are the same (up to 0 's), and the eigenfunctions (normalized in H ) v i of T H corresponding to positive eigenvalues satisfy:

Note that λ S 1 > . . . > λ S q > λ S q +1 ⩾ 0 by hypothesis. Following [42], we know that

in probability, and that in particular, with probability going to one λ i W > 0 for all i = 1 , . . . , q . Assuming this is satisfied for all i , we denote by v i W = 1 √ nλ i W ∑ j w S ( · , x j ) u W i,j ∈ H . Then, using the fact that the eigenvalues of S have single multiplicities, the proof of Theorem 12 in [42] tells us that for all m = 1 , . . . , q ,

in probability. Since these are all nonnegative quantities, all partial sums go to 0 . The first part (the term for j = m ) tells us that ∑ i ⩾ m +1 〈 v S i , v W m 〉 2 H → 0 , and the second part applied at m -1 (again the term with j = m ) gives us ∑ m -1 i =1 〈 v S i , v W m 〉 2 H → 0 . Hence for all m = 1 , . . . , q ,

̸

Since ( v S i ) i and ( v i W ) i are orthonormal basis of H , we have ∥ ∥ v W m ∥ ∥ 2 H = 1 = ∑ i 〈 v S i , v W m 〉 2 H , and thus 〈 v S m , v W m 〉 2 H → 1 . By the reproducing property

By the convergence of λ W m we obtain that ( 1 √ n 〈 u i W , ι X u S i 〉 ) 2 → 1 in probability, and choosing the sign of u i W such that 〈 u i W , ι X u S i 〉 ⩾ 0 , we get 1 √ n 〈 u i W , ι X u S i 〉 → 1 . Finally

by the LLN, ∥ ∥ ι X u S i ∥ ∥ 2 MSE → ∥ ∥ u S i ∥ ∥ 2 L 2 = 1 in probability, which concludes the proof.

By combining Theorem 5 with Lemma 6, we conclude the proof in the p.s.d. case.

B.2.2 SBM

Variants of the following result appear under (sometimes significantly) different formulations in the literature.

Theorem 6. Suppose that Assumption 2 holds, that λ S 1 , . . . , λ S q +1 are pairwise distinct, and that X is finite (Ex. a). Then:

Proof. In the SBM case, functions are represented by vectors of size K . The operator S acts as: S f = C diag( P k ) f , where C def. = [ w S ( k, ℓ )] kℓ . Therefore C diag( P k ) u S i = λ S i u S i , for i = 1 , . . . , K . Note that u S i is orthonormal in L 2 ( P ) , that is, ( u S i ) ⊤ diag( P k ) u S i = 1 and ( u S i ) ⊤ diag( P k ) u S j = 0 . In particular, ( λ S i , diag( √ P k ) u S i ) is an eigenvalue/eigenvector pair of the symmetric matrix C P def. = diag( √ P k ) C diag( √ P k ) . Denoting by u P i def. = diag( √ P k ) u S i we have C P = ∑ K i =1 λ S i u P i ( u P i ) ⊤ .

Since the space X is finite, each x i is equal to a community label 1 ⩽ k i ⩽ K . Define Θ ∈ { 0 , 1 / √ n } n × K the community matrix , as:

Then the Gram matrix is

Also note that Θ ⊤ Θ = diag( ˆ P ) , where ˆ P k = 1 n ∑ i 1 k i = k → P k almost surely by the LLN. Note that, when interpreting u S i as a function on X = { 1 , . . . , K } , we have √ n Θ u S i = ι X u S i .

Defining Θ P = Θdiag(1 / √ P k ) , we have

where v i = Θ P u P i = Θ u S i satisfies

in probability. Hence the v i are almost orthonormal but not exactly. Define their orthonormalization:

such that u 1 , . . . , u K ∈ R n are orthonormal and u i ∈ Span ( v 1 , . . . , v i ) . We define G = ∑ K i =1 λ S i u i u ⊤ i , whose eigenvalues are the λ S i and eigenvectors u i . By the properties of v i we have sup 1 ⩽ i ⩽ n ∥ v i -u i ∥ → 0 , so ∥ W -G ∥ → 0 . Hence by Kato's inequality we have sup 1 ⩽ i ⩽ n ∣ ∣ λ i W -λ S i ∣ ∣ → 0 , and since the λ S 1 , . . . , λ S q +1 have unique multiplicity, again by DavisKahan theorem for all i = 1 , . . . , q we have min s ∈{-1 , 1 } ∥ ∥ su i W -u i ∥ ∥ → 0 , and by consequence min s ∈{-1 , 1 } ∥ ∥ s √ nu i W - √ nv i ∥ ∥ MSE → 0 . We conclude by recalling that √ nv i = √ n Θ u S i = ι X u S i .

As before, we conclude by combining Theorem 6 with Lemma 6.

B.3 Distance-based encodings

Again, we start with an intermediate result, assuming some convergence in Frobenius norm that we will then show for our cases of interest.

Lemma 7 (Intermediate result for Distance-based encodings) . Suppose that Assumption 2 holds, and that:

Then the result of Theorem 3 holds.

Proof. Let f ∈ C Lip ([0 , 1] q , R d ) and ε > 0 . By the universality theorem [40], let f MLP γ 1 such that ∥ ∥ f MLP γ 1 -f ∥ ∥ ∞ ⩽ ε on [0 , 1] q . Since it is an MLP, f MLP γ 1 is uniformly Lipschitz on R q , call L its Lipschitz constant. Then, let f MLP γ 2 such that

For convenience, define M S γ 2 j = [ S γ 2 e j , . . . , ( S γ 2 ) q e j ] ∈ R n × q , M j W = [ We j , . . . , W q e j ] , J ( x, y ) = [ S δ y ( x ) , . . . , S q δ y ( x )] ∈ [0 , 1] q . We write

The third term in (31) is bounded by ∥ ∥ f MLP γ 1 -f ∥ ∥ ∞ ⩽ ε . The second term in (31) is

where we have used the Lipschitz property of f MLP γ 1 and a supremum over x i , x j for the first term. The first term converges to 0 in probability by Lemma 9, while the second goes to 0 , using the boundedness of f MLP γ 1 on [0 , 1] q , by Lemma 8.

Finally, using Schwartz inequality, the first term in (31) is bounded by

with probability going to 1 , using (30) and the fact that ∥ W ∥ F is bounded, and thus ∥ S γ 2 ∥ F as well with probability going to 1 . Gathering everything, f ∈ F PE , which concludes the proof.

We then prove this property for both p.s.d. kernel and SBM. The following Theorem is similar to Theorem 4, but under Assumption 2, which is true for ex. 1 and 2.

Theorem 7 (Theorem 4 reformulated) . Suppose that Assumption 2 holds. For both p.s.d. kernel (Ex. b) or finite X (Ex. a), for all ε > 0 , there is an MLP filter S γ = h f MLP γ ( S ) such that

Proof. Note that S is trace-class (both if w is p.s.d. or in the SBM case), such that ∑ i ∣ ∣ λ S i ∣ ∣ < ∞ . In particular, ∣ ∣ λ S i ∣ ∣ = o (1 / √ i ) . In the p.s.d. case, all λ S i are nonnegative. We define λ ε > 0 and the support T = { i | ∣ ∣ λ S i ∣ ∣ ⩾ λ ε }

∑ i ∈ T c ( λ S i ) 2 ⩽ ε/ 2 , which can be satisfied since S is trace class and therefore Hilbert-Schmidt √

ii) √ 2 | T | sup i ∈ T c ∣ ∣ λ S i ∣ ∣ ⩽ ε/ 4 , which can be satisfied since ∣ ∣ λ S i ∣ ∣ = o (1 / i )

iii) inf i ∈ T,j ∈ T c ∣ ∣ λ S i -λ S j ∣ ∣ > 0 , which can be satisfied by choosing λ ε in a gap in the spectrum of S , since all eigenvalues but 0 have finite multiplicities.

We will first start by approximating W with an ideal filtered matrix ˆ S = ∑ i ∈ T λ S i u S i ( u S i ) ⊤ . We define G = ∑ i ∈ T λ i W u i W ( u i W ) ⊤ . We decompose

For the first term, since this matrix is of rank at most 2 | T | , we have

The second term is bounded by Theorem 9, we have ∥ S -W ∥ → 0 in probability. The third term is bounded by Kato inequality

We further decompose

the last term goes to 0 in probability.

The first term is bounded by sup i ∈ T c λ S i , and by Kato's inequality

which is a combination of both previous cases.

At the end of the day, with probability going to 1 ,

Finally, we have

The first term is bounded by ε/ 2 , the second goes to 0 in probability: in the p.s.d. case, this is a result of [42], by an application of Hoeffding's inequality in the Hilbert space of Hilbert-Schmidt operators in an RKHS; in the SBM case, there is a finite number of non-zero eigenvalues for both W and S that converge to each other and the result follows.

Now we need to bound ∥ ∥ ∥ S γ -ˆ S ∥ ∥ ∥ F for a well chosen MLP. Define ( i T , j T ) = arg min i ∈ T,j ∈ T c ∣ ∣ λ S i -λ S j ∣ ∣ , τ = | λ S i T -λ S j T | 4 > 0 and ¯ λ = | λ S i T + λ S j T | 2 . Note that we have T = { i | ∣ ∣ λ S i ∣ ∣ ⩾ ¯ λ +2 τ } and T c = { i | ∣ ∣ λ S i ∣ ∣ ⩽ ¯ λ -2 τ } Define the following f MLP : R → R :

where ρ is ReLU. This is a continuous piecewise linear function that is equal to λ on ( -∞ , -¯ λ -τ ] , 0 on [ -¯ λ + τ, ¯ λ -τ ] , and λ on [ ¯ λ + τ, + ∞ ) .

Recall that with probability going to 1 , we have sup i ∣ ∣ λ S i -λ S i ∣ ∣ ⩽ τ , so in particular, for all i ∈ T we have ∣ ∣ λ S i ∣ ∣ ⩾ ¯ λ + τ and for all i ∈ T c we have ∣ ∣ λ S i ∣ ∣ ⩽ ¯ λ -τ . In that case, the MLP filtering is exactly the ideal filtering and h ( S ) = ˆ S , which concludes the proof.

B.4 Proof of Prop. 5

The case of F Dist was proven in [25, Theorem 4]. For F Eig, we consider the SBM (ex. a) with adjacency matrix (ex. 1) and

̸

We have S 1 = (1 / 3 , 1 / 3) , therefore F S (1) contains only constant functions. Moreover, u S i = (1 , 1) / √ 2 . From the orthogonality equation ( u S 2 ) ⊤ diag( P ) u S 1 = 0 we get ( u S 2 ) 1 = 2( u S 2 ) 2 . Hence it is possible to choose f such that f (( u S 2 ) 1 ) + f ( -( u S 2 ) 1 ) = f (( u S 2 ) 2 ) + f ( -( u S 2 ) 2 ) , thus F Eig contains a non-constant function, which concludes the proof.

B.5 Proof of Prop. 6

Consider the SBM case X = { 1 , . . . , K } with K even and P = 1 K /K . Adopting the notations of the section above, we have S f = C diag( P k ) f , C P = diag( √ P k ) C diag( √ P k ) = ∑ K i =1 λ S i u P i ( u P i ) ⊤ with u P i = diag( √ P k ) u S i orthonormal (in Euclidean R K ).

For F Eig, we consider the case where

where C is a constant such that ∥ ∥ u P 2 ∥ ∥ = 1 . Consider eigenvectors PEs with q = 1 . The space F Eig contains all the function g of the form

for any f . By construction of u P 1 , these functions have the property of being 2 -periodic: for all i , g i = g i +2 . Now, F S ( F Eig ) contains e.g. the following function

̸

This function is not 2 -periodic: on i odd, we have ( u P 2 ) i = ( u P 2 ) i +2 and therefore g i = g i +2 , which concludes the proof.

̸

For F Dist, taking again q = 1 , we consider K = 4 and C 01 = C 10 = 1 , and C kℓ = 0 otherwise. The space F Dist contains all the function g of the form

where f is applied element-wise on C . With this choice, for any f we have g 3 = g 4 = f (0) . However, the space F S ( F Dist ) contains the function

Here we have h 3 = f (0) f (1) 8 + 7 f (0) 2 8 and h 4 = f (0) 2 , which can be made unequal. Hence h / ∈ F Dist , which concludes the proof.

C Universality

Here we recall the universality results of [25], that were derived for an architecture called Structured GNN, in the case of non-random edges. In this paper, these results are valid for the distance-encoded PEs (10), through Theorem 3, our definition of F Dist (11). The results in [25] basically proves that F S ( F Dist ) satisfies the hypotheses of Prop. 3. We recall them here without proof. The following results are valid for adjacency matrix (ex. 1), distance-encoding PE (10), and SBM (ex. a) or p.s.d. kernel (ex. b), with other additional hypotheses in each cases.

SBM : if X is finite, C = [ w ( k, ℓ )] is invertible, and P is such that for all s ∈ {-1 , 0 , 1 } K , s ⊤ P = 0 implies s = 0 , then F S ( F PE ) = L 2 ⊔ .

Additive kernel : if w ( x, y ) = v ( u ( x ) + u ( y )) with u, v that are continuous and injective, then F S ( F PE ) = L 2 ⊔ .

Unidimensional radial kernel : if X = [ -1 , 1] , w ( x, y ) = v ( | x -y | ) with continuous injective v , and P is symmetric (that is, P ([ a, b ]) = P ([ -b, -a ]) for all intervals), then F S ( F PE ) = L 2 ⊔ ∩ S ( X ) where S are symmetric functions. If P is not symmetric, then F S ( F PE ) = L 2 ⊔ .

Spherical kernels : If X = S d -1 is the d -dimensional sphere, w ( x, y ) = v ( x ⊤ y ) with continuous injective v , and P has a density p w.r.t. the uniform distribution on the sphere such that: the unique decomposition p ( x ) = ∑ k ⩾ 0 ∑ N ( d,k ) j =1 a k,j Y k,j ( x ) where Y k,j are spherical harmonics is such that x ↦→ [ ∑ N ( d,k ) j =1 a k,j Y k,j ( x )] k is injective (see [25] and references therein), then F S ( F PE ) = L 2 ⊔ .

D Technical or third-party results

The following Lemma can be proved in a number of ways.

Lemma 8 (Lemma 4 in [24]) . Let X be a compact metric space, and Y a measurable space. Consider a bivariate measurable function U : X × Y → R that is uniformly bounded, and continuous in the first variable. Let y 1 , . . . , y n be drawn i.i.d from a distribution P on Y . Then

We extend the previous results to polynomials of the kernel.

Lemma 9. For any bounded, continuous kernel w S , we have for all k ⩾ 1 :

Proof. We prove it by induction. For k = 1 , we have

by applying Lemma 8 on X ×X , and since w S is continuous on a compact domain and therefore bounded and Lipschitz.

Then, assuming the property for k -1 , we write

The first part converge to 0 using the boundedness of w S and the recursive hypothesis, while the second converges to 0 using again Lemma 8.

Theorem 8 (Simplified Davis-Kahan, see [57]) . Let A, ˆ A ∈ R d × d be symmetric with eigenvalues λ i and ˆ λ i ordered by decreasing order and eigenvector u i , ˆ u i . Take p and assume that δ = min( λ p -1 -λ p , λ p -λ p +1 ) > 0 . Then there exists s ∈ {-1 , 1 } such that

Theorem 9. Denote W = [ w ( x i , x j ) /n ] ij , ¯ W = [ w ( x i ,x j ) n √ d ( x i ) d ( x j ) ] ij and L ( M ) = D -1 2 M MD -1 2 M with

D M = diag( M 1)

If α n ≳ (log n ) /n , then

Moreover, if d ( x ) ⩾ d min > 0 and α n ⩾ C (log n ) /n where C is a constant that depends on w , then

Proof. The first result is due to Lei and Rinaldo [28].

For the second result, we have from [24, Theorem 6] that

Then, according to Lemma 8 we have that

and in particular with probability going to one d i W def. = ( D W ) i ⩾ d min / 2 > 0 for all i . Denoting by ¯ D = diag( d ( x i )) , and noticing that W = ¯ D -1 2 W ¯ D -1 2 and ∥ W ∥ ⩽ 1 , we have

which concludes the proof.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper theoretically studies node classification on latent position random graphs, a generalization of graphons. More precisely, the family of functions that can be approximated using GNNs with positional encodings are determined in general and for the examples, sign-net (eigenvector PEs) and distance-based PEs.\n\nSTRENGTHS:\n* Very important and interesting topic. Most of current GNN theory focuses on WL and hence is mostly applicable to graph-level tasks. This paper takes a deep look on node classification.\n* Uses (and apparently requires) significant novel techniques / bounds to achieve the results (e.g., USVT estimators, pp) of independent interest.\n* Strong extension of [25]\n\nWEAKNESSES:\nSometimes the paper is not fully self-contained, e.g.:\n* the proper definition of graph shift matrices.\n* more intuition on latent position random graphs and graphons.\n* \"special properties\" of ReLU. Why don't you just say what properties you mean, Lipschitzness etc.\n\n\nMinor:\n* References are very badly maintained: A lot of references are missing either the Arxiv id or the venue (ICML, NeurIPS, ICLR etc.). Please fix this! The three references with arxiv ids are all published by now, their published versions should be used instead.\n* Please cite also Morris et al., [2019] when you meantion the GNN vs WL connection (you only cite [51], while the papers essentially came out in parallel).\n* sometimes you write \"latent position random graphs\" and sometimes \"latent position*s* random graphs\"\n* the claim \"as medium or large graphs are never isomorphic\" is a bit of an exaggeration. Perhaps say \"seldom\" instead of never.\n* While the authors explain in the text that $PE(S,Z)$ does not depend on $Z$, I wonder about the choice of notation. Wouldn't it be a lot clearer if one instead writes e.g., $PE(S)\\oplus Z$ where $\\oplus$ is concatation?\n* The paper deals with node classification. However, it is written in a way such that the reader is left unsure whether the paper might deal with graph-level tasks or not. E.g., it's not mentioned in the abstract. Only at page 2 it is almost apologetically mentioned that the authors will focus on node classification \"as this is makes more sense\" on large graphs. Node classification is an important and interesting task on its own. Please mention that this is your subject of study clearly in the abstract and in the beginning of the paper.  \n\nTypo:\n* Line 232 \"that sum to 1\" --> \"that sums to 1\"\n* Line 59 \"no not\" --> \"do not\"\n\n\n[Morris,  et al., 2019] \"Weisfeiler and leman go neural: Higher-order graph neural networks.\" AAAI 2019\n\nQUESTIONS:\n1. I am a bit confused by Assumption 1 and that the authors claim that e.g., the adjacency matrix satisfies it. If I am not mistaken the operator norm of a matrix $A$ is its largest singular value. For adjacency matrices of graphs this is known to be lower bounded in terms of the (square root of) maximum degree in the graph. Hence $||S||$ seems to be unbounded even for simple (and sparse!) graphs such as trees. Only for graphs with upper bounded degree this assumption seems to hold. Please explain.\n2. Can your results be generalized to general graph shift matrices, as defined by [46], i.e., matrices $S$ of size $n*n$ s.t. $S_{v,w}=0$ if ${v,w}$ is not an edge.  \n3. Theorem 2: What happens in the case of basis-invariance (and with non-distinct eigenvalues)?\n4. Of course on some specific graphs GNNs will only be able to predict using the constant function (cliques, etc.). Still, in general the space of functions approximable by GNNs *without* PEs seems interesting and rich. Is there a reason that there is no theorem exactly determining the set of functions learnable by GNNs without PEs? I imagine that e.g., nodes in the same orbit (i.e., nodes that can be mapped by an automorphism onto each other) must habe the same function value, but besides that arbitrary functions are possible?\n\nLIMITATIONS:\nPlease discuss the limitations of the latent position random graph model. On what kind of real-world data is it appropriate? Are there examples which cannot be modeled well with latent position random graphs?", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper provides a complete description of the function space generated by equivariant GNNs on latent position random graphs in order to study the expressive power of GNNs on large graphs for node level tasks. Then the paper investigates how node features such as positional encodings would affect the function space by studying two popular PE examples on two random graph models. The theoretical results suggest some normalization tricks that would help generalization across graphs of different sizes, and the paper validates this observation numerically in both synthetic and real data.\n\nSTRENGTHS:\n1. The paper studies the expressive power of GNNs by characterizing the function space generated by permutation-equivariant GNNs on latent position random graphs. As the paper argued in the paper, previous work on expressive power of GNNs has largely been focused on graph isomorphism testing, which is in fact a very limited framework when it comes to large graphs or node level tasks. The results established in the paper are much more relevant in those cases.\n\n2. The question about what role does node features or positional encoding (PE) plays in graph learning is important, and has not been well-studied in the literature. This paper provides a novel way to study them outside the graph isomorphism framework, filling the gap between previous theoretical studies and the practice.\n    - One particular weakness of results established under the graph isomorphism framework is that node features are often assume to be uniform/random/the unique node identifiers. This conditions are unrealistic and basically ignore the role of real node features play in learning on graphs.\n    - While PE has been widely used empirically, to the best of my knowledge, theoretical understanding of it is very limited, particularly outside of graph isomorphism testing framework.\n\n3. The paper is clearly written.\n\nWEAKNESSES:\n1. Although the objects of interest and technical results in this paper are mathematically precise, they are obscure.  It is hard to directly interpret the corresponding function space, and given a GNN architecture with a positional encoding, one cannot easily compute and compare the function space.\n\n2. The results are established for random graph models. Although random graph models have been useful in helping gaining deeper insights into many graph learning problems, they are synthetic graphs and there are some real graph characteristics they cannot capture. For example, nodes in stochastic block models have similar degrees, while real graphs exhibit degree heterogeneity. It is hard to know how robust/relevant the results are to real graphs or other random graph models that are not studied in the paper.\n\nQUESTIONS:\nPlease consider adding indices to the subplots and rewriting the caption for Figure 1, and place it around the same place as the relevant text. Currently sentence right after \"From left to right\" is very confusing to read.\n\nLIMITATIONS:\nn/a", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-05"}

comments: SUMMARY:
The paper investigates the theoretical expressiveness of Graph Neural Networks (GNNs) on large random graphs, and the role of node features and Positional Encodings (PEs) in enhancing their approximation power. The paper makes several contributions, such as:

- Providing a complete and intuitive description of the function space approximated by equivariant GNNs on random graphs, and showing that it is determined by a base set of input features and some extension rules;
- Analyzing two representative examples of PEs based on eigenvectors and distance-encoding, and extending previous universality results to more general random graph models;
- Proposing some normalization tricks to ensure convergence and consistency of PEs across different graph sizes, and demonstrating their effectiveness on synthetic and real data;
- Proving some new concentration inequalities for filtered random matrices that are of independent interest.

STRENGTHS:
he paper is well-written and organized, and the technical proofs are detailed in the appendix or supplementary material. The paper also provides some interesting insights and discussions on the properties and limitations of GNNs and PEs. The paper deepens the theoretical understanding of GNNs and PEs, which are widely used and studied in various domains of machine learning on graphs.

WEAKNESSES:
N/A

QUESTIONS:
N/A

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-09

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper analyzes the approximation capability of Graph Neural Networks (GNNs) in node-level tasks. The work presents a novel concept of expressivity that examines whether GNNs can approximate $L^2$-functions on measure spaces as the underlying graph grows and approximates operators on $L^2$ itself. The paper characterizes the precise class of functions that can be approximated and shows that positional encodings (eigenvector and distance-based) enhance the expressive power of GNNs in line with the new definition.\n\nSTRENGTHS:\n-    The authors give a new framework for investigating the expressivity of GNNs in node-level tasks.\n-    They give mathematical results and proofs, enhancing the scientific rigor of the study.\n-    The paper is well-written.\n-    Beyond the GNN result, the concentration and universality results offer interesting and novel insights that enrich the paper.\n\nWEAKNESSES:\n-    The authors occasionally do not discuss related work or overstate their findings. For instance, they give a novel universality result on L^2 functions and assert that their normalization rules apply to real-world graphs. Nonetheless, they do not discuss related work and test the normalization only on two small datasets.\n-    Similarly, they claim that their results are valid for general GNNs or MPNNs-like architecture, whereas the results are primarily applicable for GCN-like architectures with a normalized adjacency matrix or 1/n*A, which is not commonly used in practice.\n-    The primary results and the majority of intermediate results in the appendix lack self-containment, which makes it challenging to follow the paper in these technical sections.\n-    It's unclear why the broad applicability of graph matrices and operators S and $\\mathbf{S}$ is necessary when the only examples given are the normalized adjacency matrix and the adjacency matrix normalized by the graph size itself.\n-    The real-world experiments don't align with the theory as they involve graph-level predictions, but the expressivity measure introduced is at the node-level.\n\nQUESTIONS:\n-   Can the authors elaborate why the 1-WL test or the concept of separating isomorphic nodes in the node-classification context is unsuitable for large graphs? Proposition 3 uses Stone-Weierstrass, indicating that distinguishing isomorphic graphs remains key for achieving universality.\n -   The SignNet architecture studied isn't commonly employed in practice. Can the authors comment on how the function (may) change if $f$ is another permutation equivariant function, such as GIN? This is interesting as GIN was used in the original paper.\n  -  It would be beneficial to reference related works on universality results and concentration inequality, perhaps in the appendix.\n- Regarding the enhanced expressive power, it would be good to discuss other works that demonstrate that positional encodings (PEs) indeed boost expressivity at the graph level or node level. \n- Can the authors test the importance of normalizing PEs in node-level tasks? (see e.g. PPI datasets or [1])\n\n## Minor Points:\n\n   -  In Definition 1, the dependence on $n$ and what $S$ refers to is unclear. Generally one has to search too long for definition when reading the main results and results in the appendix.\n   -  Out of interest: Are there any accessible condition under which $F_S(F_{\\mathrm{PE}}) \\subsetneq F_{S'}({F}_{{\\mathrm{PE'}}})$ holds?\n   -  There is a typo in line 594. It should be $\\nu$ instead of $\\varepsilon$.\n\n## Conclusion:\n\nThis paper is a very nice and valuable read for anyone interested in GNNs and the utility of positional encodings (PEs) (from the perspective of large random graphs). Despite some overstated findings and certain ambiguity in the author's claims, the paper provides a new and valuable perspective on the expressivity of GNNs in the context of large random graphs. I recommend a score of 6, but am willing to increase my score if some issues are resolved.\n\n[1] Dwivedi, Vijay Prakash, et al. \"Long range graph benchmark.\" Advances in Neural Information Processing Systems 35 (2022): 22326-22340.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper aims to enhance the theoretical understanding of Graph Neural Networks (GNNs) concerning their expressive power, focusing on large graphs.  The paper provides a comprehensive description of the function space generated by equivariant GNNs for node tasks, incorporating notions of convergence from previous examples. The role of input node features is emphasized, along with the study of node Positional Encodings and equivariant eigen-based features. The theoretical findings suggest that certain normalization tricks positively impact GNN generalization on synthetic and real data, supported by new concentration inequalities introduced in the paper.\n\nSTRENGTHS:\n* The paper is very well written and was very enjoyable to read.\n\n* The analysis is fairly novel and studies GNNs with powerful features.\n\n* The positioning of the paper with respect to existing literature is clear and honest.\n\nWEAKNESSES:\n* The connection between the derived analysis and the practical implementation of GNNs is not clear.\n\n* The paper suggests the message-passing operations might not add up to the representation power of the GNN output features, which is not very well supported. For example, the eigenvectors and eigenvalues of the graph shift operator are able to perfectly reconstruct the graph structure. The analysis suggests that equivariant eigen-based features, which are less expressive compared to eigenvectors, not to mention eigenvectors and eigenvalues combined, do not benefit from message-passing operations. Current analysis suggests the opposite.\n\n* The analysis is mainly based on random graphs, which admit certain regularities, that do not allow concrete conclusions about the representation power of GNNs.\n\n* There are certain claims that are not properly supported, e.g., \"In some context, however, WL-based analyses have limitations: they pertain to tasks on graphs (e.g., graph classification or regression) and have limited to no connections to tasks on nodes or links; and they are mostly relevant for small-scale graphs, as medium or large graphs are never isomorphic but exhibit different characteristics.\"\n\nHow is the size of the graph related to graph isomorphism? What do the authors mean by small-scale graphs, medium or large graphs?\n\nQUESTIONS:\n* Please address the above concerns.\n\n* I would appreciate it if the authors make a direct connection between their analysis and practical implementations of GNNs. How is this work benefits deep learning on real graphs? \n\n* I would also expect some more detailed experiments with respect to the benefit of the proposed normalization.\n\nLIMITATIONS:\nThe limitations of the work are discussed in the previous sections. I am willing to raise my score if my comments are addressed properly.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-20"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper the authors provide characterisations of the class of functions that can be approximated by GNNs, possibly in the presence of positional encodings (PE). Here the notion of convergence is stated with high probability and for increasing graph size. So, in large graph limit.\n\nSTRENGTHS:\n- There is a lot to be liked in this paper since it provides a rigorous analysis of the function spaces that can be approximated.\n- The technical statements seem correct to me.\n- Presentation in the beginning of the paper is fine, but section 4 is rather dense.\n- The problem considered is quite natural, if one is interested this kind of limiting behaviour of graph learning methods.\n- Recent, state-of-the-art PE approaches are considered.\n- Giving a bit of hints in the main paper as how things are proved is appreciated.\n\nWEAKNESSES:\n- Although familiar with theoretical expressiveness results of GNNs, this material was new for me. It took some time to understand what was meant by \"GNNs become functions in the limit\". Once this was clear, I looked for insights as to what results about the limit situation could tell about capabilities of concrete practical GNN architectures. More effort could have been made to make the aim and reason why more accessible.\n- On line 57, it is mentioned that [25] does not consider random edges, but it does. This is a bit confusion. It is also mentioned the current paper generalises previous results from [25], but this could have been made more precise.\n- Throughout the paper I was looking for connections to approximation properties of GNNs (with PE) on finite graphs. It is well known that approximation capabilities are closely related to the distinguishing power of GNNs (and PE), and under conditions not unlike those in Definition 2, universality wrt functions with matching distinguishing power can be shown (see e.g., work by Azizian and Lelarge). Most of the result resemble this to some extent. A discussion on more classical approximation results in this context.\n- As mentioned earlier, section 4 is rather dense and the presentation could be improved. For example, what are are the precise assumptions on eigenvalues and vectors that you assume in Section 4.2.1? Also you define function classes in the theorems  without much explanation. It makes reading and understanding the result more difficult than it should.\n- The results in section 4 seem to depend on SBM or PSD kernels. It is unclear why this assumption is made. \n\n\n**minor**\nl6. certains functions -> certain\n\nQUESTIONS:\n- Please elaborate on what kind of practical insights one can get from the analyses. In the discrete finite case, approximation results are easier to interpret and use to build \"better\" models.\n- A gut feeling tells me that there must be a close connection to approximation properties in the finite case. Since, PEs for example, add to the distinguishing power and hence approximation capabilities. Could you comment?\n- Could you intuitively explain why SBM and PSD kernel assumptions are necessary in Section 4?\n\nLIMITATIONS:\nNot applicable.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-25"}