On the Universality of Graph Neural Networks on Large Random Graphs

1 Introduction . Graph Neural Networks ( GNNs ) are deep architectures defined over graph data that have garnered a lot of attention in recent years . They represent the state-of-the-art in many graph Machine Learning ( graph ML ) problems , and have been successfully applied to e.g . node clustering [ 7 ] , semi-supervised learning [ 24 ] , quantum chemistry [ 17 ] , and so on . See [ 5 , 44 , 18 , 21 ] for reviews . As the universality of Multi-Layers Perceptrons ( MLP ) is one of the foundational theorems in deep learning – that is , any continuous function can be approximated arbitrarily well by an MLP – in the last few years the approximation power of GNNs has been a topic of great interest . In the absence of special node features , i.e . when one has only access to the graph structure , the crux of the problem has been proven to be the capacity of GNNs to solve the graph isomorphism problem , that is , deciding when two graphs are permutations of each other or not ( a difficult problem for which no polynomial algorithm is known [ 4 ] ) . Indeed , this property is directly linked to the approximation power of GNNs [ 8 , 3 ] . In this light , the landmark paper [ 45 ] proves that classical GNNs are at best as powerful as the famous Weisfeiler-Lehman ( WL ) test [ 43 ] for graph isomorphism . Since then , several works [ 45 , 31 , 8 ] have derived new architectures , for instance involving high-order tensors [ 31 ] , with improved discriminative power equivalent to “ higher-order ” variants of the WL test . In another line of works , several recent papers have advocated the use of unique node identifiers [ 28 , 27 ] , with 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . strategies to preserve the permutation-equivariance/invariance of GNNs coined Structural Message Passing ( SMP ) in [ 41 ] ( see Sec . 2.3 ) . We call these models Structural GNN ( SGNN ) here . SGNNs have been proved to be strictly more powerful than the WL test in [ 41 ] , and even universal on graphs with bounded degrees , however for powerful layers that can not be implemented in practice . When the size of the graphs grows , the notion of graph isomorphism becomes somewhat moot : large graphs might share properties ( number of well-connected communities , and so on ) but are never isomorphic to each other . GNNs have nevertheless proven successful in identifying their large-scale structures , e.g . for node clustering [ 7 ] . Several papers have therefore used tools from random graph theory and graphons to study the behavior of GNNs in the large-graph limit . In [ 22 , 38 ] , GNNs are shown to converge to limiting “ continuous ” architectures ( coined c-GNNs in [ 22 ] ) . A few works have studied the discriminative power of c-GNNs on graphons [ 30 ] , however , analogously to how the WL test will fail on regular graphs , c-GNNs are severely limited on graph models with almost-constant degree function ( Fig . 1 ) , and the question is still largely open . Contribution and outline . In this paper , we study the convergence of SGNNs on large random graphs towards “ c-SGNNs ” , and analyze the approximation power of c-GNNs and cSGNNs . After some preliminary results in Sec . 2 , we study the convergence of SGNNs in Sec 3 . In Sec . 4 and 5 , we show that c-SGNNs are strictly more powerful than cGNNs in both permutation-invariant and equivariant case . We then prove the universality of c-SGNNs on several popular models of random graphs , including Stochastic Block Models ( SBMs ) and radial kernels . For instance , we show in Sec . 5.3 that c-SGNNs can perfectly identify the communities of most SBMs , including some for which any c-GNN provably fails ( Fig . 1 ) . The code for the numerical illustrations is available at https : //github.com/ nkeriven/random-graph-gnn . Related work . The approximation power of GNNs ( on finite graphs ) has been a topic of great interest in recent years , and we do not attempt to make an exhaustive list of all results here . The landmark paper [ 45 ] showed that permutation-invariant GNNs were at best as powerful as the WL test , and later models [ 31 , 8 , 16 ] were constructed to be as powerful as higher-order WL tests , using for instance higher-order tensors [ 31 ] . The link between the graph isomorphism problem and function approximation was made in [ 8 , 23 ] and extended in [ 3 ] . Some architectures were proven to be universal [ 32 , 23 ] but involve tensors of unbounded order . When graphs are equipped with unique node identifiers , the approximation power of GNNs can be significantly improved if relaxing permutation-invariance/equivariance [ 28 , 27 , 10 ] , or assuming that it holds only in expectation with respect to additional sources of randomness [ 39 , 1 ] . On the contrary , as we will see in Sec . 2.3 , the SGNN architecture proposed in [ 41 ] is inspired by the same ideas but still satisfies exact permutation-invariance/equivariance . As mentioned above , SGNNs are more powerful than the WL test [ 41 ] , and even universal on graphs with bounded degrees when allowing powerful layers . To our knowledge , the approximation power of SGNNs as they are implemented in practice is still open . We treat of their continuous limit in this paper . Fewer results can be found in the large-graph limit . Beyond the graph-isomorphism paradigm , authors have studied the capacity of GNNs to count graph substructures [ 6 , 13 ] or identify various graph properties [ 15 , 46 ] . Several works have studied the large-graphs limit of GNNs [ 38 , 30 , 26 , 22 ] , assuming random graph or graphon models . The degree function has been proven to be a crucial element for the discriminative power of c-GNNs , and they will not be able to distinguish graphons with close degree functions [ 30 ] . Here we show how c-SGNNs allow to overcome these limitations , and provide the first universality theorems for ( S ) GNNs in the continuous limit . Finally , we note that universal architectures exist on measures [ 11 , 49 ] , which can be seen as limits of point clouds , but , similar to the discrete case [ 48 , 32 , 23 ] , the graph case may be significantly harder to study . 2 Preliminaries . In this section , we group notations on random graphs and GNNs , present the convergence result of [ 22 ] slightly adapted to our context , and introduce the SGNNs of [ 41 ] . An undirected graph G with n nodes is represented by a symmetric adjacency matrix A ∈ Rn×n . It is isomorphic to all graphs with adjacency matrices σAσ > , for any permutation matrix σ ∈ { 0 , 1 } n×n . Matrix rows ( resp . columns ) are denoted by Ai , : ( resp . A : ,i ) . The norm ‖·‖ is the operator norm for matrices and Euclidean norm for vectors . For a compact metric space X , we denote by C ( X , R ) the space of bounded continuous functions X → R , equipped with ‖f‖∞ = supx |f ( x ) | . For a multivariate function f ∈ C ( X , Rd ) , fi denotes its ith coordinate , and ‖f‖∞ = ( ∑ i ‖fi‖ 2 ∞ ) 1/2 . 2.1 Random graph model . We consider “ latent position ” random graphs , which include several popular models such as SBMs or graphons [ 29 ] . The nodes are associated with unobserved latent random variables xi drawn i.i.d . For the edges , we will examine two cases : an “ ideal ” one with deterministic weighted edges , and a more “ realistic ” one where the edges are randomly drawn independently . In the literature , the latter is often considered as an idealized framework to study the behavior of various algorithms [ 42 , 37 ] . Let ( X , mX ) be a compact metric space that is not a singleton , and assume that the ε-covering numbers1 of X scale as O ( ε−dX ) for some dimension dX . We denote byW the set of symmetric bivariate functions in C ( X × X , [ 0 , 1 ] ) that are LW -Lipschitz in each variable , and by P the set of probability distributions over X equipped with the total variation norm ‖·‖TV . A graph with n nodes is generated according to a random graph model ( W , P ) ∈ W ×P as follows : x1 , . . . , xn i.i.d∼ P , aij = { W ( xi , xj ) deterministic edges case α−1n Bernoulli ( αnW ( xi , xj ) ) random edges case where αn is the sparsity level of the graph in the random edges case , which we assume to be known for simplicity . When αn ∼ 1 , the graph is said to be dense , when αn ∼ 1n the graph is sparse , and when α ∼ lognn the graph is relatively sparse . Note that we have normalized the Bernoulli edges by 1/αn such that Eaij = W ( xi , xj ) conditionally on xi , xj . When X is finite , the model is called an SBM ( see Sec . 4.3 ) . Like finite graphs , random graph models can be isomorphic to one another [ 22 , 29 ] . In this paper , similar to [ 22 ] we consider that , for any bijection ϕ : X → X , the model ( W , P ) is isomorphic to ( Wϕ , ϕ −1 ] P ) , where Wϕ ( x , y ) = W ( ϕ ( x ) , ϕ ( y ) ) and f ] P is the pushforward of P ( the distribution of f ( X ) ) . Indeed , it is easy to see that both produce exactly the same distribution over graphs . 2.2 Graph Neural Networks . Following [ 22 , 12 ] , in this paper we consider the so-called “ spectral ” version of GNNs , which include several message-passing models for certain aggregation functions . We consider polynomial filters h defined as h ( A ) = ∑ k βkA k for a matrix or operator A . In practice , the order of the filters is always finite , but our results are valid for infinite-order filters ( assuming that the sum always converges for simplicity ) . We consider any activation function ρ : R → R which satisfies ρ ( 0 ) = 0 and |ρ ( x ) − ρ ( y ) | 6 |x− y| for which the universality theorem of MLPs applies [ 35 ] , e.g . ReLU . Spectral GNNs are defined by successive filtering of a graph signal . Given an input signal Z ( 0 ) ∈ Rn×d0 , at each layer ` = 0 , . . . , M − 1 : Z ( ` +1 ) : ,j = ρ ( ∑d ` i=1 h ( ` ) ij ( 1 nA ) Z ( ` ) : ,i + b ( ` ) j 1n ) ∈ Rn j = 1 , . . . , d ` +1 , ( 1 ) where h ( ` ) ij ( A ) = ∑ k β ( ` ) ijkA k are trainable graph filters , b ( ` ) j ∈ R are trainable additive biases and ρ is applied pointwise . We note the normalization A/n that will be necessary for convergence . GNNs exist in two main versions : so-called “ permutation-invariant ” GNNs Φ̄ output a single vector for the entire graph , while “ permutation-equivariant ” GNNs Φ output a graph signal : ΦA ( Z ( 0 ) ) = g ( Z ( M ) ) ∈ Rn×dout Φ̄A ( Z ( 0 ) ) = g ( 1 n ∑n i=1 Z ( M ) i , : ) ∈ Rdout , ( 2 ) 1the number of balls of radius ε required to cover X where g : RdM → Rdout is an MLP applied row-wise in the equivariant case . By construction , such GNNs are indeed equivariant or invariant to node permutation : for all permutation matrices σ , we have ΦσAσ > ( σZ ( 0 ) ) = σΦA ( Z ( 0 ) ) and Φ̄σAσ > ( σZ ( 0 ) ) = Φ̄A ( Z ( 0 ) ) . Continuous GNNs . In [ 22 ] , the authors show that , in the large random graphs limit , GNNs converge to the following “ continuous ” models ( coined c-GNNs ) , which propagate functions over the latent space f ( ` ) ∈ C ( X , Rd ` ) instead of graph signals . Given an input function f ( 0 ) ∈ C ( X , Rd0 ) : f ( ` +1 ) j = ρ ( ∑d ` i=1 h ( ` ) ij ( TW , P ) f ( ` ) i + b ( ` ) j ) , where TW , P is the operator TW , P f = ∫ W ( · , x ) f ( x ) dP ( x ) . Then similar to ( 2 ) the permutationequivariant/invariant versions of c-GNNs are defined as : ΦW , P ( f ( 0 ) ) = g ◦ f ( M ) , Φ̄W , P ( f ( 0 ) ) = g ( ∫ f ( M ) ( x ) dP ( x ) ) . Remark that ΦW , P ( f ( 0 ) ) is itself a function in C ( X , Rdout ) while Φ̄W , P ( f ( 0 ) ) ∈ Rdout is a vector . Moreover , by virtue of the polynomial filters , the four architectures ΦA , Φ̄A , ΦW , P , Φ̄W , P have the exact same set of parameters . Like in the discrete case , one can prove that c-GNN are equivariant or invariant to isomorphisms of random graph models [ 22 ] : for all bijection ϕ : X → X , we have ΦWϕ , ϕ−1 ] P ( f ◦ ϕ ) = ΦW , P ( f ) ◦ ϕ and Φ̄Wϕ , ϕ−1 ] P ( f ◦ ϕ ) = Φ̄W , P ( f ) . Let us now turn to convergence of GNNs to c-GNNs . The following result is adapted2 from [ 22 ] . While the outputs of permutation-invariant GNNs and c-GNNs are vectors in Rdout that can be directly compared , the output graph signal of a permutation-equivariant GNN is compared with a sampling of the output function of the corresponding c-GNN : for a graph signal Z ∈ Rn×d , a function f : X → Rd and X = { xi } ni=1 , we define the ( square root of the ) mean-square error as MSEX ( Z , f ) = ( 1n ∑n i=1 ‖Zi − f ( xi ) ‖ 2 2 ) 1/2 . The proof of Theorem 1 with all multiplicative constants is given in App . A.1 . Recall that dX is the “ dimension ” of X . Theorem 1 . Assume G is drawn from ( W , P ) and has latent variables X . Fix ρ , ν > 0 . • In the deterministic edges case : with probability 1− ρ , for all Z ( 0 ) : MSEX ( ΦA ( Z ( 0 ) ) , ΦW , P ( f ( 0 ) ) ) 6 C ·MSEX ( Z ( 0 ) , f ( 0 ) ) +R1 ( n ) ( 3 ) for some constant C and R1 ( n ) = O ( √ ( dX + log ( 1/ρ ) ) /n ) . • In the random edges case : assume that the sparsity level is αn & n−1 log n. There is a constant Cν such that , with probability 1− ρ− n−ν , for all Z ( 0 ) : MSEX ( ΦA ( Z ( 0 ) ) , ΦW , P ( f ( 0 ) ) ) 6 C ·MSEX ( Z ( 0 ) , f ( 0 ) ) +R1 ( n ) +R2 ( n ) ( 4 ) where R2 ( n ) = O ( Cν/ √ αnn ) . • In the permutation-invariant case : The exact same results hold for ∥∥Φ̄A ( Z ( 0 ) ) − Φ̄W , P ( f ( 0 ) ) ∥∥ instead of the MSE on the left-hand-side , with an added error term R3 ( n ) = O ( √ log ( 1/ρ ) /n ) . By the theorem above , a GNN converges to its continuous counterpart if Z ( 0 ) is ( close to ) a sampling of a function f ( 0 ) at the latent variables . This is directly assumed in [ 22 ] . In the present paper however , we do not suppose that input node features are available . While several strategies have been proposed in the literature , a popular baseline is to simply take constant input Z ( 0 ) = 1n ( which , by a multiplication by A on the first layer , is also equivalent to inputing the degrees as in [ 7 ] for instance ) . In this case , there is convergence to a c-GNN with f ( 0 ) = 1 . For simplicity in the rest of the paper we drop the notation “ ( 1 ) ” and write ΦA = ΦA ( 1n ) , ΦW , P = ΦW , P ( 1 ) , and so on . It is known that GNNs are limited on regular graphs : for permutation-invariant GNNs , the WL test can not distinguish regular graphs of the same order , and for permutation-equivariant GNNs , ΦA ( 1n ) is constant over the nodes . Similarly for c-GNN , the degree function ∫ W ( · , x ) dP ( x ) is key in the discriminative power of c-GNNs [ 30 ] , and if it is constant , then ΦW , P ( 1 ) is a constant function ( see Fig . 1 ) . 2The authors in [ 22 ] proved this for the normalized Laplacian , which allows bypassing the knowledge of αn . Here we use the adjacency matrix , for our later results on approximation power .

Summary: This paper studies continuous limits of Structured Graph Neural Networks (SGNNs), which authors call c-SGNNs. Prior work studied the continuous limits of GNNs as c-GNNs. Another prior work proposed SGNNs and showed it is more powerful than GNNs. This work merges these two results and show that SGNNs converge to c-SGNNs, and also that c-SGNNs are more powerful than c-GNNs. Specifically, c-GNNs do not cover Stochastic Block Models (SBMs), whereas c-SGNNs do.