On the Universality of Graph Neural Networks on Large Random Graphs

1 Introduction . Graph Neural Networks ( GNNs ) are deep architectures defined over graph data that have garnered a lot of attention in recent years . They represent the state-of-the-art in many graph Machine Learning ( graph ML ) problems , and have been successfully applied to e.g . node clustering [ 7 ] , semi-supervised learning [ 24 ] , quantum chemistry [ 17 ] , and so on . See [ 5 , 44 , 18 , 21 ] for reviews . As the universality of Multi-Layers Perceptrons ( MLP ) is one of the foundational theorems in deep learning – that is , any continuous function can be approximated arbitrarily well by an MLP – in the last few years the approximation power of GNNs has been a topic of great interest . In the absence of special node features , i.e . when one has only access to the graph structure , the crux of the problem has been proven to be the capacity of GNNs to solve the graph isomorphism problem , that is , deciding when two graphs are permutations of each other or not ( a difficult problem for which no polynomial algorithm is known [ 4 ] ) . Indeed , this property is directly linked to the approximation power of GNNs [ 8 , 3 ] . In this light , the landmark paper [ 45 ] proves that classical GNNs are at best as powerful as the famous Weisfeiler-Lehman ( WL ) test [ 43 ] for graph isomorphism . Since then , several works [ 45 , 31 , 8 ] have derived new architectures , for instance involving high-order tensors [ 31 ] , with improved discriminative power equivalent to “ higher-order ” variants of the WL test . In another line of works , several recent papers have advocated the use of unique node identifiers [ 28 , 27 ] , with 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . strategies to preserve the permutation-equivariance/invariance of GNNs coined Structural Message Passing ( SMP ) in [ 41 ] ( see Sec . 2.3 ) . We call these models Structural GNN ( SGNN ) here . SGNNs have been proved to be strictly more powerful than the WL test in [ 41 ] , and even universal on graphs with bounded degrees , however for powerful layers that can not be implemented in practice . When the size of the graphs grows , the notion of graph isomorphism becomes somewhat moot : large graphs might share properties ( number of well-connected communities , and so on ) but are never isomorphic to each other . GNNs have nevertheless proven successful in identifying their large-scale structures , e.g . for node clustering [ 7 ] . Several papers have therefore used tools from random graph theory and graphons to study the behavior of GNNs in the large-graph limit . In [ 22 , 38 ] , GNNs are shown to converge to limiting “ continuous ” architectures ( coined c-GNNs in [ 22 ] ) . A few works have studied the discriminative power of c-GNNs on graphons [ 30 ] , however , analogously to how the WL test will fail on regular graphs , c-GNNs are severely limited on graph models with almost-constant degree function ( Fig . 1 ) , and the question is still largely open . Contribution and outline . In this paper , we study the convergence of SGNNs on large random graphs towards “ c-SGNNs ” , and analyze the approximation power of c-GNNs and cSGNNs . After some preliminary results in Sec . 2 , we study the convergence of SGNNs in Sec 3 . In Sec . 4 and 5 , we show that c-SGNNs are strictly more powerful than cGNNs in both permutation-invariant and equivariant case . We then prove the universality of c-SGNNs on several popular models of random graphs , including Stochastic Block Models ( SBMs ) and radial kernels . For instance , we show in Sec . 5.3 that c-SGNNs can perfectly identify the communities of most SBMs , including some for which any c-GNN provably fails ( Fig . 1 ) . The code for the numerical illustrations is available at https : //github.com/ nkeriven/random-graph-gnn . Related work . The approximation power of GNNs ( on finite graphs ) has been a topic of great interest in recent years , and we do not attempt to make an exhaustive list of all results here . The landmark paper [ 45 ] showed that permutation-invariant GNNs were at best as powerful as the WL test , and later models [ 31 , 8 , 16 ] were constructed to be as powerful as higher-order WL tests , using for instance higher-order tensors [ 31 ] . The link between the graph isomorphism problem and function approximation was made in [ 8 , 23 ] and extended in [ 3 ] . Some architectures were proven to be universal [ 32 , 23 ] but involve tensors of unbounded order . When graphs are equipped with unique node identifiers , the approximation power of GNNs can be significantly improved if relaxing permutation-invariance/equivariance [ 28 , 27 , 10 ] , or assuming that it holds only in expectation with respect to additional sources of randomness [ 39 , 1 ] . On the contrary , as we will see in Sec . 2.3 , the SGNN architecture proposed in [ 41 ] is inspired by the same ideas but still satisfies exact permutation-invariance/equivariance . As mentioned above , SGNNs are more powerful than the WL test [ 41 ] , and even universal on graphs with bounded degrees when allowing powerful layers . To our knowledge , the approximation power of SGNNs as they are implemented in practice is still open . We treat of their continuous limit in this paper . Fewer results can be found in the large-graph limit . Beyond the graph-isomorphism paradigm , authors have studied the capacity of GNNs to count graph substructures [ 6 , 13 ] or identify various graph properties [ 15 , 46 ] . Several works have studied the large-graphs limit of GNNs [ 38 , 30 , 26 , 22 ] , assuming random graph or graphon models . The degree function has been proven to be a crucial element for the discriminative power of c-GNNs , and they will not be able to distinguish graphons with close degree functions [ 30 ] . Here we show how c-SGNNs allow to overcome these limitations , and provide the first universality theorems for ( S ) GNNs in the continuous limit . Finally , we note that universal architectures exist on measures [ 11 , 49 ] , which can be seen as limits of point clouds , but , similar to the discrete case [ 48 , 32 , 23 ] , the graph case may be significantly harder to study . 2 Preliminaries . In this section , we group notations on random graphs and GNNs , present the convergence result of [ 22 ] slightly adapted to our context , and introduce the SGNNs of [ 41 ] . An undirected graph G with n nodes is represented by a symmetric adjacency matrix A ∈ Rn×n . It is isomorphic to all graphs with adjacency matrices σAσ > , for any permutation matrix σ ∈ { 0 , 1 } n×n . Matrix rows ( resp . columns ) are denoted by Ai , : ( resp . A : ,i ) . The norm ‖·‖ is the operator norm for matrices and Euclidean norm for vectors . For a compact metric space X , we denote by C ( X , R ) the space of bounded continuous functions X → R , equipped with ‖f‖∞ = supx |f ( x ) | . For a multivariate function f ∈ C ( X , Rd ) , fi denotes its ith coordinate , and ‖f‖∞ = ( ∑ i ‖fi‖ 2 ∞ ) 1/2 . 2.1 Random graph model . We consider “ latent position ” random graphs , which include several popular models such as SBMs or graphons [ 29 ] . The nodes are associated with unobserved latent random variables xi drawn i.i.d . For the edges , we will examine two cases : an “ ideal ” one with deterministic weighted edges , and a more “ realistic ” one where the edges are randomly drawn independently . In the literature , the latter is often considered as an idealized framework to study the behavior of various algorithms [ 42 , 37 ] . Let ( X , mX ) be a compact metric space that is not a singleton , and assume that the ε-covering numbers1 of X scale as O ( ε−dX ) for some dimension dX . We denote byW the set of symmetric bivariate functions in C ( X × X , [ 0 , 1 ] ) that are LW -Lipschitz in each variable , and by P the set of probability distributions over X equipped with the total variation norm ‖·‖TV . A graph with n nodes is generated according to a random graph model ( W , P ) ∈ W ×P as follows : x1 , . . . , xn i.i.d∼ P , aij = { W ( xi , xj ) deterministic edges case α−1n Bernoulli ( αnW ( xi , xj ) ) random edges case where αn is the sparsity level of the graph in the random edges case , which we assume to be known for simplicity . When αn ∼ 1 , the graph is said to be dense , when αn ∼ 1n the graph is sparse , and when α ∼ lognn the graph is relatively sparse . Note that we have normalized the Bernoulli edges by 1/αn such that Eaij = W ( xi , xj ) conditionally on xi , xj . When X is finite , the model is called an SBM ( see Sec . 4.3 ) . Like finite graphs , random graph models can be isomorphic to one another [ 22 , 29 ] . In this paper , similar to [ 22 ] we consider that , for any bijection ϕ : X → X , the model ( W , P ) is isomorphic to ( Wϕ , ϕ −1 ] P ) , where Wϕ ( x , y ) = W ( ϕ ( x ) , ϕ ( y ) ) and f ] P is the pushforward of P ( the distribution of f ( X ) ) . Indeed , it is easy to see that both produce exactly the same distribution over graphs . 2.2 Graph Neural Networks . Following [ 22 , 12 ] , in this paper we consider the so-called “ spectral ” version of GNNs , which include several message-passing models for certain aggregation functions . We consider polynomial filters h defined as h ( A ) = ∑ k βkA k for a matrix or operator A . In practice , the order of the filters is always finite , but our results are valid for infinite-order filters ( assuming that the sum always converges for simplicity ) . We consider any activation function ρ : R → R which satisfies ρ ( 0 ) = 0 and |ρ ( x ) − ρ ( y ) | 6 |x− y| for which the universality theorem of MLPs applies [ 35 ] , e.g . ReLU . Spectral GNNs are defined by successive filtering of a graph signal . Given an input signal Z ( 0 ) ∈ Rn×d0 , at each layer ` = 0 , . . . , M − 1 : Z ( ` +1 ) : ,j = ρ ( ∑d ` i=1 h ( ` ) ij ( 1 nA ) Z ( ` ) : ,i + b ( ` ) j 1n ) ∈ Rn j = 1 , . . . , d ` +1 , ( 1 ) where h ( ` ) ij ( A ) = ∑ k β ( ` ) ijkA k are trainable graph filters , b ( ` ) j ∈ R are trainable additive biases and ρ is applied pointwise . We note the normalization A/n that will be necessary for convergence . GNNs exist in two main versions : so-called “ permutation-invariant ” GNNs Φ̄ output a single vector for the entire graph , while “ permutation-equivariant ” GNNs Φ output a graph signal : ΦA ( Z ( 0 ) ) = g ( Z ( M ) ) ∈ Rn×dout Φ̄A ( Z ( 0 ) ) = g ( 1 n ∑n i=1 Z ( M ) i , : ) ∈ Rdout , ( 2 ) 1the number of balls of radius ε required to cover X where g : RdM → Rdout is an MLP applied row-wise in the equivariant case . By construction , such GNNs are indeed equivariant or invariant to node permutation : for all permutation matrices σ , we have ΦσAσ > ( σZ ( 0 ) ) = σΦA ( Z ( 0 ) ) and Φ̄σAσ > ( σZ ( 0 ) ) = Φ̄A ( Z ( 0 ) ) . Continuous GNNs . In [ 22 ] , the authors show that , in the large random graphs limit , GNNs converge to the following “ continuous ” models ( coined c-GNNs ) , which propagate functions over the latent space f ( ` ) ∈ C ( X , Rd ` ) instead of graph signals . Given an input function f ( 0 ) ∈ C ( X , Rd0 ) : f ( ` +1 ) j = ρ ( ∑d ` i=1 h ( ` ) ij ( TW , P ) f ( ` ) i + b ( ` ) j ) , where TW , P is the operator TW , P f = ∫ W ( · , x ) f ( x ) dP ( x ) . Then similar to ( 2 ) the permutationequivariant/invariant versions of c-GNNs are defined as : ΦW , P ( f ( 0 ) ) = g ◦ f ( M ) , Φ̄W , P ( f ( 0 ) ) = g ( ∫ f ( M ) ( x ) dP ( x ) ) . Remark that ΦW , P ( f ( 0 ) ) is itself a function in C ( X , Rdout ) while Φ̄W , P ( f ( 0 ) ) ∈ Rdout is a vector . Moreover , by virtue of the polynomial filters , the four architectures ΦA , Φ̄A , ΦW , P , Φ̄W , P have the exact same set of parameters . Like in the discrete case , one can prove that c-GNN are equivariant or invariant to isomorphisms of random graph models [ 22 ] : for all bijection ϕ : X → X , we have ΦWϕ , ϕ−1 ] P ( f ◦ ϕ ) = ΦW , P ( f ) ◦ ϕ and Φ̄Wϕ , ϕ−1 ] P ( f ◦ ϕ ) = Φ̄W , P ( f ) . Let us now turn to convergence of GNNs to c-GNNs . The following result is adapted2 from [ 22 ] . While the outputs of permutation-invariant GNNs and c-GNNs are vectors in Rdout that can be directly compared , the output graph signal of a permutation-equivariant GNN is compared with a sampling of the output function of the corresponding c-GNN : for a graph signal Z ∈ Rn×d , a function f : X → Rd and X = { xi } ni=1 , we define the ( square root of the ) mean-square error as MSEX ( Z , f ) = ( 1n ∑n i=1 ‖Zi − f ( xi ) ‖ 2 2 ) 1/2 . The proof of Theorem 1 with all multiplicative constants is given in App . A.1 . Recall that dX is the “ dimension ” of X . Theorem 1 . Assume G is drawn from ( W , P ) and has latent variables X . Fix ρ , ν > 0 . • In the deterministic edges case : with probability 1− ρ , for all Z ( 0 ) : MSEX ( ΦA ( Z ( 0 ) ) , ΦW , P ( f ( 0 ) ) ) 6 C ·MSEX ( Z ( 0 ) , f ( 0 ) ) +R1 ( n ) ( 3 ) for some constant C and R1 ( n ) = O ( √ ( dX + log ( 1/ρ ) ) /n ) . • In the random edges case : assume that the sparsity level is αn & n−1 log n. There is a constant Cν such that , with probability 1− ρ− n−ν , for all Z ( 0 ) : MSEX ( ΦA ( Z ( 0 ) ) , ΦW , P ( f ( 0 ) ) ) 6 C ·MSEX ( Z ( 0 ) , f ( 0 ) ) +R1 ( n ) +R2 ( n ) ( 4 ) where R2 ( n ) = O ( Cν/ √ αnn ) . • In the permutation-invariant case : The exact same results hold for ∥∥Φ̄A ( Z ( 0 ) ) − Φ̄W , P ( f ( 0 ) ) ∥∥ instead of the MSE on the left-hand-side , with an added error term R3 ( n ) = O ( √ log ( 1/ρ ) /n ) . By the theorem above , a GNN converges to its continuous counterpart if Z ( 0 ) is ( close to ) a sampling of a function f ( 0 ) at the latent variables . This is directly assumed in [ 22 ] . In the present paper however , we do not suppose that input node features are available . While several strategies have been proposed in the literature , a popular baseline is to simply take constant input Z ( 0 ) = 1n ( which , by a multiplication by A on the first layer , is also equivalent to inputing the degrees as in [ 7 ] for instance ) . In this case , there is convergence to a c-GNN with f ( 0 ) = 1 . For simplicity in the rest of the paper we drop the notation “ ( 1 ) ” and write ΦA = ΦA ( 1n ) , ΦW , P = ΦW , P ( 1 ) , and so on . It is known that GNNs are limited on regular graphs : for permutation-invariant GNNs , the WL test can not distinguish regular graphs of the same order , and for permutation-equivariant GNNs , ΦA ( 1n ) is constant over the nodes . Similarly for c-GNN , the degree function ∫ W ( · , x ) dP ( x ) is key in the discriminative power of c-GNNs [ 30 ] , and if it is constant , then ΦW , P ( 1 ) is a constant function ( see Fig . 1 ) . 2The authors in [ 22 ] proved this for the normalized Laplacian , which allows bypassing the knowledge of αn . Here we use the adjacency matrix , for our later results on approximation power .

Summary: This paper studies the universality of Structural Graph Neural Networks (SGNNs) and their continuous counterparts, termed c-SGNNs, in both the permutation-invariant and equivariant settings.  More specifically, the paper produces a theoretical convergence result between the recently proposed SGNNs and their continuous c-SGNNs in the limit of large graphs, analogously to existing connections between GNNs and their continuous counterparts c-GNNs. Then, the universality of SGNN is theoretically studied based on this convergence, and the expressive power of SGNNs is compared with that of standard GNNs. From this study, it is established that c-SGNNs are strictly more powerful than c-GNNs, and that c-SGNNs are universal on many random graph families, such as stochastic block models (SBMs), symmetric radial kernels, and injectively additive kernels. Finally, the overall universality of SGNNs and c-SGNNs, as well as open questions remaining in this regard, are discussed.