Expressiveness and Approximation Properties of Graph Neural Networks

1 INTRODUCTION . Graph Neural Networks ( GNNs ) ( Merkwirth & Lengauer , 2005 ; Scarselli et al. , 2009 ) cover many popular deep learning methods for graph learning tasks ( see Hamilton ( 2020 ) for a recent overview ) . These methods typically compute vector embeddings of vertices or graphs by relying on the underlying adjacency information . Invariance ( for graph embeddings ) and equivariance ( for vertex embeddings ) of GNNs ensure that these methods are oblivious to the precise representation of the graphs . Separation power . Our primary focus is on the separation power of GNN architectures , i.e. , on their ability to separate vertices or graphs by means of the computed embeddings . It has become standard to characterize GNN architectures in terms of the separation power of graph algorithms such as color refinement ( CR ) and k-dimensional Weisfeiler-Leman tests ( k-WL ) , as initiated in Xu et al . ( 2019 ) and Morris et al . ( 2019 ) . Unfortunately , understanding the separation power of any given GNN architecture requires complex proofs , geared at the specifics of the architecture . We provide a tensor language-based technique to analyze the separation power of general GNNs . Tensor languages . Matrix query languages ( Brijder et al. , 2019 ; Geerts et al. , 2021b ) are defined to assess the expressive power of linear algebra . Balcilar et al . ( 2021a ) observe that , by casting various GNNs into the MATLANG ( Brijder et al. , 2019 ) matrix query language , one can use existing separation results ( Geerts , 2021 ) to obtain upper bounds on the separation power of GNNs in terms of 1-WL and 2-WL . In this paper , we considerably extend this approach , by defining , and studying , a new general-purprpose tensor language specifically designed for modeling GNNs . As in Balcilar et al . ( 2021a ) , our focus on tensor languages allows us to obtain new insights about GNN architectures . First , since tensor languages can only define invariant and equivariant graph functions , any GNN that can be cast in our tensor language inherits these desired properties . More importantly , the separation power of our tensor language is as closely related to CR and k-WL as GNNs are . Loosely speaking , if tensor language expressions use k + 1 indices , then their separation power is bounded by k-WL . Furthermore , if the maximum nesting of summations in the expression is t , then t rounds of k-WL are needed for to obtain an upper bound on the separation power . A similar connection is obtained for CR and a fragment of tensor language that we call “ guarded ” tensor language . We thus reduce problem of assessing the separation power of any specific GNN architecture to the problem of specifying it in our tensor language , analyzing the number of indices used and counting their summation depth . As we show in the paper , this is usually much easier than dealing with intricacies of CR and k-WL . In fact , casting GNNs in our tensor language is often as simple as writing down their layer-based definition found in research articles . We believe that this provides a nice toolbox for GNN designers to assess the separation power of their architecture . We use this toolbox to recover known results about the separation power of specific GNN architectures such as GINs ( Xu et al. , 2019 ) , GCNs ( Kipf & Welling , 2017 ) , Folklore GNNs ( Maron et al. , 2019b ) , k-GNNs ( Morris et al. , 2019 ) , and several others . We also derive new results : we answer an open problem posed by Maron et al . ( 2019a ) by showing that the separation power of Invariant Graph Networks ( k-IGNs ) , introduced by Maron et al . ( 2019b ) , is bounded by ( k − 1 ) -WL . In addition , we revisit the analysis by Balcilar et al . ( 2021b ) of ChebNet ( Defferrard et al. , 2016 ) , and show that CayleyNet ( Levie et al. , 2019 ) is bounded by 2-WL . When writing down GNNs in our tensor language , the less indices needed , the stronger the bounds in terms of k-WL we obtain . After all , ( k − 1 ) -WL is known to be strictly less separating than k-WL ( Otto , 2017 ) . Thus , it is important to minimize the number of indices used in tensor language expressions . We connect this number to the notion of treewidth : expressions of treewidth k can be translated into expressions using k+1 indices . This corresponds to optimizing expressions , as done in many areas in machine learning , by reordering the summations ( a.k.a . variable elimination ) . Approximation and universality . We next consider the ability of GNNs to approximate general invariant or equivariant graph functions . Once more , instead of focusing on specific architectures , we use our tensor languages to obtain general approximation results , which naturally translate to universality results for GNNs . We show : k+1 index tensor language expressions suffice to approximate any ( invariant/equivariant ) graph function whose separating power is bounded by k-WL , and we can further refine this by comparing the number of rounds in k-WL with the summation depth of the expressions . These results provide a finer picture than the one obtained by Azizian & Lelarge ( 2021 ) . Furthermore , focusing on “ guarded ” tensor expressions yields a similar universality result for CR , a result that , to our knowledge , was not known before . We also provide the link between approximation results for tensor expressions and GNNs , enabling us to transfer our insights into universality properties of GNNs . As an example , we show that k-IGNs can approximate any graph function that is less separating than ( k−1 ) -WL . This case was left open in Azizian & Lelarge ( 2021 ) . In summary , we draw new and interesting connections between tensor languages , GNN architectures and classic graph algorithms . We provide a general recipe to bound the separation power of GNNs , optimize them , and understand their approximation power . We show the usefulness of our method by recovering several recent results , as well as new results , some of them left open in previous work . Related work . Separation power has been studied for specific classes of GNNs . ( Morris et al. , 2019 ; Xu et al. , 2019 ; Maron et al. , 2019b ; Chen et al. , 2019 ; Morris et al. , 2020 ; Azizian & Lelarge , 2021 ) . Balcilar et al . ( 2021a ) follow a more general approach in which the MATLANG matrix query language is used to obtain upper bounds on the separation power of various GNNs . MATLANG can only be used to obtain bounds up to 2-WL and is inherently limited to matrices . Our tensor language is more general and flexible and allows for reasoning over the number of indices , treewidth , and summation depth of expressions . These are all crucial for our main results . The tensor language introduced resembles sum-MATLANG ( Geerts et al. , 2021b ) , but with the added ability to represent tensors . Neither separation power nor guarded fragments were considered in Geerts et al . ( 2021b ) . See the supplementary material for a detailed comparison . For universality , Azizian & Lelarge ( 2021 ) is closest in spirit . Our tensor language approach provides an elegant way to recover and extend their results . Azizian & Lelarge ( 2021 ) describe how their work ( and hence also ours ) encompasses previous works ( Keriven & Peyré , 2019 ; Maron et al. , 2019c ; Chen et al. , 2019 ) . Our results use connections between k-WL and logics ( Immerman & Lander , 1990 ; Cai et al. , 1992 ) , and CR and guarded logics ( Barceló et al. , 2020 ) . The optimization of algebraic computations and the use of treewidth relates to the approaches by Aji & McEliece ( 2000 ) and Abo Khamis et al . ( 2016 ) . 2 BACKGROUND . We denote sets by { } and multisets by { { } } . For n ∈ N , n > 0 , [ n ] : = { 1 , . . . , n } . Vectors are denoted by v , w , . . . , matrices by A , B , . . . , and tensors by S , T , . . .. Furthermore , vi is the i-th entry of vector v , Aij is the ( i , j ) -th entry of matrix A and Si denotes the i = ( i1 , . . . , ik ) -th entry of a tensor S. If certain dimensions are unspecified , then this is denoted by a “ : ” . For example , Ai : and A : j denote the i-th row and j-th column of matrix A , respectively . Similarly for slices of tensors . We consider undirected simple graphs G = ( VG , EG , colG ) equipped with a vertex-labelling colG : VG → Rℓ . We assume that graphs have size n , so VG consists of n vertices and we often identify VG with [ n ] . For a vertex v ∈ VG , NG ( v ) : = { u ∈ VG | vu ∈ EG } . We let G be the set of all graphs of size n and let Gs be the set of pairs ( G , v ) with G ∈ G and v ∈ V sG . Note that G = G0 . The color refinement algorithm ( CR ) ( Morgan , 1965 ) iteratively computes vertex labellings based on neighboring vertices , as follows . For a graph G and vertex v ∈ VG , cr ( 0 ) ( G , v ) : = colG ( v ) . Then , for t ≥ 0 , cr ( t+1 ) ( G , v ) : = ( cr ( t ) ( G , v ) , { { cr ( t ) ( G , u ) | u ∈ NG ( v ) } } ) . We collect all vertex labels to obtain a label for the entire graph by defining gcr ( t ) ( G ) : = { { cr ( t ) ( G , v ) | v ∈ VG } } . The k-dimensional Weisfeiler-Leman algorithm ( k-WL ) ( Cai et al. , 1992 ) iteratively computes labellings of k-tuples of vertices . For a k-tuple v , its atomic type in G , denoted by atpk ( G , v ) , is a vector in R2 ( k 2 ) +kℓ . The first ( k 2 ) entries are 0/1-values encoding the equality type of v , i.e. , whether vi = vj for 1 ≤ i < j ≤ k. The second ( k 2 ) entries are 0/1-values encoding adjacency information , i.e. , whether vivj ∈ EG for 1 ≤ i < i ≤ k. The last kℓ real-valued entries correspond to colG ( vi ) ∈ Rℓ for 1 ≤ i ≤ k. Initially , for a graph G and v ∈ V kG , k-WL assigns the label wl ( 0 ) k ( G , v ) : = atpk ( G , v ) . For t ≥ 0 , k-WL revises the label according to wl ( t+1 ) k ( G , v ) : = ( wl ( t ) k ( G , v ) , M ) with M : = { { ( atpk+1 ( G , vu ) , wl ( t ) k ( G , v [ u/1 ] ) , . . . , wl ( t ) k ( G , v [ u/k ] ) ) | u ∈ VG } } , where v [ u/i ] : = ( v1 , . . . , vi−1 , u , vi+1 , . . . , vk ) . We use k-WL to assign labels to vertices and graphs by defining : vwl ( t ) k ( G , v ) : = wl ( t ) k ( G , ( v , . . . , v ) ) , for vertex-labellings , and gwl ( t ) k : = { { wl ( t ) k ( G , v ) | v ∈ V kG } } , for graph-labellings . We note that 1-WL differs from CR in that 1-WL also uses information from non-adjacent vertices . This distinction only matters for vertex embeddings ( Grohe , 2021 ) . Moreover , we use cr ( ∞ ) , gcr ( ∞ ) , vwl ( ∞ ) k , and gwl ( ∞ ) k to denote the stable labellings produced by the corresponding algorithm over an arbitrary number of rounds . We emphasize that we are using the “ folklore ” k-WL of Cai et al . ( 1992 ) , except that Cai et al . use 1-WL to refer to CR . It is equivalent to “ oblivious ” ( k + 1 ) -WL ( Grohe , 2021 ) , used in some other works on GNNs . Care is thus needed when comparing our results to related works . Let G be a graph with VG = [ n ] and let σ be a permutation of [ n ] . We denote by σ ⋆ G the isomorphic copy of G obtained by applying the permutation σ . Similarly , for v ∈ V kG , σ ⋆ v is the permuted version of v. Let F be some feature space . A function f : G0 → F is called invariant if f ( G ) = f ( σ ⋆ G ) for any permutation π . More generally , f : Gs → F is equivariant if f ( σ ⋆G , σ ⋆ v ) = f ( G , v ) for any permutation σ . The functions cr ( t ) : G1 → F and vwl ( t ) k : G1 → F are equivariant , whereas gcr ( t ) : G0 → F and gwl ( t ) k : G0 → F are invariant , for any t ≥ 0 and k ≥ 1 .

Summary: The paper proposes Tensor Language (TL), a language with which popular GNN models can be uniformly studied, so as to yield insights about their expressiveness and separation power. More specifically, the paper shows how TL corresponds to the WL hierarchy, and shows that summation depth in TL, as well as the number of index variables, correspond directly to iterations and tuple size for k-WL, respectively. Moreover, the paper establishes a direct correspondence between GNN update equations and expressions in TL, thus offering a simpler means to understand the expressive and separation power of GNNs, even considering their number of layers, by mapping their architectures to expressions in TL. TL , as well as its guarded fragment GTL, are then used to re-establish a set of known GNN results from the literature.   Beyond expressive power, TL is also used to quantify the function approximation power of GNNs, and also shows that GNNs, characterized by a TL fragment, can learn functions with separation power upper-bounded by a refinement algorithm corresponding to this fragment. Finally, the TL framework is used to establish new results. In particular, it shows that k-IGN cannot achieve expressiveness beyond (k-1)-WL, as this model corresponds to tk iterations of (k-1)-WL, and also offers insights as to the expressiveness of k-IGN with a polynomial number of layers: Indeed, model power does not increase with more standard layers, e.g., those relying on the adjacency matrix, but rather, any increase in layers can only improve expressiveness by deriving GNN functions with e.g., increased treewidth.