The Surprising Power of Graph Neural Networks with Random Node Initialization

1 INTRODUCTION . Graph neural networks ( GNNs ) ( Scarselli et al. , 2009 ; Gori et al. , 2005 ) are neural architectures designed for learning functions over graph-structured data , and naturally encode desirable properties such as permutation invariance ( resp. , equivariance ) relative to graph nodes , and node-level computation based on message passing between these nodes . These properties provide GNNs with a strong inductive bias , enabling them to effectively learn and combine both local and global graph features ( Battaglia et al. , 2018 ) . As a result , GNNs have been applied to a multitude of tasks , ranging from protein classification ( Gilmer et al. , 2017 ) and synthesis ( You et al. , 2018 ) , protein-protein interaction ( Fout et al. , 2017 ) , and social network analysis ( Hamilton et al. , 2017 ) , to recommender systems ( Ying et al. , 2018 ) and combinatorial optimization ( Bengio et al. , 2018 ; Selsam et al. , 2019 ) . However , popular GNN architectures , primarily based on message passing ( MPNNs ) , are limited in their expressive power . In particular , MPNNs are at most as powerful as the Weisfeiler-Leman ( 1-WL ) graph isomorphism heuristic ( Morris et al. , 2019 ; Xu et al. , 2019 ) , and thus can not discern between several families of non-isomorphic graphs , e.g. , sets of regular graphs ( Cai et al. , 1992 ) . To address this limitation , alternative GNN architectures with provably higher expressive power than MPNNs have been proposed . These models , which we refer to as higher-order GNNs , are inspired by the more powerful generalization of 1-WL to k−tuples of nodes , known as k-WL ( Grohe , 2017 ) . These models are the only GNNs with an established universality result , but these models are computationally very demanding . As a result , MPNNs , despite their limited expressiveness , remain the standard GNN model for graph learning applications . In a parallel development , MPNNs have recently achieved significant empirical improvements using random node initialization ( RNI ) , through which initial graph node embeddings are randomly set . Indeed , RNI has enabled MPNNs to distinguish instances that 1-WL can not distinguish , and is proven to enable better approximation of a class of combinatorial problems ( Sato et al. , 2020 ) . However , the effect of RNI on the expressive power of GNNs has not yet been comprehensively studied , and its impact on the inductive capacity and learning ability of GNNs remains unclear . In this paper , we thoroughly study the impact of RNI on MPNNs . First , we prove that MPNNs enhanced with RNI are universal , in the sense that they can approximate every function defined on graphs of any fixed order . This follows from a logical characterisation of the expressiveness of MPNNs ( Barceló et al. , 2020 ) combined with an argument on order-invariant definability . Our result strongly contrasts with existing 1-WL limitations for deterministic MPNNs , and provides a foundation for developing very expressive and memory-efficient MPNN models . To empirically verify our theoretical findings , we carry out a careful empirical study to quantify the practical impact of RNI . To this end , we design EXP , a synthetic dataset requiring 2-WL expressive power for models to achieve above-random performance , and run MPNNs with RNI on it , to observe how well and how easily this model can learn and generalize based on this dataset . Then , we propose CEXP , a modification of EXP with partially 1-WL distinguishable data , and evaluate the same questions in this more variable setting . Overall , the contributions of this paper are as follows : - We prove that MPNNs with RNI are universal , a significant improvement over the 1-WL limit of standard MPNNs and , to our knowledge , a first universality result for memory-efficient GNNs . - We introduce two carefully designed datasets , EXP and CEXP , based on graph pairs only distinguishable by 2-WL or higher , to rigorously evaluate the impact of RNI . - Using these datasets , we thoroughly analyze the effects of RNI on MPNN , and observe that ( i ) MPNNs with RNI can closely match the performance of higher-order GNNs , ( ii ) the improved performance of MPNNs with RNI comes at the cost of slower convergence ( compared to higherorder GNNs ) , and ( iii ) using a partial random initialization regime over node features typically improves convergence rate and the accuracy of the models . - We additionally perform the same experiments with analog , sparser datasets , with longer training , and observe similar behavior , but more volatility . 2 GRAPH NEURAL NETWORKS . Graph neural networks ( GNNs ) ( Gori et al. , 2005 ; Scarselli et al. , 2009 ) are neural architectures dedicated to learning functions over graph-structured data . In a GNN , nodes in the input graph are assigned vector representations , which are updated iteratively through series of invariant or equivariant computational layers . We recall message passing neural networks ( MPNNs ) ( Gilmer et al. , 2017 ) , a popular family of GNN models , and its expressive power in relation to the WeisfeilerLeman graph isomorphism heuristic . We discuss alternative GNN models in Section 3 ; for a broader coverage , we refer the reader to the literature ( Hamilton , 2020 ) . In MPNNs , node representations aggregate messages from their neighboring nodes , and use this information to iteratively update their representations . Formally , given a node x , its vector representation vx , t at time t , and its neighborhood N ( x ) , a message passing update can be written as : vx , t+1 = combine ( vx , t , aggregate ( { vy , t∣ y ∈ N ( x ) } ) ) , where combine and aggregate are functions , and aggregate is typically permutation-invariant . Once message passing is complete , the final node representations are then used to compute target outputs . Prominent message passing GNN architectures include graph convolutional networks ( GCNs ) ( Kipf & Welling , 2017 ) and gated graph neural networks ( GGNNs ) ( Li et al. , 2016 ) . It is well-known that standard MPNNs have the same power as the 1-dimensional Weisfeiler-Leman algorithm ( 1-WL ) ( Xu et al. , 2019 ; Morris et al. , 2019 ) . This entails that two nodes in a graph can not be distinguished if 1-WL does not distinguish them , and neither can two graphs be distinguished if 1-WL can not distinguish them . G H Figure 1 : G and H are indistinguishable by 1-WL and hence by ( 1-WL ) GNNs . Consider the graphsG andH shown in Figure 1 . 1-WL can not distinguish any two nodes of the graph G. Thus , for example , the invariant function f ∶ V ( G ) → R that maps all nodes in the 4-cycle of G to 1 and all nodes in the triangle to 0 is not expressible ( or approximable ) by a GNN . Moreover , 1-WL can not distinguish the graphs G , H , even though they are obviously non-isomorphic , so no classifier based on the node embeddings computed by an MPNN can distinguish the two graphs . A somewhat trivial limitation in the expressiveness of MPNNs is that information is only propagated along edges , and hence can never be shared between distinct connected components of a graph ( Barceló et al. , 2020 ; Xu et al. , 2019 ) . An easy way to overcome this limitation is by adding global readouts , that is , permutation-invariant functions that aggregate the current states of all nodes1 . Throughout the paper , we therefore focus on MPNNs with global readouts ( also called aggregatecombine GNNs with global readout , i.e. , ACR-GNNs ( Barceló et al. , 2020 ) ) . 3 RELATED WORK & MOTIVATION . Developing more expressive GNNs is an active research area due to the prominence of GNNs for relational learning ( Hamilton et al. , 2017 ) and combinatorial optimization ( Bengio et al. , 2018 ) . As mentioned earlier , standard GNN models are at most as expressive as 1-WL ( Morris et al. , 2019 ; Xu et al. , 2019 ) , and thus can not distinguish between non-isomorphic input instances . In this section , we describe theoretical results quantifying the expressive power of existing GNNs . Higher-order GNNs . We recall the following families of higher-order GNN models : - Invariant ( resp. , equivariant ) graph networks : Invariant ( resp. , equivariant ) graph networks ( Maron et al. , 2019b ) represent graphs as a tensor where node adjacency is directly encoded , and implicitly pass information between nodes through invariant ( resp. , equivariant ) computational blocks . Hence , these models are themselves invariant ( resp. , equivariant ) , a desirable property for computations on graphs . Following intermediate blocks , higher-order tensors are typically returned , and the order of these tensors correlates directly with the expressive power of the overall model . Indeed , invariant networks ( Maron et al. , 2019c ) , and later equivariant networks ( Keriven & Peyré , 2019 ) , are shown to be universal , but with tensor orders of O ( ∣V ∣2 ) , where ∣V ∣ denotes the number of graph nodes . Furthermore , invariant ( resp. , equivariant ) networks with intermediate tensor order k are shown to be equivalent in power to ( k − 1 ) -WL ( Maron et al. , 2019a ) , which is strictly more expressive as k increases ( Cai et al. , 1992 ) . Therefore , such universal higher-order models require intractably-sized intermediate tensors in practice . - Higher-order MPNNs : The k−WL hierarchy has been directly emulated in GNNs , such that these models learn embeddings for tuples of nodes , and perform messaging passing between them , as opposed to individual nodes . This approach has yielded models such as k-GNNs ( Morris et al. , 2019 ) . k-GNNs have ( k−1 ) -WL expressive power,2 but need O ( ∣V ∣k ) memory to run , leading to excessive memory requirements . - Provably powerful graph networks ( PPGNs ) : PPGN is an invariant GNN ( Maron et al. , 2019a ) , based on “ blocks ” of multilayer perceptrons ( MLPs ) and matrix multiplication , which theoretically has 2-WL expressive power , and only requires memory O ( ∣V ∣2 ) ( compared to O ( ∣V ∣3 ) for 3-GNNs ) . However , PPGN theoretically requires exponentially many samples in the number of graph nodes to learn necessary functions for 2-WL expressiveness ( Puny et al. , 2020 ) . GNNs with random node initialization . MPNNs have been enhanced with random node initialization ( Sato et al. , 2020 ) , such that the model trains and runs with partially randomized initial 1In the terminology of Maron et al . ( 2019c ) , such global readouts are tensors of order 1 that are invariant under the symmetric group of the vertex set of the input graph . 2In the literature , one can find different ( though equally expressive ) versions of the Weisfeiler Leman algorithm leading to inconsistent dimension counts . For example , the ( k + 1 ) -WL and the ( k + 1 ) -GNNs of Morris et al . ( 2019 ) are equivalent in expressiveness to the k-WL of Cai et al . ( 1992 ) ; Grohe ( 2017 ) . We follow the version of Cai et al . ( 1992 ) , as it has been adopted as the standard in the literature on graph isomorphism testing . node features . These models , denoted rGNNs , are shown to near-optimally approximate solutions to specific combinatorial optimization problems , and are able to distinguish between 1-WL indistinguishable graph pairs . rGNNs can also detect characteristic sub-graphs in an input graph with high probability . Nonetheless , it remains open as to how much expressive power is exactly gained through RNI , and , in general , whether a GNN model that is universal , scalable , and structure-preserving , can be developed . Our work strongly improves the theoretical result of Sato et al . ( 2020 ) , as it shows universality of MPNNs with RNI , and thus that arbitrary real-valued functions over graphs can be learned by MPNNs with the help of RNI . On the empirical side , we highlight the power of RNI in a significantly more challenging setting than rGNN , using a target function ( SAT ) beyond their theoretical scope . Indeed , for SAT , approximation is known to be hard , and fixed local structures are not useful for prediction . Similar work to RNI has also been conducted in terms of randomly adding features from a predetermined set of colors ( Dasoulas et al. , 2020 ) to disambiguate between nodes . This model , known as CLIP , is similar in spirit to RNI , in that it introduces randomness to node representations , but explicitly makes graphs distinguishable by construction . By contrast , we study random features produced by RNI , which ( i ) are not designed a priori to distinguish nodes , ( ii ) do not explicitly introduce a fixed underlying structure , and ( iii ) yield potentially infinitely many representations for a single graph . In this more general setting , we nonetheless show that RNI adds expressive power to distinguish between nodes with high probability , leads to a universality result , and performs strongly in challenging problem settings .

Summary: This paper studies the power of message passing neural networks (MPNNs) with random node initialization (RNI). Although the power of standard MPNNs is limited to 1-WL, the main result of the paper is to prove that RNI makes MPNNs universal. The paper also introduces two graph classification datasets where each graph is a SAT problem and the label is the satisfiability/unsatisfiability. The datasets have been created in a way that their graphs are 1-WL indistinguishable to serve as a test-bed for graph neural networks with power beyond 1-WL. The results on these two datasets show the merit of adding RNI to MPNNs.