The Surprising Power of Graph Neural Networks with Random Node Initialization

1 INTRODUCTION . Graph neural networks ( GNNs ) ( Scarselli et al. , 2009 ; Gori et al. , 2005 ) are neural architectures designed for learning functions over graph-structured data , and naturally encode desirable properties such as permutation invariance ( resp. , equivariance ) relative to graph nodes , and node-level computation based on message passing between these nodes . These properties provide GNNs with a strong inductive bias , enabling them to effectively learn and combine both local and global graph features ( Battaglia et al. , 2018 ) . As a result , GNNs have been applied to a multitude of tasks , ranging from protein classification ( Gilmer et al. , 2017 ) and synthesis ( You et al. , 2018 ) , protein-protein interaction ( Fout et al. , 2017 ) , and social network analysis ( Hamilton et al. , 2017 ) , to recommender systems ( Ying et al. , 2018 ) and combinatorial optimization ( Bengio et al. , 2018 ; Selsam et al. , 2019 ) . However , popular GNN architectures , primarily based on message passing ( MPNNs ) , are limited in their expressive power . In particular , MPNNs are at most as powerful as the Weisfeiler-Leman ( 1-WL ) graph isomorphism heuristic ( Morris et al. , 2019 ; Xu et al. , 2019 ) , and thus can not discern between several families of non-isomorphic graphs , e.g. , sets of regular graphs ( Cai et al. , 1992 ) . To address this limitation , alternative GNN architectures with provably higher expressive power than MPNNs have been proposed . These models , which we refer to as higher-order GNNs , are inspired by the more powerful generalization of 1-WL to k−tuples of nodes , known as k-WL ( Grohe , 2017 ) . These models are the only GNNs with an established universality result , but these models are computationally very demanding . As a result , MPNNs , despite their limited expressiveness , remain the standard GNN model for graph learning applications . In a parallel development , MPNNs have recently achieved significant empirical improvements using random node initialization ( RNI ) , through which initial graph node embeddings are randomly set . Indeed , RNI has enabled MPNNs to distinguish instances that 1-WL can not distinguish , and is proven to enable better approximation of a class of combinatorial problems ( Sato et al. , 2020 ) . However , the effect of RNI on the expressive power of GNNs has not yet been comprehensively studied , and its impact on the inductive capacity and learning ability of GNNs remains unclear . In this paper , we thoroughly study the impact of RNI on MPNNs . First , we prove that MPNNs enhanced with RNI are universal , in the sense that they can approximate every function defined on graphs of any fixed order . This follows from a logical characterisation of the expressiveness of MPNNs ( Barceló et al. , 2020 ) combined with an argument on order-invariant definability . Our result strongly contrasts with existing 1-WL limitations for deterministic MPNNs , and provides a foundation for developing very expressive and memory-efficient MPNN models . To empirically verify our theoretical findings , we carry out a careful empirical study to quantify the practical impact of RNI . To this end , we design EXP , a synthetic dataset requiring 2-WL expressive power for models to achieve above-random performance , and run MPNNs with RNI on it , to observe how well and how easily this model can learn and generalize based on this dataset . Then , we propose CEXP , a modification of EXP with partially 1-WL distinguishable data , and evaluate the same questions in this more variable setting . Overall , the contributions of this paper are as follows : - We prove that MPNNs with RNI are universal , a significant improvement over the 1-WL limit of standard MPNNs and , to our knowledge , a first universality result for memory-efficient GNNs . - We introduce two carefully designed datasets , EXP and CEXP , based on graph pairs only distinguishable by 2-WL or higher , to rigorously evaluate the impact of RNI . - Using these datasets , we thoroughly analyze the effects of RNI on MPNN , and observe that ( i ) MPNNs with RNI can closely match the performance of higher-order GNNs , ( ii ) the improved performance of MPNNs with RNI comes at the cost of slower convergence ( compared to higherorder GNNs ) , and ( iii ) using a partial random initialization regime over node features typically improves convergence rate and the accuracy of the models . - We additionally perform the same experiments with analog , sparser datasets , with longer training , and observe similar behavior , but more volatility . 2 GRAPH NEURAL NETWORKS . Graph neural networks ( GNNs ) ( Gori et al. , 2005 ; Scarselli et al. , 2009 ) are neural architectures dedicated to learning functions over graph-structured data . In a GNN , nodes in the input graph are assigned vector representations , which are updated iteratively through series of invariant or equivariant computational layers . We recall message passing neural networks ( MPNNs ) ( Gilmer et al. , 2017 ) , a popular family of GNN models , and its expressive power in relation to the WeisfeilerLeman graph isomorphism heuristic . We discuss alternative GNN models in Section 3 ; for a broader coverage , we refer the reader to the literature ( Hamilton , 2020 ) . In MPNNs , node representations aggregate messages from their neighboring nodes , and use this information to iteratively update their representations . Formally , given a node x , its vector representation vx , t at time t , and its neighborhood N ( x ) , a message passing update can be written as : vx , t+1 = combine ( vx , t , aggregate ( { vy , t∣ y ∈ N ( x ) } ) ) , where combine and aggregate are functions , and aggregate is typically permutation-invariant . Once message passing is complete , the final node representations are then used to compute target outputs . Prominent message passing GNN architectures include graph convolutional networks ( GCNs ) ( Kipf & Welling , 2017 ) and gated graph neural networks ( GGNNs ) ( Li et al. , 2016 ) . It is well-known that standard MPNNs have the same power as the 1-dimensional Weisfeiler-Leman algorithm ( 1-WL ) ( Xu et al. , 2019 ; Morris et al. , 2019 ) . This entails that two nodes in a graph can not be distinguished if 1-WL does not distinguish them , and neither can two graphs be distinguished if 1-WL can not distinguish them . G H Figure 1 : G and H are indistinguishable by 1-WL and hence by ( 1-WL ) GNNs . Consider the graphsG andH shown in Figure 1 . 1-WL can not distinguish any two nodes of the graph G. Thus , for example , the invariant function f ∶ V ( G ) → R that maps all nodes in the 4-cycle of G to 1 and all nodes in the triangle to 0 is not expressible ( or approximable ) by a GNN . Moreover , 1-WL can not distinguish the graphs G , H , even though they are obviously non-isomorphic , so no classifier based on the node embeddings computed by an MPNN can distinguish the two graphs . A somewhat trivial limitation in the expressiveness of MPNNs is that information is only propagated along edges , and hence can never be shared between distinct connected components of a graph ( Barceló et al. , 2020 ; Xu et al. , 2019 ) . An easy way to overcome this limitation is by adding global readouts , that is , permutation-invariant functions that aggregate the current states of all nodes1 . Throughout the paper , we therefore focus on MPNNs with global readouts ( also called aggregatecombine GNNs with global readout , i.e. , ACR-GNNs ( Barceló et al. , 2020 ) ) . 3 RELATED WORK & MOTIVATION . Developing more expressive GNNs is an active research area due to the prominence of GNNs for relational learning ( Hamilton et al. , 2017 ) and combinatorial optimization ( Bengio et al. , 2018 ) . As mentioned earlier , standard GNN models are at most as expressive as 1-WL ( Morris et al. , 2019 ; Xu et al. , 2019 ) , and thus can not distinguish between non-isomorphic input instances . In this section , we describe theoretical results quantifying the expressive power of existing GNNs . Higher-order GNNs . We recall the following families of higher-order GNN models : - Invariant ( resp. , equivariant ) graph networks : Invariant ( resp. , equivariant ) graph networks ( Maron et al. , 2019b ) represent graphs as a tensor where node adjacency is directly encoded , and implicitly pass information between nodes through invariant ( resp. , equivariant ) computational blocks . Hence , these models are themselves invariant ( resp. , equivariant ) , a desirable property for computations on graphs . Following intermediate blocks , higher-order tensors are typically returned , and the order of these tensors correlates directly with the expressive power of the overall model . Indeed , invariant networks ( Maron et al. , 2019c ) , and later equivariant networks ( Keriven & Peyré , 2019 ) , are shown to be universal , but with tensor orders of O ( ∣V ∣2 ) , where ∣V ∣ denotes the number of graph nodes . Furthermore , invariant ( resp. , equivariant ) networks with intermediate tensor order k are shown to be equivalent in power to ( k − 1 ) -WL ( Maron et al. , 2019a ) , which is strictly more expressive as k increases ( Cai et al. , 1992 ) . Therefore , such universal higher-order models require intractably-sized intermediate tensors in practice . - Higher-order MPNNs : The k−WL hierarchy has been directly emulated in GNNs , such that these models learn embeddings for tuples of nodes , and perform messaging passing between them , as opposed to individual nodes . This approach has yielded models such as k-GNNs ( Morris et al. , 2019 ) . k-GNNs have ( k−1 ) -WL expressive power,2 but need O ( ∣V ∣k ) memory to run , leading to excessive memory requirements . - Provably powerful graph networks ( PPGNs ) : PPGN is an invariant GNN ( Maron et al. , 2019a ) , based on “ blocks ” of multilayer perceptrons ( MLPs ) and matrix multiplication , which theoretically has 2-WL expressive power , and only requires memory O ( ∣V ∣2 ) ( compared to O ( ∣V ∣3 ) for 3-GNNs ) . However , PPGN theoretically requires exponentially many samples in the number of graph nodes to learn necessary functions for 2-WL expressiveness ( Puny et al. , 2020 ) . GNNs with random node initialization . MPNNs have been enhanced with random node initialization ( Sato et al. , 2020 ) , such that the model trains and runs with partially randomized initial 1In the terminology of Maron et al . ( 2019c ) , such global readouts are tensors of order 1 that are invariant under the symmetric group of the vertex set of the input graph . 2In the literature , one can find different ( though equally expressive ) versions of the Weisfeiler Leman algorithm leading to inconsistent dimension counts . For example , the ( k + 1 ) -WL and the ( k + 1 ) -GNNs of Morris et al . ( 2019 ) are equivalent in expressiveness to the k-WL of Cai et al . ( 1992 ) ; Grohe ( 2017 ) . We follow the version of Cai et al . ( 1992 ) , as it has been adopted as the standard in the literature on graph isomorphism testing . node features . These models , denoted rGNNs , are shown to near-optimally approximate solutions to specific combinatorial optimization problems , and are able to distinguish between 1-WL indistinguishable graph pairs . rGNNs can also detect characteristic sub-graphs in an input graph with high probability . Nonetheless , it remains open as to how much expressive power is exactly gained through RNI , and , in general , whether a GNN model that is universal , scalable , and structure-preserving , can be developed . Our work strongly improves the theoretical result of Sato et al . ( 2020 ) , as it shows universality of MPNNs with RNI , and thus that arbitrary real-valued functions over graphs can be learned by MPNNs with the help of RNI . On the empirical side , we highlight the power of RNI in a significantly more challenging setting than rGNN , using a target function ( SAT ) beyond their theoretical scope . Indeed , for SAT , approximation is known to be hard , and fixed local structures are not useful for prediction . Similar work to RNI has also been conducted in terms of randomly adding features from a predetermined set of colors ( Dasoulas et al. , 2020 ) to disambiguate between nodes . This model , known as CLIP , is similar in spirit to RNI , in that it introduces randomness to node representations , but explicitly makes graphs distinguishable by construction . By contrast , we study random features produced by RNI , which ( i ) are not designed a priori to distinguish nodes , ( ii ) do not explicitly introduce a fixed underlying structure , and ( iii ) yield potentially infinitely many representations for a single graph . In this more general setting , we nonetheless show that RNI adds expressive power to distinguish between nodes with high probability , leads to a universality result , and performs strongly in challenging problem settings .

Summary: The paper studies the how random initialization of node states can improve the expressivity of message passing graph neural networks. Theoretically the paper shows that RNI makes MPNNs universal approximators for invariant functions over graphs. To supplement this claim, the authors evaluate GNNs with RNI and higher-order GNNs over a carefully constructed synthetic dataset and show that RNI (even if only a fraction of the nodes are randomly initialized) are as expressive as higher order GNNs. They also highlight some drawbacks of this approach, such as slower training and sensitivity to hyperparameter tuning.