DropGNN: Random Dropouts Increase the Expressiveness of Graph Neural Networks

1 Introduction . Neural networks have been successful in handling various forms of data . Since some of the world ’ s most interesting data is represented by graphs , Graph Neural Networks ( GNNs ) have achieved state-of-the-art performance in various fields such as quantum chemistry , physics , or social networks [ 12 ; 27 ; 18 ] . On the other hand , GNNs are also known to have severe limitations and are sometimes unable to recognize even simple graph structures . In this paper , we present a new approach to increase the expressiveness of GNNs , called Dropout Graph Neural Networks ( DropGNNs ) . Our main idea is to execute not one but multiple different runs of the GNN . We then aggregate the results from these different runs into a final result . In each of these runs , we remove ( “ drop out ” ) each node in the graph with a small probability p. As such , the different runs of an episode will allow us to not only observe the actual extended neighborhood of a node for some number of layers d , but rather to observe various slightly perturbed versions of this d-hop neighborhood . We emphasize that this notion of dropouts is very different from the popular dropout regularization method ; in particular , DropGNNs remove nodes during both training and testing , since their goal is to observe a similar distribution of dropout patterns during training and testing . This dropout technique increases the expressive power of our GNNs dramatically : even when two distinct d-hop neighborhoods can not be distinguished by a standard GNN , their dropout variants ( with a few nodes removed ) are already separable by GNNs in most cases . Thus by learning to identify the dropout patterns where the two d-hop neighborhoods differ , DropGNNs can also distinguish a wide variety of cases that are beyond the theoretical limits of standard GNNs . Our contributions . We begin by showing several example graphs that are not distinguishable in the regular GNN setting but can be easily separated by DropGNNs . We then analyze the theoretical properties of DropGNNs in detail . We first show that executing Õ ( γ ) different runs is often already 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . sufficient to ensure that we observe a reasonable distribution of dropouts in a neighborhood of size γ . We then discuss the theoretical capabilities and limitations of DropGNNs in general , as well as the limits of the dropout approach when combined with specific aggregation methods . We validate our theoretical findings on established problems that are impossible to solve for standard GNNs . We find that DropGNNs clearly outperform the competition on these datasets . We further show that DropGNNs have a competitive performance on several established graph benchmarks , and they provide particularly impressive results in applications where the graph structure is really a crucial factor . 2 Related Work . GNNs apply deep learning to graph-structured data [ 30 ] . In GNNs , every node has an embedding that is shared over multiple iterations with its neighbors . This way nodes can gather their neighbors ’ features . In recent years , many different models have been proposed to realize how the information between nodes is shared [ 35 ] . Some approaches take inspiration from convolution [ 23 ; 8 ; 13 ] , others from graph spectra [ 18 ; 5 ] , others from attention [ 33 ] , and others extend previous ideas of established concepts such as skip connections [ 36 ] . Principally , GNNs are limited in their expressiveness by the Weisfeiler-Lehman test ( WL-test ) [ 37 ] , a heuristic to the graph isomorphism problem . The work of [ 37 ] proposes a new architecture , Graph Isomoprhism Networks ( GIN ) , that is proven to be exactly as powerful as the WL-test . However , even GINs can not distinguish certain different graphs , namely those that the WL-test can not distinguish . This finding [ 11 ] motivated more expressive GNN architectures . These improvements follow two main paths . The first approach augments the features of nodes or edges by additional information to make nodes with similar neighborhoods distinguishable . Several kinds of information have been used : inspired from distributed computing are port numbers on edges [ 28 ] , unique IDs for nodes [ 20 ] , or random features on nodes [ 29 ; 1 ] . Another idea is to use angles between edges [ 19 ] from chemistry ( where edges correspond to electron bonds ) . However , all of these approaches have some shortcomings . For ports and angles , there are some simple example graphs that still can not be distinguished with these extensions [ 11 ] . Adding IDs or random features helps during training , but the learned models do not generalize : GNNs often tend to overfit to the specific random values in the training set , and as such , they produce weaker results on unseen test graphs that received different random values . In contrast to this , DropGNNs observe a similar distribution of embeddings during training and testing , and hence they also generalize well to test set graphs . The second approach exploits the fact that running the WL-test on tuples , triples , or generally ktuples keeps increasing its expressiveness . Thus a GNN operating on tuples of nodes has higher expressiveness than a standard GNN [ 22 ; 21 ] . However , the downside of this approach is that even building a second-order graph blows up the graph quadratically . The computational cost quickly becomes a problem that needs to be to addressed , for example with sampling [ 22 ] . Furthermore , second-order graph creation is a global operation of the graph that destroys the local semantics induced by the edges . In contrast to this , DropGNN can reason about graphs beyond the WL-test with only a small overhead ( through run repetition ) , while also keeping the local graph structure intact . Our work is also somewhat similar to the randomized smoothing approach [ 7 ] , which has also been extended to GNNs recently [ 4 ] . This approach also conducts multiple runs on slightly perturbed variants of the data . However , in randomized smoothing , the different embeddings are combined in a smoothing operation ( e.g . majority voting ) , which specifically aims to get rid of the atypical perturbed variants in order to increase robustness . In contrast to this , the main idea of DropGNNs is exactly to find and identify these perturbed special cases which are notably different from the original neighborhood , since these allow us to distinguish graphs that otherwise seem identical . Finally , we note that removing nodes is a common tool for regularization in deep neural networks , which has also seen use in GNNs [ 26 ; 9 ] . However , as mentioned before , this is a different dropout concept where nodes are only removed during training to reduce the co-dependence of nodes . 3 DropGNN . 3.1 About GNNs . Almost all GNN architectures [ 33 ; 18 ; 37 ; 8 ; 35 ; 13 ; 36 ] follow the message passing framework [ 12 ; 3 ] . Every node starts with an embedding given by its initial features . One round of message passing has three steps . In the first MESSAGE step , nodes create a message based on their embedding and send this message to all neighbors . Second , nodes AGGREGATE all messages they receive . Third , every node UPDATEs its embedding based on its old embedding and the aggregated messages . One such round corresponds to one GNN layer . Usually , a GNN performs d rounds of message passing for some small constant d. Thus , the node ’ s embedding in a GNN reflects its features and the information within its d-hop neighborhood . Finally , a READOUT method translates these final embeddings into predictions . Usually , MESSAGE , AGGREGATE , UPDATE and READOUT are functions with learnable parameters , for instance linear layers with activation functions . This GNN paradigm is closely related to the WL-test for a pair of graphs , which is an iterative color refinement procedure . In rounds 1 , ... , d , each node looks at its own color and the multiset of colors of its direct neighbors , and uses a hash function to select a new color based on this information . As such , if the WL-test can not distinguish two graphs , then a standard GNN can not distinguish them either : intuitively , the nodes in these graphs receive the same messages and create the same embedding in each round , and thus they always arrive at the same final result . 3.2 Idea and motivation . The main idea of DropGNNs is to execute multiple independent runs of the GNN during both training and testing . In each run , every node of the GNN is removed with probability p , independently from all other nodes . If a node v is removed during a run , then v does not send or receive any messages to/from its neighbors and does not affect the remaining nodes in any way . Essentially , the GNN behaves as if v ( and its incident edges ) were not present in the graph in the specific run , and no embedding is computed for v in this run ( see Figure 1 for an illustration ) . Over the course of multiple runs , dropouts allow us to not only observe the d-hop neighborhood around any node u , but also several slightly perturbed variants of this d-hop neighborhood . In the different runs , the embedding computed for umight also slightly vary , depending on which node ( s ) are missing from its d-hop neighborhood in a specific run . This increases the expressive power of GNNs significantly : even when two different d-hop neighborhoods can not be distinguished by standard GNNs , the neighborhood variants observed when removing some of the nodes are usually still remarkably different . In Section 3.4 , we discuss multiple examples for this improved expressiveness . Our randomized approach means that in different runs , we will have different nodes dropping out of the GNN . As such , the GNN is only guaranteed to produce the same node embeddings in two runs if we have exactly the same subset of nodes dropping out . Given the d-hop neighborhood of a node u , we will refer to a specific subset of nodes dropping out as a dropout combination , or more concretely as a k-dropout in case the subset has size k. In order to analyze the d-hop neighborhood of u , the reasonable strategy is to use a relatively small dropout probability p : this ensures that in each run , only a few nodes are removed ( or none at all ) , and thus the GNN will operate on a d-hop neighborhood that is similar to the original neighborhood of u . As a result , 1-dropouts will be frequent , while for a larger k , observing a k-dropout will be unlikely . To reduce the effect of randomization on the final outcome , we have to execute multiple independent runs of our GNN ; we denote this number of runs by r. For a successful application of the dropout idea , we have to select r large enough to ensure that the set of observed dropout combinations is already reasonably close to the actual probability distribution of dropouts . In practice , this will not be feasible for k-dropouts with large k that occur very rarely , but we can already ensure for a reasonably small r that e.g . the frequency of each 1-dropout is relatively close to its expected value .

Summary: The paper proposes Dropout Graph Neural Networks (DropGNNs), an approach that utlizes dropout in a way that allows GNNs to be more expressive. In DropGNN, multiple runs  are conducted where, in each run, the nodes of the graph are dropped out with a certain small probability and passed through a  GNN to produce a distinct embedding. The embeddings of the multiple runs are aggregated (with a sum) after a non-linear  transformation of the node features. DropGNNs achieve competitive results on synthetic and real-world datasets.