DropGNN: Random Dropouts Increase the Expressiveness of Graph Neural Networks

1 Introduction . Neural networks have been successful in handling various forms of data . Since some of the world ’ s most interesting data is represented by graphs , Graph Neural Networks ( GNNs ) have achieved state-of-the-art performance in various fields such as quantum chemistry , physics , or social networks [ 12 ; 27 ; 18 ] . On the other hand , GNNs are also known to have severe limitations and are sometimes unable to recognize even simple graph structures . In this paper , we present a new approach to increase the expressiveness of GNNs , called Dropout Graph Neural Networks ( DropGNNs ) . Our main idea is to execute not one but multiple different runs of the GNN . We then aggregate the results from these different runs into a final result . In each of these runs , we remove ( “ drop out ” ) each node in the graph with a small probability p. As such , the different runs of an episode will allow us to not only observe the actual extended neighborhood of a node for some number of layers d , but rather to observe various slightly perturbed versions of this d-hop neighborhood . We emphasize that this notion of dropouts is very different from the popular dropout regularization method ; in particular , DropGNNs remove nodes during both training and testing , since their goal is to observe a similar distribution of dropout patterns during training and testing . This dropout technique increases the expressive power of our GNNs dramatically : even when two distinct d-hop neighborhoods can not be distinguished by a standard GNN , their dropout variants ( with a few nodes removed ) are already separable by GNNs in most cases . Thus by learning to identify the dropout patterns where the two d-hop neighborhoods differ , DropGNNs can also distinguish a wide variety of cases that are beyond the theoretical limits of standard GNNs . Our contributions . We begin by showing several example graphs that are not distinguishable in the regular GNN setting but can be easily separated by DropGNNs . We then analyze the theoretical properties of DropGNNs in detail . We first show that executing Õ ( γ ) different runs is often already 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . sufficient to ensure that we observe a reasonable distribution of dropouts in a neighborhood of size γ . We then discuss the theoretical capabilities and limitations of DropGNNs in general , as well as the limits of the dropout approach when combined with specific aggregation methods . We validate our theoretical findings on established problems that are impossible to solve for standard GNNs . We find that DropGNNs clearly outperform the competition on these datasets . We further show that DropGNNs have a competitive performance on several established graph benchmarks , and they provide particularly impressive results in applications where the graph structure is really a crucial factor . 2 Related Work . GNNs apply deep learning to graph-structured data [ 30 ] . In GNNs , every node has an embedding that is shared over multiple iterations with its neighbors . This way nodes can gather their neighbors ’ features . In recent years , many different models have been proposed to realize how the information between nodes is shared [ 35 ] . Some approaches take inspiration from convolution [ 23 ; 8 ; 13 ] , others from graph spectra [ 18 ; 5 ] , others from attention [ 33 ] , and others extend previous ideas of established concepts such as skip connections [ 36 ] . Principally , GNNs are limited in their expressiveness by the Weisfeiler-Lehman test ( WL-test ) [ 37 ] , a heuristic to the graph isomorphism problem . The work of [ 37 ] proposes a new architecture , Graph Isomoprhism Networks ( GIN ) , that is proven to be exactly as powerful as the WL-test . However , even GINs can not distinguish certain different graphs , namely those that the WL-test can not distinguish . This finding [ 11 ] motivated more expressive GNN architectures . These improvements follow two main paths . The first approach augments the features of nodes or edges by additional information to make nodes with similar neighborhoods distinguishable . Several kinds of information have been used : inspired from distributed computing are port numbers on edges [ 28 ] , unique IDs for nodes [ 20 ] , or random features on nodes [ 29 ; 1 ] . Another idea is to use angles between edges [ 19 ] from chemistry ( where edges correspond to electron bonds ) . However , all of these approaches have some shortcomings . For ports and angles , there are some simple example graphs that still can not be distinguished with these extensions [ 11 ] . Adding IDs or random features helps during training , but the learned models do not generalize : GNNs often tend to overfit to the specific random values in the training set , and as such , they produce weaker results on unseen test graphs that received different random values . In contrast to this , DropGNNs observe a similar distribution of embeddings during training and testing , and hence they also generalize well to test set graphs . The second approach exploits the fact that running the WL-test on tuples , triples , or generally ktuples keeps increasing its expressiveness . Thus a GNN operating on tuples of nodes has higher expressiveness than a standard GNN [ 22 ; 21 ] . However , the downside of this approach is that even building a second-order graph blows up the graph quadratically . The computational cost quickly becomes a problem that needs to be to addressed , for example with sampling [ 22 ] . Furthermore , second-order graph creation is a global operation of the graph that destroys the local semantics induced by the edges . In contrast to this , DropGNN can reason about graphs beyond the WL-test with only a small overhead ( through run repetition ) , while also keeping the local graph structure intact . Our work is also somewhat similar to the randomized smoothing approach [ 7 ] , which has also been extended to GNNs recently [ 4 ] . This approach also conducts multiple runs on slightly perturbed variants of the data . However , in randomized smoothing , the different embeddings are combined in a smoothing operation ( e.g . majority voting ) , which specifically aims to get rid of the atypical perturbed variants in order to increase robustness . In contrast to this , the main idea of DropGNNs is exactly to find and identify these perturbed special cases which are notably different from the original neighborhood , since these allow us to distinguish graphs that otherwise seem identical . Finally , we note that removing nodes is a common tool for regularization in deep neural networks , which has also seen use in GNNs [ 26 ; 9 ] . However , as mentioned before , this is a different dropout concept where nodes are only removed during training to reduce the co-dependence of nodes . 3 DropGNN . 3.1 About GNNs . Almost all GNN architectures [ 33 ; 18 ; 37 ; 8 ; 35 ; 13 ; 36 ] follow the message passing framework [ 12 ; 3 ] . Every node starts with an embedding given by its initial features . One round of message passing has three steps . In the first MESSAGE step , nodes create a message based on their embedding and send this message to all neighbors . Second , nodes AGGREGATE all messages they receive . Third , every node UPDATEs its embedding based on its old embedding and the aggregated messages . One such round corresponds to one GNN layer . Usually , a GNN performs d rounds of message passing for some small constant d. Thus , the node ’ s embedding in a GNN reflects its features and the information within its d-hop neighborhood . Finally , a READOUT method translates these final embeddings into predictions . Usually , MESSAGE , AGGREGATE , UPDATE and READOUT are functions with learnable parameters , for instance linear layers with activation functions . This GNN paradigm is closely related to the WL-test for a pair of graphs , which is an iterative color refinement procedure . In rounds 1 , ... , d , each node looks at its own color and the multiset of colors of its direct neighbors , and uses a hash function to select a new color based on this information . As such , if the WL-test can not distinguish two graphs , then a standard GNN can not distinguish them either : intuitively , the nodes in these graphs receive the same messages and create the same embedding in each round , and thus they always arrive at the same final result . 3.2 Idea and motivation . The main idea of DropGNNs is to execute multiple independent runs of the GNN during both training and testing . In each run , every node of the GNN is removed with probability p , independently from all other nodes . If a node v is removed during a run , then v does not send or receive any messages to/from its neighbors and does not affect the remaining nodes in any way . Essentially , the GNN behaves as if v ( and its incident edges ) were not present in the graph in the specific run , and no embedding is computed for v in this run ( see Figure 1 for an illustration ) . Over the course of multiple runs , dropouts allow us to not only observe the d-hop neighborhood around any node u , but also several slightly perturbed variants of this d-hop neighborhood . In the different runs , the embedding computed for umight also slightly vary , depending on which node ( s ) are missing from its d-hop neighborhood in a specific run . This increases the expressive power of GNNs significantly : even when two different d-hop neighborhoods can not be distinguished by standard GNNs , the neighborhood variants observed when removing some of the nodes are usually still remarkably different . In Section 3.4 , we discuss multiple examples for this improved expressiveness . Our randomized approach means that in different runs , we will have different nodes dropping out of the GNN . As such , the GNN is only guaranteed to produce the same node embeddings in two runs if we have exactly the same subset of nodes dropping out . Given the d-hop neighborhood of a node u , we will refer to a specific subset of nodes dropping out as a dropout combination , or more concretely as a k-dropout in case the subset has size k. In order to analyze the d-hop neighborhood of u , the reasonable strategy is to use a relatively small dropout probability p : this ensures that in each run , only a few nodes are removed ( or none at all ) , and thus the GNN will operate on a d-hop neighborhood that is similar to the original neighborhood of u . As a result , 1-dropouts will be frequent , while for a larger k , observing a k-dropout will be unlikely . To reduce the effect of randomization on the final outcome , we have to execute multiple independent runs of our GNN ; we denote this number of runs by r. For a successful application of the dropout idea , we have to select r large enough to ensure that the set of observed dropout combinations is already reasonably close to the actual probability distribution of dropouts . In practice , this will not be feasible for k-dropouts with large k that occur very rarely , but we can already ensure for a reasonably small r that e.g . the frequency of each 1-dropout is relatively close to its expected value .

Summary: The authors propose DropGNN, a GNN exension where multiple forward passes of the model are aggregated. In each of the passes, each node in the graph is dropped with probability $p$ independently. Aggregating a sufficiently large number of dropout combinations effectively breaks symmetries that lead to traditional GNNs to not be able to distinguish between certain neighborhoods. The authors present theoretical results that DropGNNs can distinguish neighborhoods that traditional GNNs (e.g., GIN) cannot. The authors evaluate their method on both synthetic and real-world datasets.