Don't stack layers in graph neural networks, wire them randomly

1 INTRODUCTION . Data defined over the nodes of graphs are ubiquitous . Social network profiles ( Hamilton et al. , 2017 ) , molecular interactions ( Duvenaud et al. , 2015 ) , citation networks ( Sen et al. , 2008 ) , 3D point clouds ( Simonovsky & Komodakis , 2017 ) are just examples of a wide variety of data types where describing the domain as a graph allows to encode constraints and patterns among the data points . Exploiting the graph structure is crucial in order to extract powerful representations of the data . However , this is not a trivial task and only recently graph neural networks ( GNNs ) have started showing promising approaches to the problem . GNNs ( Wu et al. , 2020 ) extend the deep learning toolbox to deal with the irregularity of the graph domain . Much of the work has been focused on defining a graph convolution operation ( Bronstein et al. , 2017 ) , i.e. , a layer that is well-defined over the graph domain but also retains some of the key properties of convolution such as weight reuse and locality . A wide variety of such graph convolution operators has been defined over the years , mostly based on neighborhood aggregation schemes where the features of a node are transformed by processing the features of its neighbors . Such schemes have been shown to be as powerful as the Weisfeiler-Lehman graph isomorphism test ( Weisfeiler & Lehman , 1968 ; Xu et al. , 2019 ) , enabling them to simultaneuosly learn data features and graph topology . However , contrary to classic literature on CNNs , few works ( Li et al. , 2019a ; Dehmamy et al. , 2019 ; Xu et al. , 2018 ; Dwivedi et al. , 2020 ) addressed GNNs architectures and their role in extracting powerful representations . Several works , starting with the early GCN ( Kipf & Welling , 2017 ) , noticed an inability to build deep GNNs , often resulting in worse performance than that of methods that disregard the graph domain , when trying to build anything but very shallow networks . This calls for exploring whether advances on CNN architectures can be translated to the GNN space , while understanding the potentially different needs of graph representation learning . Li et al . ( 2019b ) suggest that GCNs suffer from oversmoothing as several layers are stacked , resulting in the extraction of mostly low-frequency features . This is related to the lack of self-loop information in this specific graph convolution . It is suggested that ResNet-like architectures mitigate the problem as the skip connections supply high frequency contributions . Xu et al . ( 2018 ) point out that the size of the receptive field of a node , i.e. , which nodes contribute to the features of the node under consideration , plays a crucial role , but it can vary widely depending on the graph and too large receptive fields may actually harm performance . They conclude that for graph-based problems it would be optimal to learn how to adaptively merge contributions from receptive fields of multiple size . For this reason they propose an architecture where each layer has a skip connection to the output so that contributions at multiple depths ( hence sizes of receptive fields ) can be merged . Nonetheless , the problem of finding methods for effectively increasing the capacity of graph neural networks is still standing , since stacking many layers has been proven to provide limited improvements ( Li et al. , 2019b ; Oono & Suzuki , 2019 ; Alon & Yahav , 2020 ; NT & Maehara , 2019 ) . In this paper , we argue that the recently proposed randomly wired architectures ( Xie et al. , 2019 ) are ideal for GNNs . In a randomly wired architecture , “ layers ” are arranged according to a random directed acyclic graph and data are propagated through the paths towards the output . Such architecture is ideal for GNNs because it realizes the intuition of Xu et al . ( 2018 ) of being able of merging receptive fields of varied size . Indeed , the randomly wired network can be seen as an extreme generalization of their jumping network approach where layer outputs can not only jump to the network output but to other layers as well , continuously merging receptive fields . Hence , randomly wired architectures provide a way of effectively scaling up GNNs , mitigating the depth problem and creating richer representations . Fig . 1 shows a graphical representation of this concept by highlighting the six layers directly contributing to the output , having different receptive fields induced by the distribution of paths from the input . Our novel contributions can be summarized as follows : i ) we are the first to analyze randomly wired architectures and show that they are generalizations of ResNets when looked at as ensembles of paths ( Veit et al. , 2016 ) ; ii ) we show that path ensembling allows to merge receptive fields of varied size and that it can do so adaptively , i.e. , trainable weights on the architecture edges can tune the desired size of the receptive fields to be merged to achieve an optimal configuration for the problem ; iii ) we introduce improvements to the basic design of randomly wired architectures by optionally embedding a path that sequentially goes through all layers in order to promote larger receptive fields when needed , and by presenting MonteCarlo DropPath , which decorrelates path contributions by randomly dropping architecture edges ; iv ) we provide extensive experimental evidence , using a recently introduced benchmarking framework ( Dwivedi et al. , 2020 ) to ensure significance and reproducibility , that randomly wired architectures consistently outperform ResNets , often by large margins , for five of the most popular graph convolution definitions on three different tasks . 2 BACKGROUND . 2.1 GRAPH NEURAL NETWORKS . A major shortcoming of CNNs is that they are unable to process data defined on irregular domains . In particular , one case that is drawing attention is when the data structure can be described by a graph and the data are defined as vectors on the graph nodes . This setting can be found in many applications , including 3D point clouds ( Wang et al. , 2019 ; Valsesia et al. , 2019 ) , computational biology ( Alipanahi et al. , 2015 ; Duvenaud et al. , 2015 ) , and social networks ( Kipf & Welling , 2017 ) . However , extending CNNs from data with a regular structure , such as images and video , to graph-structured data is not straightforward if one wants to preserve useful properties such as locality and weight reuse . GNNs redefine the convolution operation so that the new layer definition can be used on domains described by graphs . The most widely adopted graph convolutions in the literature rely on message passing , where a weighted aggregation of the feature vectors in a neighborhood is computed . The GCN ( Kipf & Welling , 2017 ) is arguably the simplest definition , applying the same linear transformation to all the node features , followed by neighborhood aggregation and non-linear activation : h ( l+1 ) i = σ 1 |Ni| ∑ j∈Ni Wh ( l ) j . Variants of this definition have been developed , e.g. , GraphSage ( Hamilton et al. , 2017 ) concatenates the feature vector of node i to the feature vectors of its neighbors , so that self-information can also be exploited ; GIN ( Xu et al. , 2019 ) uses a multilayer perceptron instead of a linear transform , replaces average with sum to ensure injectivity and proposes a different way of computing the output by using all the feature vectors produced by the intermediate layers . These definitions are all isotropic because they treat every edge in the same way . It has been observed that better representation capacity can be achieved using anistropic definitions , where every edge can have a different transformation , at the cost of increased computational complexity . The Gated GCN ( Bresson & Laurent , 2017 ) and GAT ( Veličković et al. , 2017 ) definitions fall in this category . 2.2 RANDOMLY WIRED ARCHITECTURES . In recent work , Xie et al . ( 2019 ) explore whether it is possible to avoid handcrafted design of neural network architectures and , at the same time , avoid expensive neural architecture search methods ( Elsken et al. , 2019 ) , by designing random architecture generators . They show that “ layers ” performing convolution , normalization and non-linear activation can be connected in a random architecture graph . Strong performance is observed on the traditional image classification task by outperforming state-of-the-art architectures . The authors conjecture that random architectures generalize ResNets and similar constructions , but the underlying principles of their excellent performance are unclear , as well as whether the performance translates to tasks other than image recognition or to operations other than convolution on grids . 3 RANDOMLY WIRED GNNS . In this section , we first introduce randomly wired architectures and the notation we are going to use . We then analyze their behavior when viewed as ensembles of paths . ∑ RELU GCONV BN 𝜎 ( wia ) 𝜎 ( wib ) 𝜎 ( wic ) h ( i ) h ( a ) h ( b ) h ( c ) Figure 2 : An architecture node is equivalent to a GNN layer . A randomly wired architecture consists of a directed acyclic graph ( DAG ) connecting a source architecture node , which is fed with the input data , to a sink architecture node . One should not confuse the architecture DAG with the graph representing the GNN domain : to avoid any source of confusion we will use the terms architecture nodes ( edges ) and domain nodes ( edges ) , respectively . A domain node is a node of the graph that is fed as input to the GNN . An architecture node is effectively a GNN layer performing the following operations ( Fig . 2 ) : i ) aggregation of the inputs from other architecture nodes via a weighted sum as in ( Xie et al. , 2019 ) : h ( i ) = ∑ j∈Ai ωijh ( j ) = ∑ j∈Ai σ ( wij ) h ( j ) , i = 1 , ... , L− 1 ( 1 ) 3 being σ a sigmoid function , Ai the set of direct predecessors of the architecture node i , and wij a scalar trainable weight ; ii ) a non-linear activation ; iii ) a graph-convolution operation ( without output activation ) ; iv ) batch normalization . The architecture DAG is generated using a random graph generator . In this paper , we will focus on the Erdős-Renyi model where the adjacency matrix of the DAG is a strictly upper triangular matrix with entries being realizations of a Bernoulli random variable with probability p. If multiple input architecture nodes are randomly generated , they are all wired to a single global input . Multiple output architecture nodes are averaged to obtain a global output . Other random generators may be used , e.g. , small-world and scale-free random networks have been studied in ( Xie et al. , 2019 ) . However , a different generator will display a different behavior concerning the properties we study in Sec . 3.1 . 3.1 RANDOMLY WIRED ARCHITECTURES BEHAVE LIKE PATH ENSEMBLES . It has already been shown that ResNets behave like ensembles of relatively shallow networks , where one can see the ResNet architecture as a collection of paths of varied lengths ( Veit et al. , 2016 ) . More specifically , in a ResNet with n layers , where all layers have a skip connection except the first one and the last one , there are exactly 2L−2 paths , whose lengths follow a Binomial distribution ( i.e. , the number of paths of length l from layer k to the last layer is ( L−k−1 l−2 ) ) , and the average path length is L 2 + 1 ( Veit et al. , 2016 ) . In this section , we show that a randomly wired neural network can also be considered as an ensemble of networks with varied depth . However , in this case , the distribution of the path length is different from the one obtained with the ResNet , as shown in the following lemma ( proof in the supplementary material ) . Lemma 3.1 . Let us consider a randomly wired network with L architecture nodes , where the architecture DAG is generated according to a Erdős-Renyi graph generator with probability p. The average number of paths of length l from node k to the sink , where k < L , is E [ N ( k ) l ] = ( L−k−1 l−2 ) pl−1 and the average total number of paths from node k to the sink is E [ N ( k ) ] = p ( 1 + p ) L−k−1 . We can observe that if p = 1 , the randomly wired network converges to the ResNet architecture . This allows to think of randomly wired architectures as generalizations of ResNets as they enable increased flexibility in the number and distribution of paths instead of enforcing the use of all 2L−2 .

Summary: The paper proposes a new method for building graph convolutional neural networks. It shows, that during the building of the network, instead of stacking many layers and adding the residual connection between them, one could employ a randomly-wired architecture, that can be a more effective way to increase the capacity of the network and thus it could obtain richer representations. The proposed method is an interesting direction in the field of graph convolutional neural networks. The new method could be seen asa generalization of the residual networks and the jumping knowledge networks.