DeeperGCN: Training Deeper GCNs with Generalized Aggregation Functions

1 INTRODUCTION . The rise of availability of non-Euclidean data ( Bronstein et al. , 2017 ) has recently shed interest into the topic of Graph Convolutional Networks ( GCNs ) . GCNs provide powerful deep learning architectures for irregular data , like point clouds and graphs . GCNs have proven valuable for applications in social networks ( Tang & Liu , 2009 ) , drug discovery ( Zitnik & Leskovec , 2017 ; Wale et al. , 2008 ) , recommendation engines ( Monti et al. , 2017b ; Ying et al. , 2018 ) , and point clouds ( Wang et al. , 2018 ; Li et al. , 2019b ) . Recent works looked at frameworks to train deeper GCN architectures ( Li et al. , 2019b ; a ) . These works demonstrate how increased depth leads to state-of-the-art performance on tasks like point cloud classification and segmentation , and protein interaction prediction . The power of deep models become more evident with the introduction of more challenging and largescale graph datasets . Such datasets were recently introduced in the Open Graph Benchmark ( OGB ) ( Hu et al. , 2020 ) , for tasks of node classification , link prediction , and graph classification . Graph convolutions in GCNs are based on the notion of message passing ( Gilmer et al. , 2017 ) . To compute a new node feature at each GCN layer , information is aggregated from the node and its connected neighbors . Given the nature of graphs , aggregation functions must be permutation invariant . This property guarantees invariance/equivariance to isomorphic graphs ( Battaglia et al. , 2018 ; Xu et al. , 2019b ; Maron et al. , 2019a ) . Popular choices for aggregation functions are mean ( Kipf & Welling , 2016 ) , max ( Hamilton et al. , 2017 ) , and sum ( Xu et al. , 2019b ) . Recent works suggest different aggregations have different performance impact depending on the task . For example , mean and sum perform best in node classification ( Kipf & Welling , 2016 ) , while max is favorable for dealing with 3D point clouds ( Qi et al. , 2017 ; Wang et al. , 2019 ) . Currently , all works rely on empirical analysis to choose aggregation functions . In DeepGCNs ( Li et al . ( 2019b ) ) , the authors complement aggregation functions with residual and dense connections , and dilated convolutions , in order to train very deep GCNs . Equipped with these new modules , GCNs with more than 100 layers can be reliably trained . Despite the potential of these new modules ( Kipf & Welling , 2016 ; Hamilton et al. , 2017 ; Veličković et al. , 2018 ; Xu et al. , 2019a ) , it is still unclear if they are the ideal choice for DeepGCNs when handling large-scale graphs . In this work , we analyze the performance of GCNs on large-scale graphs . In particular , we look at the effect of aggregation functions in performance . We unify aggregation functions by proposing a novel Generalized Aggregation Function ( Figure 1 ) suited for graph convolutions . We show how our function covers all commonly used aggregations ( mean , max , and sum ) , and its parameters can be tuned to learn customized functions for different tasks . Our novel aggregation is fully differentiable and can be learned in an end-to-end fashion in a deep GCN framework . In our experiments , we show the performance of baseline aggregations in various large-scale graph datasets . We then introduce our generalized aggregation and observe improved performance with the correct choice of aggregation parameters . Finally , we demonstrate how learning the parameters of our generalized aggregation , in an end-to-end fashion , leads to state-of-the-art performance in several OGB benchmarks . Our analysis indicates the choice of suitable aggregations is imperative to the performance of different tasks . A differentiable generalized aggregation function ensures the correct aggregation is used for each learning scenario . We summarize our contributions as two-fold : ( 1 ) We propose a novel Generalized Aggregation Function . This new function is suitable for GCNs , as it enjoys a permutation invariant property . We show how our generalized aggregation covers commonly used functions such as mean , max , and sum in graph convolutions . Additionally , we show how its parameters can be tuned to improve performance on diverse GCN tasks . Since this new function is fully differentiable , we show how its parameters can be learned in an end-to-end fashion . ( 2 ) We run extensive experiments on seven datasets from the Open Graph Benchmark ( OGB ) . Our results show that combining depth with our generalized aggregation function achieves state-of-the-art in several of these benchmarks . 2 RELATED WORK . Graph Convolutional Networks ( GCNs ) . Current GCN algorithms can be divided into two categories : spectral-based and spatial-based . Based on spectral graph theory , Bruna et al . ( 2013 ) firstly developed graph convolutions using the Fourier basis of a given graph in the spectral domain . Later , many methods proposed to apply improvements , extensions , and approximations on spectral-based GCNs ( Kipf & Welling , 2016 ; Defferrard et al. , 2016 ; Henaff et al. , 2015 ; Levie et al. , 2018 ; Li et al. , 2018 ; Wu et al. , 2019 ) . Spatial-based GCNs ( Scarselli et al. , 2008 ; Hamilton et al. , 2017 ; Monti et al. , 2017a ; Niepert et al. , 2016 ; Gao et al. , 2018 ; Xu et al. , 2019b ; Veličković et al. , 2018 ) define graph convolution operations directly on the graph by aggregating information from neighbor nodes . To address the scalability issue of GCNs on large-scale graphs , two main categories of algorithms exist : sampling-based ( Hamilton et al. , 2017 ; Chen et al. , 2018b ; Li et al. , 2018 ; Chen et al. , 2018a ; Zeng et al. , 2020 ) and clustering-based ( Chiang et al. , 2019 ) . Training Deep GCNs . Despite the rapid and fruitful progress of GCNs , most prior work employs shallow GCNs . Several works attempt different ways of training deeper GCNs ( Hamilton et al. , 2017 ; Armeni et al. , 2017 ; Rahimi et al. , 2018 ; Xu et al. , 2018 ) . However , all these approaches are limited to 10 layers of depth , after which GCN performance would degrade because of vanishing gradient and over-smoothingLi et al . ( 2018 ) . Inspired by the merits of training deep CNN-based networks ( He et al. , 2016a ; Huang et al. , 2017 ; Yu & Koltun , 2016 ) , DeepGCNs ( Li et al. , 2019b ) propose to train very deep GCNs ( 56 layers ) by adapting residual/dense connections ( ResGCN/DenseGCN ) and dilated convolutions to GCNs . DeepGCN variants achieve state-of-the art results on S3DIS point cloud semantic segmentation ( Armeni et al. , 2017 ) and the PPI dataset . Many recent works focus on further addressing this phenomenon ( Klicpera et al. , 2019 ; Rong et al. , 2020 ; Zhao & Akoglu , 2020 ; Chen et al. , 2020 ; Gong et al. , 2020 ; Rossi et al. , 2020 ) . In particular , Klicpera et al . ( 2019 ) propose a PageRank-based message passing mechanism involving the root node in the loop . Alternatively , DropEdge ( Rong et al. , 2020 ) randomly removes edges from the graph , and PairNorm ( Zhao & Akoglu , 2020 ) develops a novel normalization layer . We find that the choice of aggregation may also limit the power of deep GCNs . In this work , we thoroughly study the important relation between aggregation functions and deep GCN architectures . Aggregation Functions for GCNs . GCNs update a node ’ s feature vector by aggregating feature information from its neighbors in the graph . Many different neighborhood aggregation functions that possess a permutation invariant property have been proposed ( Hamilton et al. , 2017 ; Veličković et al. , 2018 ; Xu et al. , 2019b ) . Specifically , Hamilton et al . ( 2017 ) examine mean , max , and LSTM aggregators , and they empirically find that max and LSTM achieve the best performance . Graph attention networks ( GATs ) ( Veličković et al. , 2018 ) employ the attention mechanism ( Bahdanau et al. , 2015 ) to obtain different and trainable weights for neighbor nodes by learning the attention between their feature vectors and that of the central node . Thus , the aggregator in GATs operates like a learnable weighted mean . Furthermore , Xu et al . ( 2019b ) propose a GCN architecture , denoted Graph Isomorphism Network ( GIN ) , with a sum aggregation that has been shown to have high discriminative power according to the Weisfeiler-Lehman ( WL ) graph isomorphism test ( Weisfeiler & Lehman , 1968 ) . In this work , we propose generalized message aggregation functions , a new family of aggregation functions , that generalizes conventional aggregators including mean , max and sum . With the nature of differentiablity and continuity , generalized message aggregation functions provide a new perspective for designing GCN architectures . 3 REPRESENTATION LEARNING ON GRAPHS . Graph Representation . A graph G is usually defined as a tuple of two sets G = ( V , E ) , where V = { v1 , v2 , ... , vN } and E ⊆ V × V are the sets of vertices and edges , respectively . If an edge eij = ( vi , vj ) ∈ E for an undirected graph , eij is an edge connecting vertices vi and vj ; for a directed graph , eij is an edge directed from vi to vj . Usually , a vertex v and an edge e in the graph are associated with vertex features hv ∈ RD and edge features he ∈ RC respectively.1 GCNs for Learning Graph Representation . We define a general graph representation learning operator F , which takes as input a graph G and outputs a transformed graph G′ , i.e . G′ = F ( G ) . The features or even the topology of the graph can be learned or updated after the transformation F . Typical graph representation learning operators usually learn latent features or representations for graphs such as DeepWalk ( Perozzi et al. , 2014 ) , Planetoid ( Yang et al. , 2016 ) , Node2Vec ( Grover & Leskovec , 2016 ) , Chebyshev graph CNN ( Defferrard et al. , 2016 ) , GCN ( Kipf & Welling , 2016 ) , Neural Message Passing Network ( MPNN ) ( Gilmer et al. , 2017 ) , GraphSage ( Hamilton et al. , 2017 ) , GAT ( Veličković et al. , 2018 ) and GIN ( Xu et al. , 2019b ) . In this work , we focus on the GCN family and its message passing framework ( Gilmer et al. , 2017 ; Battaglia et al. , 2018 ) . To be specific , message passing based on the GCN operator F operating on vertex v ∈ V at the l-th layer is defined as follows : m ( l ) vu = ρ ( l ) ( h ( l ) v , h ( l ) u , h ( l ) evu ) , ∀u ∈ N ( v ) ( 1 ) m ( l ) v = ζ ( l ) ( { m ( l ) vu | u ∈ N ( v ) } ) ( 2 ) h ( l+1 ) v = φ ( l ) ( h ( l ) v , m ( l ) v ) , ( 3 ) where ρ ( l ) , ζ ( l ) , and φ ( l ) are all learnable or differentiable functions for message construction , message aggregation , and vertex update at the l-th layer , respectively . For simplicity , we only consider the case where vertex features are updated at each layer . It is straightforward to extend it to edge features . Message construction function ρ ( l ) is applied to vertex features h ( l ) v of v , its neighbor ’ s features h ( l ) u , and the corresponding edge features hevu to construct an individual message m ( l ) vu for 1In some cases , vertex features or edge features are absent . each neighbor u ∈ N ( v ) . Message aggregation function ζ ( l ) is commonly a permutation invariant set function that takes as input a countable unordered message set { m ( l ) vu | u ∈ N ( v ) } , where m ( l ) vu ∈ RD , and outputs a reduced or aggregated message m ( l ) v ∈ RD . The permutation invariance of ζ ( l ) guarantees the invariance/equivariance to isomorphic graphs ( Battaglia et al. , 2018 ) . ζ ( l ) can simply be a symmetric function such as mean ( Kipf & Welling , 2016 ) , max ( Hamilton et al. , 2017 ) , or sum ( Xu et al. , 2019b ) . Vertex update function φ ( l ) combines the original vertex features h ( l ) v and the aggregated message m ( l ) v to obtain the transformed vertex features h ( l+1 ) v .

Summary: 	The authors propose a generalized neighborhood message aggregation function for GNNs. The proposed choice of generalized aggregation functions is SoftMax and PowerMean, which generalizes Max and Mean functions and interpolates them. Additionally, they propose a variant of these two methods, which can also encompass the Sum function. By making the components of these generalized aggregator functions differentiable, the GNNs can choose an approximate instantiation of the aggregation function that best optimizes the task.