A PAC-Bayesian Approach to Generalization Bounds for Graph Neural Networks

1 INTRODUCTION . Graph neural networks ( GNNs ) ( Gori et al. , 2005 ; Scarselli et al. , 2008 ; Bronstein et al. , 2017 ; Battaglia et al. , 2018 ) have become very popular recently due to their ability to learn powerful representations from graph-structured data , and have achieved state-of-the-art results in a variety of application domains such as social networks ( Hamilton et al. , 2017 ; Xu et al. , 2018 ) , quantum chemistry ( Gilmer et al. , 2017 ; Chen et al. , 2019a ) , computer vision ( Qi et al. , 2017 ; Monti et al. , 2017 ) , reinforcement learning ( Sanchez-Gonzalez et al. , 2018 ; Wang et al. , 2018 ) , robotics ( Casas et al. , 2019 ; Liang et al. , 2020 ) , and physics ( Henrion et al. , 2017 ) . Given a graph along with node/edge features , GNNs learn node/edge representations by propagating information on the graph via local computations shared across the nodes/edges . Based on the specific form of local computation employed , GNNs can be divided into two categories : graph convolution based GNNs ( Bruna et al. , 2013 ; Duvenaud et al. , 2015 ; Kipf & Welling , 2016 ) and message passing based GNNs ( Li et al. , 2015 ; Dai et al. , 2016 ; Gilmer et al. , 2017 ) . The former generalizes the convolution operator from regular graphs ( e.g. , grids ) to ones with arbitrary topology , whereas the latter mimics message passing algorithms and parameterizes the shared functions via neural networks . Due to the tremendous empirical success of GNNs , there is increasing interest in understanding their theoretical properties . For example , some recent works study their expressiveness ( Maron et al. , 2018 ; Xu et al. , 2018 ; Chen et al. , 2019b ) , that is , what class of functions can be represented by GNNs . However , only few works investigate why GNNs generalize so well to unseen graphs . They are either restricted to a specific model variant ( Verma & Zhang , 2019 ; Du et al. , 2019 ; Garg et al. , 2020 ) or have loose dependencies on graph statistics ( Scarselli et al. , 2018 ) . On the other hand , GNNs have close ties to standard feedforward neural networks , e.g. , multi-layer perceptrons ( MLPs ) and convolutional neural networks ( CNNs ) . In particular , if each i.i.d . sample is viewed as a node , then the whole dataset becomes a graph without edges . Therefore , GNNs can be seen as generalizations of MLPs/CNNs since they model not only the regularities within a sample but also the dependencies among samples as defined in the graph . It is therefore natural to ask if we can generalize the recent advancements on generalization bounds for MLPs/CNNs ( Harvey et al. , 2017 ; Neyshabur et al. , 2017 ; Bartlett et al. , 2017 ; Dziugaite & Roy , 2017 ; Arora et al. , 2018 ; 2019 ) to GNNs , and how would graph structures affect the generalization bounds ? In this paper , we answer the above questions by proving generalization bounds for the two primary classes of GNNs , i.e. , graph convolutional networks ( GCNs ) ( Kipf & Welling , 2016 ) and messagepassing GNNs ( MPGNNs ) ( Dai et al. , 2016 ; Jin et al. , 2018 ) . Our generalization bound for GCNs shows an intimate relationship with the bounds for MLPs/CNNs with ReLU activations ( Neyshabur et al. , 2017 ; Bartlett et al. , 2017 ) . In particular , they share the same term , i.e. , the product of the spectral norms of the learned weights at each layer multiplied by a factor that is additive across layers . The bound for GCNs has an additional multiplicative factor d ( l 1 ) /2 where d 1 is the maximum node degree and l is the network depth . Since MLPs/CNNs are special GNNs operating on graphs without edges ( i.e. , d 1 = 0 ) , the bound for GCNs coincides with the ones for MLPs/CNNs with ReLU activations ( Neyshabur et al. , 2017 ) on such degenerated graphs . Therefore , our result is a natural generalization of the existing results for MLPs/CNNs . Our generalization bound for message passing GNNs reveals that the governing terms of the bound are similar to the ones of GCNs , i.e. , the geometric series of the learned weights and the multiplicative factor dl 1 . The geometric series appears due to the weight sharing across message passing steps , thus corresponding to the product term across layers in GCNs . The term dl 1 encodes the key graph statistics . Our bound improves the dependency on the maximum node degree and the maximum hidden dimension compared to the recent Rademacher complexity based bound ( Garg et al. , 2020 ) . Moreover , we compute the bound values on four real-world graph datasets ( e.g. , social networks and protein structures ) and verify that our bounds are tighter . In terms of the proof techniques , our analysis follows the PAC-Bayes framework in the seminal work of ( Neyshabur et al. , 2017 ) for MLPs/CNNs with ReLU activations . However , we make two distinctive contributions which are customized for GNNs . First , a naive adaptation of the perturbation analysis in ( Neyshabur et al. , 2017 ) does not work for GNNs since ReLU is not 1-Lipschitz under the spectral norm , i.e. , kReLU ( X ) k2 kXk2 does not hold for some real matrix X . Instead , we construct the recursion on certain node representations of GNNs like the one with maximum ` 2 norm , so that we can perform perturbation analysis with vector 2-norm . Second , in contrast to ( Neyshabur et al. , 2017 ) which only handles the homogeneous networks , i.e. , f ( ax ) = af ( x ) when a 0 , we properly construct a quantity of the learned weights which 1 ) provides a way to satisfy the constraints of the previous perturbation analysis and 2 ) induces a finite covering on the range of the quantity so that the PAC-Bayes bound holds for all possible weights . This generalizes the analysis to non-homogeneous GNNs like typical MPGNNs . The rest of the paper is organized as follows . In Section 2 , we introduce background material necessary for our analysis . We then present our generalization bounds and the comparison to existing results in Section 3 . We also provide an empirical study to support our theoretical arguments in Section 4 . At last , we discuss the extensions , limitations and some open problems . 2 BACKGROUND . In this section , we first explain our analysis setup including notation and assumptions . We then describe the two representative GNN models in detail . Finally , we review the PAC-Bayes analysis . 2.1 ANALYSIS SETUP . In the following analysis , we consider the K-class graph classification problem which is common in the GNN literature , where given a graph sample z , we would like to classify it into one of the predefined K classes . We will discuss extensions to other problems like graph regression in Section 5 . Each graph sample z is a triplet of an adjacency matrix A , node features X 2 Rn⇥h0 and output label y 2 R1⇥K , i.e . z = ( A , X , y ) , where n is the number of nodes and h0 is the input feature dimension . We start our discussion by defining our notations . Let N+k be the first k positive integers , i.e. , N+k = { 1 , 2 , . . . , k } , | · |p the vector p-norm and k · kp the operator norm induced by the vector p-norm . Further , k · kF denotes the Frobenius norm of a matrix , e the base of the natural logarithm function log , A [ i , j ] the ( i , j ) -th element of matrix A and A [ i , : ] the i-th row . We use parenthesis to avoid the ambiguity , e.g. , ( AB ) [ i , j ] means the ( i , j ) -th element of the product matrix AB . We then introduce some terminologies from statistical learning theory and define the sample space as Z , z = ( A , X , y ) 2 Z where X 2 X ( node feature space ) and A 2 G ( graph space ) , data distribution D , z iid⇠ D , hypothesis ( or model ) fw where fw 2 H ( hypothesis class ) , and training set S with size m , S = { z1 , . . . , zm } . We make the following assumptions which also appear in the literature : A1 Data , i.e. , triplets ( A , X , y ) , are i.i.d . samples drawn from some unknown distribution D. A2 The maximum hidden dimension across all layers is h. A3 Node feature of any graph is contained in a ` 2-ball with radius B . Specifically , we have 8i 2 N+n , the i-th node feature X [ i , : ] 2 XB , h0 = { x 2 Rh0 | Ph0 j=1 x 2 j B2 } . A4 We only consider simple graphs ( i.e. , undirected , no loops1 , and no multi-edges ) with maximum node degree as d 1 . Note that it is straightforward to estimate B and d empirically on real-world graph data . 2.2 GRAPH NEURAL NETWORKS ( GNNS ) . In this part , we describe the details of the GNN models and the loss function we used for the graph classification problem . The essential idea of GNNs is to propagate information over the graph so that the learned representations capture the dependencies among nodes/edges . We now review two classes of GNNs , GCNs and MPGNNs , which have different mechanisms for propagating information . We choose them since they are the most popular variants and represent two common types of neural networks , i.e. , feedforward ( GCNs ) and recurrent ( MPGNNs ) neural networks . We discuss the extension of our analysis to other GNN variants in Section 5 . For ease of notation , we define the model to be fw 2 H : X ⇥ G ! RK where w is the vectorization of all model parameters . GCNs : Graph convolutional networks ( GCNs ) ( Kipf & Welling , 2016 ) for the K-class graph classification problem can be defined as follows , Hk = k ⇣ L̃Hk 1Wk ⌘ ( k-th Graph Convolution Layer ) Hl = 1 n 1nHl 1Wl ( Readout Layer ) , ( 1 ) where k 2 N+l 1 , Hk 2 Rn⇥hk are the node representations/states , 1n 2 R1⇥n is a all-one vector , l is the number of layers.2 and Wj is the weight matrix of the j-th layer . The initial node state is the observed node feature H0 = X . For both GCNs and MPGNNs , we consider l > 1 since otherwise the model degenerates to a linear transformation which does not leverage the graph and is trivial to analyze . Due to assumption A2 , Wj is of size at most h ⇥ h , i.e. , hk h , 8k 2 N+l 1 . The graph Laplacian L̃ is defined as , Ã = I + A , L̃ = D 12 ÃD 12 where D is the degree matrix of Ã . Note that the maximum eigenvalue of L̃ is 1 in this case . We absorb the bias into the weight by appending constant 1 to the node feature . Typically , GCNs use ReLU as the non-linearity , i.e. , i ( x ) = max ( 0 , x ) , 8i = 1 , · · · , l 1 . We use the common mean-readout to obtain the graph representation where Hl 1 2 Rn⇥hl 1 , Wl 2 Rhl 1⇥K , and Hl 2 R1⇥K . MPGNNs : There are multiple variants of message passing GNNs , e.g. , ( Li et al. , 2015 ; Dai et al. , 2016 ; Gilmer et al. , 2017 ) , which share the same algorithmic framework but instantiate a few components differently , e.g. , the node state update function . We choose the same class of models as in ( Garg et al. , 2020 ) which are popular in the literature ( Dai et al. , 2016 ; Jin et al. , 2018 ) in order to fairly compare bounds . This MPGNN model can be written in matrix forms as follows , Mk = g ( C > outHk 1 ) ( k-th step Message Computation ) M̄k = CinMk ( k-th step Message Aggregation ) Hk = XW1 + ⇢ M̄k W2 ( k-th step Node State Update ) Hl = 1 n 1nHl 1Wl ( Readout Layer ) , ( 2 ) 1Here loop means an edge that connects a vertex to itself , a.k.a. , self-loop . 2We count the readout function as a layer to be consistent with the existing analysis of MLPs/CNNs . where k 2 N+l 1 , Hk 2 Rn⇥hk are node representations/states and Hl 2 R1⇥K is the output representation . Here we initialize H0 = 0 . W.l.o.g. , we assume 8k 2 N+l 1 , Hk 2 Rn⇥h and Mk 2 Rn⇥h since h is the maximum hidden dimension . Cin 2 Rn⇥c and Cout 2 Rn⇥c ( c is the number of edges ) are the incidence matrices corresponding to incoming and outgoing nodes3 respectively . Specifically , rows and columns of Cin and Cout correspond to nodes and edges respectively . Cin [ i , j ] = 1 indicates that the incoming node of the j-th edge is the i-th node . Similarly , Cout [ i , j ] = 1 indicates that the outgoing node of the j-th edge is the i-th node . g , , ⇢ are nonlinear mappings , e.g. , ReLU and Tanh . Technically speaking , g : Rh ! Rh , : Rh ! Rh , and ⇢ : Rh ! Rh operate on vector-states of individual node/edge . However , since we share these functions across nodes/edges , we can naturally generalize them to matrix-states , e.g. , ̃ : Rn⇥h ! Rn⇥h where ̃ ( X ) [ i , : ] = ( X [ i , : ] ) . By doing so , the same function could be applied to matrices with varying size of the first dimension . For simplicity , we use g , , ⇢ to denote such generalization to matrices . We denote the Lipschitz constants of g , , ⇢ under the vector 2-norm as Cg , C , C⇢ respectively . We also assume g ( 0 ) = 0 , ( 0 ) = 0 , and ⇢ ( 0 ) = 0 and define the percolation complexity as C = CgC C⇢kW2k2 following ( Garg et al. , 2020 ) . Multiclass Margin Loss : We use the multi-class -margin loss following ( Bartlett et al. , 2017 ; Neyshabur et al. , 2017 ) . The generalization error is defined as , LD , ( fw ) = P z⇠D ✓ fw ( X , A ) [ y ] +max j 6=y fw ( X , A ) [ j ] ◆ , ( 3 ) where > 0 and fw ( X , A ) is the l-th layer representations , i.e. , Hl = fw ( X , A ) . Accordingly , we can define the empirical error as , LS , ( fw ) = 1 m X zi2S 1 ✓ fw ( X , A ) [ y ] +max j 6=y fw ( X , A ) [ j ] ◆ . ( 4 )

Summary: This paper provides results of generalization bounds for two types of GNNs: GCN and MPGNN. The presented analysis follows the framework of Neyshabur 2017 to construct posterior by adding random perturbations so that the PAC-Bayesian technique can be applied. The main contributions are the perturbation analysis for GCN and MPGNN, which results in a bound depending on the graph statistics. The paper compares the derived bounds with existing results, and examines the bounds numerically through experiments.