A PAC-Bayesian Approach to Generalization Bounds for Graph Neural Networks

1 INTRODUCTION . Graph neural networks ( GNNs ) ( Gori et al. , 2005 ; Scarselli et al. , 2008 ; Bronstein et al. , 2017 ; Battaglia et al. , 2018 ) have become very popular recently due to their ability to learn powerful representations from graph-structured data , and have achieved state-of-the-art results in a variety of application domains such as social networks ( Hamilton et al. , 2017 ; Xu et al. , 2018 ) , quantum chemistry ( Gilmer et al. , 2017 ; Chen et al. , 2019a ) , computer vision ( Qi et al. , 2017 ; Monti et al. , 2017 ) , reinforcement learning ( Sanchez-Gonzalez et al. , 2018 ; Wang et al. , 2018 ) , robotics ( Casas et al. , 2019 ; Liang et al. , 2020 ) , and physics ( Henrion et al. , 2017 ) . Given a graph along with node/edge features , GNNs learn node/edge representations by propagating information on the graph via local computations shared across the nodes/edges . Based on the specific form of local computation employed , GNNs can be divided into two categories : graph convolution based GNNs ( Bruna et al. , 2013 ; Duvenaud et al. , 2015 ; Kipf & Welling , 2016 ) and message passing based GNNs ( Li et al. , 2015 ; Dai et al. , 2016 ; Gilmer et al. , 2017 ) . The former generalizes the convolution operator from regular graphs ( e.g. , grids ) to ones with arbitrary topology , whereas the latter mimics message passing algorithms and parameterizes the shared functions via neural networks . Due to the tremendous empirical success of GNNs , there is increasing interest in understanding their theoretical properties . For example , some recent works study their expressiveness ( Maron et al. , 2018 ; Xu et al. , 2018 ; Chen et al. , 2019b ) , that is , what class of functions can be represented by GNNs . However , only few works investigate why GNNs generalize so well to unseen graphs . They are either restricted to a specific model variant ( Verma & Zhang , 2019 ; Du et al. , 2019 ; Garg et al. , 2020 ) or have loose dependencies on graph statistics ( Scarselli et al. , 2018 ) . On the other hand , GNNs have close ties to standard feedforward neural networks , e.g. , multi-layer perceptrons ( MLPs ) and convolutional neural networks ( CNNs ) . In particular , if each i.i.d . sample is viewed as a node , then the whole dataset becomes a graph without edges . Therefore , GNNs can be seen as generalizations of MLPs/CNNs since they model not only the regularities within a sample but also the dependencies among samples as defined in the graph . It is therefore natural to ask if we can generalize the recent advancements on generalization bounds for MLPs/CNNs ( Harvey et al. , 2017 ; Neyshabur et al. , 2017 ; Bartlett et al. , 2017 ; Dziugaite & Roy , 2017 ; Arora et al. , 2018 ; 2019 ) to GNNs , and how would graph structures affect the generalization bounds ? In this paper , we answer the above questions by proving generalization bounds for the two primary classes of GNNs , i.e. , graph convolutional networks ( GCNs ) ( Kipf & Welling , 2016 ) and messagepassing GNNs ( MPGNNs ) ( Dai et al. , 2016 ; Jin et al. , 2018 ) . Our generalization bound for GCNs shows an intimate relationship with the bounds for MLPs/CNNs with ReLU activations ( Neyshabur et al. , 2017 ; Bartlett et al. , 2017 ) . In particular , they share the same term , i.e. , the product of the spectral norms of the learned weights at each layer multiplied by a factor that is additive across layers . The bound for GCNs has an additional multiplicative factor d ( l 1 ) /2 where d 1 is the maximum node degree and l is the network depth . Since MLPs/CNNs are special GNNs operating on graphs without edges ( i.e. , d 1 = 0 ) , the bound for GCNs coincides with the ones for MLPs/CNNs with ReLU activations ( Neyshabur et al. , 2017 ) on such degenerated graphs . Therefore , our result is a natural generalization of the existing results for MLPs/CNNs . Our generalization bound for message passing GNNs reveals that the governing terms of the bound are similar to the ones of GCNs , i.e. , the geometric series of the learned weights and the multiplicative factor dl 1 . The geometric series appears due to the weight sharing across message passing steps , thus corresponding to the product term across layers in GCNs . The term dl 1 encodes the key graph statistics . Our bound improves the dependency on the maximum node degree and the maximum hidden dimension compared to the recent Rademacher complexity based bound ( Garg et al. , 2020 ) . Moreover , we compute the bound values on four real-world graph datasets ( e.g. , social networks and protein structures ) and verify that our bounds are tighter . In terms of the proof techniques , our analysis follows the PAC-Bayes framework in the seminal work of ( Neyshabur et al. , 2017 ) for MLPs/CNNs with ReLU activations . However , we make two distinctive contributions which are customized for GNNs . First , a naive adaptation of the perturbation analysis in ( Neyshabur et al. , 2017 ) does not work for GNNs since ReLU is not 1-Lipschitz under the spectral norm , i.e. , kReLU ( X ) k2 kXk2 does not hold for some real matrix X . Instead , we construct the recursion on certain node representations of GNNs like the one with maximum ` 2 norm , so that we can perform perturbation analysis with vector 2-norm . Second , in contrast to ( Neyshabur et al. , 2017 ) which only handles the homogeneous networks , i.e. , f ( ax ) = af ( x ) when a 0 , we properly construct a quantity of the learned weights which 1 ) provides a way to satisfy the constraints of the previous perturbation analysis and 2 ) induces a finite covering on the range of the quantity so that the PAC-Bayes bound holds for all possible weights . This generalizes the analysis to non-homogeneous GNNs like typical MPGNNs . The rest of the paper is organized as follows . In Section 2 , we introduce background material necessary for our analysis . We then present our generalization bounds and the comparison to existing results in Section 3 . We also provide an empirical study to support our theoretical arguments in Section 4 . At last , we discuss the extensions , limitations and some open problems . 2 BACKGROUND . In this section , we first explain our analysis setup including notation and assumptions . We then describe the two representative GNN models in detail . Finally , we review the PAC-Bayes analysis . 2.1 ANALYSIS SETUP . In the following analysis , we consider the K-class graph classification problem which is common in the GNN literature , where given a graph sample z , we would like to classify it into one of the predefined K classes . We will discuss extensions to other problems like graph regression in Section 5 . Each graph sample z is a triplet of an adjacency matrix A , node features X 2 Rn⇥h0 and output label y 2 R1⇥K , i.e . z = ( A , X , y ) , where n is the number of nodes and h0 is the input feature dimension . We start our discussion by defining our notations . Let N+k be the first k positive integers , i.e. , N+k = { 1 , 2 , . . . , k } , | · |p the vector p-norm and k · kp the operator norm induced by the vector p-norm . Further , k · kF denotes the Frobenius norm of a matrix , e the base of the natural logarithm function log , A [ i , j ] the ( i , j ) -th element of matrix A and A [ i , : ] the i-th row . We use parenthesis to avoid the ambiguity , e.g. , ( AB ) [ i , j ] means the ( i , j ) -th element of the product matrix AB . We then introduce some terminologies from statistical learning theory and define the sample space as Z , z = ( A , X , y ) 2 Z where X 2 X ( node feature space ) and A 2 G ( graph space ) , data distribution D , z iid⇠ D , hypothesis ( or model ) fw where fw 2 H ( hypothesis class ) , and training set S with size m , S = { z1 , . . . , zm } . We make the following assumptions which also appear in the literature : A1 Data , i.e. , triplets ( A , X , y ) , are i.i.d . samples drawn from some unknown distribution D. A2 The maximum hidden dimension across all layers is h. A3 Node feature of any graph is contained in a ` 2-ball with radius B . Specifically , we have 8i 2 N+n , the i-th node feature X [ i , : ] 2 XB , h0 = { x 2 Rh0 | Ph0 j=1 x 2 j B2 } . A4 We only consider simple graphs ( i.e. , undirected , no loops1 , and no multi-edges ) with maximum node degree as d 1 . Note that it is straightforward to estimate B and d empirically on real-world graph data . 2.2 GRAPH NEURAL NETWORKS ( GNNS ) . In this part , we describe the details of the GNN models and the loss function we used for the graph classification problem . The essential idea of GNNs is to propagate information over the graph so that the learned representations capture the dependencies among nodes/edges . We now review two classes of GNNs , GCNs and MPGNNs , which have different mechanisms for propagating information . We choose them since they are the most popular variants and represent two common types of neural networks , i.e. , feedforward ( GCNs ) and recurrent ( MPGNNs ) neural networks . We discuss the extension of our analysis to other GNN variants in Section 5 . For ease of notation , we define the model to be fw 2 H : X ⇥ G ! RK where w is the vectorization of all model parameters . GCNs : Graph convolutional networks ( GCNs ) ( Kipf & Welling , 2016 ) for the K-class graph classification problem can be defined as follows , Hk = k ⇣ L̃Hk 1Wk ⌘ ( k-th Graph Convolution Layer ) Hl = 1 n 1nHl 1Wl ( Readout Layer ) , ( 1 ) where k 2 N+l 1 , Hk 2 Rn⇥hk are the node representations/states , 1n 2 R1⇥n is a all-one vector , l is the number of layers.2 and Wj is the weight matrix of the j-th layer . The initial node state is the observed node feature H0 = X . For both GCNs and MPGNNs , we consider l > 1 since otherwise the model degenerates to a linear transformation which does not leverage the graph and is trivial to analyze . Due to assumption A2 , Wj is of size at most h ⇥ h , i.e. , hk h , 8k 2 N+l 1 . The graph Laplacian L̃ is defined as , Ã = I + A , L̃ = D 12 ÃD 12 where D is the degree matrix of Ã . Note that the maximum eigenvalue of L̃ is 1 in this case . We absorb the bias into the weight by appending constant 1 to the node feature . Typically , GCNs use ReLU as the non-linearity , i.e. , i ( x ) = max ( 0 , x ) , 8i = 1 , · · · , l 1 . We use the common mean-readout to obtain the graph representation where Hl 1 2 Rn⇥hl 1 , Wl 2 Rhl 1⇥K , and Hl 2 R1⇥K . MPGNNs : There are multiple variants of message passing GNNs , e.g. , ( Li et al. , 2015 ; Dai et al. , 2016 ; Gilmer et al. , 2017 ) , which share the same algorithmic framework but instantiate a few components differently , e.g. , the node state update function . We choose the same class of models as in ( Garg et al. , 2020 ) which are popular in the literature ( Dai et al. , 2016 ; Jin et al. , 2018 ) in order to fairly compare bounds . This MPGNN model can be written in matrix forms as follows , Mk = g ( C > outHk 1 ) ( k-th step Message Computation ) M̄k = CinMk ( k-th step Message Aggregation ) Hk = XW1 + ⇢ M̄k W2 ( k-th step Node State Update ) Hl = 1 n 1nHl 1Wl ( Readout Layer ) , ( 2 ) 1Here loop means an edge that connects a vertex to itself , a.k.a. , self-loop . 2We count the readout function as a layer to be consistent with the existing analysis of MLPs/CNNs . where k 2 N+l 1 , Hk 2 Rn⇥hk are node representations/states and Hl 2 R1⇥K is the output representation . Here we initialize H0 = 0 . W.l.o.g. , we assume 8k 2 N+l 1 , Hk 2 Rn⇥h and Mk 2 Rn⇥h since h is the maximum hidden dimension . Cin 2 Rn⇥c and Cout 2 Rn⇥c ( c is the number of edges ) are the incidence matrices corresponding to incoming and outgoing nodes3 respectively . Specifically , rows and columns of Cin and Cout correspond to nodes and edges respectively . Cin [ i , j ] = 1 indicates that the incoming node of the j-th edge is the i-th node . Similarly , Cout [ i , j ] = 1 indicates that the outgoing node of the j-th edge is the i-th node . g , , ⇢ are nonlinear mappings , e.g. , ReLU and Tanh . Technically speaking , g : Rh ! Rh , : Rh ! Rh , and ⇢ : Rh ! Rh operate on vector-states of individual node/edge . However , since we share these functions across nodes/edges , we can naturally generalize them to matrix-states , e.g. , ̃ : Rn⇥h ! Rn⇥h where ̃ ( X ) [ i , : ] = ( X [ i , : ] ) . By doing so , the same function could be applied to matrices with varying size of the first dimension . For simplicity , we use g , , ⇢ to denote such generalization to matrices . We denote the Lipschitz constants of g , , ⇢ under the vector 2-norm as Cg , C , C⇢ respectively . We also assume g ( 0 ) = 0 , ( 0 ) = 0 , and ⇢ ( 0 ) = 0 and define the percolation complexity as C = CgC C⇢kW2k2 following ( Garg et al. , 2020 ) . Multiclass Margin Loss : We use the multi-class -margin loss following ( Bartlett et al. , 2017 ; Neyshabur et al. , 2017 ) . The generalization error is defined as , LD , ( fw ) = P z⇠D ✓ fw ( X , A ) [ y ] +max j 6=y fw ( X , A ) [ j ] ◆ , ( 3 ) where > 0 and fw ( X , A ) is the l-th layer representations , i.e. , Hl = fw ( X , A ) . Accordingly , we can define the empirical error as , LS , ( fw ) = 1 m X zi2S 1 ✓ fw ( X , A ) [ y ] +max j 6=y fw ( X , A ) [ j ] ◆ . ( 4 )

Summary: In this paper, the authors propose generalization bounds for GNNs, both convolutional and standard message passing variants. The result is a generalization of those for CNN/MLP architectures with relu activation functions. The analysis method closely follows those established in the former settings as well. The specific setting they consider is where each sample in the dataset is a graph. The proof relies on ensuring small perturbations in the GNN weights don't cause large deviations in output distributions. The resulting PAC-Bayes bound is shown to be tighter than corresponding Rademacher Complexity bounds.