Learning Theory Can (Sometimes) Explain Generalisation in Graph Neural Networks

1 Introduction . Neural networks have found tremendous success in a wide range of practical applications and , in the broader society , it is often considered synonymous to machine learning . The rapid gain in popularity has , however , come at the cost of interpretability and reliability of complex neural network architectures . Hence , there has been an increasing interest in understanding generalization and other theoretical properties of neural networks in the theoretical machine learning community ( Feldman 2020 ; Arora et al . 2019a ; Ma et al . 2017 ; Nagarajan et al . 2019 ; Theisen et al . 2020 ; Ghorbani et al . 2020 ) . Most of the existing theory literature focuses on the supervised learning problem , or more precisely , the setting of inductive inference . In contrast , there is a general lack of understanding of transductive problems , in particular the role of unlabeled data in training . Consequently there has also been little progress in rigorously understanding one of widely used tools for transductive inference—Graph neural networks ( GNN ) . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Graph neural networks . GNNs were introduced by Gori et al . ( 2005 ) and Scarselli et al . ( 2009 ) , who used recurrent neural network architectures , for the purpose of transductive inference on graphs , that is , the task of labelling all the nodes of a graph given the graph structure , all node features and labels for few nodes . Broadly , GNNs use a combination of local aggregation of node features and non-linear transformations to predict on unlabelled nodes . In practice , the exact form of aggregation and combination steps varies across architectures to solve domain specific tasks ( Kipf et al . 2017 ; Bruna et al . 2014 ; Defferrard et al . 2016 ; Veličković et al . 2018 ; Xu et al . 2019 ) . While some GNNs focus on the transductive setting , sometimes referred to as semi-supervised node classification,1 GNNs have also found success in supervised learning , where the task is to label entire graphs , in contrast to labelling nodes in a graph . While the understanding of GNNs is limited , there are empirical approaches to study GNNs in the transductive ( Bojchevski et al . 2018 ) and supervised setting ( Zhang et al . 2018 ; Ying et al . 2018 ) . For an extensive survey on the state of the art of GNNs see for example Wu et al . ( 2020 ) . Leaning theoretical analysis of GNNs . While empirical studies provide some insights into the behaviour of machine learning models , rigorous theoretical analysis is the key to deep insights into a model . The focus of this paper is to provide a learning-theoretic analysis of generalisation of GNNs in the transductive setting . Vapnik first studied the problem of transductive inference and provided generalisation bounds for empirical risk minimization ( Vapnik 1982 ; Vapnik 1998 ) . Subsequent works further analysed this setting in transductive regression ( Cortes et al . 2007 ) , and derive VC Dimension and Rademacher complexity for transductive classification ( Tolstikhin et al . 2016 ; El-Yaniv et al . 2009 ) . Generalisation error bounds for 1-layer GNNs have been derived in transductive setting based on algorithmic stability ( Verma et al . 2019 ) . In contrast , the focus of the current paper is on learning-theoretic measures , which have been previously used to analyse GNNs in a supervised setting . In Scarselli et al . ( 2018 ) , VC Dimension is derived for a specific class of GNNs and a generalisation error bound is given using node representations . However , their approach of subsuming the graph convolutions under Pfaffian functions does not allow for an explicit representation in terms of the diffusion operator which is important to our presented analysis . Garg et al . ( 2020 ) derives the Rademacher complexity for GNN in a supervised setting with the focus of the equivariant structures of the input graphs and does not allow for an explicit inclusion and analysis of the graph information . Liao et al . ( 2021 ) provides PAC-Bayes bounds for GNNs that are tighter than the bounds in Garg et al . ( 2020 ) . In the context of this work , especially relevant is Oono et al . ( 2020b ) and Oono et al . ( 2020a ) . Oono et al . ( 2020a ) describes the effect of oversmoothing with increasing number of layers . A more detailed comparison to our work is presented in section 2.3 . Oono et al . ( 2020b ) analyzes GNNs in the transductive setting . However , they consider a multiscale GCN , and therefore , the analysis is based in a weak-learning/boosting framework where the focus is mostly on exploring the weak learning component , whereas this paper focuses on the specific analysis of the generalization bound and the influence of it ’ s individual components . In addition , we provide a detailed analysis of its dependence on the graph and feature information and provide a more expressive bound by considering generalization under planted models . Infinite limit analysis . In the broader deep learning , there has been a growing call for alternatives to standard learning-theoretic bounds since they do not adequately capture the behaviour of deep models ( Neyshabur et al . 2017 ) . To this end , different limiting case analysis have been introduced . In the context of GNNs , it is known that GNNs have a fundamental connection to belief propagation and message passing ( Dai et al . 2016 ; Gilmer et al . 2017 ) and some theoretical analyses of GNNs have been based on cavity methods and mean field approaches for supervised ( Zhou et al . 2020 ) and transductive settings ( Kawamoto et al . 2019 ; Chen et al . 2019 ) . The central idea of these approaches is to show results in the limit of the number of iterations . In another limiting setting , Du et al . ( 2019 ) study GNNs with infintiely wide hidden layers , and derive corresponding neural tangent kernel ( Jacot et al . 2018 ; Arora et al . 2019b ) that can provide generalisation error bounds in the supervised setting . Keriven et al . ( 2020 ) derive continuous versions of GNNs applied to large random graphs . While limiting assumptions allow for a theoretical analysis , it is difficult to infer the implications of these results for finite GNNs . 1In semi-supervised learning , the learner is given a training set of labeled and unlabeled examples and the goal is to generate a hypothesis that generates predictions on the unseen examples . In transductive learning all features are available to the learner , and the goal is to transfer knowledge from the labeled to the unlabeled data points . The focus of graph-based semi-supervised learning aligns more with the latter setting . Contributions and paper structure . We reconsider classical learning-theoretic measures to analyse GNNs , with a specific focus on explicitly characterising the influence of the graph information and the network architecture on generalisation . In the process , we show that , under careful construction of the complexity measure and distributional assumptions on the graph data , learning theory can provide insights into the behaviour of GNNs . The main contributions are the following : 1 ) We introduce a formal setup for graph-based transductive inference , and in Section 2.2 , we use this framework to show that VC Dimension based generalisation error bounds are typically loose , except for few trivial cases . This observation is along the lines of existing evidence for neural networks . 2 ) In Section 2.3 , we derive generalization bounds based on the transductive Rademacher complexity . Our results show that these bounds are more informative , suggesting that the correct choice of complexity measure is important . 3 ) We further refine the generalisation error bounds in Section 3 under a planted model for the graph and features . Such an analysis , under random graphs , is rare in GNN literature . We empirically show that the test error is consistent with the trends predicted by the theoretical bound . Our results suggest that , under distributional assumptions , learning-theoretic bounds can explain behaviour of GNNs . 4 ) In addition , we consider GNNs with residual connections in Section 4 , and demonstrate how the above analysis can be extended to other network architectures . We prove that residual connections have smaller generalisation gap in comparison with vanilla GNN , and also empirically show that the theoretical bounds explain ( to a limited extent ) the influence of network depth on performance . We conclude in Section 5 . All proofs and an overview of the notation are provided in the appendix . 2 Statistical Framework for Transductive Learning on GNN . For a rigorous analysis , we introduce a statistical learning framework for graph based transductive inference in Section 2.1 . Based on this , we derive generalisation error bounds based on VC Dimension in Section 2.2 and demonstrate that the bounds have limited expresitivity even under strong assumptions . To overcome this problem we consider transductive Rademacher complexity in Section 2.3 . While without further assumptions this bound also gives limited insight , the bound is more expressive and , in Section 3 , we show that it can provide meaningful bounds under certain distributional assumptions . 2.1 Framework for Transductive Learning . We briefly recall the framework for supervised binary classification . Let X = Rd be the domain or feature space and Y = { ±1 } be the label set . The goal is to find a predictor h : X → Y based on m training samples S , { ( xi , yi ) } mi=1 ⊂ X ×Y and a loss function ` : Y ×Y → [ 0 , ∞ ) . In a statistical framework , we assume that S consists independent labelled samples from a distributionD = DX ×η , that is , xi ∼ DX and yi ∼ η ( xi ) , where η ( · ) governs the label probability for each feature . The goal of learning is to find h that minimises the risk / generalisation error LD ( h ) , E ( x , y ) ∼D [ ` ( h ( x ) , y ) ] . Since , LD ( h ) can not be computed without the knowledge of D , one minimises the empirical risk over the training sample S as LS ( h ) , 1m ∑m i=1 ` ( h ( xi ) , yi ) . Transductive learning . In transductive inference , one restricts the domain to be X , { xi } ni=1 , a finite set of features xi ∈ Rd . Without loss of generality , one may assume that the labels y1 , . . . , ym ∈ { ±1 } are known , and the goal is to predict ym+1 , . . . yn . The problem can be reformulated in the statistical learning framework as follows . We define the feature distribution DX to be uniform over the n features , whereas yi ∼ η ( xi ) for some unknown distribution η . Hence D : = Unif ( [ n ] ) × η is the joint distribution on X × Y , and the goal is to find a predictor h : X → Y that minimises the generalisation error Lu ( h ) , 1n−m ∑n i=m+1 ` ( h ( xi ) , yi ) . In addition we define the empirical error of h to be L̂m ( h ) , 1m ∑m i=1 ` ( h ( xi ) , yi ) and the full sample error of h to be Ln ( h ) , 1n ∑n i=1 ` ( h ( xi ) , yi ) , which is defined over both labelled and unlabelled instances . The purpose of this paper is to derive generalisation error bound for graph based transduction of the form Lu ( h ) ≤ L̂m ( h ) + complexity term . The complexity term is typically characterised using learning-theoretic terms such as VC Dimension and Rademacher complexity . For the transductive setting see Tolstikhin et al . ( 2016 ) , El-Yaniv et al . ( 2009 ) , and Tolstikhin et al . ( 2014 ) . Graph-based transductive learning . A typical view of graph information in transductive inference is as a form of a regularisation ( Belkin et al . 2004 ) . In contrast , we view the graph as part of the hypothesis class and derive the impact of the graph information on the complexity term . We assume access to a graph G with n vertices , corresponding to the respective feature vectors x1 , . . . , xn , and edge ( i , j ) denoting similarity of vertices i and j . For ease of exposition , we define the matrix X ∈ Rn×d with rows being the n feature vectors of dimension d. We also abuse notation to write a predictor as h : Rn×d → { ±1 } n. Furthermore , typically neural networks output a soft predictor in R , that is further transformed into labels through sign or softmax functions . Hence , much of our analysis focuses on predictors h : Rn×d → Rn , and corresponding hypothesis class HG = { h : Rn×d → Rn : h is parametrized by G } ⊂ R [ n ] . When applicable , we denote the hypothesis class of binary predictors obtained through sign function as sign ◦HG = { sign ( h ) | h ∈ HG } . Note that sign ◦HG ⊂ HG , and hence , VC Dimension or Rademacher complexity bounds for the latter also hold for the hypothesis class of binary predictors . We also note that the presented analysis holds for both sign and sigmoid function for binarisation . Formal setup of GNNs . We next characterise the hypothesis class for graph neural networks . Consider graph-based neural network model with the propagation rule for layer k denoted by gk ( H ) : Rdk−1 → Rdk with layer wise input matrix H ∈ Rn×dk−1 . Consider a class of GNNs defined over K layers , with dimension of layer k ∈ [ K ] being dk and S ∈ Rn×n the graph diffusion operator . Let φ denote the point-wise activation function of the network , which we assume to be a Lipschitz function with Lipschitz constant Lφ . We assume φ to be the same throughout the network . We define the hypothesis class over all K-layer GNNs as : HφG , { hφG ( X ) = gK ◦ · · · ◦ g0 : R n×d → { ±1 } n } ( 1 ) with gk , φ ( bk + Sgk−1 ( H ) Wk ) , k ∈ [ K ] , g0 , X . ( 2 ) where ( 2 ) defines the layer wise transformation withWk ∈ Rdk−1×dk as the trainable weight matrix and bk ∈ Rdk the bias term . Here , the graph is treated as part of the hypothesis class , as indicated by the subscript in HφG . For ease of notation we drop the superscript for non-linearity where it is unambiguous . For the diffusion operator S , we consider two main formulations during discussions : Sloop , A+ I self loop Snor , ( D + I ) − 1 2 ( A+ I ) ( D + I ) − 1 2 , degree normalized ( Kipf et al . 2017 ) whereA denotes the graph adjacency matrix andD is the degree matrix . However , most results are stated for general S .

Summary: This paper analyzed the generalization performance of GNNs based on statistical learning theory. First, this paper derived the VC dimension-based generalization error bounds. It showed that graph structures affected the upper bound of the generalization performance gap via the rank of the adjacency matrix (and its normalized version). This paper also argued that these gaps are vacuous for some random graphs. For a more precise evaluation, this paper next derived generalization error bounds based on the notion of Transductive Rademacher Complexity (TRC). The effect of the normalization of the adjacency matrix S on the generalization bounds is explained for the general and planted model case. Finally, this paper derived the generalization bounds based on TRC for GNNs with residual connections. It argued that residual connections reduced the generalization gap.