Learning Theory Can (Sometimes) Explain Generalisation in Graph Neural Networks

1 Introduction . Neural networks have found tremendous success in a wide range of practical applications and , in the broader society , it is often considered synonymous to machine learning . The rapid gain in popularity has , however , come at the cost of interpretability and reliability of complex neural network architectures . Hence , there has been an increasing interest in understanding generalization and other theoretical properties of neural networks in the theoretical machine learning community ( Feldman 2020 ; Arora et al . 2019a ; Ma et al . 2017 ; Nagarajan et al . 2019 ; Theisen et al . 2020 ; Ghorbani et al . 2020 ) . Most of the existing theory literature focuses on the supervised learning problem , or more precisely , the setting of inductive inference . In contrast , there is a general lack of understanding of transductive problems , in particular the role of unlabeled data in training . Consequently there has also been little progress in rigorously understanding one of widely used tools for transductive inference—Graph neural networks ( GNN ) . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Graph neural networks . GNNs were introduced by Gori et al . ( 2005 ) and Scarselli et al . ( 2009 ) , who used recurrent neural network architectures , for the purpose of transductive inference on graphs , that is , the task of labelling all the nodes of a graph given the graph structure , all node features and labels for few nodes . Broadly , GNNs use a combination of local aggregation of node features and non-linear transformations to predict on unlabelled nodes . In practice , the exact form of aggregation and combination steps varies across architectures to solve domain specific tasks ( Kipf et al . 2017 ; Bruna et al . 2014 ; Defferrard et al . 2016 ; Veličković et al . 2018 ; Xu et al . 2019 ) . While some GNNs focus on the transductive setting , sometimes referred to as semi-supervised node classification,1 GNNs have also found success in supervised learning , where the task is to label entire graphs , in contrast to labelling nodes in a graph . While the understanding of GNNs is limited , there are empirical approaches to study GNNs in the transductive ( Bojchevski et al . 2018 ) and supervised setting ( Zhang et al . 2018 ; Ying et al . 2018 ) . For an extensive survey on the state of the art of GNNs see for example Wu et al . ( 2020 ) . Leaning theoretical analysis of GNNs . While empirical studies provide some insights into the behaviour of machine learning models , rigorous theoretical analysis is the key to deep insights into a model . The focus of this paper is to provide a learning-theoretic analysis of generalisation of GNNs in the transductive setting . Vapnik first studied the problem of transductive inference and provided generalisation bounds for empirical risk minimization ( Vapnik 1982 ; Vapnik 1998 ) . Subsequent works further analysed this setting in transductive regression ( Cortes et al . 2007 ) , and derive VC Dimension and Rademacher complexity for transductive classification ( Tolstikhin et al . 2016 ; El-Yaniv et al . 2009 ) . Generalisation error bounds for 1-layer GNNs have been derived in transductive setting based on algorithmic stability ( Verma et al . 2019 ) . In contrast , the focus of the current paper is on learning-theoretic measures , which have been previously used to analyse GNNs in a supervised setting . In Scarselli et al . ( 2018 ) , VC Dimension is derived for a specific class of GNNs and a generalisation error bound is given using node representations . However , their approach of subsuming the graph convolutions under Pfaffian functions does not allow for an explicit representation in terms of the diffusion operator which is important to our presented analysis . Garg et al . ( 2020 ) derives the Rademacher complexity for GNN in a supervised setting with the focus of the equivariant structures of the input graphs and does not allow for an explicit inclusion and analysis of the graph information . Liao et al . ( 2021 ) provides PAC-Bayes bounds for GNNs that are tighter than the bounds in Garg et al . ( 2020 ) . In the context of this work , especially relevant is Oono et al . ( 2020b ) and Oono et al . ( 2020a ) . Oono et al . ( 2020a ) describes the effect of oversmoothing with increasing number of layers . A more detailed comparison to our work is presented in section 2.3 . Oono et al . ( 2020b ) analyzes GNNs in the transductive setting . However , they consider a multiscale GCN , and therefore , the analysis is based in a weak-learning/boosting framework where the focus is mostly on exploring the weak learning component , whereas this paper focuses on the specific analysis of the generalization bound and the influence of it ’ s individual components . In addition , we provide a detailed analysis of its dependence on the graph and feature information and provide a more expressive bound by considering generalization under planted models . Infinite limit analysis . In the broader deep learning , there has been a growing call for alternatives to standard learning-theoretic bounds since they do not adequately capture the behaviour of deep models ( Neyshabur et al . 2017 ) . To this end , different limiting case analysis have been introduced . In the context of GNNs , it is known that GNNs have a fundamental connection to belief propagation and message passing ( Dai et al . 2016 ; Gilmer et al . 2017 ) and some theoretical analyses of GNNs have been based on cavity methods and mean field approaches for supervised ( Zhou et al . 2020 ) and transductive settings ( Kawamoto et al . 2019 ; Chen et al . 2019 ) . The central idea of these approaches is to show results in the limit of the number of iterations . In another limiting setting , Du et al . ( 2019 ) study GNNs with infintiely wide hidden layers , and derive corresponding neural tangent kernel ( Jacot et al . 2018 ; Arora et al . 2019b ) that can provide generalisation error bounds in the supervised setting . Keriven et al . ( 2020 ) derive continuous versions of GNNs applied to large random graphs . While limiting assumptions allow for a theoretical analysis , it is difficult to infer the implications of these results for finite GNNs . 1In semi-supervised learning , the learner is given a training set of labeled and unlabeled examples and the goal is to generate a hypothesis that generates predictions on the unseen examples . In transductive learning all features are available to the learner , and the goal is to transfer knowledge from the labeled to the unlabeled data points . The focus of graph-based semi-supervised learning aligns more with the latter setting . Contributions and paper structure . We reconsider classical learning-theoretic measures to analyse GNNs , with a specific focus on explicitly characterising the influence of the graph information and the network architecture on generalisation . In the process , we show that , under careful construction of the complexity measure and distributional assumptions on the graph data , learning theory can provide insights into the behaviour of GNNs . The main contributions are the following : 1 ) We introduce a formal setup for graph-based transductive inference , and in Section 2.2 , we use this framework to show that VC Dimension based generalisation error bounds are typically loose , except for few trivial cases . This observation is along the lines of existing evidence for neural networks . 2 ) In Section 2.3 , we derive generalization bounds based on the transductive Rademacher complexity . Our results show that these bounds are more informative , suggesting that the correct choice of complexity measure is important . 3 ) We further refine the generalisation error bounds in Section 3 under a planted model for the graph and features . Such an analysis , under random graphs , is rare in GNN literature . We empirically show that the test error is consistent with the trends predicted by the theoretical bound . Our results suggest that , under distributional assumptions , learning-theoretic bounds can explain behaviour of GNNs . 4 ) In addition , we consider GNNs with residual connections in Section 4 , and demonstrate how the above analysis can be extended to other network architectures . We prove that residual connections have smaller generalisation gap in comparison with vanilla GNN , and also empirically show that the theoretical bounds explain ( to a limited extent ) the influence of network depth on performance . We conclude in Section 5 . All proofs and an overview of the notation are provided in the appendix . 2 Statistical Framework for Transductive Learning on GNN . For a rigorous analysis , we introduce a statistical learning framework for graph based transductive inference in Section 2.1 . Based on this , we derive generalisation error bounds based on VC Dimension in Section 2.2 and demonstrate that the bounds have limited expresitivity even under strong assumptions . To overcome this problem we consider transductive Rademacher complexity in Section 2.3 . While without further assumptions this bound also gives limited insight , the bound is more expressive and , in Section 3 , we show that it can provide meaningful bounds under certain distributional assumptions . 2.1 Framework for Transductive Learning . We briefly recall the framework for supervised binary classification . Let X = Rd be the domain or feature space and Y = { ±1 } be the label set . The goal is to find a predictor h : X → Y based on m training samples S , { ( xi , yi ) } mi=1 ⊂ X ×Y and a loss function ` : Y ×Y → [ 0 , ∞ ) . In a statistical framework , we assume that S consists independent labelled samples from a distributionD = DX ×η , that is , xi ∼ DX and yi ∼ η ( xi ) , where η ( · ) governs the label probability for each feature . The goal of learning is to find h that minimises the risk / generalisation error LD ( h ) , E ( x , y ) ∼D [ ` ( h ( x ) , y ) ] . Since , LD ( h ) can not be computed without the knowledge of D , one minimises the empirical risk over the training sample S as LS ( h ) , 1m ∑m i=1 ` ( h ( xi ) , yi ) . Transductive learning . In transductive inference , one restricts the domain to be X , { xi } ni=1 , a finite set of features xi ∈ Rd . Without loss of generality , one may assume that the labels y1 , . . . , ym ∈ { ±1 } are known , and the goal is to predict ym+1 , . . . yn . The problem can be reformulated in the statistical learning framework as follows . We define the feature distribution DX to be uniform over the n features , whereas yi ∼ η ( xi ) for some unknown distribution η . Hence D : = Unif ( [ n ] ) × η is the joint distribution on X × Y , and the goal is to find a predictor h : X → Y that minimises the generalisation error Lu ( h ) , 1n−m ∑n i=m+1 ` ( h ( xi ) , yi ) . In addition we define the empirical error of h to be L̂m ( h ) , 1m ∑m i=1 ` ( h ( xi ) , yi ) and the full sample error of h to be Ln ( h ) , 1n ∑n i=1 ` ( h ( xi ) , yi ) , which is defined over both labelled and unlabelled instances . The purpose of this paper is to derive generalisation error bound for graph based transduction of the form Lu ( h ) ≤ L̂m ( h ) + complexity term . The complexity term is typically characterised using learning-theoretic terms such as VC Dimension and Rademacher complexity . For the transductive setting see Tolstikhin et al . ( 2016 ) , El-Yaniv et al . ( 2009 ) , and Tolstikhin et al . ( 2014 ) . Graph-based transductive learning . A typical view of graph information in transductive inference is as a form of a regularisation ( Belkin et al . 2004 ) . In contrast , we view the graph as part of the hypothesis class and derive the impact of the graph information on the complexity term . We assume access to a graph G with n vertices , corresponding to the respective feature vectors x1 , . . . , xn , and edge ( i , j ) denoting similarity of vertices i and j . For ease of exposition , we define the matrix X ∈ Rn×d with rows being the n feature vectors of dimension d. We also abuse notation to write a predictor as h : Rn×d → { ±1 } n. Furthermore , typically neural networks output a soft predictor in R , that is further transformed into labels through sign or softmax functions . Hence , much of our analysis focuses on predictors h : Rn×d → Rn , and corresponding hypothesis class HG = { h : Rn×d → Rn : h is parametrized by G } ⊂ R [ n ] . When applicable , we denote the hypothesis class of binary predictors obtained through sign function as sign ◦HG = { sign ( h ) | h ∈ HG } . Note that sign ◦HG ⊂ HG , and hence , VC Dimension or Rademacher complexity bounds for the latter also hold for the hypothesis class of binary predictors . We also note that the presented analysis holds for both sign and sigmoid function for binarisation . Formal setup of GNNs . We next characterise the hypothesis class for graph neural networks . Consider graph-based neural network model with the propagation rule for layer k denoted by gk ( H ) : Rdk−1 → Rdk with layer wise input matrix H ∈ Rn×dk−1 . Consider a class of GNNs defined over K layers , with dimension of layer k ∈ [ K ] being dk and S ∈ Rn×n the graph diffusion operator . Let φ denote the point-wise activation function of the network , which we assume to be a Lipschitz function with Lipschitz constant Lφ . We assume φ to be the same throughout the network . We define the hypothesis class over all K-layer GNNs as : HφG , { hφG ( X ) = gK ◦ · · · ◦ g0 : R n×d → { ±1 } n } ( 1 ) with gk , φ ( bk + Sgk−1 ( H ) Wk ) , k ∈ [ K ] , g0 , X . ( 2 ) where ( 2 ) defines the layer wise transformation withWk ∈ Rdk−1×dk as the trainable weight matrix and bk ∈ Rdk the bias term . Here , the graph is treated as part of the hypothesis class , as indicated by the subscript in HφG . For ease of notation we drop the superscript for non-linearity where it is unambiguous . For the diffusion operator S , we consider two main formulations during discussions : Sloop , A+ I self loop Snor , ( D + I ) − 1 2 ( A+ I ) ( D + I ) − 1 2 , degree normalized ( Kipf et al . 2017 ) whereA denotes the graph adjacency matrix andD is the degree matrix . However , most results are stated for general S .

Summary: The problem addressed in this paper is an important area in theoretical understanding graph convolution networks. This paper discusses the transductive learning using the Rademacher complexity analysis with VC-dimensions and data dependent bounds. Further, the paper analyzes the special setting of the GCNII which is known to be robust against oversmoothing. Overall, a good paper, however, some of the interpretations of bounds are not clear, which require further clarifications.