Increase and Conquer: Training Graph Neural Networks on Growing Graphs

1 INTRODUCTION . Graph Neural Networks ( GNNs ) are deep convolutional architectures formed by a succession of layers where each layer composes a graph convolution and a pointwise nonlinearity ( Wu et al. , 2021 ; Zhou et al. , 2020 ) . Tailored to network data , GNNs have been used in a variety of applications such as recommendation systems ( Fan et al. , 2019 ; Tan et al. , 2020 ; Ying et al. , 2018 ; Schlichtkrull et al. , 2018 ; Ruiz et al. , 2019a ) and Markov chains ( Qu et al. , 2019 ; Ruiz et al. , 2019b ; Li et al. , 2015 ) , and fields such as biology ( Fout et al. , 2017 ; Duvenaud et al. , 2015 ; Gilmer et al. , 2017 ; Chen et al. , 2020 ) and robotics ( Qi et al. , 2018 ; Gama & Sojoudi , 2021 ; Li et al. , 2019 ) . Their success in these fields and applications provides ample empirical evidence of the ability of GNNs to generalize to unseen data . More recently , their successful performance has also been justified by theoretical works showing that GNNs are invariant to relabelings ( Chen et al. , 2019 ; Keriven & Peyré , 2019 ) , stable to graph perturbations ( Gama et al. , 2020 ) and transferable across graphs ( Ruiz et al. , 2020a ) . One of the most important features of a GNN is that , because the linear operation is a graph convolution , its number of parameters does not depend on the number of nodes of the graph . In theory , this means that GNNs can be trained on graphs of any size . In practice , however , if the graph has large number of nodes training the GNN is costly because computing graph convolutions involves large matrix operations . While this issue could be mitigated by transferability — training the GNN on a smaller graph to execute on the large graph — , this approach does not give any guarantees on the distance between the optimal solutions on the small and on the large graph . In other words , when executing the GNN on the target graph we do not know if its error will be dominated by the transferability error or by the generalization error from training . In this paper , we address the computational burden of training a GNN on a large graph by progressively increasing the size of the network . We consider the limit problem of learning an “ optimal ” neural network for a graphon , which is both a graph limit and a random graph model ( Lovász , 2012 ) . We postulate that , because sequences of graphs sampled from the graphon converge to it , the so-called graphon neural network ( Ruiz et al. , 2020a ) can be learned by sampling graphs of growing size and training a GNN on these graphs ( Algorithm 1 ) . We prove that this is true in two steps . In Theorem 1 , we bound the expected distance between the gradient descent steps on the GNN and on the graphon neural network by a term that decreases asymptotically with the size of the graph . A consequence of this bias bound is that it allows us to quantify the trade-off between a more accurate gradient and one that could be obtained with less computational power . We then use this theorem to prove our main result in Theorem 2 , which is stated in simplified form below . Theorem ( Graphon neural network learning , informal ) Let W be a graphon and let { Gn } be a sequence of growing graphs sampled from W. Consider the graphon neural network Φ ( W ) and assume that it is learned by training the GNN Φ ( Gn ) with loss function ` ( yn , Φ ( Gn ) ) on the sequence { Gn } . Over a finite number of training steps , we obtain ‖∇ ` ( Y , Φ ( W ) ‖ ≤ with probability 1 . The most important implication of this result is that the learning iterates computed in the sequence of growing graphs follow the direction of the graphon gradient up to a small ball , which provides theoretical validation to our cost efficient training methodology . We also validate our algorithm in two numerical experiments . In the first , we learn a GNN-based recommendation system on increasingly large subnetworks of a movie similarity graph , and compare it with the recommendation system trained on the full graph . In the second , we consider the problem of flocking and train GNNs to learn the actions agents need to take to flock . We compare the results obtained when progressively increasing the number of agents during training and when training directly on the target graph . 2 RELATED WORK . GNNs are data processing architectures that follow from the seminal works in the areas of deep learning applied to graph theory ( Bruna et al. , 2013 ; Defferrard et al. , 2016 ; Gori et al. , 2005 ; Lu & Getoor , 2003 ) . They have been successfully used in a wide variety of statistical learning problems ( Kipf & Welling , 2016 ; Scarselli et al. , 2018 ) , where their good performance is generally attributed to the fact that they exploit invariances present in network data ( Maron et al. , 2019 ; Gama et al. , 2018 ; Chami et al. , 2021 ) . More recently , a number of works show that GNNs can be transferred across graphs of different sizes ( Ruiz et al. , 2020a ; Levie et al. , 2019 ; Keriven et al. , 2020 ) . Specifically , ( Ruiz et al. , 2020a ) leverages graphons to define families of graphs within which GNNs can be transferred with an error bound that decreases asymptotically with the size of the graph . The papers by Levie et al . ( 2019 ) and Keriven et al . ( 2020 ) offer similar results by considering the graph limit to be a generic topological space and a random graph model respectively . In this paper , we use an extension of the transferability bound derived in Ruiz et al . ( 2020a ) to propose a novel learning algorithm for GNNs . 3 PRELIMINARY DEFINITIONS . 3.1 GRAPH NEURAL NETWORKS . Graph neural networks exploit graph symmetries to extract meaningful information from network data ( Ruiz et al. , 2020c ; Gama et al. , 2020 ) . Graphs are represented as triplets Gn = ( V , E , W ) , where V , |V| = n , is the set of nodes , E ⊆ V × V is the set of edges and W : E → R is a map assigning weights to each edge . The graph Gn can also be represented by the graph shift operator ( GSO ) S ∈ Rn×n , a square matrix that respects the sparsity of the graph . Examples of GSOs include the adjacency matrix A , the graph Laplacian L = diag ( A1 ) −A and their normalized counterparts Gama et al . ( 2018 ) . In this paper we consider the graph Gn to be undirected and fix S = A/n . Graph data is represented in the form of graph signals . A graph signal x = [ x1 , . . . , xn ] T ∈ Rn is a vector whose i-th component corresponds to the information present at the i-th node of graph Gn . A basic data aggregation operation can be defined by applying the GSO S to graph signals x . The resulting signal z = Sx is such that the data at node i is a weighted average of the information in the 1-hop neighborhood of i , zi = ∑ j∈Ni [ S ] ijxj where Ni = { j | [ S ] ij 6= 0 } . Information coming from further neighborhoods can be aggregated by successive applications of the GSO , also called shifts . Using this notion of shift , graph convolutions are defined by weighting the contribution of each successive application of S to define a polynomial in the GSO . Explicitly , the graph convolutional filter with coefficients h = [ h0 , . . . , hK−1 ] is given by y = h ∗S x = K−1∑ k=0 hkS kx ( 1 ) where ∗S denotes the convolution operation with GSO S. Since the adjacency matrix of an undirected graph is always symmetric , the GSO admits an eigendecomposition S = VΛVH . The columns of V are the graph eigenvectors and the diagonal elements of Λ are the graph eigenvalues , which take values between −1 and 1 and are ordered as −1 ≤ λ−1 ≤ λ−2 ≤ . . . ≤ 0 ≤ . . . ≤ λ2 ≤ λ1 ≤ 1 . Since the eigenvectors of S form an orthonormal basis of Rn , we can project ( 1 ) onto this basis to obtain the spectral representation of the graph convolution , which is given by h ( λ ) = K−1∑ k=0 hkλ k. ( 2 ) Note that ( 2 ) only depends on the hk and on the eigenvalues of the GSO . Hence , as a consequence of the Cayley-Hamilton theorem , convolutional filters may be used to represent any graph filter with spectral representation h ( λ ) = f ( λ ) where f is analytic ( Strang , 1976 ) . Graph neural networks are layered architectures where each layer consists of a graph convolution followed by a pointwise nonlinearity ρ , and where each layer ’ s output is the input to the following layer . At layer l , a GNN can output multiple features xfl , 1 ≤ f ≤ Fl which we stack in a matrix Xl = [ x1l , . . . , x Fl l ] ∈ Rn×Fl . Each column of the feature matrix is the value of the graph signal at feature f . To map the Fl−1 features coming from layer l − 1 into Fl features , Fl−1 × Fl convolutions need to be computed , one per input-output feature pair . Stacking their weights in K matrices Hlk ∈ RFl−1×Fl , we write the l-th layer of the GNN as Xl = ρ ( K−1∑ k=0 SkXl−1Hlk ) . ( 3 ) In an L-layer GNN , the operation in ( 3 ) is cascaded L times to obtain the GNN output Y = XL . At the first layer , the GNN input is given by X0 = X ∈ Rn×F0 . In this paper we assume F0 = FL = 1 so that Y = y and X = x . A more concise representation of this GNN can be obtained by grouping all learnable parameters Hlk in a tensor H = { Hlk } l , k and defining the map y = Φ ( x ; H , S ) . Due to the polynomial nature of the graph convolution , the dimensions of the learnable parameter tensor H are independent from the size of the graph ( K is typically much smaller than n ) . Ergo , a GNN trained on a graph Gn can be deployed on a network Gm with m 6= n . 3.2 GRAPHON INFORMATION PROCESSING . A graphon is a bounded , symmetric , and measurable function W : [ 0 , 1 ] 2 → [ 0 , 1 ] which has two theoretical interpretations — it is both a graph limit and a generative model for graphs . In the first interpretation , sequences of dense graphs converge to a graphon in the sense that the densities of adjacency-preserving graph motifs converge to the same densities on the graphon Lovász ( 2012 ) . In the second , graphs can be generated from a graphon by sampling points ui , uj from the unit interval and either assigning weight W ( ui , uj ) to edges ( i , j ) , or sampling edges ( i , j ) with probability W ( ui , uj ) . In this paper , we focus on stochastic graphs Gn where the points ui are defined as ui = ( i− 1 ) /n for 1 ≤ i ≤ n and where the adjacency matrix Sn is sampled from W as [ Sn ] ij ∼ Bernoulli ( W ( ui , uj ) ) . ( 4 ) Sequences of graphs generated in this way can be shown to converge to W with probability one ( Lovász , 2012 ) [ Chapter 11 ] . In practice , the two theoretical interpretations of a graphon allow thinking of it as an identifying object for a family of graphs of different sizes that are structurally similar . Hence , given a network we can use its family ’ s identifying graphon as a continuous proxy for the graph . This is beneficial because it is typically easier to operate in continuous than in discrete domains , even more so if the network is large . We will leverage these ideas to consider graphon data and graphon neural networks as proxies for graph data and GNNs supported on graphs of arbitrary size .

Summary: In this paper, the authors propose a method that learns a large graph neural network (GNN) starting from a relatively smaller GNN which would increase its number of nodes step by step (epoch by epoch). Most importantly, the paper gives mathematical proof to show that the gradient descent steps on GNN are close to the graphon neural network (WNN) which is the limit object of the GNN. This paper also demonstrates its proposed method on a recommendation system and a decentralized control problem and shows reduced computational cost.