Increase and Conquer: Training Graph Neural Networks on Growing Graphs

1 INTRODUCTION . Graph Neural Networks ( GNNs ) are deep convolutional architectures formed by a succession of layers where each layer composes a graph convolution and a pointwise nonlinearity ( Wu et al. , 2021 ; Zhou et al. , 2020 ) . Tailored to network data , GNNs have been used in a variety of applications such as recommendation systems ( Fan et al. , 2019 ; Tan et al. , 2020 ; Ying et al. , 2018 ; Schlichtkrull et al. , 2018 ; Ruiz et al. , 2019a ) and Markov chains ( Qu et al. , 2019 ; Ruiz et al. , 2019b ; Li et al. , 2015 ) , and fields such as biology ( Fout et al. , 2017 ; Duvenaud et al. , 2015 ; Gilmer et al. , 2017 ; Chen et al. , 2020 ) and robotics ( Qi et al. , 2018 ; Gama & Sojoudi , 2021 ; Li et al. , 2019 ) . Their success in these fields and applications provides ample empirical evidence of the ability of GNNs to generalize to unseen data . More recently , their successful performance has also been justified by theoretical works showing that GNNs are invariant to relabelings ( Chen et al. , 2019 ; Keriven & Peyré , 2019 ) , stable to graph perturbations ( Gama et al. , 2020 ) and transferable across graphs ( Ruiz et al. , 2020a ) . One of the most important features of a GNN is that , because the linear operation is a graph convolution , its number of parameters does not depend on the number of nodes of the graph . In theory , this means that GNNs can be trained on graphs of any size . In practice , however , if the graph has large number of nodes training the GNN is costly because computing graph convolutions involves large matrix operations . While this issue could be mitigated by transferability — training the GNN on a smaller graph to execute on the large graph — , this approach does not give any guarantees on the distance between the optimal solutions on the small and on the large graph . In other words , when executing the GNN on the target graph we do not know if its error will be dominated by the transferability error or by the generalization error from training . In this paper , we address the computational burden of training a GNN on a large graph by progressively increasing the size of the network . We consider the limit problem of learning an “ optimal ” neural network for a graphon , which is both a graph limit and a random graph model ( Lovász , 2012 ) . We postulate that , because sequences of graphs sampled from the graphon converge to it , the so-called graphon neural network ( Ruiz et al. , 2020a ) can be learned by sampling graphs of growing size and training a GNN on these graphs ( Algorithm 1 ) . We prove that this is true in two steps . In Theorem 1 , we bound the expected distance between the gradient descent steps on the GNN and on the graphon neural network by a term that decreases asymptotically with the size of the graph . A consequence of this bias bound is that it allows us to quantify the trade-off between a more accurate gradient and one that could be obtained with less computational power . We then use this theorem to prove our main result in Theorem 2 , which is stated in simplified form below . Theorem ( Graphon neural network learning , informal ) Let W be a graphon and let { Gn } be a sequence of growing graphs sampled from W. Consider the graphon neural network Φ ( W ) and assume that it is learned by training the GNN Φ ( Gn ) with loss function ` ( yn , Φ ( Gn ) ) on the sequence { Gn } . Over a finite number of training steps , we obtain ‖∇ ` ( Y , Φ ( W ) ‖ ≤ with probability 1 . The most important implication of this result is that the learning iterates computed in the sequence of growing graphs follow the direction of the graphon gradient up to a small ball , which provides theoretical validation to our cost efficient training methodology . We also validate our algorithm in two numerical experiments . In the first , we learn a GNN-based recommendation system on increasingly large subnetworks of a movie similarity graph , and compare it with the recommendation system trained on the full graph . In the second , we consider the problem of flocking and train GNNs to learn the actions agents need to take to flock . We compare the results obtained when progressively increasing the number of agents during training and when training directly on the target graph . 2 RELATED WORK . GNNs are data processing architectures that follow from the seminal works in the areas of deep learning applied to graph theory ( Bruna et al. , 2013 ; Defferrard et al. , 2016 ; Gori et al. , 2005 ; Lu & Getoor , 2003 ) . They have been successfully used in a wide variety of statistical learning problems ( Kipf & Welling , 2016 ; Scarselli et al. , 2018 ) , where their good performance is generally attributed to the fact that they exploit invariances present in network data ( Maron et al. , 2019 ; Gama et al. , 2018 ; Chami et al. , 2021 ) . More recently , a number of works show that GNNs can be transferred across graphs of different sizes ( Ruiz et al. , 2020a ; Levie et al. , 2019 ; Keriven et al. , 2020 ) . Specifically , ( Ruiz et al. , 2020a ) leverages graphons to define families of graphs within which GNNs can be transferred with an error bound that decreases asymptotically with the size of the graph . The papers by Levie et al . ( 2019 ) and Keriven et al . ( 2020 ) offer similar results by considering the graph limit to be a generic topological space and a random graph model respectively . In this paper , we use an extension of the transferability bound derived in Ruiz et al . ( 2020a ) to propose a novel learning algorithm for GNNs . 3 PRELIMINARY DEFINITIONS . 3.1 GRAPH NEURAL NETWORKS . Graph neural networks exploit graph symmetries to extract meaningful information from network data ( Ruiz et al. , 2020c ; Gama et al. , 2020 ) . Graphs are represented as triplets Gn = ( V , E , W ) , where V , |V| = n , is the set of nodes , E ⊆ V × V is the set of edges and W : E → R is a map assigning weights to each edge . The graph Gn can also be represented by the graph shift operator ( GSO ) S ∈ Rn×n , a square matrix that respects the sparsity of the graph . Examples of GSOs include the adjacency matrix A , the graph Laplacian L = diag ( A1 ) −A and their normalized counterparts Gama et al . ( 2018 ) . In this paper we consider the graph Gn to be undirected and fix S = A/n . Graph data is represented in the form of graph signals . A graph signal x = [ x1 , . . . , xn ] T ∈ Rn is a vector whose i-th component corresponds to the information present at the i-th node of graph Gn . A basic data aggregation operation can be defined by applying the GSO S to graph signals x . The resulting signal z = Sx is such that the data at node i is a weighted average of the information in the 1-hop neighborhood of i , zi = ∑ j∈Ni [ S ] ijxj where Ni = { j | [ S ] ij 6= 0 } . Information coming from further neighborhoods can be aggregated by successive applications of the GSO , also called shifts . Using this notion of shift , graph convolutions are defined by weighting the contribution of each successive application of S to define a polynomial in the GSO . Explicitly , the graph convolutional filter with coefficients h = [ h0 , . . . , hK−1 ] is given by y = h ∗S x = K−1∑ k=0 hkS kx ( 1 ) where ∗S denotes the convolution operation with GSO S. Since the adjacency matrix of an undirected graph is always symmetric , the GSO admits an eigendecomposition S = VΛVH . The columns of V are the graph eigenvectors and the diagonal elements of Λ are the graph eigenvalues , which take values between −1 and 1 and are ordered as −1 ≤ λ−1 ≤ λ−2 ≤ . . . ≤ 0 ≤ . . . ≤ λ2 ≤ λ1 ≤ 1 . Since the eigenvectors of S form an orthonormal basis of Rn , we can project ( 1 ) onto this basis to obtain the spectral representation of the graph convolution , which is given by h ( λ ) = K−1∑ k=0 hkλ k. ( 2 ) Note that ( 2 ) only depends on the hk and on the eigenvalues of the GSO . Hence , as a consequence of the Cayley-Hamilton theorem , convolutional filters may be used to represent any graph filter with spectral representation h ( λ ) = f ( λ ) where f is analytic ( Strang , 1976 ) . Graph neural networks are layered architectures where each layer consists of a graph convolution followed by a pointwise nonlinearity ρ , and where each layer ’ s output is the input to the following layer . At layer l , a GNN can output multiple features xfl , 1 ≤ f ≤ Fl which we stack in a matrix Xl = [ x1l , . . . , x Fl l ] ∈ Rn×Fl . Each column of the feature matrix is the value of the graph signal at feature f . To map the Fl−1 features coming from layer l − 1 into Fl features , Fl−1 × Fl convolutions need to be computed , one per input-output feature pair . Stacking their weights in K matrices Hlk ∈ RFl−1×Fl , we write the l-th layer of the GNN as Xl = ρ ( K−1∑ k=0 SkXl−1Hlk ) . ( 3 ) In an L-layer GNN , the operation in ( 3 ) is cascaded L times to obtain the GNN output Y = XL . At the first layer , the GNN input is given by X0 = X ∈ Rn×F0 . In this paper we assume F0 = FL = 1 so that Y = y and X = x . A more concise representation of this GNN can be obtained by grouping all learnable parameters Hlk in a tensor H = { Hlk } l , k and defining the map y = Φ ( x ; H , S ) . Due to the polynomial nature of the graph convolution , the dimensions of the learnable parameter tensor H are independent from the size of the graph ( K is typically much smaller than n ) . Ergo , a GNN trained on a graph Gn can be deployed on a network Gm with m 6= n . 3.2 GRAPHON INFORMATION PROCESSING . A graphon is a bounded , symmetric , and measurable function W : [ 0 , 1 ] 2 → [ 0 , 1 ] which has two theoretical interpretations — it is both a graph limit and a generative model for graphs . In the first interpretation , sequences of dense graphs converge to a graphon in the sense that the densities of adjacency-preserving graph motifs converge to the same densities on the graphon Lovász ( 2012 ) . In the second , graphs can be generated from a graphon by sampling points ui , uj from the unit interval and either assigning weight W ( ui , uj ) to edges ( i , j ) , or sampling edges ( i , j ) with probability W ( ui , uj ) . In this paper , we focus on stochastic graphs Gn where the points ui are defined as ui = ( i− 1 ) /n for 1 ≤ i ≤ n and where the adjacency matrix Sn is sampled from W as [ Sn ] ij ∼ Bernoulli ( W ( ui , uj ) ) . ( 4 ) Sequences of graphs generated in this way can be shown to converge to W with probability one ( Lovász , 2012 ) [ Chapter 11 ] . In practice , the two theoretical interpretations of a graphon allow thinking of it as an identifying object for a family of graphs of different sizes that are structurally similar . Hence , given a network we can use its family ’ s identifying graphon as a continuous proxy for the graph . This is beneficial because it is typically easier to operate in continuous than in discrete domains , even more so if the network is large . We will leverage these ideas to consider graphon data and graphon neural networks as proxies for graph data and GNNs supported on graphs of arbitrary size .

Summary: This paper shows that a graph neural network (GNN) trained on an increasingly larger graph behaves like a graphon neural network (WNN), the "limit object of a GNN", which arises when considering a continuous graphon instead of a discrete graph (the graphon can be seen as a generative model from which the discrete graph is sampled).   This observation is translated into two theorems that bound the expected distance between the GNN and the WNN as a function of the number of nodes used to train the GNN.   The authors then propose an algorithm to train a GNN by gradually adding nodes to the training data, leading to a computational advantage compared to training the GNN on the full dataset from the start.