Constant Time Graph Neural Networks

1 INTRODUCTION . Machine learning on graph structures has various applications such as chemo-informatics ( Gilmer et al. , 2017 ) , question answering systems ( Schlichtkrull et al. , 2018 ) , and recommender systems ( Fan et al. , 2019 ) . Recently , a novel machine learning model for graph data called graph neural networks ( GNNs ) ( Gori et al. , 2005 ; Scarselli et al. , 2009 ; Kipf & Welling , 2017 ; Hamilton et al. , 2017 ) demonstrated state-of-the-art performances in various graph learning tasks . However , large scale graphs such as social network graphs and web graphs contain billions of nodes , and even a linear time computation cost per iteration is prohibited . Therefore , applying GNNs to huge graphs is challenging . Although Ying et al . ( 2018 ) succeeded in applying GNNs to a web-scale network using MapReduce , it still requires massive computational resources . There are several node sampling techniques to reduce GNN computation . For example , an empirical neighbor sampling scheme is used to speed up GraphSAGE ( Hamilton et al. , 2017 ) . FastGCN employs a random layer-wise node sampling ( Chen et al. , 2018b ) . Huang et al . ( 2018 ) further improved FastGCN by using an adaptive sampling technique to reduce the variance of estimators . Chen et al . ( 2018a ) proposed a variant of neighbor sampling , which used historical activations to reduce the estimator variance . Overall , the existing sampling techniques for GNNs work well in practice . However , these techniques are either not theoretically guaranteed in terms of approximation error , or thy require at least a linear time computation cost . In this study , we consider the problem of approximating the embedding of one node using GNNs in constant time with maximum precision1 . We analyze the neighbor sampling technique ( Hamilton et al. , 2017 ) to show that a constant number of samples are needed to guarantee the approximation error . It should be noted that the neighbor sampling was introduced as a heuristic method originally , and they did not provide any theoretical guarantees . Specifically , given an error tolerance ε and 1e.g. , predicting whether a user of an SNS clicks an advertisement by GNNs in real-time ( i.e. , when the user accesses ) . A user may have many neighbors , but GNNs must respond in limited time , which prohibits exact computation . It motivates us to approximate the exact computation in limited time with theoretical guarantee . confidence probability 1 − δ , our analysis shows that the estimate ẑv of the exact embedding zv of a node v such that Pr [ ‖ẑv − zv‖2 ≥ ε ] ≤ δ and the estimate ∂̂zv∂θ of the exact gradient ∂zv ∂θ of the embedding zv with respect to the network parameters θ , such that Pr [ ‖ ∂̂zv∂θ − ∂zv ∂θ ‖F ≥ ε ] ≤ δ can be computed in a constant time . Especially , the uniform node sampling can approximate the exact embedding and its gradients within O ( 1 ε2L ( log 1ε + log 1 δ ) L−1 log 1δ ) time , where L denotes the number of layers . This complexity is completely independent of the number of nodes , edges , and neighbors of the input , which enables us to deal with graphs irrespective of their size . Moreover , the complexity is a polynomial with respect to 1ε and log 1 δ . We demonstrate that the time complexity is optimal when L = 1 with respect to the error tolerance ε . Through experiments , we show that the approximation error between the exact computation and its approximation rapidly converges to zero . To the best of our knowledge , this is the first constant time approximation algorithm for GNNs with a theoretical guarantee in terms of approximation error . Contributions : The contributions of this paper are summarized as follows : • We analyze the neighbor sampling technique for GraphSAGE , GAT , and GCN to provide theoretical justification . Especially , our analysis shows that the complexity is completely independent of the number of nodes , edges , and neighbors of the input . • We show that some existing GNNs , including the original GraphSAGE ( Hamilton et al. , 2017 ) , can not be approximated in constant time by any algorithm ( see Table 1 for details ) . • We empirically validate our theorems using synthetic and real-world datasets . 2 RELATED WORK . 2.1 GRAPH NEURAL NETWORKS . Graph neural networks ( GNNs ) were first introduced by Gori et al . ( 2005 ) and Scarselli et al . ( 2009 ) . They obtained node embedding by recursively applying the propagation function until convergence . Kipf & Welling ( 2017 ) proposed graph convolutional networks ( GCN ) , which significantly outperformed the existing methods , including non-neural network based approaches . Gilmer et al . ( 2017 ) proposed the message passing neural networks ( MPNNs ) , a general framework of GNNs using the message passing mechanism . Veličković et al . ( 2018 ) proposed the graph attention networks ( GAT ) , which incorporate the attention mechanism into GNNs . With the advent of GAT , various GNN models with the attention mechanism have been recently proposed ( Wang et al. , 2019 ; Park et al. , 2019 ) . GraphSAGE ( Hamilton et al. , 2017 ) is another GNN model , which employs neighbor sampling to reduce the computational costs of training and inference . Owing to neighbor sampling , GraphSAGE can deal with large graphs . However , neighbor sampling was introduced without any theoretical guarantee , and the number of samples is chosen empirically . An alternative computationally efficient GNN would be FastGCN ( Chen et al. , 2018b ) , which employs layer-wise random node sampling to speed up training and inference . Huang et al . ( 2018 ) further improved FastGCN by using an adaptive node sampling technique to reduce the variance of estimators . Thanks to the adaptive sampling technique , it reduces the computational costs and outperforms neighbor sampling in terms of classification accuracy and convergence speed . Chen et al . ( 2018a ) proposed an alternative neighbor sampling technique , which uses historical activations to reduce the estimator variance . Additionally , it could achieve zero variance after a certain number of iterations . However , because it used the same sampling technique of GraphSAGE to obtain the initial solution , the approximation error was not theoretically bounded until the Ω ( n ) -th iteration . Overall , the existing sampling techniques work well in practice . However , these techniques are either not theoretically guaranteed in terms of approximation error , or they require at least a linear time computation cost to calculate the embedding of a node and its gradient of GNN models . Moreover , it is not clear whether we can apply the sampling techniques to state-of-the-art GNN models such as GAT . 2.2 SUBLINEAR TIME ALGORITHMS . The sublinear time algorithms were originally proposed for property testing ( Rubinfeld & Sudan , 1996 ) . Sublinear property testing algorithms check whether the input has some property π or the input is sufficiently far from the property π with high probability in sublinear time with respect to the input size . Sublinear time approximation algorithms are another type of sublinear time algorithms . More specifically , they calculate a value sufficiently close to the exact value with high probability in sublinear time . Constant time algorithms are a subclass of sublinear time algorithms . They work not only in sublinear time with respect to the input size but also in constant time . The proposed algorithm is classified as a constant time approximation algorithm . The examples of sublinear time approximation algorithms include minimum spanning tree in metric space ( Czumaj & Sohler , 2004 ) and minimum spanning tree with integer weights ( Chazelle et al. , 2005 ) . Parnas & Ron ( 2007 ) proposed a method to convert distributed local algorithms into constant time approximation algorithms . In their study , they proposed a method to construct constant time algorithms for the minimum vertex cover problem and dominating set problem . A classic example of sublinear time algorithms related to machine learning includes clustering ( Indyk , 1999 ; Mishra et al. , 2001 ) . Examples of recent work in this stream include constant time approximation of the minimum value of quadratic functions ( Hayashi & Yoshida , 2016 ) and constant time approximation of the residual error of the Tucker decomposition ( Hayashi & Yoshida , 2017 ) . They adopted simple sampling strategies to obtain theoretical guarantee similar to our work . In this paper , we provide theoretical guarantee for approximation of GNNs within a constant time for the first time . 3 BACKGROUND . 3.1 NOTATIONS . Let G be the input graph , V = { 1 , 2 , . . . , n } be the set of nodes , n = |V| be the number of nodes , E be the set of edges , m = |E| be the number of edges , deg ( v ) be the degree of a node v , N ( v ) be the set of neighbors of a node v , xv ∈ Rd0 be the feature vector associated to a node v ∈ V , X = ( x1 , x2 . . . , xn ) > ∈ Rn×d0 be the stacked feature vectors , and > denotes the matrix transpose . 3.2 NODE EMBEDDING MODEL Algorithm 1 Oz : Exact embedding Require : Graph G = ( V , E ) ; Features X ∈ Rn×d0 ; Node index v ∈ V ; Model parameters θ . Ensure : Exact embedding zv 1 : z ( 0 ) i ← xi ( ∀i ∈ V ) 2 : for l ∈ { 1 , . . . , L } do 3 : for i ∈ V do 4 : h ( l ) i ← ∑ u∈N ( i ) Mliu ( z ( l−1 ) i , z ( l−1 ) u , eiu , θ ) 5 : z ( l ) i ← Ul ( z ( l−1 ) i , h ( l ) i , θ ) 6 : end for 7 : end for 8 : return z ( L ) v We consider the node embedding problem using GNNs . Especially , we employ the message passing neural networks ( MPNNs ) framework ( Gilmer et al. , 2017 ) . This framework includes many GNN models , such as GraphSAGE and GCN . Algorithm 1 shows the algorithm of MPNNs . We refer to z ( L ) v as zv simply . The aim of this study is to develop a constant time approximation algorithm for calculating the embedding vector zv and gradients ∂zv∂θ with the given model parameters θ and node v . 3.3 COMPUTATIONAL MODEL ASSUMPTIONS . We have to specify how to access the input to design constant time algorithms because the constant time algorithms can not read the entire input . We follow the standard convention of sublinear time algorithms ( Parnas & Ron , 2007 ; Nguyen & Onak , 2008 ) . We model our algorithm as an oracle machine that can generate queries regarding the input , and we measure the complexity by query complexity . Algorithms can access the input only by querying the following oracles : ( 1 ) Odeg ( v ) : the degree of node v , ( 2 ) OG ( v , i ) : the i-th neighbor of node v , and ( 3 ) Ofeature ( v ) : the feature of node v. We assume that our algorithm can query the oracles in constant time per query .

Summary: In this paper, the authors provide a theoretical framework for characterizing the approximation guarantees provided by node sampling to estimate embeddings in various GNN architectures. In particular, they prove several PAC learning-style bounds on the embedding and gradient estimation when using node sampling approaches. They also observe that since the number of nodes selected for sampling is not dependent on the size of the graph, this amounts to a constant time operation for determining embedding and gradient estimates.