EIGNN: Efficient Infinite-Depth Graph Neural Networks

1 Introduction . Graph-structured data are ubiquitous in the real world . To model and learn from such data , graph representation learning aims to produce meaningful node representations by simultaneously considering the graph topology and node attributes . It has attracted growing interest in recent years , as well as numerous real-world applications [ 32 ] . In particular , graph neural networks ( GNNs ) are a widely used approach for node , edge , and graph prediction tasks . Recently , many GNN models have been proposed ( e.g. , graph convolutional network [ 14 ] , graph attention network [ 26 ] , simple graph convolution [ 30 ] ) . Most modern GNN models follow a “ message passing ” scheme : they iteratively aggregate the hidden representations of every node with those of the adjacent nodes to generate new hidden representations , where each iteration is parameterized as a neural network layer with learnable weights . Despite the success existing GNN models achieve on many different scenarios , they lack the ability to capture long-range dependencies . Specifically , for a predefined number of layers T , these models can not capture dependencies with a range longer than T -hops away from any given node . A straightforward strategy to capture long-range dependencies is to stack a large number of GNN layers for receiving “ messages ” from distant nodes . However , existing work has observed poor empirical performance when stacking more than a few layers [ 16 ] , which has been referred to as oversmoothing . This has been attributed to various reasons , including node representations becoming indistinguishable as depth increases . Besides oversmoothing , GNN models with numerous layers require excessive computational cost in practice since they need to repeatedly propagate representations across many layers . For these two reasons , simply stacking many layers for GNNs is not a suitable way to capture long-range dependencies . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) , virtual . Recently , instead of stacking many layers , several works have been proposed to capture long-range dependencies in graphs . Pei et al . [ 21 ] propose a global graph method Geom-GCN which builds structural neighborhoods based on the graph and the embedding space , and uses a bi-level aggregation to capture long-range dependencies . However , as Geom-GCN still only has finite layers , it still fails to capture very long range dependencies . To model longer range dependencies , Gu et al . [ 10 ] propose the implicit graph neural network ( IGNN ) , which can be considered as a GNN model with infinite layers . Thus , IGNN does not suffer from any a priori limitation on the range of information it can capture . To achieve this , IGNN generates predictions as the solution to a fixed-point equilibrium equation . Specifically , the solution is obtained by an iterative solver . Moreover , they require additional conditions to ensure well-posedness of the model and use projected gradient descent method to train the model . However , the practical limitations of iterative solvers are widely recognized : they can lack robustness ; the generated solution is approximated ; and the number of iterations can not be known in advance [ 23 ] . In our experiments , we found that IGNN sometimes experiences non-convergence in its iterative solver . Besides the non-convergence issue , the iterative solver of IGNN can be inefficient as IGNN needs to run it once per forward or backward pass . Motivated by these limitations , we propose our Efficient Infinite-Depth Graph Neural Network ( EIGNN ) approach , which can effectively capture very long range dependencies in graphs . Instead of relying on iterative solvers to generate the solution like in IGNN [ 10 ] , we derive a closed-form solution for EIGNN without additional conditions , avoiding the need for projected gradient descent to train the model . Furthermore , we propose to use eigendecomposition to improve the efficiency of EIGNN , without affecting its accuracy . The contributions of this work are summarized as follows : • To capture long-range dependencies , we propose our infinite-depth EIGNN model . To do this , we first define our model as the limit of an infinite sequence of graph convolutions , and theoretically prove its convergence . Then , we derive tractable forward and backward computations for EIGNN . • We then further derive an eigendecomposition-based approach to improve the computational/memory complexity of our approach , without affecting its accuracy . • We empirically compare our model to recent baseline GNN models on synthetic and real-world graph datasets . The results show that EIGNN has a better ability to capture very long range dependencies and provides better performance compared with other baselines . Moreover , the empirical results of noise sensitivity experiments demonstrate that EIGNN is more robust against both noise and adversarial perturbations . Paper outline In Section 2 , we provide an overview of GNNs , implicit models , and the oversmoothing problem . Section 3 introduces the background and major mathematics symbols used in this paper . In Section 4 , we first present the EIGNN model and discuss how to train this infinitely deep model in practice . In addition , we show that with eigendecomposition we can reduce the complexity and achieve more efficient computation for EIGNN . In Section 5 , we empirically compare EIGNN with other representative GNN methods . 2 Related work . Graph neural network models GNN models have been successfully used on various graph related tasks [ 32 , 34 , 27 , 28 ] . Pioneered by Graph Convolutional Networks ( GCNs ) [ 14 ] , many convolutional GNN models [ 33 , 30 , 26 , 4 , 15 , 11 , 29 ] use different aggregation schemes and components ( e.g. , attention [ 26 ] , skip connection [ 33 , 4 ] ) . They only have a finite number of aggregation layers , which makes them unable to capture long-range dependencies . Recurrent GNN models [ 24 , 9 , 17 , 6 ] generally share the same parameters in each aggregation step and potentially allow infinite steps until convergence . However , as pointed out in Gu et al . [ 10 ] , their sophisticated training process and conservative convergence conditions limit the use of these previous methods in practice . Recently , Gu et al . [ 10 ] proposes an implicit graph neural network ( IGNN ) with infinite depth . IGNN uses an iterative solver for forward and backward computations , and requires additional conditions to ensure the convergence of the iterative method . With infinite depth , they aim to capture very long range dependencies on graphs . Another global method Geom-GCN [ 21 ] is also proposed for capturing long-range dependencies using structural neighborhoods based on the graph and the embedding space . However , as Geom-GCN only has finite layers , it fails to capture very long range dependencies . Implicit models Implicit networks use implicit hidden layers which are defined through an equilibrium point of an infinite sequence of computation . This makes an implicit network equivalent to a feedforward network with infinite depth . Several works [ 1 , 2 , 5 , 7 ] show the potential advantages of implicit models on many applications , e.g. , language modeling , image classification , and semantic segmentation . Besides practical applications , Kawaguchi [ 13 ] provides a theoretical analysis on the global convergence of deep implicit linear models . Oversmoothing For capturing long-range dependencies , a straightforward method is to stack more GCN layers . However , Li et al . [ 16 ] found that stacking many layers make the learned node representations indistinguishable , which is named the oversmoothing phenomenon . To mitigate oversmoothing and allow for deeper GNN models , several empirical and theoretical works [ 16 , 15 , 4 , 35 , 19 ] have been proposed . However , these models still can not effectively capture long-range dependencies , as shown in our empirical experiments ( see Section 5 ) . 3 Preliminaries . Let G = ( V , E ) be an undirected graph with node set V and edge set E , where the number of nodes n = |V| . In practice , a graph G can be represented as its adjacency matrix A 2 Rn⇥n and its node feature matrix X 2 Rm⇥n where the feature vector of node i is xi 2 Rm . In node classification task , given graph data ( G , X ) , graph models are required to produce the prediction ŷi for node i to match the true label yi . Simple graph convolution The Simple Graph Convolution ( SGC ) was recently proposed by Wu et al . [ 30 ] . The pre-softmax output of SGC of depth H can be written as : fSGC ( X , W ) = WXS H 2 Rmy⇥n , ( 1 ) where X 2 Rm⇥n is the node feature matrix , SH = Q H i=1 S 2 Rn⇥n is the product of H normalized adjacency matrix with added self-loops S 2 Rn⇥n , and W 2 Rmy⇥m is the matrix of the trainable weight parameters . After the graph convolution , the softmax operation is applied to obtain the prediction ŷ . Notation We use ⌦ to represent Kronecker product . is used to represent element-wise product between two matrices with the same shape . For a matrix V 2 Rx⇥y , by stacking all columns , we can get vec [ V ] 2 Rxy as the vectorized form of V . The Frobenius norm of V is denoted as kV kF . kV k2 denotes the 2-norm . We use In to represent the identity matrix with size n⇥ n . 4 Efficient Infinite-Depth Graph Neural Networks . To capture long-range dependencies , we propose Efficient Infinite-Depth Graph Neural Networks ( EIGNN ) . The pre-softmax output of EIGNN is defined as : f ( X , F , B ) = B ⇣ lim H ! 1 Z ( H ) X , F ⌘ , ( 2 ) where F 2 Rm⇥m and B 2 Rmy⇥m represent the trainable weight parameters , and Z ( H ) = Z ( H ) X , F is the output of the H-th hidden layer : Z ( l+1 ) = g ( F ) Z ( l ) S +X ( 3 ) where we are given an arbitrary 2 ( 0 , 1 ] and g ( F ) = 1 kF > FkF + ✏F F > F ( 4 ) with an arbitrary small ✏F > 0 . Note that g ( F ) is constrained by Equation ( 4 ) to lie within a Frobenius norm ball of radius < 1 , which prevents divergence of the infinite sequence . EIGNN extends SGC to an infinite depth model with learnable propagation while additionally adding skip connections . By the designs , EIGNN have several helpful properties : 1 ) residual connection , which is also used in finite-depth GNN models ( e.g. , APPNP [ 15 ] and GCNII [ 4 ] ) and has shown its importance in graph learning ; 2 ) learnable weights for propagation on graphs ( i.e. , g ( F ) ) . In contrast , APPNP [ 15 ] only directly propagates information without learnable weights . As EIGNN is a model with infinite depth , how to conduct forward and backward computation for training is not obvious . In the rest of this section , we first show how to perform forward and backward computation for EIGNN . Then we propose to use eigendecomposition to improve the efficiency of this computation .

Summary: This paper presents an interesting idea on designing a novel graph neural network to capture the long-range dependency, which is the lack of ability of most existing works. A linear model that extends SGC with implicit infinite layers, EIGNN, is proposed to mitigate this issue. A closed form of the infinite sequences is used to define the layer-wise update rule and eigendecomposition is combined to achieve more efficient training. Experimental results on both synthetic and real-world datasets demonstrate the effectiveness of the proposed EIGNN for capturing the long-range dependency.