EIGNN: Efficient Infinite-Depth Graph Neural Networks

1 Introduction . Graph-structured data are ubiquitous in the real world . To model and learn from such data , graph representation learning aims to produce meaningful node representations by simultaneously considering the graph topology and node attributes . It has attracted growing interest in recent years , as well as numerous real-world applications [ 32 ] . In particular , graph neural networks ( GNNs ) are a widely used approach for node , edge , and graph prediction tasks . Recently , many GNN models have been proposed ( e.g. , graph convolutional network [ 14 ] , graph attention network [ 26 ] , simple graph convolution [ 30 ] ) . Most modern GNN models follow a “ message passing ” scheme : they iteratively aggregate the hidden representations of every node with those of the adjacent nodes to generate new hidden representations , where each iteration is parameterized as a neural network layer with learnable weights . Despite the success existing GNN models achieve on many different scenarios , they lack the ability to capture long-range dependencies . Specifically , for a predefined number of layers T , these models can not capture dependencies with a range longer than T -hops away from any given node . A straightforward strategy to capture long-range dependencies is to stack a large number of GNN layers for receiving “ messages ” from distant nodes . However , existing work has observed poor empirical performance when stacking more than a few layers [ 16 ] , which has been referred to as oversmoothing . This has been attributed to various reasons , including node representations becoming indistinguishable as depth increases . Besides oversmoothing , GNN models with numerous layers require excessive computational cost in practice since they need to repeatedly propagate representations across many layers . For these two reasons , simply stacking many layers for GNNs is not a suitable way to capture long-range dependencies . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) , virtual . Recently , instead of stacking many layers , several works have been proposed to capture long-range dependencies in graphs . Pei et al . [ 21 ] propose a global graph method Geom-GCN which builds structural neighborhoods based on the graph and the embedding space , and uses a bi-level aggregation to capture long-range dependencies . However , as Geom-GCN still only has finite layers , it still fails to capture very long range dependencies . To model longer range dependencies , Gu et al . [ 10 ] propose the implicit graph neural network ( IGNN ) , which can be considered as a GNN model with infinite layers . Thus , IGNN does not suffer from any a priori limitation on the range of information it can capture . To achieve this , IGNN generates predictions as the solution to a fixed-point equilibrium equation . Specifically , the solution is obtained by an iterative solver . Moreover , they require additional conditions to ensure well-posedness of the model and use projected gradient descent method to train the model . However , the practical limitations of iterative solvers are widely recognized : they can lack robustness ; the generated solution is approximated ; and the number of iterations can not be known in advance [ 23 ] . In our experiments , we found that IGNN sometimes experiences non-convergence in its iterative solver . Besides the non-convergence issue , the iterative solver of IGNN can be inefficient as IGNN needs to run it once per forward or backward pass . Motivated by these limitations , we propose our Efficient Infinite-Depth Graph Neural Network ( EIGNN ) approach , which can effectively capture very long range dependencies in graphs . Instead of relying on iterative solvers to generate the solution like in IGNN [ 10 ] , we derive a closed-form solution for EIGNN without additional conditions , avoiding the need for projected gradient descent to train the model . Furthermore , we propose to use eigendecomposition to improve the efficiency of EIGNN , without affecting its accuracy . The contributions of this work are summarized as follows : • To capture long-range dependencies , we propose our infinite-depth EIGNN model . To do this , we first define our model as the limit of an infinite sequence of graph convolutions , and theoretically prove its convergence . Then , we derive tractable forward and backward computations for EIGNN . • We then further derive an eigendecomposition-based approach to improve the computational/memory complexity of our approach , without affecting its accuracy . • We empirically compare our model to recent baseline GNN models on synthetic and real-world graph datasets . The results show that EIGNN has a better ability to capture very long range dependencies and provides better performance compared with other baselines . Moreover , the empirical results of noise sensitivity experiments demonstrate that EIGNN is more robust against both noise and adversarial perturbations . Paper outline In Section 2 , we provide an overview of GNNs , implicit models , and the oversmoothing problem . Section 3 introduces the background and major mathematics symbols used in this paper . In Section 4 , we first present the EIGNN model and discuss how to train this infinitely deep model in practice . In addition , we show that with eigendecomposition we can reduce the complexity and achieve more efficient computation for EIGNN . In Section 5 , we empirically compare EIGNN with other representative GNN methods . 2 Related work . Graph neural network models GNN models have been successfully used on various graph related tasks [ 32 , 34 , 27 , 28 ] . Pioneered by Graph Convolutional Networks ( GCNs ) [ 14 ] , many convolutional GNN models [ 33 , 30 , 26 , 4 , 15 , 11 , 29 ] use different aggregation schemes and components ( e.g. , attention [ 26 ] , skip connection [ 33 , 4 ] ) . They only have a finite number of aggregation layers , which makes them unable to capture long-range dependencies . Recurrent GNN models [ 24 , 9 , 17 , 6 ] generally share the same parameters in each aggregation step and potentially allow infinite steps until convergence . However , as pointed out in Gu et al . [ 10 ] , their sophisticated training process and conservative convergence conditions limit the use of these previous methods in practice . Recently , Gu et al . [ 10 ] proposes an implicit graph neural network ( IGNN ) with infinite depth . IGNN uses an iterative solver for forward and backward computations , and requires additional conditions to ensure the convergence of the iterative method . With infinite depth , they aim to capture very long range dependencies on graphs . Another global method Geom-GCN [ 21 ] is also proposed for capturing long-range dependencies using structural neighborhoods based on the graph and the embedding space . However , as Geom-GCN only has finite layers , it fails to capture very long range dependencies . Implicit models Implicit networks use implicit hidden layers which are defined through an equilibrium point of an infinite sequence of computation . This makes an implicit network equivalent to a feedforward network with infinite depth . Several works [ 1 , 2 , 5 , 7 ] show the potential advantages of implicit models on many applications , e.g. , language modeling , image classification , and semantic segmentation . Besides practical applications , Kawaguchi [ 13 ] provides a theoretical analysis on the global convergence of deep implicit linear models . Oversmoothing For capturing long-range dependencies , a straightforward method is to stack more GCN layers . However , Li et al . [ 16 ] found that stacking many layers make the learned node representations indistinguishable , which is named the oversmoothing phenomenon . To mitigate oversmoothing and allow for deeper GNN models , several empirical and theoretical works [ 16 , 15 , 4 , 35 , 19 ] have been proposed . However , these models still can not effectively capture long-range dependencies , as shown in our empirical experiments ( see Section 5 ) . 3 Preliminaries . Let G = ( V , E ) be an undirected graph with node set V and edge set E , where the number of nodes n = |V| . In practice , a graph G can be represented as its adjacency matrix A 2 Rn⇥n and its node feature matrix X 2 Rm⇥n where the feature vector of node i is xi 2 Rm . In node classification task , given graph data ( G , X ) , graph models are required to produce the prediction ŷi for node i to match the true label yi . Simple graph convolution The Simple Graph Convolution ( SGC ) was recently proposed by Wu et al . [ 30 ] . The pre-softmax output of SGC of depth H can be written as : fSGC ( X , W ) = WXS H 2 Rmy⇥n , ( 1 ) where X 2 Rm⇥n is the node feature matrix , SH = Q H i=1 S 2 Rn⇥n is the product of H normalized adjacency matrix with added self-loops S 2 Rn⇥n , and W 2 Rmy⇥m is the matrix of the trainable weight parameters . After the graph convolution , the softmax operation is applied to obtain the prediction ŷ . Notation We use ⌦ to represent Kronecker product . is used to represent element-wise product between two matrices with the same shape . For a matrix V 2 Rx⇥y , by stacking all columns , we can get vec [ V ] 2 Rxy as the vectorized form of V . The Frobenius norm of V is denoted as kV kF . kV k2 denotes the 2-norm . We use In to represent the identity matrix with size n⇥ n . 4 Efficient Infinite-Depth Graph Neural Networks . To capture long-range dependencies , we propose Efficient Infinite-Depth Graph Neural Networks ( EIGNN ) . The pre-softmax output of EIGNN is defined as : f ( X , F , B ) = B ⇣ lim H ! 1 Z ( H ) X , F ⌘ , ( 2 ) where F 2 Rm⇥m and B 2 Rmy⇥m represent the trainable weight parameters , and Z ( H ) = Z ( H ) X , F is the output of the H-th hidden layer : Z ( l+1 ) = g ( F ) Z ( l ) S +X ( 3 ) where we are given an arbitrary 2 ( 0 , 1 ] and g ( F ) = 1 kF > FkF + ✏F F > F ( 4 ) with an arbitrary small ✏F > 0 . Note that g ( F ) is constrained by Equation ( 4 ) to lie within a Frobenius norm ball of radius < 1 , which prevents divergence of the infinite sequence . EIGNN extends SGC to an infinite depth model with learnable propagation while additionally adding skip connections . By the designs , EIGNN have several helpful properties : 1 ) residual connection , which is also used in finite-depth GNN models ( e.g. , APPNP [ 15 ] and GCNII [ 4 ] ) and has shown its importance in graph learning ; 2 ) learnable weights for propagation on graphs ( i.e. , g ( F ) ) . In contrast , APPNP [ 15 ] only directly propagates information without learnable weights . As EIGNN is a model with infinite depth , how to conduct forward and backward computation for training is not obvious . In the rest of this section , we first show how to perform forward and backward computation for EIGNN . Then we propose to use eigendecomposition to improve the efficiency of this computation .

Summary: The authors propose a linear infinite-depth GNN, whose forward pass is a convergent series whose limit can be written in closed-form. Since the series involves the powers of the normalised adjacency matrix, the resulting model captures interactions between all pairs of connected nodes. At the same time, the existence of a closed-form expression makes the gradients easy to compute and provides an easy way to train the model. The authors show how the computational complexity of the model can be improved by diagonalising some of the matrices involved in the computation. The paper validates the hypothesis that the model can capture long-range interactions better than existent models in a synthetic setting and also on small real-world heterophilic graphs.