RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
Paper outline
In Section 2 , we provide an overview of GNNs, implicit models, and the oversmoothing problem.
Section 3 introduces the background and major mathematics symbols used in this paper.
In Section 4 , we first present the EIGNN model and discuss how to train this infinitely deep model in practice. In addition, we show that with eigendecomposition we can reduce the complexity and achieve more efficient computation for EIGNN.
In Section 5 , we empirically compare EIGNN with other representative GNN methods.
Graph neural network models
GNN models have been successfully used on various graph related tasks [ 32 , 34 , 27 , 28 ] .
Pioneered by Graph Convolutional Networks (GCNs) [ 14 ] , many convolutional GNN models [ 33 , 30 , 26 , 4 , 15 , 11 , 29 ] use different aggregation schemes and components (e.g., attention [ 26 ] , skip connection [ 33 , 4 ] ).
They only have a finite number of aggregation layers, which makes them unable to capture long-range dependencies.
Recurrent GNN models [ 24 , 9 , 17 , 6 ] generally share the same parameters in each aggregation step and potentially allow infinite steps until convergence.
However, as pointed out in Gu et al. [ 10 ] , their sophisticated training process and conservative convergence conditions limit the use of these previous methods in practice.
Recently, Gu et al. [ 10 ] proposes an implicit graph neural network (IGNN) with infinite depth.
IGNN uses an iterative solver for forward and backward computations, and requires additional conditions to ensure the convergence of the iterative method.
With infinite depth, they aim to capture very long range dependencies on graphs.
Another global method Geom-GCN [ 21 ] is also proposed for capturing long-range dependencies using structural neighborhoods based on the graph and the embedding space.
However, as Geom-GCN only has finite layers, it fails to capture very long range dependencies.
Implicit models
Implicit networks use implicit hidden layers which are defined through an equilibrium point of an infinite sequence of computation. This makes an implicit network equivalent to a feedforward network with infinite depth.
Several works [ 1 , 2 , 5 , 7 ] show the potential advantages of implicit models on many applications, e.g., language modeling, image classification, and semantic segmentation.
Besides practical applications, Kawaguchi [ 13 ] provides a theoretical analysis on the global convergence of deep implicit linear models.
Oversmoothing
For capturing long-range dependencies, a straightforward method is to stack more GCN layers. However, Li et al. [ 16 ] found that stacking many layers make the learned node representations indistinguishable, which is named the oversmoothing phenomenon.
To mitigate oversmoothing and allow for deeper GNN models, several empirical and theoretical works [ 16 , 15 , 4 , 35 , 19 ] have been proposed.
However, these models still cannot effectively capture long-range dependencies, as shown in our empirical experiments (see Section 5 ).
Preliminaries
Let \mathcal{G}=(\mathcal{V},\mathcal{E}) be an undirected graph with node set \mathcal{V} and edge set \mathcal{E} , where the number of nodes n=|\mathcal{V}| .
In practice, a graph \mathcal{G} can be represented as its adjacency matrix A\in\mathbb{R}^{n\times n} and its node feature matrix X\in\mathbb{R}^{m\times n} where the feature vector of node i is x_{i}\in\mathbb{R}^{m} .
In node classification task, given graph data (\mathcal{G},X) , graph models are required to produce the prediction \hat{y_{i}} for node i to match the true label y_{i} .
Simple graph convolution
The Simple Graph Convolution (SGC) was recently proposed by Wu et al. [ 30 ] . The pre-softmax output of SGC of depth H can be written as:

f_{SGC}(X,W)=WXS^{H}\in\mathbb{R}^{m_{y}\times n}, (1)

where X\in\mathbb{R}^{m\times n} is the node feature matrix, S^{H}=\prod_{i=1}^{H}S\in\mathbb{R}^{n\times n} is the product of H normalized adjacency matrix with added self-loops S\in\mathbb{R}^{n\times n} , and W\in\mathbb{R}^{m_{y}\times m} is the matrix of the trainable weight parameters.
After the graph convolution, the softmax operation is applied to obtain the prediction \hat{y} .
Notation
We use \otimes to represent Kronecker product. \circ is used to represent element-wise product between two matrices with the same shape.
For a matrix V\in\mathbb{R}^{x\times y} , by stacking all columns, we can get \operatorname{vec}[V]\in\mathbb{R}^{xy} as the vectorized form of V .
The Frobenius norm of V is denoted as \|V\|_{\mathrm{F}} . \|V\|_{2} denotes the 2-norm.
We use I_{n} to represent the identity matrix with size n\times n .
Efficient Infinite-Depth Graph Neural Networks
To capture long-range dependencies, we propose Efficient Infinite-Depth Graph Neural Networks (EIGNN).
The pre-softmax output of EIGNN is defined as:

f(X,F,B)=B\left(\lim_{H\rightarrow\infty}Z^{(H)}_{X,F}\right), (2)

where F\in\mathbb{R}^{m\times m} and B\in\mathbb{R}^{m_{y}\times m} represent the trainable weight parameters, and Z^{(H)}=Z^{(H)}_{X,F} is the output of the H -th hidden layer:

Z^{(l+1)}=\gamma g(F)Z^{(l)}S+X (3)

where we are given an arbitrary \gamma\in(0,1] and

g(F)=\frac{1}{\|F^{\top}F\|_{\mathrm{F}}+\epsilon_{F}}F^{\top}F (4)

with an arbitrary small \epsilon_{F}>0 .

Note that g(F) is constrained by Equation ( 4 ) to lie within a Frobenius norm ball of radius <1 , which prevents divergence of the infinite sequence.
EIGNN extends SGC to an infinite depth model with learnable propagation while additionally adding skip connections.
By the designs, EIGNN have several helpful properties: 1) residual connection, which is also used in finite-depth GNN models (e.g., APPNP [ 15 ] and GCNII [ 4 ] ) and has shown its importance in graph learning; 2) learnable weights for propagation on graphs (i.e., g(F)). In contrast, APPNP [ 15 ] only directly propagates information without learnable weights.

As EIGNN is a model with infinite depth, how to conduct forward and backward computation for training is not obvious.
In the rest of this section, we first show how to perform forward and backward computation for EIGNN.
Then we propose to use eigendecomposition to improve the efficiency of this computation.
Forward and backward computation
At first glance, it may seem that we cannot compute the infinite sequence of (Z^{(l)})_{l} without iterative solvers in practice.
However, in Proposition 1 , we show that the infinite sequence of (Z^{(l)})_{l} is guaranteed to converge and is computable without iterative solvers.

Given any matrix X, F and normalized symmetric adjacency matrix S, the infinite sequence of (Z^{(l)})_{l} is convergent and the limit of the sequence can be written as follows:

\lim_{H\rightarrow\infty}\operatorname{vec}[Z^{(H)}]=\left(I-\gamma[S\otimes g(F)]\right)^{-1}\operatorname{vec}[X]

This can be proved by using the triangle inequality of a norm and some properties of the Kronecker product. The complete proof can be found in Appendix B.1 .
Computing f(X,F,B)
After getting the limit of the sequence, we can conduct forward computation without iterative solvers as shown in Proposition 2 .
In contrast, IGNN [ 10 ] heavily relies on iterative solvers for both forward and backward computation.
The motivation to avoid iterative solvers is that iterative solvers suffer from some commonly known issues, e.g., approximated solutions and sensitive convergence criteria [ 23 ] .

Given any matrices X, F and normalized symmetric adjacency matrix S, with an arbitrary \gamma\in(0,1] and an arbitrary small \epsilon_{F}>0 , f(X,F,B) of Equation ( 2 ) can be obtained without iterative solvers:

\operatorname{vec}[f(X,F,B)]=\operatorname{vec}\left[B\left(\lim_{H\rightarrow\infty}Z^{(H)}\right)\right]=[I_{n}\otimes B]\left(I-\gamma[S\otimes g(F)]\right)^{-1}\operatorname{vec}[X]

Combining Equation ( 5 ) with the property of the vectorization (i.e., \operatorname{vec}[AB]=(I_{m}\otimes A)\operatorname{vec}[B] ), Equation ( 6 ) can be obtained.
∎
Computing \frac{\partial(\lim_{H\rightarrow\infty}\operatorname{vec}[Z^{(H)}_{X,F}])}{\partial\operatorname{vec}[F]}
In Proposition 3 , we show that backward computation can be done without iterative solvers as well.
Without iterative solvers in both forward and backward computation, we can avoid the issue that the solution of iterative methods is not exact.
Specifically, as in IGNN [ 10 ] , the forward computation can yield an error through the iterative solver and the backward pass with the iterative solver can amplify the error from the forward computation. The approximation errors can lead to degradation performance, which we can avoid in our method.

Given any matrices X, F and normalized symmetric adjacency matrix S,
assuming an arbitrary \gamma\in(0,1] and an arbitrary small \epsilon_{F}>0 , the gradient \frac{\partial(\lim_{H\rightarrow\infty}\operatorname{vec}[Z^{(H)}_{X,F}])}{\partial\operatorname{vec}[F]} can be computed without iterative solver:

\frac{\partial(\lim_{H\rightarrow\infty}\operatorname{vec}[Z^{(H)}_{X,F}])}{\partial\operatorname{vec}[F]}=\gamma U^{-1}\left[S\left(\lim_{H\rightarrow\infty}Z^{(H)}_{X,F}\right)^{\top}\otimes I_{m}\right]\frac{\partial\operatorname{vec}[g(F)]}{\partial\operatorname{vec}[F]}, (7)

where U=I-\gamma[S\otimes g(F)] and the limit can be computed by

\lim_{H\rightarrow\infty}\operatorname{vec}[Z^{(H)}_{X,F}]=U^{-1}\operatorname{vec}[X].

The proof of Proposition 3 is strongly related to the implicit function theorem. See Appendix B.2 for the complete proof.
Computing the gradients of objective functions
With the gradients of the infinite sequence, the gradients of objective functions can be obtained.
Define the objective function to be minimized by:

L(B,F)=\ell_{Y}(f(X,F,B)).

Here, \ell_{Y} is an arbitrary differentiable function; for example, it can be set to the cross-entropy loss or square loss with the label matrix Y\in\mathbb{R}^{m_{y}\times n} .
Let \mathbf{f}_{X,F,B}=f(X,F,B) and Z_{X,F}=\lim_{H\rightarrow\infty}Z^{(H)}_{X,F} .
Then using the chain rule with Equation ( 7 ), we have

\frac{\partial L(B,F)}{\partial\operatorname{vec}[F]}=\gamma\frac{\partial\ell_{Y}(\mathbf{f}_{X,F,B})}{\partial\operatorname{vec}[\mathbf{f}_{X,F,B}]}[I_{n}\otimes B]U^{-1}\left[SZ_{X,F}^{\top}\otimes I_{m}\right]\frac{\partial\operatorname{vec}[g(F)]}{\partial\operatorname{vec}[F]} (8)

and

\frac{\partial L(B,F)}{\partial\operatorname{vec}[B]}=\frac{\partial\ell_{Y}(\mathbf{f}_{X,F,B})}{\partial\operatorname{vec}[\mathbf{f}_{X,F,B}]}[Z_{X,F}^{\top}\otimes I_{m_{y}}], (9)

where Z_{X,F}=(I-\gamma[S\otimes g(F)])\operatorname{vec}[X] .
The full derivation of Equation ( 8 ) and ( 9 ) can be found in Appendix B.3 .
More efficient computation via eigendecomposition
In the previous subsection, we show that even with the infinite depth of EIGNN, forward and backward computations can be conducted without iterative solvers.
However, this requires us to calculate and store the matrix U=I-\gamma[S\otimes g(F)]\in\mathbb{R}^{mn\times mn} , which can be expensive in terms of memory consumption. For example, when m=100,n=1000 , the memory requirement of U becomes around 40 GB which usually cannot be handled by a commodity GPU.
In this subsection, we show that we can achieve more efficient computation via eigendecomposition.
We avoid the matrix U by using eigendecomposition of much smaller matrices g(F)=Q_{F}\Lambda_{F}Q_{F}^{\top}\in\mathbb{R}^{m\times m} and S=Q_{S}\Lambda_{S}Q_{S}^{\top}\in\mathbb{R}^{n\times n} separately.
Therefore, we reduce the memory complexity from an mn\times mn matrix to two matrices with sizes m\times m and n\times n respectively.
We first define G\in\mathbb{R}^{m\times n} where G_{ij}=1/(1-\gamma(\bar{\Lambda}_{F}\bar{\Lambda}_{S}^{\top})_{ij}) .
The output of EIGNN and the gradient with respect to F can be computed as follows:

f(X,F,B)=BQ_{F}(G\circ(Q_{F}^{\top}XQ_{S}))Q_{S}^{\top}, (10)

abla_{\operatorname{vec}[F]}L(B,F)=\gamma\left(\frac{\partial\operatorname{vec}[g(F)]}{\partial\operatorname{vec}[F]}\right)^{\top}\operatorname{vec}\left[Q_{F}\left(G\circ\left(Q_{F}^{\top}B^{\top}\frac{\partial\ell_{Y}(\mathbf{f}_{X,F,B})}{\partial\mathbf{f}_{X,F,B}}Q_{S}\right)\right)Q_{S}^{\top}SZ_{X,F}^{\top}\right], (11)

where Z_{X,F}=Q_{F}(G\circ(Q_{F}^{\top}XQ_{S}))Q_{S}^{\top}\in\mathbb{R}^{m\times n} .
The complete derivations of Equation ( 10 ) and ( 11 ) are presented in Appendix B.4 .
Further improvements via avoiding \frac{\partial\operatorname{vec}[g(F)]}{\partial\operatorname{vec}[F]}
As shown in Equation ( 10 ) and ( 11 ), we can avoid the mn\times mn matrix, but this still requires us to deal with the mm\times mm matrix \frac{\partial\operatorname{vec}[g(F)]}{\partial\operatorname{vec}[F]} in Equation ( 11 ).
We show that we can further avoid this as well.
Therefore, the memory complexity for the computation of the gradient with respect to F reduces from O(m^{4}) to O(n^{2}) if n>m , or O(m^{2}) otherwise.

abla_{F}L(B,F)=\frac{\gamma}{\|F^{\top}F\|_{\mathrm{F}}+\epsilon_{F}}F\left((R+R^{\top})-\frac{2\left\langle F^{\top}F,R\right\rangle_{\mathrm{F}}}{\|F^{\top}F\|_{\mathrm{F}}^{2}+\epsilon_{F}\|F^{\top}F\|_{\mathrm{F}}}F^{\top}F\right)\in\mathbb{R}^{m\times m}, (12)

where

R=Q_{F}\left(G\circ\left(Q_{F}^{\top}B^{\top}\frac{\partial\ell_{Y}(\mathbf{f}_{X,F,B})}{\partial\mathbf{f}_{X,F,B}}Q_{S}\right)\right)Q_{S}^{\top}SZ_{X,F}^{\top}\in\mathbb{R}^{m\times m}.

Appendix B.5 shows the full derivation of Equation ( 12 ).
Note that \frac{\partial\ell_{Y}(\mathbf{f}_{X,F,B})}{\partial\mathbf{f}_{X,F,B}} can be easily calculated by modern autograd frameworks (e.g., PyTorch [ 20 ] ). For the gradient \nabla_{B}L(B,F) , it can be directly obtained by applying chain rule with \frac{\partial\ell_{Y}(\mathbf{f}_{X,F,B})}{\partial\mathbf{f}_{X,F,B}} since B is not within the infinite sequence. Thus, we omit the formula here.
Comparison with other GNNs with infinite depth
Comparing with IGNN [ 10 ] , our model EIGNN can directly obtain a closed-form solution rather than relying on an iterative solver. Thus, we avoid some common issues of iterative solvers [ 23 ] , e.g., approximated solutions and sensitivity to hyper-parameters.
IGNN requires additional conditions to ensure convergence, whereas EIGNN is also an infinite-depth GNN model but avoids such conditions. Without restrictions on the weight matrix, even as a linear model, EIGNN achieves better performance than IGNN as shown in the empirical experiments.
Previous work also shows that a linear GNN can achieve comparable performance compared with GCNs [ 30 ] .
APPNP and PPNP [ 15 ] separate predictions from the propagation scheme where PPNP can be treated as considering infinitely many aggregation layers as well via the personalized pagerank matrix.
However, the propagation of APPNP and PPNP is not learnable.
This is completely different from IGNN and EIGNN, and limits the model’s representation power.
Besides that, PPNP requires performing a costly matrix inverse operation for a dense n\times n matrix during training.
Time complexity analysis
Besides the aforementioned advantages of EIGNN over IGNN, using eigendecomposition also has an efficiency advantage over using iterative solvers (like in IGNN).
If we were to use iterative methods in EIGNN, we could iterate Equation ( 3 ) until convergence.
Each computation of gradients has a theoretical computational complexity of O(K(m^{2}n+mn^{2})) where K is the number of iterations of an iterative method.
With eigendecomposition, the time complexity of computing the limit of the infinite sequence is O(m^{3}+m^{2}n) . During training, the eigendecomposition of g(F) requires the cost of O(m^{3}) . Thus, the complexity is O(m^{3}+m^{2}n) in total.
In practice, for large real-world graph settings, the number of nodes n is generally larger than the number of feature dimensions m , which makes m^{3}<mn^{2} .
Therefore, using eigendecomposition for training is more efficient than using iterative methods for infinite-depth GNNs.

In the above analysis, we omit the time complexity of the eigendecomposition of S since it is a one-time preprocessing operation for each graph. Specifically, after obtaining the result of eigendecomposition of S , it can always be reused for training and inference. The time complexity of a plain full eigendecomposition of S is O(n^{3}) . In some cases where the complexity of the preprocessing is of importance, we can consider using truncated eigendecomposition or graph coarsening [ 12 ] to mitigate the cost. Regarding this, we provide more discussions in Appendix D .
Experiments
In this section, we demonstrate that EIGNN can effectively learn representations which have the ability to capture long-range dependencies in graphs.
Therefore, EIGNN achieves state-of-the-art performance for node classification task on both synthetic and real-world datasets.
Specifically, we conduct experiments 1 1 1 The implementation can be found at https://github.com/liu-jc/EIGNN to compare EIGNN with representative baselines on seven graph datasets (Chains, Chameleon, Squirrel, Cornell, Texas, Wisconsin, and PPI), where Chains is a synthetic dataset used in Gu et al. [ 10 ] . Chameleon, Squirrel, Cornell, Texas, and Wisconsin are real-world datasets with a single graph each [ 21 ] while PPI is a real-world dataset with multiple graphs [ 11 ] .
Detailed descriptions of datasets and settings about experiments can be found in Appendix C .
Synthetic experiments & setup
In order to directly test existing GNN models’ abilities for capturing long-range dependencies of graphs, like in Gu et al. [ 10 ] , we use the Chains dataset, in which the model is supposed to classify nodes on the graph constructed by several chains.
The label information is only encoded in the starting end node of each chain. Suppose we have c classes, n_{c} chains for each class, and l nodes in each chain, then the graph has c\times n_{c}\times l nodes in total.
For training/validation/testing split, we consider 5%/10%/85% which is similar with the semi-supervised node classification setting [ 14 ] .
We select several representative baselines to compare (i.e., IGNN [ 10 ] , GCN [ 14 ] , SGC [ 30 ] , GAT [ 26 ] , JKNet [ 33 ] , APPNP [ 15 ] , GCNII [ 4 ] , and H2GCN [ 36 ] ).
The hyper-parameter setting and details about baselines’ implementation can be found in Appendix C.2 .
Results and analysis
We first consider binary classification ( c=2 ) with 20 chains of each class as in Gu et al. [ 10 ] .
We conduct experiments on graphs with chains of different lengths (10 to 200 with the interval 10). The averaged accuracies with respect to the length of chains are illustrated in Figure 1(a) .
In general, EIGNN and IGNN always outperform the other baselines.
GCN, SGC and GAT provide similarly poor performances, which indicates that these three classic GNN models cannot effectively capture long-range dependencies.
H2GCN has an even worse performance than GAT, SGC, GCN, which is caused by its ego- and neighbor-embedding separation focusing more on ego-embedding rather than distant information.
APPNP, JKNet, GCNII are either designed to consider different range of neighbors or to mitigate oversmoothing [ 16 , 33 ] .
Despite that they outperform GCN, SGC and GAT, they still perform worse than implicit models with infinitely deep layers (i.e., EIGNN and IGNN) since finite layers for aggregations cannot effectively capture underlying dependencies along extremely long chains (e.g., with length > 30).
Intuitively, increasing the number of layers should improve the model’s ability to capture long-range dependencies. However, stacking layers requires excessive computational cost and causes oversmoothing which degrades the performance.
Appendix C.5 shows the results of APPNP, JKNet and GCNII with a increased number of layers, which illustrates that simply stacking layers cannot effectively help to capture long-range dependencies.
Comparing EIGNN and IGNN, they both perfectly capture the long-range dependency when the length is less than 30, while the performance of IGNN generally decreases when the length increases.

In addition to binary classification used in Gu et al. [ 10 ] , we also conduct experiments on multi-class classification ( c=5 ) to evaluate the ability of capturing long-range dependencies on a generalized setting.
Figure 1(b) shows that EIGNN still easily maintains 100% test accuracy on multi-class classification. In contrast, all other baselines experience performance drops compared to binary classification.
On multi-class classification, EIGNN and IGNN consistently outperform the other baselines, verifying that models with infinitely deep layers have advantages for capturing long-range dependencies.

On both classification and multiclass classification settings, as shown in Figure 1 , IGNN cannot always achieve 100% test accuracy, especially when the length is longer than 30.
However, IGNN is supposed to capture dependencies on infinite-length chains as it is an implicit model with infinite depth.
We conjecture that the reason for the inferior performance is that the iterative solver in IGNN cannot easily capture all dependencies on extremely long chains.
Finite-depth IGNN v.s. Infinite-depth IGNN
To verify the hypothesis mentioned above, we conduct an additional experiment considering two kinds of IGNN, i.e., finite-depth IGNN and infinite-depth IGNN (the version proposed in Gu et al. [ 10 ] ).
On a dataset with chain length l , for finite-depth IGNN, we use l hidden layers which makes it able to capture dependencies with up to l -hops.
As shown in Figure 2 , IGNN-finite always outperforms IGNN when the length of chains becomes longer.
This confirms that the iterative solver used in infinite-depth IGNN is unable to capture all dependencies in longer chains.
Note that we use the official implementation 2 2 2 https://github.com/SwiftieH/IGNN of IGNN and follow the exact same setting (e.g., hyperparameters) as in Gu et al. [ 10 ] .
This can be caused by the approximated solutions generated by iterative solvers and the sensitivity to hyper-parameters of iterative solvers (e.g., convergence criteria).
To further support this hypothesis, we illustrate train/test loss trajectories of IGNN-finite and IGNN in Appendix C.3 .
Efficiency comparison
Besides the performance comparison, on Table 1 , we also report the training time per epoch of different models (i.e., EIGNN, IGNN, and IGNN-finite) on datasets with varying length l and the number of chains n_{c} .
On datasets with different chain length l , we set the number of hidden layers to l for IGNN-finite as mentioned in the last paragraph.
EIGNN requires less training time than IGNN and IGNN-finite.
IGNN-finite with many hidden layers (i.e., >100) spends much more training time than implicit models (i.e., IGNN and EIGNN), which shows the efficiency advantage of implicit models when capturing long-range dependencies is desired.
Note that EIGNN requires precomputation for eigendecomposition on adjacency matrix.
After that, the results can be saved to the disk for further use.
Thus, this operation is only conducted once for each dataset, which is tractable and generally requires less than 30 seconds in the experiment.
Real-world datasets & setup
Besides the experiments on synthetic graphs, following Pei et al. [ 21 ] , we also conduct experiments on several real-world heterophilic graphs to evaluate the model’s ability to capture long-range dependencies.
As nodes with the same class are far away from each other in heterophilic graphs, this requires models to aggregate information from distant nodes.
Cornell, Texas, and Wisconsin are web-page graphs of the corresponding universities while Chameleon and Squirrel are web-page graphs of Wikipedia of the corresponding topic.

In addition, to show that EIGNN is applicable to multi-label multi-graph inductive learning, we also evaluate EIGNN on a commonly used dataset Protein-Protein Interation (PPI). PPI contains multiple graphs where nodes are proteins and edges are interactions between proteins. We follow the train/validation/test split used in GraphSAGE [ 11 ] .