RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
Results and analysis
As shown in Table 2 , in general, EIGNN outperforms other baselines on most datasets.
On Texas, EIGNN provides a similar mean accuracy compared with H2GCN which generally provides the second best performance on other real-world datasets.
H2GCN focuses more on different designs about aggregations [ 36 ] .
In contrast, EIGNN performs well through a different approach (i.e., capturing long-range dependencies).
In terms of capturing long-range dependencies, H2GCN lacks this ability as demonstrated in our synthetic experiments (see Figure 1 ).
Geom-GCN outperforms several baselines (i.e., SGC, GCN, GAT, APPNP and JKNet), which is attributed to its ability to capture long-range dependencies.
However, GCNII provides better mean accuracies than Geom-GCN.
GCNII with many aggregation layers stacked is designed for mitigating the oversmoothing issue.
Thus, this result suggests that alleviating the oversmoothing issue can also improve the model’s ability to capture long-range dependencies.
Note that Geom-GCN and IGNN are both significantly outperformed by EIGNN. Therefore, it indicates that they are still not effective enough in capturing long-range dependencies on heterophilic graphs.

Table 3 reports the micro-averaged F1 scores of EIGNN and other popular baseline methods on PPI dataset. As we follow the exactly same setting used in Gu et al. [ 10 ] , the results of baselines are obtained from Gu et al. [ 10 ] . From Table 3 , EIGNN achieves better performance compared with other baselines, which can be attributed to the ability of EIGNN to capture long-range dependencies between proteins on PPI dataset. Additionally, we also conduct the efficiency comparison between EIGNN and IGNN. On average, for training an epoch, EIGNN requires 2.23 seconds, whereas IGNN spends 35.03 seconds. As suggested in IGNN [ 10 ] , the training generally requires more than 1000 epochs. Besides the training time, EIGNN needs a one-time preprocessing for eigendecomposition on S which costs 40.59 seconds. Therefore, considering both preprocessing and training, EIGNN still requires less time compared with IGNN.
Noise sensitivity
Recent research [ 31 , 37 ] shows that GNNs are vulnerable to perturbations, and El Ghaoui et al. [ 7 ] analyze the potential robustness properties of an implicit model. Thus, in this subsection, we empirically examine the potential benefit of EIGNN on robustness against noise.
For simplicity, we add perturbations only to node features.
Experiments on synthetic datasets and real-world datasets are both conducted.
Synthetic experiments
On synthetic chain datasets, we add random perturbations to node features as in Wu et al. [ 31 ] .
Specifically, we add uniform noise \epsilon\sim\mathcal{U}(-\alpha,\alpha) to each node’s features for constructing noisy datasets.
Figure 3 shows that across different noise levels, EIGNN consistently outperforms IGNN.
This indicates that EIGNN is less vulnerable to noisy features.
Real-world experiments
On real-world datasets, we use two classic adversarial attacks (i.e., FGSM [ 8 ] and PGD [ 18 ] ) to add perturbations on node features. See the detailed setting in Appendix C.2 .
We compare three models (i.e., EIGNN, H2GCN, IGNN) on Cornell, Texas, and Wisconsin under different level perturbations.
As we can observe from Table 4 , EIGNN consistently achieves the best performance under different perturbation rates on all three datasets.
Specifically, EIGNN outperforms H2GCN by a larger margin when the perturbation rate is higher.
Under FGSM with a perturbation rate larger than 0.001, H2GCN is even outperformed by IGNN which has worse performance under 0.0001 perturbation rate.

In summary, the above results show that EIGNN is generally more robust against perturbations compared with H2GCN. The intuitive reason is that EIGNN is not/less sensitive to local neighborhoods, while finite-depth GNNs (e.g., H2GCN) could be very sensitive to local neighborhoods. Finite-depth GNNs usually only have a few layers and the node representations would be changed a lot when aggregating from perturbed node features. In contrast, EIGNN, as an infinite-depth GNN model, can consider more about global information and become less sensitive to perturbations from local neighborhoods.
Conclusion
In this paper, we address an important limitation of existing GNNs: their lack of ability to capture long-range dependencies. We propose EIGNN, a GNN model with implicit infinite layers.
Instead of using iterative solvers like in previous work, we derive a closed-form solution of EIGNN with rigorous proofs, which makes training tractable.
We further achieve more efficient computation for training our model by using eigendecomposition.
On synthetic experiments, we demonstrate that EIGNN has a better ability to capture long-range dependencies. In addition, it also provides state-of-the-art performance on real-world datasets.
Moreover, our model is less susceptible to perturbations on node features compared with other models.
One of the potential limitations of our work is that EIGNN can be slow when the number of feature dimensions is very large (e.g., 1 million), which is rare in practical GNN settings.
Future work could propose a new model which allows for efficient training even with large feature dimension.
Acknowledgments and Disclosure of Funding
We would like to thank Keke Huang and Tianyuan Jin for helpful discussions. We also gratefully acknowledge the insightful feedback and suggestions from the anonymous reviewers.
This paper is supported by the Ministry of Education, Singapore (Grant No: MOE2018-T2-2-091).
The views and conclusions contained in this paper are those of the authors and should not be interpreted as representing any funding agencies.
B.3 Derivations of Equation ( 8 ) and ( 9 )
By the chain rule,

\frac{\partial L(B,F)}{\partial\operatorname{vec}[F]} =\frac{\partial\ell_{Y}(\mathbf{f}_{X,F,B})}{\partial\operatorname{vec}[\mathbf{f}_{X,F,B}]}\frac{\partial\operatorname{vec}[\mathbf{f}_{X,F,B}]}{\partial\operatorname{vec}[F]}.

Since \operatorname{vec}[\mathbf{f}_{X,F,B}]=[I_{n}\otimes B]\operatorname{vec}[Z_{X,F}] ,

\frac{\partial L(B,F)}{\partial\operatorname{vec}[F]} =\frac{\partial\ell_{Y}(\mathbf{f}_{X,F,B})}{\partial\operatorname{vec}[\mathbf{f}_{X,F,B}]}\frac{\partial\operatorname{vec}[\mathbf{f}_{X,F,B}]}{\partial\operatorname{vec}[F]}=\frac{\partial\ell_{Y}(\mathbf{f}_{X,F,B})}{\partial\operatorname{vec}[\mathbf{f}_{X,F,B}]}[I_{n}\otimes B]\frac{\partial\operatorname{vec}[Z_{X,F}]}{\partial\operatorname{vec}[F]}.

Using the formula of \frac{\partial\operatorname{vec}[Z_{X,F}]}{\partial\operatorname{vec}[F]} from Appendix B.2 ,

\frac{\partial L(B,F)}{\partial\operatorname{vec}[F]}=\gamma\frac{\partial\ell_{Y}(\mathbf{f}_{X,F,B})}{\partial\operatorname{vec}[\mathbf{f}_{X,F,B}]}[I_{n}\otimes B]U^{-1}\left[SZ_{X,F}^{\top}\otimes I_{m}\right]\frac{\partial\operatorname{vec}[g(F)]}{\partial\operatorname{vec}[F]}.

Similarly, since \operatorname{vec}[\mathbf{f}_{X,F,B}]=[Z_{X,F}^{\top}\otimes I_{m_{y}}]\operatorname{vec}[B] ,

\frac{\partial L(B,F)}{\partial\operatorname{vec}[B]} =\frac{\partial\ell_{Y}(\mathbf{f}_{X,F,B})}{\partial\operatorname{vec}[\mathbf{f}_{X,F,B}]}\frac{\partial\operatorname{vec}[\mathbf{f}_{X,F,B}]}{\partial\operatorname{vec}[B]}=\frac{\partial\ell_{Y}(\mathbf{f}_{X,F,B})}{\partial\operatorname{vec}[\mathbf{f}_{X,F,B}]}[Z_{X,F}^{\top}\otimes I_{m_{y}}]
Synthetic chains datasets
To evaluate the ability of models to capture information from distant nodes, we construct synthetic chains datasets as in Gu et al. [ 10 ] .
In our experiments, we consider both binary classification and multiclass classification. Assuming the number of classes is c , we then have c types of chains and the information of the label class is only encoded as a one-hot vector in the first c dimensions of the node feature vector of the starting end node of the chain.
With c classes, n_{c} chains for each class, and l nodes in each chains, the chain dataset has c\times n_{c}\times l nodes in total.
We choose 20 chains for each class and c=5 for multiclass classification setting. The train set consists 5% nodes while the validation and test set contain 10% and 85% nodes respectively.
Real-world datasets
In our real-world experiments, following Pei et al. [ 21 ] , we use the following real-world datasets:

Cornell, Texas and Wisconsin are web-page graphs of the corresponding universities, where nodes are web pages and edges represent hyper-links between web pages. There are 5 label classes: faculty, student, course, project and staff. These datasets are originally collected by the CMU WebKB project 3 3 3 http://www.cs.cmu.edu/~webkb/ . In our experiments, we use the preprocessed version in Pei et al. [ 21 ] .

Chameleon and Squirrel are graphs of web pages in Wikipedia of the corresponding topic, originally collected by [ 22 ] . We use the labels generated by Pei et al. [ 21 ] , where the nodes is classified into 5 categories using the amount of their average monthly traffic.

In addition to the above single-graph datasets, we also use Protein-Protein Interaction (PPI) dataset, which contains multiple graphs, to show that EIGNN is applicable to multi-label multi-graph inductive learning setting. PPI dataset has 24 graphs in total and each graph corresponds to a different human issue. In a graph, nodes represents proteins and edges indicates interaction between proteins. Each node can have at most 121 labels, which is originally collected from the Molecular Signatures Database [ 25 ] by Hamilton et al. [ 11 ] . We also follow the data splits used in [ 11 ] , i.e., 20 graphs for training, two graphs for validation, and the rest two graphs for testing.
Node classification
For node classification task on synthetic and real-world datasets, we mainly choose 9 representative baselines to compare with EIGNN: Graph Convolution Network (GCN) [ 14 ] , Simple Graph Convolution (SGC) [ 30 ] , Graph Attention Network [ 26 ] , Jumping Knowledge Network (JKNet) [ 33 ] , APPNP [ 15 ] , GCNII [ 4 ] , and H2GCN [ 36 ] . The presented results are averaged by 20 different runs.
Noise sensitivity
In synthetic experiments, we add uniform noise \epsilon\sim\mathcal{U}(-\alpha,\alpha) to node features for constructing synthetic datasets with noises.
Then, EIGNN and IGNN are trained and evaluated on these datasets with \alpha=0.01,0.1 . The results are averaged by 10 different runs.

In real-world experiments, we compare the robustness of H2GCN, EIGNN, and IGNN against adversarial perturbations on node features.
We evaluate the models on evasive setting, i.e., the perturbations are added after the model is trained.
We use the trained models of H2GCN, EIGNN, and IGNN with their best performance on Cornell, Texas, and Wisconsin.
For each node which is correctly classified, we use two classic methods, i.e., Fast Gradient Sign Method (FGSM) [ 8 ] and Projected Gradient Descent (PGD) [ 18 ] , to add perturbations to node features.
For FGSM, we add perturbation \epsilon\in(0.0001,0.001,0.01) .
For PGD, we run 15 iterations with different step sizes. Different combinations of perturbation \epsilon and step size \alpha are used: (0.01,0.001),(0.001,0.0001),(0.0001,10^{-5}) .
Hyperparameter setting
To avoid bias, we tune the hyperparameters for each baseline on each real-world dataset. We report the best performance, using the set of hyperparameters which performs the best on the validation set, for each method.
For baselines, besides the hyperparameters suggested in their papers, we also conduct a hyperparameter search on learning rate {0.001, 0.05, 0.01, 0.1} and weight decay {5e-4, 5e-6}.
For GAT, 8 attention heads are used.
For APPNP, 10 propagation layers are used as suggested in [ 15 ] .
On university datasets (i.e., Cornell, Texas, and Wisconsin), for our model EIGNN, we set the learning rate as 0.5, the weight decay as 5e-6, and \gamma=0.8 .
On Wikipedia datasets (i.e., Chameleon and Squirrel), we employ batch normalization between the infinite-depth layer and the final linear transformation. After that, the dropout is used with parameter 0.5. The hyperparameter search space is set as follows: learning rate {1e-4, 1e-3, 1e-2}, weight decay {5e-4, 5e-6}.
Hardware specifications
We run experiments on a machine with Intel(R) Xeon(R) Gold 6240 CPU @ 2.60GHz and a single GeForce RTX 2080 Ti GPU with 11 GB GPU memory.
C.3 Finite-depth IGNN v.s. Infinite-depth IGNN
In Figure 4 , we show the train and test loss trajectories of IGNN and IGNN-finite on datasets with different chain lengths.
It is clear that IGNN-finite can achieve both lower train and test loss compared with IGNN.
For IGNN, the train and test losses both cannot effectively decrease and their trajectories are quite similar, which indicates the issue is that IGNN cannot be effectively learned instead of overfitting problem.
The reason why IGNN cannot be effectively learned is that the iterative solver usually generated approximated solutions.
To be specific, the error yielded by the iterative solver in the forward pass would be further amplified in the backward pass.
Thus, approximation errors make IGNN cannot be optimized well, which leads to the inferior performance.
C.4 Statistical significance
In our experiments, we conduct multiple runs to get the averaged results. However, for clarity purpose, we omit the error bars on figures represented in Section 5 . To test whether the differences are statistically significant, we calculate T-tests for two result series. For Figure 1(b) , comparing IGNN and EIGNN, the p-values are all smaller than 0.001 for all chain lengths. Comparing IGNN and APPNP (the third-best one), the p-values are all smaller than 0.05 when the chain length is larger than 10. For Figure 1(a) , comparing IGNN and EIGNN, the p-values are all smaller than 0.05 when the chain length is larger than 20. Comparing IGNN and APPNP (the third-best one), the p-values are all smaller than 1e-4 for all chain lengths. For Figure 2 , comparing IGNN and IGNN-fininte, the p-values are smaller than 0.05 when the chain length is larger than 40. Based on the above results, we can conclude the differences are statistically significant.
C.5 Oversmoothing
A straightforward way to help GNNs capture long-range dependencies is stacking layers. With T hidden layers, GNN models are supposed to capture the information within T -hop neighborhoods.
However, stacking many layers for traditional GNN models like GCN, SGC, GAT cause oversmoothing.
Several works, i.e., APPNP [ 15 ] , JKNet [ 33 ] , and GCNII [ 4 ] , are proposed to mitigate oversmoothing problem.
Nevertheless, we demonstrate that they still cannot perfectly capture long-range dependencies.
Figure 5 shows the results of APPNP, JKNet and GCNII on datasets with different chain lengths. On the dataset with T -length chains, we set the number of propagation layers of these models as T . Ideally, with T layers, they should capture the label information from distant nodes and achieve 100% test accuracy.
As shown in Figure 5 , their performance drops when the chain length increases.
The reason might be these model still cannot completely avoid oversmoothing, which makes them ineffectively capture long-range dependencies.
(a) Results of binary classification; (b) Results of multiclass classification; Averaged accuracies with respect to the length of chains.
Figure 2 : Mean accuracies comparison between the finite-depth and infinite-depth IGNN.; Mean accuracies comparison between the finite-depth and infinite-depth IGNN.
Figure 3 : Accuracy on datasets with different feature noise.; Accuracy on datasets with different feature noise.
(a) On the dataset with 50-length chains.; (b) On the dataset with 100-length chains.; Train and Test losses versus the number of epochs for IGNN and IGNN-finite on datasets with different chain length.
Figure 5 : oversmoothness; oversmoothness
Training time per epoch with respect to different l and n_{c}
Results on real-world datasets: mean accuracy (%) \pm stdev over different data splits. “*" denotes that the results are obtained from [ 21 ] .
Multi-label node classification on PPI: Micro-F1 (%).
Accuracy (%) under attacks with different perturbations.