SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Theorem 5.3. Let  $\mathcal{G}$  be an undirected graph with SNA L. Consider the initial value problem in (2) with  $\mathbf{W} \in \mathbb{R}^{K \times K}$  and  $\alpha \in \mathbb{R}$ . Then, for almost all initial values  $\mathbf{x}_0 \in \mathbb{R}^{N \times K}$  the following holds.
$(\alpha >0)$  The solution to (2) is either HFD or LFD.
$(\alpha < 0)$  Let  $\lambda_{+}(\mathbf{L})$  and  $\lambda_{-}(\mathbf{L})$  be the smallest positive and negative non-zero eigenvalue of  $\mathbf{L}$ , respectively. The solution to (2) is either  $(1 - \lambda_{+}(\mathbf{L}))$ -FD or  $(1 - \lambda_{-}(\mathbf{L}))$ -FD.
Furthermore, the previous results also hold for solutions to the Schrödinger equation (3) if  $\mathbf{W} \in \mathbb{C}^{K \times K}$  has at least one eigenvalue with non-zero imaginary part.
Theorem 5.3 ( $\alpha > 0$ ) generalizes the result by Di Giovanni et al. (2023) for  $\alpha = 1$  to arbitrary positive values of  $\alpha$ . The convergence speed in Theorem 5.3 ( $\alpha > 0$ ) depends on the choice of  $\alpha \in \mathbb{R}$ . By selecting a variable  $\alpha$  (e.g., as a learnable parameter), we establish a flexible learning framework capable of adapting the convergence speed of the Dirichlet energy. A slower or more adjustable convergence speed facilitates broader frequency exploration as it converges more gradually to its maximal or minimal value. Consequently, the frequency component contributions (for finite time, i.e., in practice) are better balanced, which is advantageous for graphs with different homophily levels. Theorem 5.3 ( $\alpha < 0$ ) shows that solutions of the fractional neural ODEs in (2) and (3) are not limited to be LFD or HFD. To demonstrate this and the other results of Theorem 5.3, we solve (2) using an explicit Euler scheme for different choices of  $\alpha$  and  $\mathbf{W}$  on the Cora and Chameleon graphs. The resulting evolution of the Dirichlet energy with respect to time is illustrated in Figure 5. Finally, we refer to Theorem D.5 in Appendix D.1 for the full statement and proof of Theorem 5.3.
Remark 5.4. Theorem 5.3 is stated for the analytical solutions of (2) and (3), respectively. As noted in Section 5, calculating the analytical solution is infeasible in practice. However, we show in Appendices D.2 to D.3 that approximations of the solution of (2) and (3) via explicit Euler schemes satisfy the same Dirichlet energy convergence properties if the step size is sufficiently small.
Remark 5.5. Theorem 5.3 can be generalized to all directed graphs with normal SNA, i.e., satisfying the condition  $\mathbf{L}\mathbf{L}^{\top} = \mathbf{L}^{\top}\mathbf{L}$ . For the complete statement, see Appendix D.1.
(a) Cora (undirected).
(b) chameleon (directed).
Figure 5: Convergence of Dirichlet energy for the solution of equation (2) using an explicit Euler scheme with a step size of  $h = 10^{-1}$ . We consider different  $\alpha$ -FGL in (2) and choose  $\mathbf{W}$  as a random diagonal matrix. In the left plot,  $\mathbf{W}$  has only a negative spectrum, while in the right plot,  $\mathbf{W}$  has only a positive spectrum. The black horizontal line represents the theoretical limit based on Theorem 5.3.
5.2 Frequency Dominance for Directed Graphs
Section 5.1 analyzes the Dirichlet energy in graphs with normal SNA. However, the situation becomes significantly more complex when considering generic directed graphs. In our experiments (see Figure 5), we observe that the solution to (2) and (3) does not necessarily lead to oversmoothing. On the contrary, the solution can be controlled to exhibit either LFD or HFD for  $\alpha > 0$ , and mid-frequency-dominance for  $\alpha < 0$  as proven for undirected graphs in Theorem 5.3. We present an initial theoretical result for directed graphs, specifically in the case of  $\alpha = 1$ .
Theorem 5.6. Let  $\mathcal{G}$  be a directed graph with SNA  $\mathbf{L}$ . Consider the initial value problem in (2) with diagonal channel mixing matrix  $\mathbf{W} \in \mathbb{R}^{K \times K}$  and  $\alpha = 1$ . Suppose  $\lambda_1(\mathbf{L})$  is unique. For almost all initial values  $\mathbf{x}_0 \in \mathbb{R}^{N \times K}$ , the solution to (2) is either HFD or LFD.
The proof of Theorem 5.6 is given in Appendix E.1. Finally, we refer to Appendix E.2 for the analogous statement and proof when the solution of (2) is approximated via an explicit Euler scheme.
This section evaluates the fractional Laplacian ODEs in node classification by approximating (2) and (3) with an explicit Euler scheme. This leads to the following update rules
$$
\mathbf {x} _ {t + 1} = \mathbf {x} _ {t} - h \mathbf {L} ^ {\alpha} \mathbf {x} _ {t} \mathbf {W}, \mathbf {x} _ {t + 1} = \mathbf {x} _ {t} + i h \mathbf {L} ^ {\alpha} \mathbf {x} _ {t} \mathbf {W}, \tag {4}
$$
for the heat and Schrödinger equation, respectively. In both cases,  $\mathbf{W}$ ,  $\alpha$  and  $h$  are learnable parameters,  $t$  is the layer, and  $\mathbf{x}_0$  is the initial nodes' feature matrix. In accordance with the results in Section 5, we select  $\mathbf{W}$  as a diagonal matrix. The initial features  $\mathbf{x}_0$  in (4) are encoded through a MLP, and the output is decoded using a second MLP. We refer to the resulting model as FLODE (fractional Laplacian ODE). In Appendix A, we present details on the baseline models, the training setup, and the exact hyperparameters.
Table 1: Test accuracy (Film, Squirrel, Chameleon, Citeseer) and test AUROC (Minesweeper, Tolokers, Questions) on node classification, top three models. The thorough comparison is reported in Table 4, Appendix A: FLODE consistently outperforms the baseline models GCN and GRAFF, and it achieves results comparable to state-of-the-art.
(a) Undirected graphs.
Squirrel	Chameleon	Citeseer
1st	FLODE	FLODE	FLODE
64.23 ± 1.84	73.60 ± 1.55	78.07 ± 1.62
2nd	GREAD	GREAD	Geom-GCN
59.22 ± 1.44	71.38 ± 1.30	78.02 ± 1.15
3rd	GRAFFNL	GRAFFNL	GREAD
59.01 ± 1.31	71.38 ± 1.47	77.60 ± 1.81
(b) Heterophily-specific graphs.
Minesweeper	Tolokers	Questions
GAT-sep	FLODE	ESGNN
93.91 ± 0.35	84.17 ± 0.58	78.86 ± 0.92
GraphSAGE	GAT-sep	FLODE
93.51 ± 0.57	83.78 ± 0.43	78.39 ± 1.22
FLODE	GAT	GT-sep
92.43 ± 0.51	83.70 ± 0.47	78.05 ± 0.93
(c) Directed graphs.
Film	Squirrel	Chameleon
1st	FLODE	HLP	FSGNN
37.41 ± 1.06	74.17 ± 1.83	78.14 ± 1.25
2nd	GRAFF	FLODE	FLODE
37.11 ± 1.08	74.03 ± 1.58	77.98 ± 1.05
3rd	ACM	FSGNN	HLP
36.89 ± 1.18	73.48 ± 2.13	77.48 ± 1.50
Ablation Study. In Appendix A.3, we investigate the influence of each component (learnable exponent, ODE framework, directionality via the SNA) on the performance of FLODE. The adjustable fractional power in the FGL is a crucial component of FLODE, as it alone outperforms the model employing the ODE framework with a fixed  $\alpha = 1$ . Further, Appendix A.3 includes ablation studies that demonstrate FLODE's capability to efficiently scale to large depths, as depicted in Figure 8.
Real-World Graphs. We report results on 6 undirected datasets consisting of both homophilic graphs, i.e., Cora (McCallum et al., 2000), Citeseer (Sen et al., 2008) and Pubmed (Namata et al., 2012), and heterophilic graphs, i.e., Film (Tang et al., 2009), Squirrel and Chameleon (Rozemberczki et al., 2021). We evaluate our method on the directed and undirected versions of Squirrel, Film, and Chameleon. In all datasets, we use the standard 10 splits from (Pei et al., 2019). The choice of the baseline models and their results are taken from (Di Giovanni et al., 2023). Further, we test our method on heterophily-specific graph datasets, i.e., Roman-empire, Minesweeper, Tolokers, and Questions (Platonov et al., 2023). The splits, baseline models, and results are taken from (Platonov et al., 2023). The top three models are shown in Table 1, and the thorough comparison is reported in Table 4. Due to memory limitations, we compute only  $30\%$  of singular values for Pubmed, Roman-Empire, and Questions, which serve as the best low-rank approximation of the original SNA.
Synthetic Directed Graph. We consider the directed stochastic block model (DSBM) datasets (Zhang et al., 2021). The DSBM divides nodes into 5 clusters and assigns probabilities for interactions between vertices. It considers two sets of probabilities:  $\{\alpha_{i,j}\}$  for undirected edge creation and  $\{\beta_{i,j}\}$  for assigning edge directions,  $i,j\in \{1,\ldots 5\}$ . The objective is to classify vertices based on their clusters. In the first experiment,  $\alpha_{i,j} = \alpha^{*}$  varies, altering neighborhood information's importance. In the second experiment,  $\beta_{i,j} = \beta^{*}$  varies, changing directional information. The results are shown in Figure 6 and Table 6. The splits, baseline models, and results are taken from (Zhang et al., 2021).
Results. The experiments showcase the flexibility of FLODE, as it can accommodate various types of graphs, both directed and undirected, as well as a broad range of homophily levels. While other methods, such as MagNet (Zhang et al., 2021), perform similarly to our approach, they face limitations when applied to certain graph configurations. For instance, when applied to undirected graphs, MagNet reduces to ChebNet, making it unsuitable for heterophilic graphs. Similarly, GRAFF
Figure 6: Experiments on directed stochastic block model. Unlike other models, FLODE's performances do not deteriorate as much when changing the inter-cluster edge density  $\alpha^{*}$ .
Figure 7: Effect of truncated SVD on test accuracy (orange) for standard directed real-world graphs. The explained variance, defined as  $\sum_{i=1}^{k} \sigma_i^2 / \sum_{j=1}^{N} \sigma_j^2$ , measures the variability the first  $k$  singular values explain. For chameleon, the accuracy stabilized after 570 (25%) singular values, corresponding to an explained variance of 0.998. For squirrel, after 1600 (31%) singular values, which correspond to an explained variance 0.999, the improvement in test accuracy is only marginal.
(Di Giovanni et al., 2023) performs well on undirected graphs but falls short on directed graphs. We note that oftentimes FLODE learns a non-trivial exponent  $\alpha \neq 1$ , highlighting the advantages of FGL-based GNNs (see, e.g., Table 5). Furthermore, as shown in Table 9 and Appendix A.3, our empirical results align closely with the theoretical results in Section 5.
In this work, we introduce the concepts of Dirichlet energy and oversmoothing for directed graphs and demonstrate their relation with the SNA. Building upon this foundation, we define fractional graph Laplacians in the singular value domain, resulting in matrices capable of capturing long-range dependencies. To address oversmoothing in directed graphs, we propose fractional Laplacian-based graph ODEs, which are provably not limited to LFD behavior. We finally show the flexibility of our method to accommodate various graph structures and homophily levels in node-level tasks.
Limitations and Future Work. The computational cost of the SVD grows cubically in  $N$ , while the storage of the singular vectors grows quadratically in  $N$ . Both costs can be significantly reduced by computing only  $k \ll N$  singular values via truncated SVD (Figure 7), giving the best  $k$ -rank approximation of the SNA. Moreover, the SVD can be computed offline as a preprocessing step.
The frequency analysis of  $\alpha$ -FGL neural ODEs in directed graphs is an exciting future direction. It would also be worthwhile to investigate the impact of choosing  $\alpha \neq 1$  on the convergence speed of the Dirichlet energy. Controlling the speed could facilitate the convergence of the Dirichlet energy to an optimal value, which has been shown to exist in synthetic settings (Keriven, 2022; X. Wu et al., 2022). Another interesting future direction would be to analyze the dynamics when approximating the solution to the FGL neural ODEs using alternative numerical solvers, such as adjoint methods.
S. M. acknowledges partial support by the NSF-Simons Research Collaboration on the Mathematical and Scientific Foundations of Deep Learning (MoDL) (NSF DMS 2031985) and DFG SPP 1798, KU 1446/27-2.
G. K. acknowledges partial support by the DAAD programme Konrad Zuse Schools of Excellence in Artificial Intelligence, sponsored by the Federal Ministry of Education and Research. G. Kutyniok also acknowledges support from the Munich Center for Machine Learning (MCML) as well as the German Research Foundation under Grants DFG-SPP-2298, KU 1446/31-1 and KU 1446/32-1 and under Grant DFG-SFB/TR 109 and Project C09.
Zhao, L. and Akoglu, L. (2019). "PairNorm: Tackling Oversmoothing in GNNs". In: International Conference on Learning Representations.
Zhu, J., Rossi, R. A., Rao, A., Mai, T., Lipka, N., Ahmed, N. K., and Koutra, D. (2021). "Graph Neural Networks with Heterophily". In: Proceedings of the AAAI Conference on Artificial Intelligence 35.12, pp. 11168-11176.
Zhu, J., Yan, Y., Zhao, L., Heimann, M., Akoglu, L., and Koutra, D. (2020). "Beyond Homophily in Graph Neural Networks: Current Limitations and Effective Designs". In: Advances in Neural Information Processing Systems. Vol. 33. Curran Associates, Inc., pp. 7793-7804.
Zou, C., Han, A., Lin, L., and Gao, J. (2022). A Simple Yet Effective SVD-GCN for Directed Graphs. arXiv: 2205.09335 [cs].
AUROC Area under the ROC curve
DSBM Directed Stochastic Block Model
FD Frequency Dominant
FGL Fractional Graph Laplacian
GAT Graph Attention Network
GCN Graph Convolutional Network
GNN Graph Neural Network
HFD Highest Frequency Dominant
LCC Largest Connected Components
LFD Lowest Frequency Dominant
MLP Multi-Layer Perceptron
ODE Ordinary Differential Equation
SNA Symmetrically Normalized Adjacency
SVD Singular Value Decomposition
$i$  Imaginary unit
$\Re (z)$  Real part of  $z\in \mathbb{C}$
$\Im (z)$  Imaginary part of  $z\in \mathbb{C}$
$\operatorname {diag}(\mathbf{x})$  Diagonal matrix with  $\mathbf{x}$  on the diagonal.
1 Constant vector of all 1s.
$\mathbf{M}^{\mathrm{T}}$  Transpose of  $\mathbf{M}$
$\mathbf{M}^{*}$  Conjugate of M
$\mathbf{M}^{\mathrm{H}}$  Conjugate transpose of  $\mathbf{M}$
$\| \mathbf{M}\|$  Spectral norm of M
$\| \mathbf{M}\| _2$  Frobenius norm of M
$\lambda (\mathbf{M})$  Spectrum of M
$\sigma (\mathbf{M})$  Singular values of  $\mathbf{M}$
$\mathcal{E}(\mathbf{x})$  Dirichlet energy computed on  $\mathbf{x}$
$\mathcal{H}(\mathcal{G})$  Homophily coefficient of the graph  $\mathcal{G}$
$\mathbf{A}\otimes \mathbf{B}$  Kronecker product between A and B
vec  $(\mathbf{M})$  Vector obtained stacking columns of M.
In this section, we give the details on the numerical results in Section 6. We begin by describing the exact model.
Model architecture. Let  $\mathcal{G}$  be a directed graph and  $\mathbf{x}_0\in \mathbb{R}^{N\times K}$  the node features. Our architecture first embeds the input node features  $\mathbf{x}$  via a multi-layer perceptron (MLP). We then evolve the features  $\mathbf{x}_0$  according to a slightly modified version of (3), i.e.,  $\mathbf{x}'(t) = -i\mathbf{L}^{\alpha}\mathbf{x}(t)\mathbf{W}$  for some time  $t\in [0,T]$ . In our experiments, we approximate the solution with an Explicit Euler scheme with step size  $h > 0$ . This leads to the following update rule
$$
\mathbf {x} _ {t + 1} = \mathbf {x} _ {t} - i h \mathbf {L} ^ {\alpha} \mathbf {x} _ {t} \mathbf {W}.
$$
The channel mixing matrix is a diagonal learnable matrix  $\mathbf{W} \in \mathbb{C}^{K \times K}$ , and  $\alpha \in \mathbb{R}$ ,  $h \in \mathbb{C}$  are also learnable parameters. The features at the last time step  $\mathbf{x}_T$  are then fed into a second MLP, whose output is used as the final output. Both MLPs use LeakyReLU as non-linearity and dropout (Srivastava et al., 2014). On the contrary, the graph layers do not use any dropout nor non-linearity. A sketch of the algorithm is reported in fLode.
Algorithm 1: fLode
$\% \mathbf{A}$  ,  $\mathbf{x}_0$  are given.   
% Preprocessing   
1  $\mathbf{D}_{\mathrm{in}} = \mathrm{diag}(\mathbf{A}\mathbf{1})$    
2  $\mathbf{D}_{\mathrm{out}} = \mathrm{diag}\left(\mathbf{A}^{\mathrm{T}}\mathbf{1}\right)$    
3  $\mathbf{L} = \mathbf{D}_{\mathrm{in}}^{-1 / 2}\mathbf{A}\mathbf{D}_{\mathrm{out}}^{-1 / 2}$    
4 U,  $\Sigma ,\mathbf{V}^{\mathsf{H}} = \mathsf{svd}(\mathbf{L})$    
% The core of the algorithm is very simple   
5 def training_step(xo):   
6  ${\bf x}_0 = {\bf input\_MLP}({\bf x}_0)$    
7 for  $t\in \{1,\dots,T\}$  do   
8  $\begin{array}{r}\lfloor \mathbf{x}_t = \mathbf{x}_{t - 1} - ih\mathbf{U}\pmb{\Sigma}^\alpha \mathbf{V}^\mathsf{H}\mathbf{x}_{t - 1}\mathbf{W}\\ \mathbf{x}_T = \mathbf{output\_MLP}(\mathbf{x}_T)\\ \mathbf{return}\mathbf{x}_T \end{array}$    
9   
10
Complexity. The computation of the SVD is  $\mathcal{O}(N^3)$ . However, one can compute only the first  $p \ll N$  singular values: this cuts down the cost to  $\mathcal{O}(N^2 p)$ . The memory required to store the singular vectors is  $\mathcal{O}(N^2)$ , since they are not sparse in general. Each training step has a cost of  $\mathcal{O}(N^2 K)$ .
Experimental details. Our model is implemented in PyTorch (Paszke et al., 2019), using PyTorch geometric (Fey et al., 2019). The computation of the SVD for the fractional Laplacian is implemented using the library linalg provided by PyTorch. In the case of truncated SVD, we use the function randomized_svd provided by the library extmath from sklearnn. The code and instructions to reproduce the experiments are available on GitHub. Hyperparameters were tuned using grid search. All experiments were run on an internal cluster with NVIDIA GeForce RTX 2080 Ti and NVIDIA TITAN RTX GPUs with 16 and 24 GB of memory, respectively.
Training details. All models were trained for 1000 epochs using Adam (Kingma et al., 2015) as optimizer with a fixed learning rate. We perform early stopping if the validation metric does not increase for 200 epochs.
A.1 Real-World Graphs
Undirected graphs We conducted 10 repetitions using data splits obtained from (Pei et al., 2019). For each split,  $48\%$  of the nodes are used for training,  $32\%$  for validation and  $20\%$  for testing. In all datasets, we considered the largest connected component (LCC). Chameleon, Squirrel, and Film are
directed graphs; hence, we converted them to undirected. Cora, Citeseer, and Pubmed are already undirected graphs: to these, we added self-loops. We normalized the input node features for all graphs.
As baseline models, we considered the same models as in (Di Giovanni et al., 2023). The results were provided by Pei et al. (2019) and include standard GNNs, such as GAT (Velickovic et al., 2018), GCN (Kipf et al., 2017), and GraphSAGE (Hamilton et al., 2017). We also included models designed to address oversmoothing and heterophilic graphs, such as PairNorm (L. Zhao et al., 2019), GGCN (Yan et al., 2022), Geom-GCN (Pei et al., 2019),  $\mathrm{H}_2\mathrm{GCN}$  (Zhu, Yan, et al., 2020), GPRGNN (Chien et al., 2021), and Sheaf (Bodnar et al., 2022). Furthermore, we included the graph neural ODE-based approaches, CGNN (Xhonneux et al., 2020) and GRAND (Chamberlain et al., 2021), as in (Di Giovanni et al., 2023), and the model GRAFF from (Di Giovanni et al., 2023) itself. Finally, we included GREAD (Choi et al., 2023), GraphCON (Rusch et al., 2022), ACMP (Y. Wang et al., 2022) and GCN and GAT equipped with DropEdge (Rong et al., 2020).
Heterophily-specific Models For heterophily-specific datasets, we use the same models and results as in (Platonov et al., 2023). As baseline models we considered the topology-agnostic ResNet (He et al., 2016) and two graph-aware modifications: ResNet+SGC(F. Wu et al., 2019) where the initial node features are multiplied by powers of the SNA, and ResNet+adj, where rows of the adjacency matrix are used as additional node features; GCN (Kipf et al., 2017), GraphSAGE (Hamilton et al., 2017); GAT (Velickovic et al., 2018) and GT (Shi et al., 2021) as well as their modification GAT-sep and GT-sep which separate ego- and neighbor embeddings;  $\mathrm{H}_2\mathrm{GCN}$  (Zhu, Yan, et al., 2020), CPGNN (Zhu, Rossi, et al., 2021), GPRGNN (Chien et al., 2021), FSGNN (Maurya et al., 2021), GloGNN (X. Li et al., 2022), FAGCN (Bo et al., 2021), GBK-GNN (Du et al., 2022), and JacobiConv (X. Wang et al., 2022).
The exact hyperparameters for FLODE are provided in Table 5.
A.2 Synthetic Directed Graphs
The dataset and code are taken from (Zhang et al., 2021). As baseline models, we considered the ones in (Zhang et al., 2021) for which we report the corresponding results. The baseline models include standard GNNs, such as ChebNet (Defferrard et al., 2016), GCN (Kipf et al., 2017), GraphSAGE (Hamilton et al., 2017), APPNP (Gasteiger et al., 2018), GIN (Xu et al., 2018), GAT (Velickovic et al., 2018), but also models specifically designed for directed graphs, such as DGCN (Tong, Liang, Sun, Rosenblum, et al., 2020), DiGraph and DiGraphIB (Tong, Liang, Sun, X. Li, et al., 2020), MagNet (Zhang et al., 2021)).
The DSBM dataset. The directed stochastic block model (DSBM) is described in detail in (Zhang et al., 2021, Section 5.1.1). To be self-contained, we include a short explanation.
The DSBM model is defined as follows. There are  $N$  vertices, which are divided into  $n_c$  clusters  $(C_1,C_2,\ldots C_{n_c})$ , each having an equal number of vertices. An interaction is defined between any two distinct vertices,  $u$  and  $v$ , based on two sets of probabilities:  $\{\alpha_{i,j}\}_{i,j=1}^{n_c}$  and  $\{\beta_{i,j}\}_{i,j=1}^{n_c}$ .
The set of probabilities  $\{\alpha_{i,j}\}$  is used to create an undirected edge between any two vertices  $u$  and  $v$ , where  $u$  belongs to cluster  $C_i$  and  $v$  belongs to cluster  $C_j$ . The key property of this probability set is that  $\alpha_{i,j} = \alpha_{j,i}$ , which means the chance of forming an edge between two clusters is the same in either direction.
The set of probabilities  $\{\beta_{i,j}\}$  is used to assign a direction to the undirected edges. For all  $i, j \in \{1, \ldots, n_c\}$ , we assume that  $\beta_{i,j} + \beta_{j,i} = 1$  holds. Then, to the undirected edge  $(u,v)$  is assigned the direction from  $u$  to  $v$  with probability  $\beta_{i,j}$  if  $u$  belongs to cluster  $C_i$  and  $v$  belongs to cluster  $C_j$ , and the direction from  $v$  to  $u$  with probability  $\beta_{j,i}$ .
The primary objective here is to classify the vertices based on their respective clusters.
There are several scenarios designed to test different aspects of the baseline models and our model. In the experiments, the total number of nodes is fixed at  $N = 2500$  and the number of clusters is fixed at  $n_c = 5$ . In all experiments, the training set contains 20 nodes per cluster, 500 nodes for validation, and the rest for testing. The results are averaged over 5 different seeds and splits.