Title: A Fractional Graph Laplacian Approach to Oversmoothing
Venue: NeurIPS 2023
Accepted: yes
Keywords: Graph Neural Networks

Graph Neural ODE

Fractional Laplacian

Oversmoothing
Abstract: Graph neural networks (GNNs) have shown state-of-the-art performances in various applications. However, GNNs often struggle to capture long-range dependencies in graphs due to oversmoothing. In this paper, we generalize the concept of oversmoothing from undirected to directed graphs. To this aim, we extend the notion of Dirichlet energy by considering a directed symmetrically normalized Laplacian. As vanilla graph convolutional networks are prone to oversmooth, we adopt a neural graph ODE framework. Specifically, we propose fractional graph Laplacian neural ODEs, which describe non-local dynamics. We prove that our approach allows propagating information  between distant nodes while maintaining a low probability of long-distance jumps. Moreover, we show that our method is more flexible with respect to the convergence of the graph’s Dirichlet energy, thereby mitigating oversmoothing. We conduct extensive experiments on synthetic and real-world graphs, both  directed and undirected, demonstrating our method’s versatility across diverse graph homophily levels.  Our
code is available at https://github.com/RPaolino/fLode

A Fractional Graph Laplacian Approach to Oversmoothing

Abstract

Graph neural networks (GNNs) have shown state-of-the-art performances in various applications. However, GNNs often struggle to capture long-range dependencies in graphs due to oversmoothing. In this paper, we generalize the concept of oversmoothing from undirected to directed graphs. To this aim, we extend the notion of Dirichlet energy by considering a directed symmetrically normalized Laplacian. As vanilla graph convolutional networks are prone to oversmooth, we adopt a neural graph ODE framework. Specifically, we propose fractional graph Laplacian neural ODEs, which describe non-local dynamics. We prove that our approach allows propagating information between distant nodes while maintaining a low probability of long-distance jumps. Moreover, we show that our method is more flexible with respect to the convergence of the graph's Dirichlet energy, thereby mitigating oversmoothing. We conduct extensive experiments on synthetic and real-world graphs, both directed and undirected, demonstrating our method's versatility across diverse graph homophily levels. Our code is available on GitHub.

1 Introduction

Graph neural networks (GNNs) (Gori et al., 2005; Scarselli et al., 2009; Bronstein et al., 2017) have emerged as a powerful class of machine learning models capable of effectively learning representations of structured data. GNNs have demonstrated state-of-the-art performance in a wide range of applications, including social network analysis (Monti et al., 2019), molecular property prediction (Gilmer et al., 2017), and recommendation systems (J. Wang et al., 2018; Fan et al., 2019). The majority of existing work on GNNs has focused on undirected graphs (Defferrard et al., 2016; Kipf et al., 2017; Hamilton et al., 2017), where edges have no inherent direction. However, many real-world systems, such as citation networks, transportation systems, and biological pathways, are inherently directed, necessitating the development of methods explicitly tailored to directed graphs.

Despite their success, most existing GNN models struggle to capture long-range dependencies, which can be critical for specific tasks, such as node classification and link prediction, and for specific graphs, such as heterophilic graphs. This shortcoming also arises from the problem of oversmoothing , where increasing the depth of GNNs results in the node features converging to similar values that only convey information about the node's degree (Oono et al., 2019; Cai et al., 2020). Consequently,

scaling the depth of GNNs is not sufficient to broaden receptive fields, and other approaches are necessary to address this limitation. While these issues have been extensively studied in undirected graphs (Q. Li et al., 2018; G. Li et al., 2019; Luan, M. Zhao, et al., 2019; D. Chen et al., 2020; Rusch et al., 2022), their implications for directed graphs remain largely unexplored. Investigating these challenges and developing effective solutions is crucial for applying GNNs to real-world scenarios.

Over-smoothing has been shown to be intimately related to the graph's Dirichlet energy , defined as

where A = ( a i,j ) N i,j =1 represents the adjacency matrix of the underlying graph, x ∈ R N × K denotes the node features, and d i ∈ R the degree of node i . Intuitively, the Dirichlet energy measures the smoothness of nodes' features. Therefore, a GNN that minimizes the Dirichlet energy is expected to perform well on homophilic graphs, where similar nodes are likely to be connected. Conversely, a GNN that ensures high Dirichlet energy should lead to better performances on heterophilic graphs, for which the nodes' features are less smooth.

This paper aims to bridge the gap in understanding oversmoothing for directed graphs. To this aim, we generalize the concept of Dirichlet energy, providing a rigorous foundation for analyzing oversmoothing. Specifically, we consider the directed symmetrically normalized Laplacian, which accommodates directed graph structures and recovers the usual definition in the undirected case. Even though the directed symmetrically normalized Laplacian has been already used (Zou et al., 2022), its theoretical properties remain widely unexplored.

However, a vanilla graph convolutional network (GCN) (Kipf et al., 2017) implementing this directed Laplacian alone is not able to prevent oversmoothing. For this reason, we adopt a graph neural ODE framework, which has been shown to effectively alleviate oversmoothing in undirected graphs (Bodnar et al., 2022; Rusch et al., 2022; Di Giovanni et al., 2023).

1.1 Graph Neural ODEs

The concept of neural ODE was introduced by Haber et al. (2018) and R. T. Q. Chen et al. (2018), who first interpreted the layers in neural networks as the time variable in ODEs. Building on this foundation, Poli et al. (2021), Chamberlain et al. (2021), and Eliasof et al. (2021) extended the connection to the realm of GNNs, resulting in the development of graph neural ODEs. In this context, each node i of the underlying graph is described by a state variable x i ( t ) ∈ R K , representing the node i at time t . We can define the dynamics of x ( t ) via the node-wise ODE

subject to the initial condition x (0) = x 0 ∈ R N × K , where the function f w : R N × K → R N × K is parametrized by the learnable parameters w .

The graph neural ODE can be seen as a continuous learnable architecture on the underlying graph, which computes the final node representation x ( T ) from the input nodes' features x 0 . Typical choices for f w include attention-based functions (Chamberlain et al., 2021), which generalize graph attention networks (GATs) (Velickovic et al., 2018), or convolutional-like functions (Di Giovanni et al., 2023) that generalize GCNs (Kipf et al., 2017).

How can we choose the learnable function f w to accommodate both directed and undirected graphs, as well as different levels of homophily? We address this question in the following subsection.

1.2 Fractional Laplacians

The continuous fractional Laplacian, denoted by ( -∆) α for α > 0 , is used to model non-local interactions. For instance, the fractional heat equation ∂ t u +( -∆) α u = 0 provides a flexible and accurate framework for modeling anomalous diffusion processes. Similarly, the fractional diffusionreaction, quasi-geostrophic, Cahn-Hilliard, porous medium, Schrödinger, and ultrasound equations are more sophisticated models to represent complex anomalous systems (Pozrikidis, 2018).

Similarly to the continuous case, the fractional graph Laplacian (FGL) (Benzi et al., 2020) models non-local network dynamics. In general, the FGL does not inherit the sparsity of the underlying

graph, allowing a random walker to leap rather than walk solely between adjacent nodes. Hence, the FGL is able to build long-range connections, making it well-suited for heterophilic graphs.

1.3 Main Contributions

We present a novel approach to the fractional graph Laplacian by defining it in the singular value domain, instead of the frequency domain (Benzi et al., 2020). This formulation bypasses the need for computing the Jordan decomposition of the graph Laplacian, which lacks reliable numerical methods. We show that our version of the FGL can still capture long-range dependencies, and we prove that its entries remain reasonably bounded.

We then propose two FGL-based neural ODEs: the fractional heat equation and the fractional Schrödinger equation. Importantly, we demonstrate that solutions to these FGL-based neural ODEs offer increased flexibility in terms of the convergence of the Dirichlet energy. Notably, the exponent of the fractional graph Laplacian becomes a learnable parameter, allowing our network to adaptively determine the optimal exponent for the given task and graph. We show that this can effectively alleviate oversmoothing in undirected and directed graphs.

To validate the effectiveness of our approach, we conduct extensive experiments on synthetic and real-world graphs, with a specific focus on supervised node classification. Our experimental results indicate the advantages offered by fractional graph Laplacians, particularly in non-homophilic and directed graphs.

2 Preliminaries

We denote a graph as G = ( V , E ) , where V is the set of nodes, E is the set of edges, and N = |V| is the number of nodes. The adjacency matrix A := { a i,j } encodes the edge information, with a i,j = 1 if there is an edge directed from node j to i , and 0 otherwise. The in- and out-degree matrices are then defined as D in = diag( A1 ) , D out = diag( A T 1 ) , respectively. The node feature matrix x ∈ R N × K contains for every node its feature in R K .

Given any matrix M ∈ C n × n , we denote its spectrum by λ ( M ) := { λ i ( M ) } n i =1 in ascending order w.r.t. to the real part, i.e., glyph[Rfractur] λ 1 ( M ) ≤ glyph[Rfractur] λ 2 ( M ) ≤ . . . ≤ glyph[Rfractur] λ n ( M ) . Furthermore, we denote by ‖ M ‖ 2 and ‖ M ‖ the Frobenius and spectral norm of M , respectively. Lastly, we denote by I n the identity matrix, where we omit the dimension n when it is clear from the context.

Homophily and Heterophily Given a graph G = ( V , E ) with labels y = { y i } i ∈V , the homophily of the graph indicates whether connected nodes are likely to have the same labels; formally,

where the numerator represents the number of neighbors of node i ∈ V that have the same label y i (Pei et al., 2019). We say that G is homophilic if H ( G ) ≈ 1 and heterophilic if H ( G ) ≈ 0 .

3 Dirichlet Energy and Laplacian for (Directed) Graphs

In this section, we introduce the concept of Dirichlet energy and demonstrate its relationship to a directed Laplacian, thereby generalizing well-known results for undirected graphs.

Definition 3.1. The Dirichlet energy is defined on the node features x ∈ R N × K of a graph G as

The Dirichlet energy measures how much the features change over the nodes of G , by quantifying the disparity between the normalized outflow of information from node j and the normalized inflow of information to node i .

Definition 3.2. We define the symmetrically normalized adjacency (SNA) as L := D -1 / 2 in AD -1 / 2 out .

glyph[Rfractur]

λ

glyph[Ifractur]

λ

Chameleon

glyph[Rfractur]

λ

glyph[Ifractur]

λ

glyph[negationslash]

Note that L is symmetric if and only if G is undirected; the term 'symmetrically" refers to the both-sided normalization rather than the specific property of the matrix itself.

It is well-known that the SNA's spectrum of a connected undirected graph lies within [ -1 , 1] (Chung, 1997). We extend this result to directed graphs, which generally exhibit complex-valued spectra.

Proposition 3.3. Let G be a directed graph with SNA L . For every λ ∈ λ ( L ) , it holds | λ | ≤ 1 .

Proposition 3.3 provides an upper bound for the largest eigenvalue of any directed graph, irrespective of its size. However, many other spectral properties do not carry over easily from the undirected to the directed case. For example, the SNA may not possess a one-eigenvalue, even if the graph is strongly connected (see, e.g., Figures 1 to 2). The one-eigenvalue is of particular interest since its eigenvector v corresponds to zero Dirichlet energy E ( v ) = 0 . Therefore, studying when 1 ∈ λ ( L ) is crucial to understanding the behavior of the Dirichlet energy. We fully characterize the set of graphs for which 1 ∈ λ ( L ) ; this is the scope of the following definition.

glyph[negationslash]

Definition 3.4. A graph G = ( V , E ) is said to be balanced if d in i = d out i for all i ∈ { 1 , . . . , N } , and weakly balanced if there exists k ∈ R N such that k = 0 and

It is straightforward to see that a balanced graph is weakly balanced since one can choose k i = √ d in i . Hence, all undirected graphs are also weakly balanced. However, as shown in Figure 2, the set of balanced graphs is a proper subset of the set of weakly balanced graphs.

Proposition 3.5. Let G be a directed graph with SNA L . Then, 1 ∈ λ ( L ) if and only if the graph is weakly balanced. Suppose the graph is strongly connected, then -1 ∈ λ ( L ) if and only if the graph is weakly balanced with an even period.

Proposition 3.5 generalizes a well-known result for undirected graphs: -1 ∈ λ ( L ) if and only if the graph is bipartite, i.e., has even period. The next result shows that the Dirichlet energy defined in (1) and the SNA are closely connected.

glyph[negationslash]

Proposition 3.6. For every x ∈ C N × K , it holds E ( x ) = 1 2 glyph[Rfractur] ( trace ( x H ( I -L ) x )) . Moreover, there exists x = 0 such that E ( x ) = 0 if and only if the graph is weakly balanced.

Proposition 3.6 generalizes the well-known result from the undirected (see, e.g., Cai et al., 2020, Definition 3.1) to the directed case. This result is an important tool for analyzing the evolution of the Dirichlet energy in graph neural networks.

α

= 2

0

-

0

.

5

0

0

.

5

α

= 2

-

3

α

= 2

-

6

(a) Synthetic cycle graph. The values of the fractional Laplacian can also be negative.

1

5

10

15

10

0

10

3

10

6

Distance in the original graph

N. of virtual edges

1

5

10

15

20

10

0

10

2

10

4

Distance in the original graph

Chameleon

α

1

0

.

75

0

.

5

0

.

25

0

(b) Real-world graphs. We count the number of virtual edges built by the fractional Laplacian based on the distance d ( i, j ) in the original graph. The number of virtual edges increases as α decreases.

4 Fractional Graph Laplacians

We introduce the fractional graph Laplacian through the singular value decomposition (SVD). This approach has two key advantages over the traditional definition (Pozrikidis, 2018; Benzi et al., 2020) in the spectral domain. First, it allows defining the fractional Laplacian based on any choice of graph Laplacian, including those with negative or complex spectrum such as the SNA. Secondly, the SVD is computationally more efficient and numerically more stable than the Jordan decomposition, which would be necessary if the fractional Laplacian was defined in the spectral domain.

Consider a directed graph with SNA L and its SVD L = UΣV H , where U , V ∈ C N × N are unitary matrices and Σ ∈ R N × N is a diagonal matrix. Given α ∈ R , we define the α -fractional graph Laplacian 2 ( α -FGL in short) as

In undirected graphs, the α -FGL preserves the sign of the eigenvalues λ of L while modifying their magnitudes, i.e., λ ↦→ sign( λ ) | λ | α . 3

The α -FGL is generally less sparse than the original SNA, as it connects nodes that are not adjacent in the underlying graph. The next theorem proves that the weight of such 'virtual' edges is bounded.

Theorem 4.1. Let G be a directed graph with SNA L . For α > 0 , if the distance d ( i, j ) between nodes i and j is at least 2, then

We provide a proof of Theorem 4.1 in Appendix C. In Figure 3a, we visually represent the cycle graph with eight nodes and the corresponding α -FGL entries. We also refer to Figure 3b, where we

depict the distribution of α -FGL entries for the real-world graphs Cora (undirected) and Chameleon (directed) with respect to the distance in the original graph. Our empirical findings align with our theoretical results presented in Theorem 4.1.

5 Fractional Graph Laplacian Neural ODE

This section explores two fractional Laplacian-based graph neural ODEs. First, we consider the fractional heat equation,

where x 0 ∈ R N × K is the initial condition , x ( t ) ∈ R N × K for t > 0 and α ∈ R . We assume that the channel mixing matrix W ∈ R K × K is a symmetric matrix. Second, we consider the fractional Schrödinger equation,

where x 0 , x ( t ) ∈ C N × K and W ∈ C K × K is unitary diagonalizable. Both (2) and (3) can be analytically solved. For instance, the solution of (2) is given by vec( x )( t ) = exp( -t W ⊗ L α )vec( x 0 ) , where ⊗ denotes the Kronecker product and vec( · ) represents the vectorization operation. However, calculating the exact solution is computationally infeasible since the memory required to store W ⊗ L α alone grows as ( NK ) 2 . Therefore, we rely on numerical schemes to solve (2) and (3).

In the remainder of this section, we analyze the Dirichlet energy for solutions to (2) and (3). We begin with the definition of oversmoothing.

Definition 5.1. Neural ODE-based GNNs are said to oversmooth if the normalized Dirichlet energy decays exponentially fast. That is, for any initial value x 0 , the solution x ( t ) satisfies for every t > 0

Definition 5.1 captures the actual smoothness of features by considering the normalized Dirichlet energy, which mitigates the impact of feature amplitude (Cai et al., 2020; Di Giovanni et al., 2023). Additionally, Proposition 3.6 shows that the normalized Dirichlet energy is intimately related to the numerical range of I -L of the underlying graph. This shows that the Dirichlet energy and eigenvalues (or frequencies ) of the SNA are intertwined, and one can equivalently talk about Dirichlet energy or frequencies (see also Lemma D.2). In particular, it holds that

As seen in Section 3, the minimal possible value attained by the normalized Dirichlet energy is often strictly greater than 0 for directed graphs. This indicates that GNNs on general directed graphs inherently cannot oversmooth to the same extent as in undirected. However, we prove that a vanilla GCN implementing the directed SNA oversmooths with respect to Definition 5.1, see Appendix E.3.

5.1 Frequency Analysis for Graphs with Normal SNA

This subsection focuses on the frequency analysis of FGL-based Neural ODEs for undirected graphs. Most classical GNNs (Kipf et al., 2017; Velickovic et al., 2018) and also graph neural ODEs (Chamberlain et al., 2021; Eliasof et al., 2021) have been shown to oversmooth. Di Giovanni et al. (2023) proved that the normalized Dirichlet energy for GNNs based on (2) with α = 1 can not only converge to its minimal value but also to its maximal possible value. A GNN exhibiting this property is then termed Highest-Frequency-Dominant (HFD) .

However, in real-world scenarios, most graphs are not purely homophilic nor purely heterophilic but fall somewhere in between. Intuitively, this suggests that mid-range frequencies might be more suitable. To illustrate this intuition, consider the cycle graph as an example. If we have a homophily of 1 , low frequencies are optimal; with a homophily equal to 0 , high frequencies are optimal. Interestingly, for a homophily of 1 / 2 , the mid-range frequency is optimal, even though the eigendecomposition is label-independent. More information on this example can be found in Figure 4 and Appendix F. Based on this observation, we propose the following definition to generalize the concept of HFD, accommodating not only the lowest or highest frequency but all possible frequencies.

√

λ

=

-

1

-

1

-

√

2

/

2

0

√

2

/

2

1

λ

=

-

2

/

2

λ

= 0

λ

=

2

/

2

λ

= 1

H

= 0

H

=

1

/

2

H

= 1

√

Definition 5.2. Let λ ≥ 0 . Neural ODE-based GNNs initialized at x 0 are λ -Frequency-Dominant ( λ -FD ) if the solution x ( t ) satisfies

Suppose λ is the smallest or the largest eigenvalue with respect to the real part. In that case, we call it Lowest-Frequency-Dominant (LFD) or Highest-Frequency-Dominant (HFD) , respectively.

In the following theorem, we show that (2) and (3) are not limited to being LFD or HFD, but can also be mid-frequency dominant.

Theorem 5.3. Let G be an undirected graph with SNA L . Consider the initial value problem in (2) with W ∈ R K × K and α ∈ R . Then, for almost all initial values x 0 ∈ R N × K the following holds.

( α > 0) The solution to (2) is either HFD or LFD.

( α < 0) Let λ + ( L ) and λ -( L ) be the smallest positive and negative non-zero eigenvalue of L , respectively. The solution to (2) is either (1 -λ + ( L )) -FD or (1 -λ -( L )) -FD.

Furthermore, the previous results also hold for solutions to the Schrödinger equation (3) if W ∈ C K × K has at least one eigenvalue with non-zero imaginary part.

Theorem 5.3 ( α > 0) generalizes the result by Di Giovanni et al. (2023) for α = 1 to arbitrary positive values of α . The convergence speed in Theorem 5.3 ( α > 0) depends on the choice of α ∈ R . By selecting a variable α (e.g., as a learnable parameter), we establish a flexible learning framework capable of adapting the convergence speed of the Dirichlet energy. A slower or more adjustable convergence speed facilitates broader frequency exploration as it converges more gradually to its maximal or minimal value. Consequently, the frequency component contributions (for finite time, i.e., in practice) are better balanced, which is advantageous for graphs with different homophily levels. Theorem 5.3 ( α < 0) shows that solutions of the fractional neural ODEs in (2) and (3) are not limited to be LFD or HFD. To demonstrate this and the other results of Theorem 5.3, we solve (2) using an explicit Euler scheme for different choices of α and W on the Cora and Chameleon graphs. The resulting evolution of the Dirichlet energy with respect to time is illustrated in Figure 5. Finally, we refer to Theorem D.5 in Appendix D.1 for the full statement and proof of Theorem 5.3.

Remark 5.4. Theorem 5.3 is stated for the analytical solutions of (2) and (3) , respectively. As noted in Section 5, calculating the analytical solution is infeasible in practice. However, we show in Appendices D.2 to D.3 that approximations of the solution of (2) and (3) via explicit Euler schemes satisfy the same Dirichlet energy convergence properties if the step size is sufficiently small.

Remark 5.5. Theorem 5.3 can be generalized to all directed graphs with normal SNA, i.e., satisfying the condition LL glyph[latticetop] = L glyph[latticetop] L . For the complete statement, see Appendix D.1.

10

0

10

1

10

2

0

0

.

25

0

.

5

Time

Dirichlet energy

10

0

10

1

10

2

0

.

25

0

.

5

0

.

75

1

Time

α

1

0

.

75

0

.

5

0

.

25

0

-

0

.

1

(a) Cora (undirected).

10

0

10

1

10

2

0

0

.

25

0

.

5

Time

Dirichlet energy

10

0

10

1

10

2

0

.

25

0

.

5

0

.

75

1

Time

α

1

0

.

75

0

.

5

0

.

25

0

-

0

.

1

(b) chameleon (directed).

5.2 Frequency Dominance for Directed Graphs

Section 5.1 analyzes the Dirichlet energy in graphs with normal SNA. However, the situation becomes significantly more complex when considering generic directed graphs. In our experiments (see Figure 5), we observe that the solution to (2) and (3) does not necessarily lead to oversmoothing. On the contrary, the solution can be controlled to exhibit either LFD or HFD for α > 0 , and midfrequency-dominance for α < 0 as proven for undirected graphs in Theorem 5.3. We present an initial theoretical result for directed graphs, specifically in the case of α = 1 .

Theorem 5.6. Let G be a directed graph with SNA L . Consider the initial value problem in (2) with diagonal channel mixing matrix W ∈ R K × K and α = 1 . Suppose λ 1 ( L ) is unique. For almost all initial values x 0 ∈ R N × K , the solution to (2) is either HFD or LFD.

The proof of Theorem 5.6 is given in Appendix E.1. Finally, we refer to Appendix E.2 for the analogous statement and proof when the solution of (2) is approximated via an explicit Euler scheme.

6 Numerical Experiments

This section evaluates the fractional Laplacian ODEs in node classification by approximating (2) and (3) with an explicit Euler scheme. This leads to the following update rules

for the heat and Schrödinger equation, respectively. In both cases, W , α and h are learnable parameters, t is the layer, and x 0 is the initial nodes' feature matrix. In accordance with the results in Section 5, we select W as a diagonal matrix. The initial features x 0 in (4) are encoded through a MLP, and the output is decoded using a second MLP. We refer to the resulting model as FLODE (fractional Laplacian ODE). In Appendix A, we present details on the baseline models, the training setup, and the exact hyperparameters.

Ablation Study. In Appendix A.3, we investigate the influence of each component (learnable exponent, ODE framework, directionality via the SNA) on the performance of FLODE. The adjustable fractional power in the FGL is a crucial component of FLODE, as it alone outperforms the model employing the ODE framework with a fixed α = 1 . Further, Appendix A.3 includes ablation studies that demonstrate FLODE's capability to efficiently scale to large depths, as depicted in Figure 8.

Real-World Graphs. We report results on 6 undirected datasets consisting of both homophilic graphs, i.e., Cora (McCallum et al., 2000), Citeseer (Sen et al., 2008) and Pubmed (Namata et al., 2012), and heterophilic graphs, i.e., Film (Tang et al., 2009), Squirrel and Chameleon (Rozemberczki et al., 2021). We evaluate our method on the directed and undirected versions of Squirrel, Film, and Chameleon. In all datasets, we use the standard 10 splits from (Pei et al., 2019). The choice of the baseline models and their results are taken from (Di Giovanni et al., 2023). Further, we test our method on heterophily-specific graph datasets, i.e., Roman-empire, Minesweeper, Tolokers, and Questions (Platonov et al., 2023). The splits, baseline models, and results are taken from (Platonov et al., 2023). The top three models are shown in Table 1, and the thorough comparison is reported in Table 4. Due to memory limitations, we compute only 30% of singular values for Pubmed, Roman-Empire, and Questions, which serve as the best low-rank approximation of the original SNA.

Synthetic Directed Graph. We consider the directed stochastic block model (DSBM) datasets (Zhang et al., 2021). The DSBM divides nodes into 5 clusters and assigns probabilities for interactions between vertices. It considers two sets of probabilities: { α i,j } for undirected edge creation and { β i,j } for assigning edge directions, i, j ∈ { 1 , . . . 5 } . The objective is to classify vertices based on their clusters. In the first experiment, α i,j = α ∗ varies, altering neighborhood information's importance. In the second experiment, β i,j = β ∗ varies, changing directional information. The results are shown in Figure 6 and Table 6. The splits, baseline models, and results are taken from (Zhang et al., 2021).

Results. The experiments showcase the flexibility of FLODE, as it can accommodate various types of graphs, both directed and undirected, as well as a broad range of homophily levels. While other methods, such as MagNet (Zhang et al., 2021), perform similarly to our approach, they face limitations when applied to certain graph configurations. For instance, when applied to undirected graphs, MagNet reduces to ChebNet, making it unsuitable for heterophilic graphs. Similarly, GRAFF

0.05

0.08

0.1

0

50

100

α

∗

0.1

0.2

0.3

0.4

β

∗

MagNet

DiGraphIB

DiGraph

DGCN

GAT-D

GIN-D

GraphSAGE-D

GCN-D

ChebNet

FLODE

10

0

10

1

10

2

10

3

10

-

2

10

-

1

10

0

570

0

.

5

0

.

6

0

.

7

0

.

8

N. of singular values

Explained Variance

Chameleon

Test accuracy

10

0

10

1

10

2

10

3

10

-

2

10

-

1

10

0

1600

0

.

35

0

.

45

0

.

55

0

.

65

0

.

75

N. of singular values

Explained Variance

Test accuracy

glyph[negationslash]

(Di Giovanni et al., 2023) performs well on undirected graphs but falls short on directed graphs. We note that oftentimes FLODE learns a non-trivial exponent α = 1 , highlighting the advantages of FGL-based GNNs (see, e.g., Table 5). Furthermore, as shown in Table 9 and Appendix A.3, our empirical results align closely with the theoretical results in Section 5.

7 Conclusion

In this work, we introduce the concepts of Dirichlet energy and oversmoothing for directed graphs and demonstrate their relation with the SNA. Building upon this foundation, we define fractional graph Laplacians in the singular value domain, resulting in matrices capable of capturing long-range dependencies. To address oversmoothing in directed graphs, we propose fractional Laplacian-based graph ODEs, which are provably not limited to LFD behavior. We finally show the flexibility of our method to accommodate various graph structures and homophily levels in node-level tasks.

Limitations and Future Work. The computational cost of the SVD grows cubically in N , while the storage of the singular vectors grows quadratically in N . Both costs can be significantly reduced by computing only k glyph[lessmuch] N singular values via truncated SVD (Figure 7), giving the best k -rank approximation of the SNA. Moreover, the SVD can be computed offline as a preprocessing step.

glyph[negationslash]

The frequency analysis of α -FGL neural ODEs in directed graphs is an exciting future direction. It would also be worthwhile to investigate the impact of choosing α = 1 on the convergence speed of the Dirichlet energy. Controlling the speed could facilitate the convergence of the Dirichlet energy to an optimal value, which has been shown to exist in synthetic settings (Keriven, 2022; X. Wu et al., 2022). Another interesting future direction would be to analyze the dynamics when approximating the solution to the FGL neural ODEs using alternative numerical solvers, such as adjoint methods.

Acknowledgments

S. M. acknowledges partial support by the NSF-Simons Research Collaboration on the Mathematical and Scientific Foundations of Deep Learning (MoDL) (NSF DMS 2031985) and DFG SPP 1798, KU 1446/27-2.

G. K. acknowledges partial support by the DAAD programme Konrad Zuse Schools of Excellence in Artificial Intelligence, sponsored by the Federal Ministry of Education and Research. G. Kutyniok also acknowledges support from the Munich Center for Machine Learning (MCML) as well as the German Research Foundation under Grants DFG-SPP-2298, KU 1446/31-1 and KU 1446/32-1 and under Grant DFG-SFB/TR 109 and Project C09.

References

Benzi, M., Bertaccini, D., Durastante, F., and Simunec, I. (2020). 'Non-Local Network Dynamics via Fractional Graph Laplacians'. In: Journal of Complex Networks 8.3.

Bo, D., Wang, X., Shi, C., and Shen, H. (2021). 'Beyond Low-frequency Information in Graph Convolutional Networks'. In: Proceedings of the AAAI Conference on Artificial Intelligence 35.5, pp. 3950-3957.

Bodnar, C., Di Giovanni, F., Chamberlain, B., Lió, P., and Bronstein, M. (2022). 'Neural Sheaf Diffusion: A Topological Perspective on Heterophily and Oversmoothing in GNNs'. In: Advances in Neural Information Processing Systems 35, pp. 18527-18541.

Bronstein, M. M., Bruna, J., LeCun, Y., Szlam, A., and Vandergheynst, P. (2017). 'Geometric Deep Learning: Going beyond Euclidean Data'. In: IEEE Signal Processing Magazine 34.4, pp. 18-42.

Cai, C. and Wang, Y. (2020). A Note on Over-Smoothing for Graph Neural Networks . arXiv: 2006. 13318 [cs, stat] .

Chamberlain, B., Rowbottom, J., Gorinova, M. I., Bronstein, M., Webb, S., and Rossi, E. (2021). 'GRAND: Graph Neural Diffusion'. In: Proceedings of the 38th International Conference on Machine Learning . PMLR, pp. 1407-1418.

Chen, D., Lin, Y., Li, W., Li, P., Zhou, J., and Sun, X. (2020). 'Measuring and Relieving the OverSmoothing Problem for Graph Neural Networks from the Topological View'. In: Proceedings of the AAAI Conference on Artificial Intelligence 34.04, pp. 3438-3445.

Chen, M., Wei, Z., Huang, Z., Ding, B., and Li, Y. (2020). 'Simple and Deep Graph Convolutional Networks'. In: Proceedings of the 37th International Conference on Machine Learning . PMLR, pp. 1725-1735.

Chen, R. T. Q., Rubanova, Y., Bettencourt, J., and Duvenaud, D. K. (2018). 'Neural Ordinary Differential Equations'. In: Advances in Neural Information Processing Systems . Vol. 31. Curran Associates, Inc.

Chien, E., Peng, J., Li, P., and Milenkovic, O. (2021). 'Adaptive Universal Generalized PageRank Graph Neural Network'. In: International Conference on Learning Representations .

Choi, J., Hong, S., Park, N., and Cho, S.-B. (2023). 'GREAD: Graph Neural Reaction-Diffusion Networks'. In: Proceedings of the 40th International Conference on Machine Learning . Vol. 202. ICML'23. Honolulu, Hawaii, USA: JMLR.org, pp. 5722-5747.

Chung, F. R. K. (1997). Spectral Graph Theory . Regional Conference Series in Mathematics no. 92. Providence, R.I: Published for the Conference Board of the mathematical sciences by the American Mathematical Society.

Defferrard, M., Bresson, X., and Vandergheynst, P. (2016). 'Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering'. In: Advances in Neural Information Processing Systems . Vol. 29. Curran Associates, Inc.

Di Giovanni, F., Rowbottom, J., Chamberlain, B. P., Markovich, T., and Bronstein, M. M. (2023). 'Understanding convolution on graphs via energies'. In: Transactions on Machine Learning Research .

Du, L., Shi, X., Fu, Q., Ma, X., Liu, H., Han, S., and Zhang, D. (2022). 'GBK-GNN: Gated Bi-Kernel Graph Neural Networks for Modeling Both Homophily and Heterophily'. In: Proceedings of the ACM Web Conference 2022 . WWW'22. New York, NY, USA: Association for Computing Machinery, pp. 1550-1558.

Eliasof, M., Haber, E., and Treister, E. (2021). 'PDE-GCN: Novel Architectures for Graph Neural Networks Motivated by Partial Differential Equations'. In: Advances in Neural Information Processing Systems . Vol. 34. Curran Associates, Inc., pp. 3836-3849.

Fan, W., Ma, Y., Li, Q., He, Y., Zhao, E., Tang, J., and Yin, D. (2019). 'Graph Neural Networks for Social Recommendation'. In: The World Wide Web Conference . San Francisco CA USA: ACM, pp. 417-426.

Fey, M. and Lenssen, J. E. (2019). 'Fast Graph Representation Learning with PyTorch Geometric'. In: ICLR 2019 Workshop on Representation Learning on Graphs and Manifolds .

Gasteiger, J., Bojchevski, A., and Günnemann, S. (2018). 'Predict Then Propagate: Graph Neural Networks Meet Personalized PageRank'. In: International Conference on Learning Representations .

Gilmer, J., Schoenholz, S. S., Riley, P. F., Vinyals, O., and Dahl, G. E. (2017). 'Neural Message Passing for Quantum Chemistry'. In: Proceedings of the 34th International Conference on Machine Learning . PMLR, pp. 1263-1272.

Gori, M., Monfardini, G., and Scarselli, F. (2005). 'A New Model for Learning in Graph Domains'. In: Proceedings. 2005 IEEE International Joint Conference on Neural Networks, 2005. Vol. 2, 729-734 vol. 2.

Haber, E. and Ruthotto, L. (2018). 'Stable Architectures for Deep Neural Networks'. In: Inverse Problems 34.1, p. 014004. eprint: 1705.03341 (cs, math).

Hamilton, W., Ying, Z., and Leskovec, J. (2017). 'Inductive Representation Learning on Large Graphs'. In: Advances in Neural Information Processing Systems . Vol. 30. Curran Associates, Inc.

He, K., Zhang, X., Ren, S., and Sun, J. (2016). 'Deep Residual Learning for Image Recognition'. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pp. 770-778.

Keriven, N. (2022). 'Not Too Little, Not Too Much: A Theoretical Analysis of Graph (over)Smoothing'. In: Advances in Neural Information Processing Systems .

Kingma, D. P. and Ba, J. (2015). Adam: A Method for Stochastic Optimization . Ed. by Y. Bengio and Y. LeCun.

Kipf, T. N. and Welling, M. (2017). 'Semi-Supervised Classification with Graph Convolutional Networks'. In: International Conference on Learning Representations .

Li, G., Muller, M., Thabet, A., and Ghanem, B. (2019). 'DeepGCNs: Can GCNs Go As Deep As CNNs?' In: Proceedings of the IEEE/CVF International Conference on Computer Vision , pp. 9267-9276.

Li, Q., Han, Z., and Wu, X.-m. (2018). 'Deeper Insights Into Graph Convolutional Networks for Semi-Supervised Learning'. In: Proceedings of the AAAI Conference on Artificial Intelligence 32.1.

Li, X., Zhu, R., Cheng, Y., Shan, C., Luo, S., Li, D., and Qian, W. (2022). 'Finding Global Homophily in Graph Neural Networks When Meeting Heterophily'. In: Proceedings of the 39th International Conference on Machine Learning . PMLR, pp. 13242-13256.

Lingam, V., Ragesh, R., Iyer, A., and Sellamanickam, S. (2021). Simple Truncated SVD Based Model for Node Classification on Heterophilic Graphs . eprint: 2106.12807 (cs).

Luan, S., Hua, C., Lu, Q., Zhu, J., Zhao, M., Zhang, S., Chang, X.-W., and Precup, D. (2022). 'Revisiting Heterophily For Graph Neural Networks'. In: Advances in Neural Information Processing Systems 35, pp. 1362-1375.

Luan, S., Zhao, M., Chang, X.-W., and Precup, D. (2019). 'Break the Ceiling: Stronger Multi-scale Deep Graph Convolutional Networks'. In: Advances in Neural Information Processing Systems . Vol. 32. Curran Associates, Inc.

Maurya, S. K., Liu, X., and Murata, T. (2021). Improving Graph Neural Networks with Simple Architecture Design . arXiv: 2105.07634 [cs, stat] .

McCallum, A. K., Nigam, K., Rennie, J., and Seymore, K. (2000). 'Automating the Construction of Internet Portals with Machine Learning'. In: Information Retrieval 3.2, pp. 127-163.

Monti, F., Frasca, F., Eynard, D., Mannion, D., and Bronstein, M. M. (2019). Fake News Detection on Social Media Using Geometric Deep Learning . arXiv: 1902.06673 .

Namata, G. M., London, B., Getoor, L., and Huang, B. (2012). 'Query-driven Active Surveying for Collective Classification'. In: Workshop on Mining and Learning with Graphs .

Oono, K. and Suzuki, T. (2019). 'Graph Neural Networks Exponentially Lose Expressive Power for Node Classification'. In: International Conference on Learning Representations .

Paszke, A. et al. (2019). 'PyTorch: An Imperative Style, High-Performance Deep Learning Library'. In: Advances in Neural Information Processing Systems . Vol. 32. Curran Associates, Inc.

Pei, H., Wei, B., Chang, K. C.-C., Lei, Y., and Yang, B. (2019). 'Geom-GCN: Geometric Graph Convolutional Networks'. In: International Conference on Learning Representations .

Perko, L. (2001). Differential Equations and Dynamical Systems . Ed. by J. E. Marsden, L. Sirovich, and M. Golubitsky. Vol. 7. Texts in Applied Mathematics. New York, NY: Springer New York.

Platonov, O., Kuznedelev, D., Diskin, M., Babenko, A., and Prokhorenkova, L. (2023). 'A critical look at the evaluation of GNNs under heterophily: Are we really making progress?' In: The Eleventh International Conference on Learning Representations .

Poli, M., Massaroli, S., Park, J., Yamashita, A., Asama, H., and Park, J. (2021). Graph Neural Ordinary Differential Equations . arXiv: 1911.07532 [cs, stat] .

Pozrikidis, C. (2018). The Fractional Laplacian . First. Boca Raton : Taylor & Francis, 2016. | 'A CRC title.': Chapman and Hall/CRC.

Rong, Y., Huang, W., Xu, T., and Huang, J. (2020). 'DropEdge: Towards Deep Graph Convolutional Networks on Node Classification'. In: International Conference on Learning Representations .

Rozemberczki, B., Allen, C., and Sarkar, R. (2021). 'Multi-Scale Attributed Node Embedding'. In: Journal of Complex Networks 9.2.

Rusch, T. K., Chamberlain, B., Rowbottom, J., Mishra, S., and Bronstein, M. (2022). 'Graph-Coupled Oscillator Networks'. In: Proceedings of the 39th International Conference on Machine Learning . PMLR, pp. 18888-18909.

Scarselli, F., Gori, M., Tsoi, A. C., Hagenbuchner, M., and Monfardini, G. (2009). 'The Graph Neural Network Model'. In: IEEE Transactions on Neural Networks 20.1, pp. 61-80.

Sen, P., Namata, G., Bilgic, M., Getoor, L., Galligher, B., and Eliassi-Rad, T. (2008). 'Collective Classification in Network Data'. In: AI Magazine 29.3, p. 93.

Shi, Y., Huang, Z., Feng, S., Zhong, H., Wang, W., and Sun, Y. (2021). 'Masked Label Prediction: Unified Message Passing Model for Semi-Supervised Classification'. In: Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence . Montreal, Canada: International Joint Conferences on Artificial Intelligence Organization, pp. 1548-1554.

Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., and Salakhutdinov, R. (2014). 'Dropout: A Simple Way to Prevent Neural Networks from Overfitting'. In.

Tang, J., Sun, J., Wang, C., and Yang, Z. (2009). 'Social Influence Analysis in Large-Scale Networks'. In: Proceedings of the 15th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining . Paris France: ACM, pp. 807-816.

Tong, Z., Liang, Y., Sun, C., Li, X., Rosenblum, D., and Lim, A. (2020). 'Digraph Inception Convolutional Networks'. In: Advances in Neural Information Processing Systems . V ol. 33. Curran Associates, Inc., pp. 17907-17918.

Tong, Z., Liang, Y., Sun, C., Rosenblum, D. S., and Lim, A. (2020). Directed Graph Convolutional Network . arXiv: 2004.13970 [cs, stat] .

Velickovic, P., Cucurull, G., Casanova, A., Romero, A., Liò, P., and Bengio, Y. (2018). 'Graph Attention Networks'. In: International Conference on Learning Representations .

Wang, J., Huang, P., Zhao, H., Zhang, Z., Zhao, B., and Lee, D. L. (2018). 'Billion-Scale Commodity Embedding for E-commerce Recommendation in Alibaba'. In: Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining . KDD '18. New York, NY, USA: Association for Computing Machinery, pp. 839-848.

Wang, X. and Zhang, M. (2022). 'How Powerful Are Spectral Graph Neural Networks'. In: Proceedings of the 39th International Conference on Machine Learning . PMLR, pp. 23341-23362.

Wang, Y., Yi, K., Liu, X., Wang, Y. G., and Jin, S. (2022). 'ACMP: Allen-Cahn Message Passing with Attractive and Repulsive Forces for Graph Neural Networks'. In: The Eleventh International Conference on Learning Representations .

Wu, F., Souza, A., Zhang, T., Fifty, C., Yu, T., and Weinberger, K. (2019). 'Simplifying Graph Convolutional Networks'. In: Proceedings of the 36th International Conference on Machine Learning . PMLR, pp. 6861-6871.

Wu, X., Chen, Z., Wang, W. W., and Jadbabaie, A. (2022). 'A Non-Asymptotic Analysis of Oversmoothing in Graph Neural Networks'. In: The Eleventh International Conference on Learning Representations .

Xhonneux, L.-P., Qu, M., and Tang, J. (2020). 'Continuous Graph Neural Networks'. In: Proceedings of the 37th International Conference on Machine Learning . PMLR, pp. 10432-10441.

Xu, K., Hu, W., Leskovec, J., and Jegelka, S. (2018). 'How Powerful Are Graph Neural Networks?' In: International Conference on Learning Representations .

Yan, Y., Hashemi, M., Swersky, K., Yang, Y., and Koutra, D. (2022). 'Two Sides of the Same Coin: Heterophily and Oversmoothing in Graph Convolutional Neural Networks'. In: 2022 IEEE International Conference on Data Mining (ICDM) . Orlando, FL, USA: IEEE, pp. 1287-1292.

Zhang, X., He, Y., Brugnone, N., Perlmutter, M., and Hirn, M. (2021). 'MagNet: A Neural Network for Directed Graphs'. In: Advances in Neural Information Processing Systems . Vol. 34. Curran Associates, Inc., pp. 27003-27015.

Zhao, L. and Akoglu, L. (2019). 'PairNorm: Tackling Oversmoothing in GNNs'. In: International Conference on Learning Representations .

Zhu, J., Rossi, R. A., Rao, A., Mai, T., Lipka, N., Ahmed, N. K., and Koutra, D. (2021). 'Graph Neural Networks with Heterophily'. In: Proceedings of the AAAI Conference on Artificial Intelligence 35.12, pp. 11168-11176.

Zhu, J., Yan, Y., Zhao, L., Heimann, M., Akoglu, L., and Koutra, D. (2020). 'Beyond Homophily in Graph Neural Networks: Current Limitations and Effective Designs'. In: Advances in Neural Information Processing Systems . Vol. 33. Curran Associates, Inc., pp. 7793-7804.

Zou, C., Han, A., Lin, L., and Gao, J. (2022). A Simple Yet Effective SVD-GCN for Directed Graphs . arXiv: 2205.09335 [cs] .

Acronyms

A Implementation Details

In this section, we give the details on the numerical results in Section 6. We begin by describing the exact model.

Model architecture. Let G be a directed graphs and x 0 ∈ R N × K the node features. Our architecture first embeds the input node features x via a multi-layer perceptron (MLP). We then evolve the features x 0 according to a slightly modified version of (3), i.e, x ′ ( t ) = -i L α x ( t ) W for some time t ∈ [0 , T ] . In our experiments, we approximate the solution with an Explicit Euler scheme with step size h > 0 . This leads to the following update rule

The channel mixing matrix is a diagonal learnable matrix W ∈ C K × K , and α ∈ R , h ∈ C are also learnable parameters. The features at the last time step x T are then fed into a second MLP, whose output is used as the final output. Both MLPs use LeakyReLU as non-linearity and dropout (Srivastava et al., 2014). On the contrary, the graph layers do not use any dropout nor non-linearity. A sketch of the algorithm is reported in fLode.

Complexity. The computation of the SVD is O ( N 3 ) . However, one can compute only the first p glyph[lessmuch] N singular values: this cuts down the cost to O ( N 2 p ) . The memory required to store the singular vectors is O ( N 2 ) , since they are not sparse in general. Each training step has a cost of O ( N 2 K ) .

Experimental details. Our model is implemented in PyTorch (Paszke et al., 2019), using PyTorch geometric (Fey et al., 2019). The computation of the SVD for the fractional Laplacian is implemented using the library linalg provided by PyTorch . In the case of truncated SVD, we use the function randomized_svd provided by the library extmath from sklearn . The code and instructions to reproduce the experiments are available on GitHub . Hyperparameters were tuned using grid search. All experiments were run on an internal cluster with NVIDIA GeForce RTX 2080 Ti and NVIDIA TITAN RTX GPUs with 16 and 24 GB of memory, respectively.

Training details. All models were trained for 1000 epochs using Adam (Kingma et al., 2015) as optimizer with a fixed learning rate. We perform early stopping if the validation metric does not increase for 200 epochs.

A.1 Real-World Graphs

Undirected graphs We conducted 10 repetitions using data splits obtained from (Pei et al., 2019). For each split, 48% of the nodes are used for training, 32% for validation and 20% for testing. In all datasets, we considered the largest connected component (LCC). Chameleon, Squirrel, and Film are

directed graphs; hence, we converted them to undirected. Cora, Citeseer, and Pubmed are already undirected graphs: to these, we added self-loops. We normalized the input node features for all graphs.

As baseline models, we considered the same models as in (Di Giovanni et al., 2023). The results were provided by Pei et al. (2019) and include standard GNNs, such as GAT (Velickovic et al., 2018), GCN (Kipf et al., 2017), and GraphSAGE (Hamilton et al., 2017). We also included models designed to address oversmoothing and heterophilic graphs, such as PairNorm (L. Zhao et al., 2019), GGCN (Yan et al., 2022), Geom-GCN (Pei et al., 2019), H 2 GCN (Zhu, Yan, et al., 2020), GPRGNN (Chien et al., 2021), and Sheaf (Bodnar et al., 2022). Furthermore, we included the graph neural ODE-based approaches, CGNN (Xhonneux et al., 2020) and GRAND (Chamberlain et al., 2021), as in (Di Giovanni et al., 2023), and the model GRAFF from (Di Giovanni et al., 2023) itself. Finally, we included GREAD (Choi et al., 2023), GraphCON (Rusch et al., 2022), ACMP (Y. Wang et al., 2022) and GCN and GAT equipped with DropEdge (Rong et al., 2020).

Heterophily-specific Models For heterophily-specific datasets, we use the same models and results as in (Platonov et al., 2023). As baseline models we considered the topology-agnostic ResNet (He et al., 2016) and two graph-aware modifications: ResNet+SGC(F. Wu et al., 2019) where the initial node features are multiplied by powers of the SNA, and ResNet+adj, where rows of the adjacency matrix are used as additional node features; GCN (Kipf et al., 2017), GraphSAGE (Hamilton et al., 2017); GAT (Velickovic et al., 2018) and GT (Shi et al., 2021) as well as their modification GAT-sep and GT-sep which separate ego- and neighbor embeddings; H 2 GCN (Zhu, Yan, et al., 2020), CPGNN (Zhu, Rossi, et al., 2021), GPRGNN (Chien et al., 2021), FSGNN (Maurya et al., 2021), GloGNN (X. Li et al., 2022), FAGCN (Bo et al., 2021), GBK-GNN (Du et al., 2022), and JacobiConv (X. Wang et al., 2022).

The exact hyperparameters for FLODE are provided in Table 5.

A.2 Synthetic Directed Graphs

The dataset and code are taken from (Zhang et al., 2021). As baseline models, we considered the ones in (Zhang et al., 2021) for which we report the corresponding results. The baseline models include standard GNNs, such as ChebNet (Defferrard et al., 2016), GCN (Kipf et al., 2017), GraphSAGE (Hamilton et al., 2017), APPNP (Gasteiger et al., 2018), GIN (Xu et al., 2018), GAT (Velickovic et al., 2018), but also models specifically designed for directed graphs, such as DGCN (Tong, Liang, Sun, Rosenblum, et al., 2020), DiGraph and DiGraphIB (Tong, Liang, Sun, X. Li, et al., 2020), MagNet (Zhang et al., 2021)).

The DSBM dataset. The directed stochastic block model (DSBM) is described in detail in (Zhang et al., 2021, Section 5.1.1). To be self-contained, we include a short explanation.

The DSBM model is defined as follows. There are N vertices, which are divided into n c clusters ( C 1 , C 2 , ...C n c ) , each having an equal number of vertices. An interaction is defined between any two distinct vertices, u and v , based on two sets of probabilities: { α i,j } n c i,j =1 and { β i,j } n c i,j =1 .

The set of probabilities { α i,j } is used to create an undirected edge between any two vertices u and v , where u belongs to cluster C i and v belongs to cluster C j . The key property of this probability set is that α i,j = α j,i , which means the chance of forming an edge between two clusters is the same in either direction.

The set of probabilities { β i,j } is used to assign a direction to the undirected edges. For all i, j ∈ { 1 , . . . , n c } , we assume that β i,j + β j,i = 1 holds. Then, to the undirected edge ( u, v ) is assigned the direction from u to v with probability β i,j if u belongs to cluster C i and v belongs to cluster C j , and the direction from v to u with probability β j,i .

The primary objective here is to classify the vertices based on their respective clusters.

There are several scenarios designed to test different aspects of the baseline models and our model. In the experiments, the total number of nodes is fixed at N = 2500 and the number of clusters is fixed at n c = 5 . In all experiments, the training set contains 20 nodes per cluster, 500 nodes for validation, and the rest for testing. The results are averaged over 5 different seeds and splits.

(b) Directed graphs.

(c) Heterophily-specific graphs. For Minesweeper, Tolokers and Questions the evaluation metric is the AUROC.

(c) Heterophily-specific graphs.

Following Zhang et al. (2021), we train our model in both experiments for 3000 epochs and use early-stopping if the validation accuracy does not increase for 500 epochs. We select the best model based on the validation accuracy after sweeping over a few hyperparameters. We give exact numerical values for the experiments with the standard error in Table 6a and refer to Appendix A.2 for the chosen hyperparameters.

glyph[negationslash]

DSBM with varying edge density. In the first experiment, the model is evaluated based on its performance on the DSBM with varying α i,j = α ∗ , α ∗ ∈ { 0 . 1 , 0 . 08 , 0 . 05 } for i = j , which essentially changes the density of edges between different clusters. The other probabilities are fixed at α i,i = 0 . 5 , β i,i = 0 . 5 and β i,j = 0 . 05 for i > j . The results are shown in Figure 6 with exact numerical values in Table 6a.

DSBM with varying net flow. In the other scenario, the model is tested on how it performs when the net flow from one cluster to another varies. This is achieved by keeping α i,j = 0 . 1 constant for all i and j , and allowing β i,j to vary from 0 . 05 to 0 . 4 . The other probabilities are fixed at α i,i = 0 . 5 and β i,i = 0 . 5 . The results are shown in Figure 6 with exact numerical values in Table 6b.

A.3 Ablation Study

We perform an ablation study on Chameleon and Squirrel (directed, heterophilic), and Citeseer (undirected, homophilic). For this, we sweep over different model options using the same hyperparameters

via grid search. The test accuracy corresponding to the hyperparameters that yielded maximum validation accuracy is reported in Table 8.

The ablation study on Chameleon demonstrates that all the components of the model (learnable exponent, ODE framework with the Schrödinger equation, and directionality via the SNA) contribute to the performance of FLODE. The fact that performance drops when any of these components are not used suggests that they all play crucial roles in the model's ability to capture the structure and evolution of heterophilic graphs. It is important to note that the performance appears to be more dependent on the adjustable fraction in the FGL than on the use of the ODE framework, illustrating that the fractional Laplacian alone can effectively capture long-range dependencies. However, when the ODE framework is additionally employed, a noticeable decrease in variance is observed.

From Theory to Practice. We conduct an ablation study to investigate the role of depth on Chameleon, Citeseer, Cora, and Squirrel datasets. The results, depicted in Figure 8, demonstrate that the neural ODE framework enables GNNs to scale to large depths ( 256 layers). Moreover, we see that the fractional Laplacian improves over the standard Laplacian in the heterophilic graphs which is supported by our claims in Section 5.2. We highlight that using only the fractional Laplacian without the neural ODE framework oftentimes outperforms the standard Laplacian with the neural ODE framework. This indicates the importance of the long-range connections built by the fractional Laplacian.

We further demonstrate the close alignment of our theoretical and experimental results, which enables us to precisely anticipate when the models will exhibit HFD or LFD behaviors. In this context, we calculate parameters (according to Theorem D.5) and illustrate at each depth the expected and observed behaviors. For Squirrel and Chameleon, which are heterophilic graphs, we observe that both their theoretical and empirical behaviors are HFD. Additionally, the learned exponent is small. In contrast, for Cora and Citeseer, we see the opposite.

Finally, we employ the best hyperparameters in Table 5a to solve both fractional heat and Schrödinger graph ODEs, further substantiating the intimate link between our theoretical advancements and practical applications.

(b) Squirrel (directed, heterophilic).

(c) Citeseer (undirected, homphilic).

0

.

2

0

.

4

0

.

6

0

.

8

1

Test accuracy

x

l

+1

=

x

l

-

h

L

α

x

l

W x

l

+1

=

x

l

-

h

l

W

x

l

+1

=

L

α

x

l

W

x

l

+1

=

l

W

Citeseer

Chameleon

10

-

2

10

-

1

10

0

10

1

E

(

x

L

‖

x

L

‖

)

/

E

(

x

0

‖

x

0

‖

)

-

1

0

1

LFD

HFD

FD

LFD

HFD

0

0

.

5

1

α

B Appendix for Section 3

Proposition 3.3. Let G be a directed graph with SNA L . For every λ ∈ λ ( L ) , it holds | λ | ≤ 1 and λ ( I -L ) = 1 -λ ( L ) .

Proof. We show that the numerical range W ( L ) = { x H Lx : x H x = 1 } satisfies W ( L ) ⊂ [ -1 , 1] . As W ( L ) contains all eigenvalues of L the thesis follows.

Let A be the adjacency matrix of G and x ∈ C N with x H x = 1 . Applying the Cauchy-Schwartz inequality in (2) and (3) , we get

where we used a 2 i,j = a i,j . We have ∑ N i =1 | x i | 2 = x H x = 1 such that W ( L ) ⊂ [ -1 , 1] follows. The second claim follows directly by ( I -L ) v = v -λ v = (1 -λ ) v .

Proposition 3.5. Let G be a directed graph with SNA L . Then 1 ∈ λ ( L ) if and only if the graph is weakly balanced. Suppose the graph is strongly connected; then -1 ∈ λ ( L ) if and only if the graph is weakly balanced with an even period.

Proof. Since the numerical range is only a superset of the set of eigenvalues, we cannot simply consider when the inequalities (1) -(3) in the previous proof are actual equalities. Therefore, we have to find another way to prove the statement. Suppose that the graph is weakly balanced, then

We will prove that k = ( k i ) N i =1 is an eigenvector corresponding to the eigenvalue 1 ,

glyph[negationslash]

For the other direction, suppose that there exists x ∈ R N such that x = 0 and x = Lx . Then for all i ∈ { 1 , . . . , N }

hence, the graph is weakly balanced.

By Perron-Frobenius theorem for irreducible non-negative matrices, one gets that L has exactly h eigenvalues with maximal modulus corresponding to the h roots of the unity, where h is the period of L . Hence, -1 is an eigenvalue of L if and only if the graph is weakly balanced and h is even.

Proposition 3.6. For every x ∈ C N × K , we have

glyph[negationslash]

Moreover, there exists x = 0 such that E ( x ) = 0 if and only if the graph is weakly balanced.

Proof. By direct computation, it holds

glyph[negationslash]

The last claim can be proved as follows. For simplicity, suppose x ∈ R N . The ' ⇐ = ' is clear since one can choose x to be k . To prove the ' = ⇒ ', we reason by contradiction. Suppose there exists a x = 0 such that E ( x ) = 0 and the underlying graph is not weakly connected, i.e., glyph[negationslash]

glyph[negationslash]

Then, since x = 0 ,

where we used Cauchy-Schwartz and triangle inequalities.

We give the following simple corollary.

Corollary B.1. For every x ∈ R N × K , it holds E ( x ) = 1 2 glyph[Rfractur] ( vec( x ) H ( I ⊗ ( I -L ))vec( x ) ) .

C Appendix for Section 4

In this section, we provide some properties about FGLs. The first statement shows that the FGL of a normal SNA L only changes the magnitude of the eigenvalues of L .

LemmaC.1. Let M be a normal matrix with eigenvalues λ 1 , . . . , λ N and corresponding eigenvectors v 1 , . . . , v N . Suppose M = LΣR H is its singular value decomposition. Then it holds

Proof. By hypothesis, there exist a unitary matrix V such that M = VΛV H , then

Therefore, Σ = | Λ | and L = V

Finally, we note that it must hold R = V exp( i Θ ) where Θ = diag ( { atan2( glyph[Rfractur] λ i , glyph[Ifractur] λ i ) } N i =1 ) and atan2 is the 2-argument arctangent.

We proceed by proving Theorem 4.1, which follows the proof of a similar result given in (Benzi et al., 2020) for the fractional Laplacian defined in the spectral domain of an in-degree normalized graph Laplacian. However, our result also holds for directed graphs and in particular for fractional Laplacians that are defined via the SVD of a graph SNA.

Lemma C.2. Let M ∈ R n × n with singular values σ ( M ) ⊂ [ a, b ] . For f : [ a, b ] → R , define f ( M ) = U f ( Σ ) V H , where M = UΣV H is the singular value decomposition of M . If f has modulus of continuity ω and d ( i, j ) ≥ 2 , it holds

Proof. Let g : [ a, b ] → R be any function, then

The second equation holds since the 2 -norm is invariant under unitary transformations. By Jackson's Theorem, there exists for every m ≥ 1 a polynomial p m of order m such that

Fix i, j ∈ { 1 , . . . , n } . If d ( i, j ) = m +1 , then any power of M up to order m has a zero entry in ( i, j ) , i.e., ( M m ) i,j = 0 . Hence, f ( M ) i,j = f ( M ) i,j -p m ( M ) i,j , and we get

Finally, we give a proof of Theorem 4.1, which is a consequence of the previous statement.

Proof of Theorem 4.1. The eigenvalues of L are in the unit circle, i.e., ‖ L ‖ ≤ 1 . Hence, ∥ ∥ LL H ∥ ∥ ≤ 1 , and the singular values of L are in [0 , 1] . By Lemma C.2 and the fact that f ( x ) = x α has modulus of continuity ω ( t ) = t α the thesis follows.

D Appendix for Section 5

In this section, we provide the appendix for Section 5. We begin by analyzing the solution of linear matrix ODEs. For this, let M ∈ C N × N . For x 0 ∈ C N , consider the initial value problem

Theorem D.1 (Existence and uniqueness of linear ODE solution) . The initial value problem given by (5) has a unique solution x ( t ) ∈ C N for any initial condition x 0 ∈ C N .

The solution of (5) can be expressed using matrix exponentials, even if M is not symmetric. The matrix exponential is defined as:

where M k is the k -th power of the matrix M . The solution of (5) can then be written as

D.1 Appendix for Section 5.1

In this section, we analyze the solution to (2) and (3). We further provide a proof for Theorem 5.3. We begin by considering the solution to the fractional heat equation (2). The analysis for the Schrödinger equation (3) follows analogously.

The fractional heat equation x ′ ( t ) = -L α xW can be vectorized and rewritten via the Kronecker product as

In the undirected case L and I -L are both symmetric, and the eigenvalues satisfy the relation λ i ( I -L ) = 1 -λ i ( L ) . The corresponding eigenvectors ψ i ( L ) and ψ i ( I -L ) can be chosen to be the same for L and I -L . In the following, we assume that these eigenvectors are orthonormalized.

If L is symmetric, we can decompose it via the spectral theorem into L = UDU T , where U = [ ψ 1 ( L ) , . . . , ψ N ( L )] is an orthogonal matrix containing the eigenvectors of L , and D is the diagonal matrix of eigenvalues.

Due to Lemma C.1, the fractional Laplacian L α can be written as L α = U f α ( D ) U T , where f α : R → R is the map x ↦→ sign( x ) | x | α and is applied element-wise. Clearly, the eigendecomposition of L α is given by the eigenvalues { f α ( λ 1 ( L )) , . . . , f α ( λ N ( L )) } and the corresponding eigenvectors { ψ 1 ( L ) , . . . , ψ N ( L ) } .

Now, by well-known properties of the Kronecker product, one can write the eigendecomposition of W ⊗ L α as

Note that 1 ∈ λ ( L ) and, since trace ( L ) = 0 , the SNA has at least one negative eigenvalue. This property is useful since it allows to retrieve of the indices ( r, l ) corresponding to eigenvalues with minimal real (or imaginary) parts in a simple way.

The initial condition vec( x 0 ) can be decomposed as

Then, the solution vec( x )( t ) of (7) can be written as

The following result shows the relationship between the frequencies of I -L and the Dirichlet energy and serves as a basis for the following proofs.

Lemma D.2. Let G be a graph with SNA L . Consider x ( t ) ∈ C N × K such that there exists ϕ ∈ C N × K \ { 0 } with

and ( I ⊗ ( I -L ))vec( ϕ ) = λ vec( ϕ ) . Then,

Proof. As vec( ϕ ) is the limit of unit vectors, vec( ϕ ) is a unit vector itself. We calculate its Dirichlet energy,

Since x ↦→ E ( x ) is continuous, the thesis follows.

Another useful result that will be extensively used in proving Theorem 5.3 is presented next.

Lemma D.3. Suppose x ( t ) can be expressed as

for some choice of c k,n , λ k,n , { v k } , { w n } . Let ( a, b ) be the unique index of λ k,n with minimal real part and corresponding non-null coefficient c k,n , i.e.

glyph[negationslash]

Then

Proof. The key insight is to separate the addend with index ( a, b ) . It holds glyph[negationslash]

We note that

glyph[negationslash]

for all ( k, n ) = ( a, b ) , since glyph[Rfractur] ( λ k,n -λ a,b ) > 0 . Therefore, one gets

where the normalization removes the dependency on exp( -t λ a,b )

When λ a,b is not unique, it is still possible to derive a convergence result. In this case, x will converge to an element in the span generated by vectors corresponding to λ a,b , i.e.,

glyph[negationslash]

where A := { ( k, n ) : glyph[Rfractur] ( λ k,n ) = glyph[Rfractur] ( λ a,b ) , c k,n = 0 } .

A similar result to Lemma D.3 holds for a slightly different representation of x ( t ) .

Lemma D.4. Suppose x ( t ) can be expressed as

for some choice of c k,n , λ k,n , { v k } , { w n } . Let ( a, b ) be the unique index of λ k,n with minimal imaginary part and corresponding non-null coefficient c k,n , i.e.

glyph[negationslash]

Proof. The proof follows the same reasoning as in the proof of Lemma D.3. The difference is that the dominating frequency is the one with the minimal imaginary part, since

Then and, consequently,

D.1.1 Proof of Theorem 5.3

We denote the eigenvalues of L closest to 0 from above and below as

We assume that the channel mixing W ∈ R K × K and the graph Laplacians L , I -L ∈ R N × N are real matrices. Finally, we suppose the eigenvalues of a generic matrix M are sorted in ascending order, i.e., λ i ( M ) ≤ λ j ( M ) , i < j .

We now reformulate Theorem 5.3 for the fractional heat equation (2) and provide its full proof, which follows a similar frequency analysis to the one in (Di Giovanni et al., 2023, Theorem B.3)

Theorem D.5. Let G be an undirected graph with SNA L . Consider the initial value problem in (2) with channel mixing matrix W ∈ R K × K and α ∈ R . Then, for almost all initial conditions x 0 ∈ R N × K the following is satisfied.

( α > 0) The solution to (2) is HFD if

and LFD otherwise.

( α < 0) The solution to (2) is (1 -λ -( L )) -FD if

and (1 -λ + ( L )) -FD otherwise.

Proof of ( α > 0) . As derived in (8), the solution of (2) with initial condition x 0 can be written in a vectorized form as

where λ r ( W ) are the eigenvalues of W with corresponding eigenvectors ψ r ( W ) , and λ l ( L ) are the eigenvalues of L with corresponding eigenvectors ψ l ( L ) . The coefficients c r,l are the Fourier coefficients of x 0 , i.e.,

The key insight is to separate the eigenprojection corresponding to the most negative frequency. By Lemma D.3, this frequency component dominates for t going to infinity.

Suppose

In this case, λ K ( W ) f α ( λ 1 ( L )) is the most negative frequency. Assume for simplicity that λ K ( W ) has multiplicity one; the argument can be applied even if this is not the case, since the corresponding eigenvectors are orthogonal for higher multiplicities.

For almost all initial conditions x 0 , the coefficient c K, 1 is not null; hence

By standard properties of the Kronecker product, we have

i.e., ψ K ( W ) ⊗ ψ 1 ( L ) is an eigenvector of I ⊗ L corresponding to the eigenvalue λ 1 ( L ) . Then, by Proposition 3.3, ψ K ( W ) ⊗ ψ 1 ( L ) is also an eigenvector of I ⊗ I -L corresponding to the eigenvalue 1 -λ 1 ( L ) = λ N ( I -L ) . An application of Lemma D.2 finishes the proof.

Similarly, we can show that if α > 0 and λ K ( W ) f α ( λ 1 ( L )) > λ 1 ( W ) the lowest frequency component λ 1 ( I -L ) is dominant.

Proof of ( α < 0) . In this case either f α ( λ + ( L )) λ 1 ( W ) or f α ( λ -( L )) λ K ( W ) are the most negative frequency components. Hence, if f α ( λ -( L )) λ K ( W ) > f α ( λ + ( L )) λ 1 ( W ) the frequency f α ( λ + ( L )) λ 1 ( W ) is dominating and otherwise the frequency f α ( λ -( L )) λ K ( W ) . We can see this by following the exact same reasoning of (i) .

Remark D.6. In the proof of ( α < 0) , we are tacitly assuming that L has only non-zero eigenvalues. If not, we can truncate the SVD and remove all zeros singular values (which correspond to zeros eigenvalues). In doing so, we obtain the best invertible approximation of L to which the theorem can be applied.

We now generalize the previous result to all directed graphs with normal SNA.

Theorem D.7. Let G be a a strongly connected directed graph with normal SNA L such that λ 1 ( L ) ∈ R . Consider the initial value problem in (2) with channel mixing matrix W ∈ R K × K and α > 0 . Then, for almost all initial values x 0 ∈ R N × K the solution to (2) is HFD if

and LFD otherwise.

Proof. Any normal matrix is unitary diagonalizable, i.e., there exist eigenvalues λ 1 , . . . , λ N and corresponding eigenvectors v 1 , . . . , v N such that L = VΛV H . Then, by Lemma C.1, the singular value decomposition of L is given by L = UΣV H , where

Hence,

Then, equivalent to the derivation of (8), the solution to the vectorized fractional heat equation

is given by with

Now, equivalent to the proof of Theorem 5.3, we apply Lemma D.3. Therefore, the dominating frequency is given by the eigenvalue of W ⊗ L α with the most negative real part. The eigenvalues of W ⊗ L α are given by λ r ( W ) f α ( λ l ( L )) for r = 1 , . . . , K , l = 1 , . . . , N . The corresponding real parts are given by

By Perron-Frobenius, the eigenvalue of L with the largest eigenvalues is given by λ N ( L ) ∈ R . Hence, for all l = 1 , . . . , N ,

Similarly, for all l = 1 , . . . , N with glyph[Rfractur] ( λ ( L ) l ) < 0 ,

Thus, the frequency with the most negative real part is either given by λ K ( W ) f α ( λ 1 ( L )) or λ 1 ( W ) f α ( λ N ( L )) . The remainder of the proof is analogous to the proof of Theorem D.7.

In the following, we provide the complete statement and proof for the claims made in Theorem 5.3 when the underlying ODE is the Schrödinger equation as presented in (3).

Theorem D.8. Let G be a undirected graph with SNA L . Consider the initial value problem in (3) with channel mixing matrix W ∈ C K × K and α ∈ R . Suppose that W has at least one eigenvalue with non-zero imaginary part and sort the eigenvalues of W in ascending order with respect to their imaginary part. Then, for almost initial values x 0 ∈ C N × K , the following is satisfied.

( α > 0) Solutions of (3) are HFD if

and LFD otherwise.

( α < 0) Let λ + ( L ) and λ -( L ) be the smallest positive and biggest negative non-zero eigenvalue of L , respectively. Solutions of (3) are (1 -λ -( L )) -FD if

Otherwise, solutions of (3) are (1 -λ + ( L )) -FD.

Proof. The proof follows the same reasoning as the proof for the heat equation in Theorem D.5. The difference is that we now apply Lemma D.4 instead of Lemma D.3.

Therefore, the dominating frequency is either λ K ( W ) f α ( λ 1 ( L )) or λ 1 ( W ) f α ( λ N ( L )) if α > 0 , and λ K ( W ) f α ( λ -( L )) or λ 1 ( W ) f α ( λ + ( L )) if α < 0 .

D.2 Frequency Dominance for Numerical Approximations of the Heat Equation

For n ∈ N and h ∈ R , h > 0 , the solution of (2) at time nh > 0 can be approximated with an explicit Euler scheme

which can be further simplified via the binomial theorem as

Hence, it holds the representation formula

In this case, the dominating frequency maximizes | 1 -hλ r ( W ) f α ( λ l ( L )) | . When h < ‖ W ‖ -1 , the product hλ r ( W ) f α ( λ l ( L )) is guaranteed to be in [ -1 , 1] , and

Therefore, the dominating frequency minimizes hλ r ( W ) f α ( λ l ( L )) . This is the reasoning behind the next result.

Proposition D.9. Let h ∈ R , h > 0 . Consider the fractional heat equation (2) with α ∈ R . Let { x ( nh ) } n ∈ N be the trajectory of vectors derived by approximating (2) with an explicit Euler scheme with step size h . Suppose h < ‖ W ‖ -1 . Then, for almost all initial values x 0

Proof. Define

By the hypothesis on h , this is equivalent to

Therefore, ( λ a , λ b ) is either ( λ 1 ( W ) , λ N ( L )) or ( λ K ( W ) , λ 1 ( L )) . Hence,

If the condition λ K ( W ) f α ( λ 1 ( L )) < λ 1 ( W ) is satisfied, we have b = 1 . Then by (10), the normalized vec( x ) converges to the eigenvector of I ⊗ I -L corresponding to the largest frequency 1 -λ 1 ( L ) = λ N ( I -L ) . An application of Lemma D.2 finishes the proof.

If λ K ( W ) f α ( λ 1 ( L )) < λ 1 ( W ) is not satisfied, we have b = N , and the other direction follows with the same argument.

Similarly to Proposition D.9 one can prove the following results for negative fractions.

Proposition D.10. Let h ∈ R , h > 0 . Consider the fractional heat equation (2) with α < 0 . Let { x ( nh ) } n ∈ N be the trajectory of vectors derived by approximating the solution of (2) with an explicit Euler scheme with step size h . Suppose that h < ‖ W ‖ -1 . The approximated solution is (1 -λ -( L )) -FD if

and (1 -λ + ( L )) -FD otherwise.

Proof. The proof follows the same reasoning as the proof of Proposition D.9 by realizing that the dominating frequencies ( λ a , λ b ) are either given by ( λ 1 ( W ) , λ + ( L )) or ( λ K ( W ) , λ -( L )) .

D.3 Frequency Dominance for Numerical Approximations of the Schrödinger Equation

For n ∈ N and h ∈ R , h > 0 , the solution of (3) at time nh > 0 can be approximated with an explicit Euler scheme as well. Similarly to the previous section, we can write

and

The dominating frequency will be discussed in the following theorem.

Proposition D.11. Let h ∈ R , h > 0 . Let { x ( nh ) } n ∈ N be the trajectory of vectors derived by approximating (3) with an explicit Euler scheme with sufficiently small step size h . Sort the eigenvalues of W in ascending order with respect to their imaginary part. Then, for almost all initial values x 0

Proof. Define

By definition of a and b , for all r and l it holds

Hence,

We continue by determining the indices a and b . To do so, we note that (12) is equivalent to

for all r , l . Denote by ε the gap glyph[negationslash]

Noting that

one gets that (12) is satisfied for h < ε ‖ W ‖ -2 . Therefore, for sufficiently small h , the dominating frequencies are the ones with minimal imaginary part, i.e., either f α ( λ 1 ( L )) glyph[Ifractur] ( λ K ( W )) or f α ( λ N ( L )) glyph[Ifractur] ( λ 1 ( W )) . If f α ( λ 1 ( L )) glyph[Ifractur] ( λ K ( W )) < f α ( λ N ( L )) glyph[Ifractur] ( λ 1 ( W )) , then b = 1 , and the normalized vec ( x ) converges to the eigenvector corresponding to the smallest frequency λ 1 ( L ) . By (10), this is also the eigenvector of I ⊗ I -L corresponding to the largest frequency 1 -λ 1 ( L ) = λ N ( I -L ) . An application of Lemma D.2 finishes the proof.

Finally, we present a similar result for negative powers.

Proposition D.12. Let h ∈ R , h > 0 . Consider the fractional Schrödinger equation (3) with α < 0 . Let { x ( nh ) } n ∈ N be the trajectory of vectors derived by approximating the solution of (3) with an explicit Euler scheme with step size h . Suppose that h is sufficiently small. Sort the eigenvalues of W in ascending order with respect to their imaginary part. The approximated solution is (1 -λ + ( L )) -FD if

and (1 -λ -( L )) -FD otherwise.

Proof. Similar to Proposition D.11, we can prove the statement by realizing that the dominating frequencies ( λ a , λ b ) in (12) are either given by ( λ 1 ( W ) , λ + ( L )) or ( λ K ( W ) , λ -( L )) .

E Appendix for Section 5.2

We begin this section by describing the solution of general linear matrix ODEs of the form (6) in terms of the Jordan decomposition of M . This is required when M is not diagonalizable. For instance, the SNA of a directed graph is not in general a symmetric matrix, hence, not guaranteed to be diagonalizable. We then proceed in Appendix E.1 with the proof of Theorem 5.6.

For a given matrix M ∈ C N × N , the Jordan normal form is given by

where P ∈ C N × N is an invertible matrix whose columns are the generalized eigenvectors of M , and J ∈ C N × N is a block-diagonal matrix with Jordan blocks along its diagonal. Denote with λ 1 , . . . , λ m the eigenvalues of M and with J 1 , . . . , J m the corresponding Jordan blocks. Let k l be the algebraic multiplicity of the eigenvalue λ l , and denote with { ψ i l ( M ) } i ∈{ 1 ,...,k l } the generalized eigenvectors of the Jordan block J l .

We begin by giving the following well-known result, which fully characterizes the frequencies for the solution of a linear matrix ODE.

Lemma E.1. Let M = PJP -1 ∈ C N × N be the Jordan normal form of M . Let x : [0 , T ] → R n be a solution to

and { e i l : i ∈ { 1 , . . . k l } , l ∈ { 1 , . . . , m } } is the standard basis satisfying Pe i l = ψ i l ( M ) .

Then, x is given by where

Proof. By (Perko, 2001, Section 1.8), the solution can be written as

where exp( J t ) = diag ( { exp( J l t ) } l =1 ) and

Since exp( J t ) = ⊕ m l =1 exp( J l t ) , we can focus on a single Jordan block. Fix l ∈ { 1 , . . . , m } , it holds

Bringing the direct sums together, we get

from which the thesis follows.

In the following, we derive a formula for the solution of ODEs of the form

for a diagonal matrix W ∈ C K × K and a general square matrix M ∈ C N × N with Jordan normal form PJP -1 . By vectorizing, we obtain the equivalent linear system

Then, by properties of the Kronecker product, there holds

Note that ( I ⊗ P )( W ⊗ J )( I ⊗ P ) -1 is not the Jordan normal form of D ⊗ M . However, we can characterize the Jordan form of W ⊗ M as follows.

Lemma E.2. The Jordan decomposition of W ⊗ J is given by W ⊗ J = ˜ P ˜ J ˜ P -1 where ˜ J is a block diagonal matrix with blocks

and ˜ P is a diagonal matrix obtained by concatenating ˜ P j,l = diag ( { w -n +1 j } k l n =1 ) .

Proof. As J = ⊕ m l =1 J l , we can focus on a single Jordan block. Fix l ∈ { 1 , . . . , m } . We have

hence, we can focus once again on a single block. Fix j ∈ { 1 , . . . , K } ; the Jordan decomposition of w j J l is given by ˜ P l = diag ( { w -n +1 j } k l n =1 ) and

To verify it, compute the ( n, m ) element

Since ˜ P l is a diagonal matrix, the only non-null entries are on the diagonal; therefore, i = n and k = m

and the only non-null entries of ˜ J l are when m = n or m = n +1 , hence

The thesis follows from assembling the direct sums back.

Lemma E.2 leads to the following result that fully characterizes the solution of (14) in terms of the generalized eigenvectors and eigenvalues of M and W .

Proposition E.3. Consider (14) with M = PJP -1 and W ⊗ J = ˜ P ˜ J ˜ P -1 , where ˜ J and ˜ P are given in Lemma E.2. The solution of (14) is

where the coefficients c i l 1 ,l 2 are given by

where { e i l 2 : l 2 ∈ { 1 , . . . , m } , i ∈ { 1 , . . . , k l 2 } } is the standard basis satisfying Pe i l 2 = ψ i l 2 ( M ) .

Proof. By Lemma E.2, the eigenvalues of W ⊗ M and the corresponding eigenvectors and generalized eigenvectors are

for l 1 ∈ { 1 , . . . , K } , l 2 ∈ { 1 , . . . , m } and i ∈ { 2 , . . . , k l } . Hence, by Lemma E.1, the solution of (14) is given by

where the coefficients c i l 1 ,l 2 are given by

E.1 Proof of Theorem 5.6

In the following, we reformulate and prove Theorem 5.6.

Corollary E.4. Let G be a strongly connected directed graph with SNA L ∈ R N × N . Consider the initial value problem in (2) with diagonal channel mixing matrix W ∈ R K × K and α = 1 . Then, for almost all initial values x 0 ∈ R N × K , the solution to (2) is HFD if

and λ 1 ( L ) is the unique eigenvalue that minimizes the real part among all eigenvalues of L . Otherwise, the solution is LFD.

Proof. Using the notation from Proposition E.3 and its proof, we can write the solution of the vectorized form of (2) as

As done extensively, we separate the terms corresponding to the frequency with minimal real part. This frequency dominates as the exponential converges faster than polynomials for t going to infinity. Consider the case λ K ( W ) glyph[Rfractur] ( λ 1 ( L )) < λ 1 ( W ) glyph[Rfractur] ( λ N ( L )) . As λ 1 ( L ) is unique, the product λ K ( W ) glyph[Rfractur] ( λ 1 ( L )) is the unique most negative frequency. Assume without loss of generality that λ K ( W ) has multiplicity one. The argument does not change for higher multiplicities as the corresponding eigenvectors are orthogonal since W is diagonal. Then, λ K ( W ) λ 1 ( L ) has multiplicity one, and we calculate vec( x )( t ) as

We can then write the normalized solution as

All summands, except the first, converge to zero for t going to infinity. Hence,

We apply Lemma D.2 to finish the proof for the HFD case. Note that ψ 1 1 ( L ) is an eigenvector corresponding to λ 1 ( L ) . The LFD case is equivalent. By Perron-Frobenius for irreducible nonnegative matrices, there is no other eigenvalue with the same real part as 1 -λ N ( L ) = λ 1 ( I -L ) .

Remark E.5. If the hypotheses are met, the convergence result also holds for L α . With the same reasoning, we can prove that the normalized solution converges to the eigenvector corresponding to the eigenvalue of L α with minimal real part. It suffices to consider the eigenvalues and generalized eigenvectors of L α . However, we do not know the relationship between the singular values of L α , where we defined the fractional Laplacian, and the eigenvalues of L . Hence, it is much more challenging to draw conclusions on the Dirichlet energy.

E.2 Explicit Euler

In this subsection, we show that the convergence properties of the Dirichlet energy from Theorem 5.6 are also satisfied when (2) is approximated via an explicit Euler scheme.

As noted in (11), the vectorized solution to (2) can be written as

when α = 1 . We thus aim to analyze the Jordan decomposition of L n for L ∈ C n × n and n ∈ N . Let L = PJP -1 , where J is the Jordan form, and P is a invertible matrix of generalized eigenvectors.

Consider a Jordan block J i associated with the eigenvalue λ i ( M ) . For a positive integer n , the n -th power of the Jordan block can be computed as:

We compute the n -th power of L as L n = ( PJP -1 ) n = PJ n P -1 , and we expand x 0 as

where { e i l : i ∈ { 1 , . . . k l } , l ∈ { 1 , . . . , m } } is the standard basis and Pe i l = ψ i l ( L ) are the generalized eigenvectors of L . It is easy to see that

As J n = ⊕ m l =1 J n l , we can focus on a single Jordan block. Fix l ∈ { 1 , . . . , m } , and compute

We can summarize our findings in the following lemma.

Lemma E.6. For any L = PJP -1 ∈ R N × N and x 0 = m ∑ l =1 k l ∑ i =1 c i l ψ i l ( L ) , we have

We proceed with the main result of this subsection.

Proposition E.7. Let G be a strongly connected directed graph with SNA L ∈ R N × N . Consider the initial value problem in (2) with diagonal channel mixing matrix W ∈ R K × K and α = 1 . Approximate the solution to (2) with an explicit Euler scheme with a sufficiently small step size h . Then, for almost all initial values x 0 ∈ C N × K the following holds. If λ 1 ( L ) is unique and

the approximated solution is HFD. Otherwise, the solution is LFD.

Proof. As noted in (11), the vectorized solution to (2) with α = 1 , can be written as

Consider the Jordan decomposition of L = PJP -1 and the Jordan decomposition of W ⊗ J = ˜ P ˜ J ˜ P -1 , where ˜ J and ˜ P are specified in Lemma E.2. Then,

Now, decompose x 0 into the basis of generalized eigenvectors, i.e.,

Now, consider the maximal frequency, i.e.,

Then, the solution vec( x )( nh ) can be written as

With a similar argument as in the proof of Theorem 5.6, we can then see that

glyph[negationslash]

where ψ 1 L 2 ( L ) is the eigenvector corresponding to λ L 2 ( L ) . Note that for almost all x , we have c 1 L 1 ,L 2 = 0 . Then ψ 1 L 2 ( L ) is also an eigenvector of I -L corresponding to the eigenvalue 1 -λ L 2 ( L ) . By Lemma D.2, we have that the approximated solution is (1 -λ L 2 ( L )) -FD.

We finish the proof by showing that L 2 = 1 if (15) is satisfied, and L 2 = N otherwise. First, note that either λ K ( W ) glyph[Rfractur] λ 1 ( L ) or λ 1 ( W ) glyph[Rfractur] λ N ( L ) are the most negative real parts among all { λ l ( W ) glyph[Rfractur] λ r ( L ) } l ∈{ 1 ,...,K } ,r ∈{ 1 ...,N } . Assume first that λ K ( W ) glyph[Rfractur] λ 1 ( L ) has the most negative real part, i.e., (15) holds. Then, define

and assume h < ε ‖ W ‖ 2 . Now it is easy to see that

which is equivalent to ( K, 1) = ( L 1 , L 2 ) . Hence, the dynamics are (1 -λ 1 ( L )) -FD. As (1 -λ 1 ( L )) is highest frequency of I -L , we get HFD dynamics. Similary, we can show that if λ 1 ( W ) glyph[Rfractur] λ N ( L ) is the most negative frequency, we get LFD dynamics. Note that for the HFD argument, we must assume that λ 1 ( L ) is the unique eigenvalue with the smallest real part. For the LFD argument, it is already given that λ N ( L ) has multiplicity one by Perron-Frobenius Theorem.

E.3 GCN oversmooths

Proposition E.8. Let G be a strongly connected and aperiodic directed graph with SNA L ∈ R N × N . A GCN with the update rule

where x 0 ∈ R N × K are the input node features, always oversmooths.

Proof. The proof follows similarly to the proof of Proposition E.7. The difference is that instead of (16), we can write the node features after t layers as

Now note that by Perron-Frobenius the eigenvalue λ N ( L ) with the largest absolute value is real and has multiplicity one. Then, max l 1 ,l 2 | λ l 1 ( W ) λ l 2 ( L ) | is attained at either λ 1 ( W ) λ N ( L ) or λ K ( W ) λ N ( L ) . Equivalently to the proof of Proposition E.7, we can show that the corresponding GCN is 1 -λ N ( L ) -FD. Now 1 -λ N ( L ) = λ 1 ( I -L ) , and λ 1 ( I -L ) -FD corresponds to LFD, hence the GCN oversmooths.

F Appendix for the Cycle Graph Example

Consider the cycle graph with N nodes numbered from 0 to N -1 . Since each node has degree 2 , the SNA L = A / 2 is a circulant matrix produced by the vector v = ( e 1 + e N -1 ) / 2 . Denote ω = exp(2 πi/N ) , the eigenvectors can be computed as

asociated to the eigenvalue λ j = cos(2 πj/N ) . First, we can note that λ j = λ N -j for all j ∈ { 1 , . . . , N/ 2 } ; therefore, the multiplicity of each eigenvalue is 2 except λ 0 and, if N is even, λ N/ 2 . Since the original matrix is symmetric, there exists a basis of real eigenvectors. A simple calculation

shows that glyph[Rfractur] v j and glyph[Ifractur] v j , defined as

are two eigenvectors of the same eigenvalue λ j . To show that they are linearly independent, we compute under which conditions

We note that the previous condition implies that for all n / ∈ { 0 , N/ 2 }

Suppose a, b = 0 , then it must be

glyph[negationslash]

which is equivalent to

The left-hand side is always an integer, while the right-hand side is an integer if and only if b = 0 . This reduces the conditions to

which is true if and only if a = 0 . Consider now an even number of nodes N ; the eigenspace of λ N/ 2 = -1 is

hence, the maximal eigenvector of I -L guarantees homophily 0 . Consider now a number of nodes N divisible by 4 ; the eigenspace of λ N/ 4 = 0 has basis

Their sum is then equivalent to

hence, the mid eigenvector of L guarantees homophily 1 / 2 . A visual explanation is shown in Figure 4.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose two novel Fractional Graph Laplacian (FGL)-based neural Ordinary Differential Equations (ODEs): the fractional heat equation and the fractional Schrödinger equation. These solutions provide enhanced flexibility in the convergence of the Dirichlet energy and make the exponent of the fractional graph Laplacian a learnable parameter. This allows the network to adaptively decide the optimal exponent for a specific task and graph. The fractional graph Laplacian operator generalizes the Laplacian operator. The experimental results highlight the improvements of using fractional graph Laplacians, but the benefit is limited.\n\nSTRENGTHS:\n1. The proposed FGL Neural ODE can be $\\lambda-FD$ which extends the Neural ODE-based GNNs that are limited to being either LFD or HFD.\n2. This paper generalizes the concepts related oversmoothing from undirected graphs to directed graphs.\n\nWEAKNESSES:\n1. The application of the fractional graph Laplacian operator results in the loss of the graph's sparse property, transforming it into a dense even complete graph.\n2. The paper's primary contribution appears to target oversmoothing issues, as suggested by the title. However, the evidence supporting the claim that the fractional graph Laplacian can mitigate oversmoothing in GNN is insufficient.The paper posits that the theory and experimental results presented counteract the issue of oversmoothness. However, based on [1], oversmoothness can still occur even when node features exist within an invariant space that has a rank greater than one. In this context, the paper's conclusions about oversmoothness mitigation defined using Dirichlet energy appear quite constrained. How would the proposed model fare with 256 layers stacked? How does it measure against the ODE-based methods (some are also designed to address oversmoothing), such as the recent work like GRAND[2], GraphCON [3], GRAFF [4], GRAND++[5] and GREAD [6]?\n3. While the paper introduces concepts pertaining to directed graphs, the significant Theorem 5.3 only applies to undirected graphs, and Theorem 5.5 is solely restricted to the standard Laplacian where $\\alpha=1$. Although it seems that the Fractional Graph Laplacian (FGL) may bridge the gap between homophilic and heterophilic graphs, the benefits of using FGL for directed graphs aren't clearly established.  \n4. The authors introduce several definitions, notably the Dirichlet energy for a directed graph (equation 1), but fail to provide solid justification for their choice. For instance, one could question why the energy is not defined in the following way:\n$$\\sum_{i, j=1}^N a_{i, j}\\left(\\left\\|\\frac{\\mathbf{x}_i}{\\sqrt{d_i^{\\text {in }}}}-\\frac{\\mathbf{x}_j}{\\sqrt{d_j^{\\text {in }}}}\\right\\|_2^2 +\\left\\|\\frac{\\mathbf{x}_i}{\\sqrt{d_i^{\\text {out }}}}-\\frac{\\mathbf{x}_j}{\\sqrt{d_j^{\\text {out }}}}\\right\\|_2^2\\right).$$\nThe experimental results provided do not sufficiently substantiate these definitions or the theoretical assertions made in the paper.  \n5. Very poor experimental results: While the paper focuses on directed graph Laplacian, the enhancements it introduces seem to be quite narrow, especially when considering Table 2, the only table for directed graphs. The inclusion of several definitions and theories seems to add complexity to the paper without providing a substantial contribution to the GNN community. The authors also need to include the SOTA results on those datasets.  \n6. The usage of complex numbers, as in equation (5), should be justified convincingly. While this may be commonplace in pure mathematics or physics, it needs to be shown why such a model is necessary for tackling oversmoothness in GNNs. Recent developments, such as Pytorch's support for complex numbers, do not automatically warrant their use in all contexts. If equation (5) is utilized, it only increases the model's complexity and memory consumption. Therefore, a robust justification is needed.    \n\n[1] Oono, Kenta, and Taiji Suzuki. \"Graph neural networks exponentially lose expressive power for node classification.\" arXiv preprint arXiv:1905.10947 (2019).\n\n \n\n[2] Chamberlain, Ben, et al. \"Grand: Graph neural diffusion.\" International Conference on Machine Learning. PMLR, 2021.\n\n \n\n[3] Rusch, T. Konstantin, et al. \"Graph-coupled oscillator networks.\" International Conference on Machine Learning. PMLR, 2022.\n\n \n\n[4] Di Giovanni, F., Rowbottom, J., Chamberlain, B. P., Markovich, T., and Bronstein, M. M. (2022). Graph Neural Networks as Gradient Flows: Understanding Graph Convolutions via Energy. arXiv: 2206.10991 [cs, stat].\n\n \n\n[5] Thorpe, Matthew, et al. \"GRAND++: Graph neural diffusion with a source term.\" International Conference on Learning Representations. 2021.\n\n \n\n[6] Choi, Jeongwhan, et al. \"GREAD: Graph Neural Reaction-Diffusion Networks.\" (2023).\n\nQUESTIONS:\n1. In Table 1, could you provide the learned values for $\\lambda_{K}(W)$ and $\\lambda_{1}(W)$ for each dataset? Moreover, could you offer a detailed analysis of how these results relate to Theorem 5.3? Also, all the learned $\\alpha$ values in Table 4 are greater than 0, so according to part(i) of Theorem 5.3, the FGL-ODE is still either L-FD or H-FD. Could you clarify under which circumstances the proposed $\\lambda$-FD could be applied?\n2. The paper doesn't clearly illustrate how the fractional Laplacian operator mitigates the oversmoothing issue, both from theoretical and experimental perspectives. More in-depth explanations and experiments are needed.\n3. Is the FGL-based Neural ODE applicable to large-scale graph datasets, such as those in references [1] and [2]?\n4. The results for the Cora dataset presented in Table 1 are inferior to MLP. Could you also display the results for the homophilic dataset using the data split method from reference [5]?\n5. Table 1 seems to miss important baselines for heterophilic graphs, such as ACM-GNN[3] and ACMP[4]. It would be beneficial to include these.\n6. The paper lacks ablation studies. Could you provide the node classification results for $\\alpha=1$ in equations (4) and (5) of FLODE? As the main contribution appears to be the FGL, are there any ablation studies that use layer-wise FGL-based GNNs instead of the FGL Neural ODE?    \n7. Other questions please refer to weakness part.   \n\n  \n\n\n\n\n\n[1].Hu W, Fey M, Zitnik M, et al. Open graph benchmark: Datasets for machine learning on graphs[J]. Advances in neural information processing systems, 2020, 33: 22118-22133.  \n[2].Lim D, Hohne F, Li X, et al. Large scale learning on non-homophilous graphs: New benchmarks and strong simple methods[J]. Advances in Neural Information Processing Systems, 2021, 34: 20887-20902.  \n[3].Luan S, Hua C, Lu Q, et al. Revisiting heterophily for graph neural networks[J]. arXiv preprint arXiv:2210.07606, 2022.  \n[4].Wang Y, Yi K, Liu X, et al. ACMP: Allen-Cahn Message Passing for Graph Neural Networks with Particle Phase Transition[J]. arXiv preprint arXiv:2206.05437, 2022.  \n[5]. Shchur O, Mumme M, Bojchevski A, et al. Pitfalls of graph neural network evaluation[J]. arXiv preprint arXiv:1811.05868, 2018.\n\nLIMITATIONS:\nYes,  the authors addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper extends the notion of Dirichlet energy to define oversmoothing in directed graphs. It introduces a fractional Laplacian operator that is used to create graph neural ODE architectures. The resulting designs are non-local and can alleviate oversmoothing. The paper provides empirical results (accuracy) on nine real-world benchmarks for node classification.\n\nSTRENGTHS:\n* Overall, the paper is well-written and easy to follow. Also, the proposed framework (FLODE) is simple;\n* The proposed method works relatively well for both directed and undirected graphs (flexibility).\n\nWEAKNESSES:\n- Novelty: I rank the novelty as moderate. First, directed symmetrically normalized laplacian has been previously used. Moreover, it seems that the advantages come mainly from the choice of diffusion operator. However, it is clear that the choice of diffusion operator impacts oversmoothing, and we can design, for instance, high-pass (or non-local) graph filters to alleviate oversmoothing. Last, there is no novelty in the evaluation setup.\n- Theory: Some theoretical results focus on the properties of the fractional Laplacian, which appears only complementary to the paper. As a result, the paper does not discuss empirical results in light of the theoretical ones.\n- Experiments: The empirical results are not strong. Except for Squirrel, the performance gains (over the second-best baseline) do not seem to be statistically significant. Also, the proposed model is the best-performing model only in 4 out of 9 datasets. In addition, the paper does not consider large-scale datasets from OGB, for instance.\n\nQUESTIONS:\n**Questions and comments**\n\n1. Why is it called Fractional graph laplacian if it is based on the decomposition of the normalized adjacency (SNA)?\n2. Do the authors have any intuition to explain why the proposed method works so well on the Squirrel dataset?\n3. Some parts could be improved for precision and clarity, e.g.,\n    - In the paragraph 'Homophily and Heterophily', the paper uses $i \\in V$ to denote the distribution used in the expectation. What does it mean (e.g., uniform dist. over the nodes)?\n    - In the introduction, the paper says: \"a GNN that minimizes the Dirichlet energy is expected to perform well on homophilic graphs...\". However, there is no dependency on the GNN --- x is defined as initial node features.\n5. Running the ablation study on Appendix F3 to more datasets would be interesting and helpful to validate some intuitions behind the proposal.\n6. There is a vast literature on methods for tackling oversmoothing in GNNs (DropEdge, etc.), enabling such approaches for heterophilic datasets. Shouldn't these approaches also be considered as baselines for comparison?\n7. Have you tried employing an NN instead of using just a channel mixing matrix?\n\nLIMITATIONS:\nThe authors discuss the limitations of the proposal, mainly mentioning the computation cost associated with SVD.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

comments: SUMMARY:
The authors consider the problem of classification on attributed graphs with a geometrical approach. They introduce a fractional graph Laplacian for undirected and directed graphs. They generalize the notion of Dirichlet energy to directed graphs. They study the ODEs based on this Laplacian and prove that their solutions converge to high-, low- or middle-frequency patterns, for undirected graphs and for directed graphs in a particular case. Finally they implement these ODEs as GNNs and assess their performances.

STRENGTHS:
I do not know well this topic and can hardly evaluate how significant are the improvements this article brings.

WEAKNESSES:
To my understanding a main point of this article is fractional Laplacian for directed graphs, but theorem 5.5 is restricted to the SNL.

QUESTIONS:
On formatting: in the current state of the manuscript many intra-links are broken. Some figures of interest for understanding are not in the main text (I did not look at them). Maybe for table 1 the authors could use font formatting (bold, italic, ...) instead of colors.

L. 245 « In accordance with the results in Section 5, we select W as a diagonal matrix. » could the authors develop?

Part 6, for the directed SBM, what are the features?

Out of curiosity: what happens if W is untied, i.e. taken time-dependent or different at each layer? Does one have the learned alpha small (or negative) for heterophilic datasets?

LIMITATIONS:
The authors address a main limitation of their model, its computational complexity. They may conduct an experiment showing that truncating the SVD is correct.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

comments: SUMMARY:
The authors propose a neural ODE (FLODE) that uses a fractional graph Laplacian as an alternative to GNNs, which famously suffers from oversmoothing after a few layers. The heat equation $x'(t) = -L^{\alpha}x(t)W$ is shown to possess nice qualities wrt Dirichlet energy such that it does not always end up with a low frequency solution (i.e. oversmoothing as $t \to \infty$). Several experiments are done with synthetic, real, directed, and undirected graphs, and the proposed method are shown to perform comparably with existing method.

STRENGTHS:
Even though I am not an expert in neural-ODE based GNNs, I found the paper clearly written and easy to follow. The strength of this paper is in its theoretical analysis of the spectral values of their solutions. Based on a cursory reading of Di Giovanni et al (2022), and given that the learned $\alpha$s are often not 1 (Appendix Table 4), this does not seem to be a trivial extension of existing work.

WEAKNESSES:
Please see questions below.

QUESTIONS:
1. If Film is a heterophilic graph, I expected the learned exponent to be closer to 0, which is the case with Squirrel and Chameleon. Do the authors have a hypothesis about why this might be the case?

2. Similarly, do the authors have a hypothesis about why FLODE was not in the top three performing models for Film, Pubmed and Cora?

3. Have the authors actually inspected the eigenvalues and singular values of their models and compared them to the theoretical results?

4. I recommend using boldface and italics in addition to color in Table 1. In Figure 3, the line for FLODE should be thicker or have a different line style.

LIMITATIONS:
Yes.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-07