Neural Fractional Attention Differential Equations
Authors: Qiyu Kang
Wenjun Cui
Xuhao Li
Yuxin Ma
Xueyang Fu
Wee Peng Tay
Yidong Li
Zheng-Jun Zha
Venue: NeurIPS 2025 poster ()
Keywords: neural differential equations
Primary area: general_machine_learning

Abstract: The integration of differential equations with neural networks has created powerful tools for modeling complex dynamics effectively across diverse machine learning applications. While standard integer-order neural ordinary differential equations (ODEs) have shown considerable success, they are limited in their capacity to model systems with memory effects and historical dependencies. Fractional calculus offers a mathematical framework capable of addressing this limitation, yet most current fractional neural networks use static memory weightings that cannot adapt to input-specific contextual requirements. This paper proposes a generalized neural Fractional Attention Differential Equation (FADE), which combines the memory-retention capabilities of fractional calculus with contextual learnable attention mechanisms. Our approach replaces fixed kernel functions in fractional operators with neural attention kernels that adaptively weight historical states based on their contextual relevance to current predictions. This allows our framework to selectively emphasize important temporal dependencies while filtering less relevant historical information. Our theoretical analysis establishes solution boundedness, problem well-posedness, and numerical equation solver convergence properties of the proposed model. Furthermore, through extensive evaluation on tasks such as fluid flow, graph learning problems and spatio-temporal traffic flow forecasting, we demonstrate that our adaptive attention-based fractional framework outperforms both integer-order neural ODE models and existing fractional approaches. The results confirm that our framework provides superior modeling capacity for complex dynamics with varying temporal dependencies. The code is available at \url{https://github.com/cuiwjTech/NeurIPS2025_FADE}.

Neural Fractional Attention Differential Equations
Qiyu Kang",, Wenjun Cui Xuhao Li Yuxin Ma',
Xueyang Fu', Wee Peng Tay², Yidong Li", Zhengjun Zha
University of Science and Technology of China Shanxi University 'Anhui University Nanyang Technological University BBijjing Jiaotong University
Abstract
The integration differential equations with neural networks has created powerful tools for modeling complex dynamics effectively across diverse machine learning applications. While standard integer-order neural ordinary differential equations (ODEs) have shown considerable success, they are limited in their capacity model systems with memory effects and historical dependencies. Fractional cal- culus offers mathematical framework capable of addressing this limitation, yet most current fractional neural networks use static memory weightings that cannot adapt to input-specific contextual requirements. This paper proposes generalized neural Fractional Attention Differential Equation (FADE), which combines the memory-retention capabilities of fractional calculus with contextual learnable tention mechanisms. Our approach replaces fixed kernel functions in fractional operators with neural attention kernels that adaptively weight historical states based on their contextual relevance to current predictions. This allows our framework to selectively emphasize important temporal dependencies while filtering less relevant historical information. Our theoretical analysis establishes solution boundedness, problem well- polsedossdssss and numerical equation solver convergence properties of the proposed model. Furthermore, through extensive evaluation tasks such fluid flow, graph learning problems and spatio-temporal traffic flow forecasting, we demonstrate that our adaptive attention-based fractional framework outper- forms both integer-order neural ODE models and existing fractional approaches. The results confirm that our framework provides superior modeling capacity for complex dynamics with varying temporal dependencies. The code is available at /etps://giteee cwwm cuixjTeech/MeuurIPS2002...
Introduction
Neural differential equations have emerged as powerful tools for modeling continuous-time dynamics in various machine learning tasks. By integrating neural networks with differential equations, these models can effectively learn the underlying dynamics of complex systems directly from data. The most representative model in this category is the integer-order neural Ordinary Differential Equation (ODE) [1-3], which formulates the evolution of hidden states as an integer-order ODE system. This approach has demonstrated success in various machine learning tasks, including time series forecasting [4-7), graph representation learning [8-10], physics modeling [11-13), adversarial robustness [14-16], and generative modeling [17, 18).
Despite their success, traditional neural ODEs based on integer-order differential equations exhibit inherent limitations in modeling long-range dependencies and memory effects. These stem from the Markovian nature ODEs, where the future state depends solely on the current state rather than on
First two authors contributed equally to this work. i "Coresponnence to: Xuhao Li diixh@ahuedd..com
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

the trajectory's history [19]. Such locality constrains the model's ability to represent complex temporal patterns essential in many real-world tasks. To address these limitations, researchers have explored fractional-order differential equations (FDFs) as an alternative general mathematical framework. FDEs extend the classical derivative operator d introducing a non-local memory term that naturally integrates past information. Thanks this non- local property, FDE-based models can capture long-term dependencies, making them well-suited for systems with memory or long-range interactions. They have found applications across diverse fields, including anomalous diffusion processes [20]. viscoelastic materials [21], electromagnetic wave propagation [22], and financial time series analysis [23]. In physics-informed machine learning, fractional physics-informed neural networks (fPINNs) [24] enforce underlying physical laws via FDEs, spawning further developments [25, 26]. Furthermore, recent works have incorporated FDEs into neural network architectures, giving rise to fractional neural ODE models [19, 27-30J. For instance, [19, 27] propose fractional graph neural ODE models with fractional diffusion and oscillator mechanisms to propagate information over graphs, demonstrating improved task performance and increased robustness compared to their integer-order counterparts [8, 9]. The work by [31] on generative fractional diffusion models has indicated that the fractional method can achieve greater pixel-Ievel variation and overall image fidelity.
d de to non-integer orders denoted des thereby
However, most existing fractional operators and derived neural models employ fixed fractional kernels that assign predefined weights to historical states, thereby limiting adaptability to diverse data patterns. For example, [19, 27, 29] use the Caputo fractional operator D with fixed kernel (f (f-T)-a (where a is the fractional derivative order), which applies a non-adaptive weighting to the trajectory. The main challenge in developing effective fractional neural models lies in designing appropriate kernel functions that can adapt to the complex dynamics of different systems. Fixed kemelss with constant weights might not be optimal for capturing the varying importance of historical states across different features and time points. Several works have adopted more flexible frameworks. For example, the work [32] proposes the generalized vi-Caputo fractional derivative with the kernel in the form of (vb(1) - (bTT) -a where (b(.) is an increasing continuously differentiable function. [33] proposes neural variable-order FDE network using the variable-order Caputo fractional derivative
D (r(t) This approach assumes a kernel of the form (t T) ()-)((((T)) where the derivative order att, (()) depends on hidden features x(r). capturing complex feature-updating dynamics with enhanced flexibility. However, those kernel functions still have limited capacity and does not account for correlations between historical states. The work [7] combines the modeling capabilities of integer-order neural ODEs with an attention mechanism applied to temporal partitions, capturing the dynamic changes of continuous-time systems. This is distinct from our research, wherein we focus on incorporating fractional operators for updating hidden features, modeled as memory-inclusive
dynamical process.
To overcome the limitations of fixed kernels, this paper introduces the generalized fractional attention differential equation (FADE). FADE generalizes fractional neural ODEs by incorporating learnable attention mechanisms [34, 35] directly into the fractional operator's 's kernel function. Instead of relying on predefined, static/uncorrelltted weightings for historical states, FADE employs neural attention kernels that adaptively assign weights based on the contextual relevance of past information to the current system dynamics. This allows the model learn feature-dependent representations of history, selectively emphasizing salient temporal dependencies while filtering less relevant information, leading to more expressive and flexible modeling.
Main contributions. This paper introduces a new continuous neural network framework based on generalized attention-basee fractional operator kernels. Our key contributions include:
We propose FADE, a novel fractional neural equation framework that integrates FDEs with learnable neural attention kernels. This enables adaptive, context-aware memory mechanisms that can selectively emphasize relevant historical dependencies while filtering less important temporal information, advancing beyond existing approaches with static/uncorrelated memory weightings.
We provide rigorous theoretical foundations for FADE by analyzing essential kernel properties, including boundedness under both singular and nonsingular cases. We establish the well-posedness of the resulting neural integral equations using Banach fixed-point arguments, ensuring solution uniqueness uniqueness under appropriate conditions. Additionally, we present convergence analysis for the numerical discretization solver, ensuring the framework's practical implementability.

---

We conduct extensive experiments on fluid flow, graph learning problems, spatio-temporal traffic forecasting, urban population mobility and biological neural spike trains tasks, demonstrating that FADE consistently outperforms integer-order neural ODE models and existing fractional approaches, confirming its superior capacity for modeling complex dynamics.
Related Work. Our work mainly focuses on Neural Differential Equations and neural network attention mechanisms. The related work is presented below. For a detailed discussion, please refer to Appendix A.
Neural Differential Equations Neural Differential Equations (NDEs) unify neural networks and differential equations for modeling continuous dynamics. Neural ODEs (NODEs) [2] and their variants [36, 37] learn hidden state evolution by parameterizing ODEs. Yet, these use integer-order calculus, limiting their ability to capture memory effects and historical dependencies. To address this, FDE-based models [19, 33] have consequently been developed and widely applied. FRONDD [19] employs FDEs in graph learning, alleviating oversmoothing. Subsequently, NvoFDE [33] and DRAGON [38] were introduced capture the distinct memory characteristics inherent in FDEs, but still rely on fixed kernels with static historical weights, Our work advances this by designing adaptive kernels within the FDE framework for richer spatio-temporal modeling.
Attention Mechanisms in Neural Networks Attention mechanisms are central to modern deep learning. allowing models focus on informative inputs. The Transformer [34] introduced self- attention, enabling efficient modeling of long-range dependencies across domains, In continuous- time models, attention mechanisms have been integrated with NDEs to enhance their expressive- ness, allowing the model to selectively focus on relevant parts of the input sequence [39]. Attention mechanisms have also been applied to graph neural networks, resulting in Graph Attention Net- works (GATs) [40]. However, the integration of attention with fractional-order models remains largely underexplored. Attention-based kernels in fractional NDEs offer a promising direction toward more flexible and adaptive modeling.
2 Preliminaries and Motivations
A distinctive strength of fractional calculus lies in its ability to model systems with non-local interactions, where the future state of a system depends significantly on its historical trajectory through specialized kernel functions. This section provides concise overview of fractional calculus and illustrates its relationship to traditional calculus. Additionally, we present a brief introduction to both integer-order and fractional-order neural ODEs foundational concepts for our work.
2.1 Fractional Calculus
We begin with review of traditional calculus before introducing classical fractional-order integrals and derivatives. Additionally, we present the recently developed generalized fractional derivative for- mulation that incorporates supplementary function (-(-) within the kernel. For a more comprehensive exploration of fractional calculus theory, please see Appendix B.
2.1.1 Traditional Calculus
Consider an d-dimensional function x(t) Rd with respect to (w.r.t)) time t. The traditional first-order derivative, which quantifies the instantancous rate of change, is defined as:
dx(t) lim A4-10
x(t + At) x(t)
dt
At
Let J denote the classical integration operator over an interval [a, b). defined as Jx(t)= = f x(r) dr.
For any positive integer m E N we define the iterated integral operator J by J' = J and
Jm = J- for m 2 Equivalently, applying integration by parts yields [41][Lemma 1.1.]:
Jm (() = (n - 1)1 J (t- (1- (--7)=--xp ((() dr with E N
(2)
2.1.2 Fractional Operators
The concepts of fractional-order integrals and derivatives generalize their integer-order counterparts.
The commonly used Riemann-Liouvill fractional integral [41], denoted by RL jo for positive real
3

---

order a E RR, is defined as
T(a) [. (t-)))-ix(T)dd
1
RL Ja x(t
(3)
where ((a) the gamma function, Unlike the integer-order m traditional integrals, the order a can take any positive real value.
The literature offers various definitions for fractional integrals and derivatives, e.g-, Riemann- Liouville, Hadamard, Erdělyi-Kober, and Caputo [42], which differ in how they weight historical trajectories. We mainly consider the Caputo fractional derivative because it preserves the same initial conditions as traditional integer-order differential equations. Throughout this work, we focus on the left-sided fractional derivative; the corresponding results for the right-sided derivative can be obtained analogously with appropriate modifications. We consider the case where the fractional order (0, 1]. and the formulation for any a > is presented in Appendix B.
Definition 1 (Classic Caputo Fractional Derivative). The Caputo fractional derivative of order (0, for a function xt) over an interval (a, bj is defined as follows [41]:
f. (t-r))ax((1)) (x)
*D+xx(t) T(1 a))
(4)
where x' (r) is the first-order derivative of x(T)-
Remark 1. When a = 1, the Caputo fractional derivative DD coincides with the standard first- order derivative d.
di (see [41] (Theorem 7.1)). Therefore, (4) generalizes (1). It is evident from (4) that the fractional derivative incorporates the historical states of the function x(I) via the stationary power-law kernel (t-T)-= highlighting its memory-dependent nature. In contrast, the integer-order derivative only represents the local rate of change of the function.
Definition (Variable- -rader Caputo Fractional Derivative). When the fractional order a is allowed vary with time t. the generalized variable-orde Caputo fractional derivative of x(t) is defined as:
DDll x(1) == r(1-a(t)) ( (t-r)--att)((())) dr, 0 < att)
1
0<a(t)= 1.
Remark 2. Compared to the classic Caputo fractional derivative (4), the variable-order derivative (5) employs a non-stationary power-law kernel. This allows it to dynamically adjust its memory structure, enabling the characterization of more complex processes than (4) [43, 44]. In more general settings, a(t) can be extended to depend on other parameters, e.g., att, x(t)).
Definition 3 (t)-Caputo Fractional Derivative). Let 1b E (la be a continuously differentiable scalar function such that rb(t) is increasing on [a, b). The ap-Caputo fractional derivative of x(f of order a E (0, 1) is defined by [32]:
C DD x(1) r(1-a) f( (()(0)~(())-0
dr.
(6)
Remark 3. When (t) t, the th- Caputo fractional derivative (6) reduces to the classic Caputo fractional derivative (4). Therefore, the vh- Caputtt fractional derivative is also natural generalization of (4). The main difference is that the vb-Caputo definition utilizes a generalized kernel (v(()- (T))) -a, where the function ap()) offers greater flexibility in weighting past values of the function.
Observation and Motivation: The preceding review of various Caputo fractional derivatives highlights their defining feature: the use of distinct weighting kernels, which can be static or designed to vary dynamically with time t. However, since these kernels depend only on f and and not on past states x(T), they cannot adjust their weighting based on the states correlation in the trajectory. In this paper, we propose to overcome this limitation by developing a more generalizable learnable attention kernel that extends beyond the capabilities of the above approaches, enabling memory weightings based on both temporal information and the contextual relationships between past and current states.
2.2 Integer- and Fractional-Order Neural ODEs
In an integer-order neural ODE model, the process of transforming an initial feature vector x(0) = Xy R. into an output feature vector x(T) R² is dictated by the following first-order ODE:
dx(t) dt
fo(t, x(t)
(7)
4

---

In this equation, the neural network function fe. which is parameterized by andmaps from [0, oo) x Rd to RR defines the learnable dynamics responsible for updating the hidden features. The trajectory x(t) illustrates the continuous change of the system's hidden state over time.
When equation (7) is solved using the Euler discretization method, the update rule takes the form x(t + Af) = x(t) + (t,, This structure shows strong parallel to that of ResNet with skip connections [45]. where the variable can be seen as analogous to network depth or layer count. Furthermore, when addressing tasks with temporal component, the dimension t can directly correspond to the actual time dimension. such cases, the neural ODE can be interpreted as a continuous-time recurrent neural network, as explored in studies such as [46-48].
Analogously, a fractional-order neural ODE model [19, 29), feature dynamics are governed by: ((t),, 0 < S 1.
"Dox(('
(8)
In this formulation, fe characterizes the trainable fractional derivatives governing the hidden state dynamics. Starting from the initial condition x() = Xo. the system state x(t) evolves up to a predefined terminal time T. Computation of x(T) is accomplished using forward fractional differential equation solver, such as the fractional explicit Adams Bashforth-Mouule method [49]. Similar to integer-orf order models, the dimension t can serve as continuous analog discrete layer indices or align directly with actual time dimensions in temporal tasks. However, rather than employing simple skip connections as integer-order ODEs, fractional-order ODEs exhibit inherently dense connectivity patterns [50, 19].
Remark 4. Both the integer- and fractional-orde neural ODEs presented in (7) and (8) can be equivalently reformulated as integral equations, as we will demonstrate in Section 3.1. Motivated by this equivalence, we propose the generalized FADE framework, which not only encupsulates these formulations as special cases but also exlends to scenarios where the fractional derivatives in (8) replaced by variable-order or ta-Caputo fractional derivatives shown in (5) and (6). This unified framework offers unprecedented flexibility in modeling complex continuous dynamics with adaptive memory mechanisms.
flexiibi
3 Generalized Neural Fractional Attention Differential Equation
In this section, we introduce the FADE framework, novel approach that generalizes fractional- and integer-order neural ODEs by incorporating learnable attention mechanisms directly into the fractional operator's kernel function. We first present the comprehensive framework formulation in Section 3.1, followed by a rigorous analysis of its theoretical properties in Section 3.2. Subsequently, we develop the numerical discretization solver necessary for implementing FADE and provide detailed convergence analysis in Section 3.3. This systematic development establishes both the theoretical foundations and practical implementability of our proposed framework for modeling complex dynamics with adaptive memory mechanisms. All proofs are provided in Appendix C.
3.1 Framework
While it is possible to directly incorporate attention mechanisms that generalize the kernel functions presented in the Caputo fractional derivative definitions (4) to (6), we instead opt for a more elegant presentation by first transforming the integer-order and fractional-order neural ODEs in Section 2.2 into equivalent integration equations with different kernels. Specifically, the integer-order neural ODE model (7) can be equivalently expressed as:
dx(t) dt
= fo(t,x(t)) = x(t) x(a) L' fo(t,x(T))d
(9)
with initial input x(a). Similarly, based on the relationship between fractional-order ODE and its integral form [41][Lemma 6.2.], we have:
C " = fo(t,x(t) - x(t)= L' [x (x=t)a-1
T(a) fol (()) dr.
(10)
The neural variable-order fractional differential equation networks presented [33] adopt the following variable-order differential and integral equations:
x(()) x(t) ((t) and x(t) = x (a) + /' [t-r)o(tx(0)-1
C (a((xx(0)
T(a(lx((1)) fo(t,x(1)) dr. (11)
5

---

Furthermore, if we consider the vi-Caputo fractional derivative (6) with (t) being learnable function w.r.t. t, we can propose the following di-fractional-rdde differential and integral equations:
DD - So(l, So(, - x(1) - x(a) + [
V (t)(((())) - ((t) So(1, Ia(1,x(T))dr, (12) ((a)
where the equivalence can be established from [32].
To gain further insight, we rewrite the aforementioned integral equations in a unified formulation as:
x(t) = x(aa)+ + S. (((e,.(())))))((((()))
(13)
where (t, x, kernel function, with examples (9) to (12). This motivates us to study this integral equation and unveil its underlying relation to neural network architecture design. We find that the integral equation is essentially continuous form of the following discrete iterative relation:
n-1
x(tx,) x(to)
K ((t.))((tn)) dr x fo(t,, x(t,),
(14)
where (ty 1 is grid of [a, b] and x(tz). 0 S i n 1, are sampled points from the trajectory x(t). It is observed that this discrete iterative relation shares similar structure with the attention mechanism in Transformers [34] if we consider (x(t)))7=o input, {fo(t), {(,))) as values and f4++1 K K(n, 1, (( dr as weights derived from query and key.
From the discussions above, natural relationship between the integral equations and the attention mechanism in Transformers established. However, this direct analogy highlights several limitations. The kernels in (9) and (10) utilize fixed, non-learnable weights. While the kernel in (11) allows its fractional order (t,, x(t) to respond to the current time t and state x(t). its structure does not inherently learn complex correlations between different states along the trajectory. Similarly, the kernel in (12), despite incorporating learnable temporal features through (t),, primarily accounts for temporal differences and does not adapt its weighting based the evolving state vectors (e.g.. x(T) or xf).. These factors can potentially restrict their modeling capacity and performance in practical applications.
Inspired by this insight, we propose the FADE framework, which is based on the following neural integral equation featuring an attention kernel:
x(t) = x((a)= + S. ((t,:. dr,
(15)
where K(t, T, x(f(, x(f),x(r)) 2 0. This formulation also corresponds to proper masking in self-attention layers in the Transformer decoder [34], ensuring that each step only attends to previous steps and thereby preserving the causal, auto-regressive nature of the integral equation.
Remark 5. Instead of relying on static or uncorrelated weightings for historical states, FADE employs neural attention kernels that adaptively assign weights by assessing the contextual relevance of past information to the current system dynamics. Compared to the specific kernels presented in (9) (12), FADE provides a unified formulation where the kernels are constructed bused on both temporal information and the contextual relationships between past and current states. This approach allows the model selectively emphasize salient temporal dependencies while filtering out less relevant information, leading to more expressive and flexible modeling capabilities than (9) to (12).
3.1.1 Attention Kernel Examples
In contrast to the predefined kernels from (9) to (12), which possess limited adaptivity, we now introduce several flexible forms of the attention kernel K (t, T, x(t) x(T)) that explicitly leverage attention mechanisms. Scaled Dot-Product Attention: Motivated by [34], the kernel K can be set as
x x(1) x(T)
x(t,
(16)
Vd

---

where d is the dimension of x(t). a is the softmax function. In the continuous setting, it implies a normalization of attention weights across the relevant range of for a given t. Specifically, exp(s(t.r))
vd, is the score, then T)) could be interpreted T exp(s(t,s")dde to ensure weights integrate to 1. In discretized numerical solvers, the softmax is applied over the set of sampled past states fx (T,)}-
stt, 7) x(t) x(r)
Fractional Attention Kernel: Motivated by the structure of fractional integral operators seen in (10) to (12), the kernel K can be set
(0(((()-((7)>>>))) ((x(t),x(T)))
((t,, (((,)x(())))))))))))))
(17)
T(a)
Here, the term xt ((()) introduces fractional memory component and be regarded as I(a)
positional encoding weighting. (t) is a learnable, monotonically increasing function of time, which reparameterization of the time axis, allowing the model to learn problem-
can be interpreted specific memory decay rates. The parameter represents the fractional order. K (x(t), x(r)) is itself
an attention function that depends the states x(t) and x(T). This K can be, for instance, the scaled dot-product attention described above or an additive attention mechanism [51]. This hybrid kernel thus allows the model to leverage both the structured memory of fractional operators and the dynamic, input-dependent weighting of attention mechanisms.
Position Augmented Attention Kernel: A common method for incorporating positional information, introduced in the original Transformer [34], involves adding positional encodings directly to the input representations. For given position t, the sinusoidal positional encoding vector PF(t) is defined in [34][Sec 3. In this approach, the state vectors x(t) and x(т) are first augmented as (t) xt) + PE(() and x(7) x(r) + PÉ(T). They can then be projected using learnable linear transformation matrices, Wg for the query and W. for the key. The attention kernel is computed as:
(W (W))(1) (W,X(0)((Wxx(1)) (WxX(T))
K(t, xt), x())((()))
(18)
vn
The product of Wg and Wi can be initialized as a single learnable matrix in the implementation.
3.2 Well-posedness of FADE
We begin with properties of kernel integral. The attention kernel employed in this work depends explicitly on the system states x(t) and x(T). We analyze the properties of the integral operator:
T xxx(1)= = S. (,I,,, (((,,,)()))x()))x(()))))))
(19)
This integral encompasses several well-known aforementioned operators as special cases. These special cases result in linear, bounded operators that satisfy the semigroup property, crucial for deriving equivalent differential forms [41, 32]. Generally, the semigroup property does not hold for the attention kernels defined here, typically leading to nonlinear operators. Nonetheless, boundedness, which is essential for ensuring well-posedness remains valid. Below, we discuss an important singular scenario and leave other interesting cases in Appendix D. That is, we foucs the kernel that admits the decomposition
(t)(t) - ((T))) (T(()) ((x(t),x(T)), 0<a<l.
K X(t,T,x(t)xx
l(aa)
Lemma 1 (Boundedness of the Kernel Integral Operator (19)). Suppose that (t)) monotone increas- ing and K continuous and bounded, then In remains bounded and satisfies ||xx||| S C'|[x||
Now we are ready to consider well-posedness Ensuring well- poseedees (i.e., uniqueness and robust- ness) of neural integral equations is crucial for their practical reliability and theoretical robustness.
Theorem (Uniqueness and Stability). Suppose that K satisfying the Lipschitz condition, fo(T, x) is bounded and globally Lipschitz continuous in x. The integral equation (15) admits unique continuous solution in interval (a, a + e] some > Moreover, we have
||x(t) x(t)lla < Cl|x(a)= x(a)l12, a<t<b. b,
which implies that the solution is stable robust with respect to the perturbation of initial data.
7

---

3.3 Solving FADE
The integral equation (15) nonlinear equation which can be solved using linearized technique or iterative method. Here, we adopt the latter one. Taking ty and approximating the integral in (15) using the trapezoidal rule yields
x (t,))= (x(()), (x(())) = (to)
Σ k=
K
fo xt,, (tx)) h. (20)
For above nonlinear problems, we apply basic iteration method solve it. The procedure of this method and related analysis are discussed below.
Basic Iteration Method. Given an initial guess x()) (tz) x (t)-1), we iteratively compute x(L) (42) = (x(4-1) (1,)), where 2
Convergence Criterion. The convergence of the above iterative method relies on the following condition: If o Rd _d satisfies Lipschitz condition with a Lipschitz constant C' 1, then the iterative methods converge for any initial guess x( [a, bjd.
Given that the assumptions of Theorem are satisfied with appropriate generic constant C, can be directly verified that basic iteration method proposed here will converge.
Convergence Rate, Define the iteration error eLL) = xL)) (L)-)), An iterative method has order of convergence if there exists a constant C * such that lim,,->oo ||e( ||e( [| / = C.
For Basic Iteration method, applying the Lipschitz property of Φ, we obtain [|e(5] ||(l-1)||
( (x((-1)) db (x(1-2)) (((-1) -x 2| < 1, indicating linear convergence rate.
3.4 FADE Examples
FADE is a general framework that extends the applicability of both integer- and fractional-order models. This section mainly presents several FADE variants designed for graph learning tasks. The details of the new continuous models based on FADE are presented in Appendix E.
Experiments
We carry out extensive experiments to validate the performance of our proposed approach on fluid flow, node classification on graphs, traffic forecasting, urban population mobility and biological neural spike trains. Experiments for urban population mobility and biological neural spike train dynamics are shown in Appendix I. All implementations are developed using the PoTrc framework(52) on a single NVIDIA RTX4090 24GB GPU.
4.1 Fluid Flow Prediction
Datasets and Methods. We evaluated the proposed FADE on turbulent houndary-laye flow [53]. with velocity fields measured using particle image velocimetry at five Reynolds numbers (Re= 600. 980, 1370, 1780, 2220), each containing approximately 6,000 snapshots. We trained on four Reynolds numbers and tested on the fifth. Models observed 6 consecutive snapshots to predict the next 6 using a CNN decoder for spatial feature extraction and LSTM, Transformer, Neural ODE, or our FADE for latent state temporal evolution. Turbulent flows exhibit strong memory effects due to the cascade of energy across different scales and the persistence coherent structures. The non-local temporal dependencies in turbulence make it ideal testbed for attention-based fractional framework, which can adaptively weight historical flow states based on their relevance to current dynamics.
Performance and Analysis. The preliminary prediction results are summarized in Table 1, demon- strating that FADE's adaptive memory mechanism effectively captures the nonlinear multi-scale temporal dependencies inherent in turbulent flows, outperforming other approaches. The GPU mem- ory usage of FADE is slightly higher than the baseline LSTM and ODE models, but comparable to the Transformer. This is expected since the extra computational complexity and memory usage from the attention mechanism are inevitable and standard. Importantly, with nearly the same GPU memory

---

Table 1: Average prediction error and memory consumption of different models turbulent vector field prediction.
Model
LSTM+CNN
Transformer+CNN
ODE+CNN
FADE+CNN (ours)
RMSE
1.5035
0,4701
0.5962
0.3215
MAE
0.6027
0.3532
0.2799
0.2041
Training Memory
11372MiB
14142MiB
11250MiB
14722MiB
usage, FADE performs much better than the Transformer. In fact, due to the flexibility of FADE, one can incorporate efficient attention mechanisms such as [54] to alleviate GPU memory usage, which can reduce the computational cost from O (L2) to OLL log L) in practice.
4.2 Node Classification on Graph
Datasets and Methods. We evaluate our approach on variety of datasets with different graph structures. For the Airport dataset, we adopt preprocessing protocols from [55]. For the others, we follow the protocols from GRAND [56], specifically using random splits applied to the largest connected component. We compare our method against several established GNN models, including
GCN [57], HGCN [55], GIL [58], GRAND [56], FROND [19], DRAGON [38] and NvoFDE [33], which are detailed in Appendix F. GRAND, FROND, DRAGON, and NvoFDE adopt specific kernel functions listed in (9) to (12). We primarily utilize the Fractional Attention Kernel presented in Section 3.1.1. We examine two variants of FADE: FADE-1 and FADE-nl provided in Appendix E. Taking the Cora dataset as an example to illustrate the experimental parameter settings, we set time 25, step size = 1, learning rate 0.01, weight decay 0.05, epoch = 800 and dim = 256. For more details regarding the dataset and extended experimental results, please refer Appendix G.
Table 2: Node classification results (%) for random train-val-test splits. The best result is highlighted
in red.
Model
Cora
Citeseer
Pubmed
CoauthorCS Computer
Photo
CoauthorPhy ogba-arxiv
Airport
GCN[57]
81.5.1.3
71.9119 77.812.9
91.1.0.55
82.6.12.4 91.2.11.2
92.8.11.0 90.8.1..5
72.2:10.3
81.6.10.6
HGCN[55]
78.7=1100
65.812.0 76.410.8
90.610.3
80.6.11.8
88.2:11.4
59.610.4
85.4100.7
GIL[58]
82.1=1.1
71.1=1.2 77.8+0.6
89.4
89.611..
91.5
GRAND-[[S6]
83.6#1.0
73.4+0.5
78.8
92.9=0.4
83.7+11.2
92.3++099
93.5+009
71.9+0.2
80.5+99.6
F-GRAND-I[19]
84.8+1111
74,0+1.5
7..4+
93.0++.3
84.4+1.5
92.8++0.6
94.5+00.4
72.6+001
98.1+-0.2
D.GRAND-[[3]
85.1:11.3
1. 79.612.6
93.2.1..
87.341.3
93.110088
94.610.2
98.
Nvo-GRAND-1(33]
86.0.10.5
75.6108
80.811.2
93.410..
87.910.8
94.1100.2
94.71002
71.810.1
98.7100.2
FADE-1 (ours)
86,410.5
76.120.6
80.710.77
93.5.0.1
88.3:0099
94.410.2
94.7100.2
72.010.2
98.810.11
GRAND-nI[56]
82.3111.6
70.91100 7..511.8
92.410.3
82.4.2.11
92.410.8
91.4.1..
71.2:10.2
90.9.1..6
F-GRAND-al[19]
83.2+1.1 74.7+1.9 79.2+00.7
92.9=0.4
84.1+-0.9
93.1+009
93.9+0055
71.4+0.3
96.1+0.77
D-GRAND-al(38
83,9+1.3
1.6 79.5+-2.6
93.1+0.3
87.1+1.0
93.4+0.5
94.3+00.6
97.7+00.4
Nvo-GRAND-mll333
85.411.0
75.910.6 80.6110.7
93.410.2
87.2.114 94.0.1..3
94.6.10.2
72.010.2
98.4:10.2
FADE-nl (ours)
86.011..4 7..2.10.8 80.610.7 93.410.2
87.710.9 94.110.3
94.8-10.2
72.010.1
98.6110.1
Performance and Analysis. The results presented in Table show that our FADE model achieves the best results on most datasets. Compared to GRAND, F-GRAND and D-GRAND models, FADE demonstrates superior performance. For example, on Cora, Pubmed and Photo datasets, FADE- 2.1% respectively, while FADE-nl
shows performance gains of 1 1.3% 2.8%, 1.1% 1%-.0%. and achieves improvements of 2. --77%, 1.1%- ,, and 0.7%-1.7%, respectively. Against Nvo- GRAND, both FADE variants consistently deliver competitive or better results across all datasets. These results highlight the strong performance advantage of FADE, demonstrating its ability to effectively capture complex graph dynamics.
4.3 Traffic Forecasting
Datasets and Methods. To further validate the effectiveness of the FADE model, we conduct experiments on time-series forecasting tasks using four real-world traffic datasets: PcMSD7(M), PeMSD7(L), PeMS04, and PeMS08. These datasets are collected in real time every 30 seconds by the Caltrans Performance Measurement System (PeMS) [59]. more detailed description can be found in Appendix H. Maintaining consistency with [4], we train all datasets for 200 epochs

---

using the Adam optimizer and a batch size of 64. An early stopping strategy is applied, with patience of 15 iterations on the validation dataset. For performance evaluation, we employ three widely-used metrics: Mean Absolute Error (MAF), Root Mean Square Error (RMSE), and Mean Absolute Percentage Error (MAPE), which collectively provide a comprehensive assessment of the prediction accuracy. For comparative analysis, we select several baseline models, including HA[60), VAR[60], TCN[61], D6ANe((62), AGCRN[63], STFGNN(64), Z-GCNETs]65), STGODE[6] and STG-NCDE(4]. Following [4], we refer to model spatio-temporal FADE (STG-FADE) in this part.
Performance and Analysis. The results presented in Table 3 reveal that the proposed STG-FADE gen- erally outperforms other models in forecasting accuracy across the PeMSD4, PeMSD8, PcMSD7(M), and PeMSD7(L) datasets. STG-FADE achieves the lowest MAE and RMSE values on PeMSD4, PeMSDD, and PeMSD7(M) alongside competitive MAPE performance. shows excellent results in all three metrics with the best MAPE value, highlighting its generalization on large-scale traffic datasets. More experimental results are presented in Appendix Overall, STG- FADE exhibits superior and stable performance, effectively capturing the spatio-temporal information inherent in traffic forecasting.
PEMSI7(L), STG-FADE
Table Forecasting error on PeMSD4, PeMSD8, PcMSD7(M), and PeMSD7(L). The best and the second- best results are highlighted in red and blue, respectively.
PeMSD4
PeMSDS
PcMSD7(M)
PeMSD7(L)
Model
MAE
RMSE
MAPE MAE RMSE
MAPE MAE
RMSE
MAPE
MAE
RMSE
MAPE
HA[60]
38.03
59.24
27.88%
34.86
59.24
27.88%
4.59
8.63
14.35%
4.84
9.03
14.90%
VAR[60]
24.54
38.61
17.24 %
19.19
29.81
13.10%
4.25
7.61
10.28%
4.45
8.09
11.62%
TCN[61]
23.22
37.26
15.59%
22.72
35.79
14.03%
4.36
7.20
9.71%
4.05
7.29
10.43%
DSANe((62]
22.79
35.77
16.03%
17.14
26.96
11.32%
3.52
6.98
8.78%
3.66
7.20
9.02%
AGCRN[63]
19.83
32.26
12.97%
15.95
25.22
10.09%
2.79
5.54
7.02%
2.99
5.92
7.59%
STFGNN[64]
20.48
32.51
16.77%
16.94
26.25
10.60%
2.90
5.79
7.23%
2.99
5.91
7.69%
Z-GCNETs[65]
19.50
31.61
12.78%
15.76
25.11
10.01%
2.75
5.62
6.89%
2.91
5.83
7.33%
STGODE16]
20.84
32.82
13.77%
16,81
25.97
10.62%
2.97
5.66
7.36%
3.22
5.98
7.94%
STG-NCDE[4]
19.21
31.09
12.76%
15.45
24.81
9.92%
2.68
5.39
6.76%
2.87
5.76
7.31%
STG-FADE (ours)
19.17
31.01
12.77%
15.29
24.67
10.12%
2.69
5.39
6.71%
2.91
5.81
7.20%
Conclusion
In this paper, we propose the generalized neural fractional attention differential equation (FADE). a novel continuous neural network framework that integrates fractional calculus with learnable neural attention kernels. By replacing the fixed kernel with adaptive attention, FADE effectively emphasizes relevant historical dependencies while filtering out less important information. Our analysis establishes boundedness, well posseneesssss and convergence, ensuring the soundness of the proposed framework. Extensive experiments confirm FADE superior performance and stability. Overall, FADE advances continuous-depth learning by combining fractional calculus long- mong-memory modeling with adaptive attention, enabling effective learning from complex temporal data.
Acknowledgments and Disclosure of Funding
This work is supported by the National Natural Science Foundation of China under Grants 62225207, 62436008, 62576326, 12301491, U2268203, 62433005, 62272036, 62173167, To improve the readability, parts of this paper have been grammatically revised using ChatGPT [66].
References
[1] E. Weinan, "A proposal on machine learning via dynamical systems," Commun. Math. Statist., vol. 1, no. 5, pp. 1-11, 2017.
[2] R. T. Chen, Y. Rubanova, Bettencourt, and D. Duvenaud, "Neural ordinary differential equations," in Advances Neural Inf. Process. Syst., 2018.
10

---

[3] P. Kidger, R. T. Q. Chen, and T. J. Lyons, ""hey, that's not an ode": Faster ode adjoints via seminorms." International Conference on Machine Learning. 2021.
[4] J. Choi, H. Choi, J. Hwang, and N. Park. "Graph neural controlled differential equations for traffic forecasting," " in Proceedings of the AAAI conference on artificial intelligence, vol. 36, no. 6, 2022, pp. 6367-6374.
[5] M. Jin, Y. Zheng, Y.-F. L., S. Chen, B. Yang, and S. Pan, "Multivariate time series forecasting with dynamic graph neural odes," IEEE Transactions on Knowledge and Data Engineering, vol. 35, no. 9, pp. 9168-91800 2022.
[6] Z. Fang, Q. Long, G. Song, and K. Xie, "Spatial-temporal graph ode networks for traffic flow forecasting," in Proceedings of the 27th ACM SIGKDD conference on knowledge discovery & data mining, 2021, pp. 364-373.
[7] Y. Chen, K. Ren, Y. Wang, Y. Fang, W. Sun, and D. Li, "Contiformer: Continuous time transformer for irregular time series modeling," Advances Neural Information Processing Systems, vol. 36, pp. 47 143-47 175, 2023.
[8] B. Chamberlain, Rowbottom, M. Goronova, S. Webb, E. Rossi, and M. M. Bronstein, "Grand: Graph neural diffusion," in Proc. Int. Conf. Mach. Learn., 2021.
[9] M. Thorpe, H. Xia, T. Nguyen, T. Strohmer, A. Bertozzi, S. Osher, and B. Wang. "Grand++: Graph neural diffusion with a source term," in Proc. Int. Conf. Learn. Representations, 2022.
[10] K. Zhao, Q. Kang, Y. Song, R. She, S. Wang, and W. P. Tay, "Graph neural convection-diffusin with heterophily," in Proc. Inter. Joint Conf. Artificial Intell., China, 2023.
[1]] W. Ji, W. Qiu, Z. Shi, S. Pan, and S. Deng, "Stiff-pinn: Physics-informed neural network for stiff chemical kinetics," " The Journal of Physical Chemistry A, vol. 125, no. 36, pp. 8098-8106, 2021.
[12] S. JANNY, M. Nadri, J. Digne, and C. Wolf, "Space and time continuous physics simulation from partial observations," in International Conference on Learning Representations, 2024.
[13] Y.-Y. Yu, J. Choi, W. Cho, K. Lee, N. Kim, K. Chang, C. Woo, I. KIM, S. Lee, J. Y. Yang et al., "Learning flexible body collision dynamics with hierarchical contact mesh transformer," in International Conference Learning Representations, 2024.
[14] Q. Kang, Y. Song, Q. Ding, and W. Tay, "Stable neural ode with Lyapunov-stable equilibrium points for defending against adversarial attacks," Advances in Neural Information Processing Systems, vol. 34, pp. 14 925 937, 2021.
[15] H. Yan, J. Du, V. Y. Tan, J.d Feng, "On robustness of neural ordinary differential equations," " in Advances Neural Inf. Process. Syst., 2018, pp. 1-13.
[16] W. Cui, H. Zhang. H. Chu, P. Hu, and Y. Li, "On robustness of neural odes image classifiers," " Information Sciences, vol. 632, pp. 576-593 2023.
[17] L. Yang, Z. Zhang, Y. Song, Hong, Xu, Y. Zhao, W. Zhang, B. Cui, and M.-H. Yang. "Diffusion models: A comprehensive survey of methods and applications," ACM Computing Surveys, vol. 56, no. 4, pp. 1-39, 2023.
[18) Y. Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole, "Score-hased generative modeling through stochastic differential equations," arXiv preprint arX1v:2011.13456. 2020.
[19] Q. Kang, K. Zhao, Q. Ding, F. Ji, X. Li, W. Liang. Y. Song, and W. P. Tay, "Unleashing the potential of fractional calculus in graph neural networks with FROND," in Proc. International Conference Learning Representations, 2024.
[20] W. Deng, R. Hou, W. Wang, and P. Xu, Modeling Anomalous Diffusion: From Statistics to Mathematics. New Jersey: World Scientific, 2020.

---

[21] R. L. Bagley and P. Torvik, "A theoretical basis for the application of fractional calculus to viscoclasticityy" J. Rheology, vol. 27, no. 3. pp. 201-210, 1983.
[22] M. Zubair, M. J. Mughal, Q. A. Naqvi, M. Zubair, M. Mughal, and Q. A. Naqvi, "Electro- magnetic wave propagation in fractional space," Electromagnetic fields and waves in fractional dimensional space, pp. 27-60, 2012.
[23] E. Scalas, R. Gorenflo, and F. Mainardi, "Fractional calculus and continuous-time finance," Physica A: Statistical Mechanics and its Applications, vol. 284, no. 1-4, pp. 376-384, 2000.
[24] G. Pang, L. Lu, and G. E. Karniadakis, "fpinns: Fractional physics inferse- neural networks," SIAM Sci. Comput., vol. 41, no. 4, pp. A2603-A2626, 2019.
[25] L. Guo, H. Wu, X. Yu, and T. Zhou, "Moute carlo fpinns: Deep learning method for forward and inverse problems involving high dimensional fractional partial differential equations," Comput. Methods Appl. Mechanics Eng. vol. 400, p. 115523, 2022.
[26] R. Javadi, H. Mesgarani, O. Nikan, and Z. Avazzadeh, "Solving fractional order differential equations by using fractional radial basis function neural network," Symmetry, vol. 15, no. 6, p. 1275, 2023.
[27] K. Zhao, Q. Kang, Ji, X. Li, Q. Ding, Y. Zhao, W. Liang, and W. P. Tay, "Distributed fractional graph operating network," in Advances Neural Inf. Process. Syst. Dec. 2024.
[28] Q. Kang, K. Zhao, Y. Song, Y. Xie, Y. Zhao, S. Wang, R. She, and W. P. Tay, "Coupling graph neural networks with fractional order continuous dynamics: A robustness study," in Proc. AAAI Conference Artificial Intelligence, Vancouver, Canada, Feb. 2024.
[29] Q. Kang, X. Li, K. Zhao, W. Cui, Y. Zhao, W. Deng, and W. P. Tay, "Efficient training of neural fractional-order differential equation via adjoint backpropagation; " in Proceedings the Conference Artificial Intelligence, vol. 39, no. 17, 2025, pp. 17 750-17 759.
the AAAI
[30] C. Coelho, M. F. P. Costa, and L. L. Ferrás, "Neural fractional differential equations," Applied Mathematical Modelling, vol. 144, p. 1 116060, 2025.
[31] G. Nobis, M. Aversa, M. Springenberg, M. Detzel, S. Ermon, S. Nakajima, R. Murray-Smith S. Lapuschkin, C. Knochenhauer, L. Oala et al., "Generative fractional diffusion models," " arXiv preprint arXiv:2310.1763: 2023.
[32] R. Almeida, "A caputo fractional derivative of a function with respect to another function," " Communications in Nonlinear Science and Numerical Simulation, vol. 44, pp. 460-481, 2017.
[33] W. Cui, Q. Kang, X. Li, K. Zhao, W. P. Tay, W. Deng, and Y. Li, "Neural variable-order fractional differential equation networks," " in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 39, no. 15, 2025, pp. 16 109-16
[34] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, "Attention all you need," in Advances Neural Inf. Process. Syst., 2017, pp. 1-11.
[35] 7. Lin, M. Feng, C. N. d. Santos, M. Yu, B. Xiang, B. Zhou, and Y. Bengio, "A structured self-attentive sentence embedding." arXiv preprint arXiv: wtw::7703.311 0300,, 2017.
[36] E. Dupont, A. Doucet, and Y. W. Teh, "Augmented neural odes," in Advances Neural Inf. Process. Syst. 2019, pp. 1-11.
[37] M. Poli, S. Massaroli, J. Park, A. Yamashita, H. Asama, and J. Park, "Graph neural ordinary differential equations," " arXiv preprint 1eXXv:1911..72. 2019.
[38] K. Zhao, X. Li, Q. Kang, F. Ji, Q. Ding, Y. Zhao, W. Liang, and W. P. Tay, "Distributed-order fractional graph operating network," in Advances Neural Inf. Process. Syst., 2024, pp. 1-14.
[39] J. Zhang, P. Zhang, B. Kong, J. Wei, and X. Jiang, "Continuous self-attention models with neural ode networks," in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, no. 16. 2021, pp. 14 14393-14401.
12

---

[40] P. Veličkovićć G. Cucurull, A. Casanova, A. Romero, P. Liò, and Y. Bengio, "Graph attention networks," in Proc. Int. Conf. Learn. Representations. 2018. pp. 1-12.
[41] K. Diethelm, The analysis of fractional differential equations: an applicatio-orrented exposi- lion using differential operalors of Capulo type. Springer, 2010, vol. 2004.
[42] V. E. Tarasov, Fractional dynamics: applications of fractional calculus to dynamics of particles, fields and media. Springer Science & Business Media, 2011.
[43] C. E. Coimbra, "Mechanics with variable-order differential operators," Annalen der Physik, vol. 515, no. 11-12, pp. 692-703, 2003.
[44] A. Obembe, M. E. Hossain, and S. A. Abu-Khamsin, "Variable-order derivative time fractional diffusion model for heterogeneous porous media," Journal of Petroleum Science and Engineering, vol. 152, pp. 391-405, 2017.
[45] K. He, X. Zhang. S. Ren, and J. Sun, "Deep residual learning for image recognition," in Proc. Conf. Comput. Vision Pattern Recognition, 2016.
[46] Y. Rubanova, R. T. Chen, and D. K. Duvenaud, "Latent ordinary differential equations for irregularlysaappled time series," Advances in neural information processing systems, vol. 32, 2019.
[47] E. De Brouwer, J. Simm, A. Arany, and Y. Moreau, "Gru-ode-bayes: Continuous modeling of sporadically-ossee obsserved time series," " Advances in neural information processing systems, vol. 32, 2019.
[48] P. Kidger, J. Morrill, Foster, and T. Lyons, "Neural controlled differential equations for irregular time series," Advances in neural information processing systems, vol. 33, pp. 6696- 6707, 2020.
[49] K. Diethelm, Ford, and A. D. Freed, "Detailed error analysis for a fractional adams method," Numer. Algorithms,, vol. 36, pp. 31-52, 2004.
[50] G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger, "Densely connected convolutional networks," " in Proc. Conf. Comput. Vision Pattern Recognition, 2017.
[51] D. Bahdanau, K. Cho, and Y. Bengio, "Neural machine translation by jointly learning to align and translate," arXiv preprint arXiv: 0400,, 2014.
[52] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga al., "Pytorch: An imperative style, performeffmm deep learning library," Advances Neural Inf. Process. Syst., 2019.
[53] A. Towne, S. T. Dawson, G. A. Bres, A. Lozano- T. Saxton-Fox, A. Parthasarathy, A. R. Jones, H. Biler, C.-A. Ych, H. D. Patel et al., "A database for reduced complexxty modeling of fluid flows," AIAA journal, vol. 61, no. 7, pp. 2867-2892, 2023.
[54] H. Zhou, Zhang, Peng, S. Zhang, J. Li, H. Xiong, and W. Zhang, "Informer: Beyond efficient transformer for long sequence time-series forecasting," in Proceedings of the AAAI conference on artificial intelligence, vol. 35, no. 12, 2021, pp. 106-11 115.
[55] I. Chami, Z. Ying, C. Ré, and J. Leskovec, "Hyperbolic graph convolutional neural networks," in Advances Neural Inf. Process. Syst. 2019.
[56] B. Chamberlain, J. Rowbottom, M. I. Gorinova, M. Bronstein, S. Webb, and E. Rossi, "Grand: Graph neural diffusion." in Proc. Int. Conf. Mach. Learn., 2021, pp. 1407-1418.
[57] T. N. Kipf and M. Welling, "Semi-supervised classification with graph convolutional networks." in Proc. Int. Conf. Learn. Representations, 2017.
[58] 7hu, S. Pan, C. Zhou, J. Wu, Y. Cao, and B. Wang, "Graph geometry interaction learning," in Advances Neural Inf. Process. Syst., 2020.
13

---

[59] C. Chen, K. Petty, A. Skabardonis, P. Varaiya, and Z. Jia, "Freeway performance measurement system: mining loop detector data," Transportation research record. vol. 1748. no. pp. 96-102, 2001.
[60] J. D. Hamilton, Time series analysis. Princeton university press, 2020.
[61] S. Bai, Z. Kolter, and V. Koltun, "An empirical evaluation of generic convolutional and recurrent networks for sequence modeling." arXiv preprint arXiv: 2018.
[62] S. Huang, D. Wang, X. Wu, and A. Tang, "Dsanet: Dual self-attention network for multivariate time series forecasting," in Proceedings the 28th ACM international conference on information and knowledge management, 2019, pp. 2132-
[63] L. Bai, L. Yao. C. Li, X. Wang, and C. Wang. "Adaptive graph convolutional recurrent network for traffic forecasting," in Advances in neural information processing systems, 2020, pp. 17 804- 17 815.
[64] M. Li and Z. Zhu, "Spatial-temporal fusion graph neural networks for traffic flow forecasting," " in Proceedings of the AAAI conference on artificial intelligence, vol. 35, no. 5, 2021, pp.
4189-419 4196.
[65] Y. Chen, I. Segovia, and Y. R. Gel, "Z-genets: Time zigzags at graph convolutional networks for time series forecasting," " International Conference on Machine Learning. PMLR, 2021,
pp. 1684-1694.
[66] OpenAI, "Chatgpt-4," 2022, available at: https//wwwwoppnni.comm (Accessed: 10 April 2024).
[67] Podlubby, Fractional Differential Equations. Academic Press, 1999.
[68] M. Caputo and M. Fabrizio, "A new definition of fractional derivative without singular kernel," Progress in Fractional Differentiation & Applications, vol. 1, no. 2, pp. 73-85, 2015.
[69] K. Diethelm and N. J. Ford, "Analysis of fractional differential equations." " Math. Anal. Appl., vol. 265, no. 2, pp. 229-248, 2002.
[70] R. Hilfer, Applications of fractional calculus in physics. World scientific, 2000.
[71] A. Atangana and D. Baleanu, "New fractional derivatives with nonlocal and non-singular kernel: theory and application to heat transfer model," arXiv preprint arXiv:/602.000 /602.03408, 2016.
[72] S. G. Samko and B. Ross, "Integration and differentiation variable fractional order," Integral transforms and special functions, vol. 1, no. 4, pp. 277-300, 1993.
[73] I. Podlubny, "An introduction to fractional derivatives, fractional differential equations, to methods of their solution and some of their applications," Math. Sei. Eng, vol. 198, no. 340, pp. 0924-34 008, 1999.
[74] Y. Kang, S. Mao, and Y. Zhang, "Fractional time-varying grey traffic flow model based on viscoelastic fluid and its application," Transportation research part B: methodological, vol. 157, pp. 149-174, 2022.
[75] F. Mainardi, Fractional calculus and waves in linear viscoelasticity: introduction to mathe- matical models. World Scientific, 2022.
[76] H. Antil, R. Khatri, R. Löhner, and D. Verma, "Fractional deep neural network via constrained optimization," Mach. Learn.: Sei. Technol. vol. 2, no. 1, p. 015003, 2020.
[77] A. Dosovitskiy, L.. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly al., "An image is worth 16x16 words: Transformers for image recognition at scale," in International Conference on Learning Representations, 2020.
[78] B. Lim, S. Ó. Ank, N. Loeff, and T. Pfister, "Temporal fusion transformers for interpretable multi-horizon time series forecasting." International Journal of Forecasting, vol. 37, no. 4, pp. 1748-1764, 2021.
14

---

[79] P. Wang. T. Zhang, H. Zhang, S. Cheng, and W. Wang, "Adding attention to the neural ordinary differential equation for spatio-tempora prediction," International Journal of Geographical Information Science, vol. 38, no. 1, pp. 156-181, 2024.
[80] H. Kim, A. Mnih, J. Schwarz, M. Garnelo, A. Eslami, D. Rosenbaum, O. Vinyals, and Y. W. Teh, "Attentive neural processes," arXiv preprint arXiv: 1011:1901.00761,, 2019.
[81] X. Wang, H. Ji, C. Shi, B. Wang, Y. Ye, P. Cui, and P. S. Yu, "Heterogeneous graph attention network," in The world wide web conference, 2019, pp. 2032.
[82] Jarad, T. Abdeljawad, and D. Baleanu, "Caputo-type modification of the hadamard fractional derivatives," Advances in Difference Equations, vol. 2012, pp. 1-8, 2012.
[83] E. Zeidler, Applied Functional Analysis, ser. Applied Mathematical Sciences, J. E. Marsden and I.. Sirovich, Eds. New York, NY: Springer New York, 1995, vol. 108.
[84] Shen, T. Tang, and L.-L. Wang, Spectral Methods: Algorithms, Analysis and Applications, ser. Springer Series in Computational Mathematics. Berlin, Heidelberg: Springer Berlin Heidelberg, 2011, vol. 41.
[85] Y. Song. Q. Kang, S. Wang, K. Zhao, and W. P. Tay, "On the robustness of graph neural diffusion topology perturbations," in Advances Neural Inf. Process. Syst., 2022.
[86] O. Platonov, D. Kuznedelev, M. Diskin, A. Babenko, and Prokhorenkova, "A critical look at the evaluation of gnns under heterophily: Are we really making progress?" arXiv preprint ar3iv:2300. 1640, 2023.
[87] M. Fliasof, F. Haber, E. Treister, and C.-B. B. Schönlieh, "On the temporal domain of differential equation inspired graph neural networks," in International Conference on Artificial Intelligence and Statistics. PMLR, 2024, pp. 1792-1800.
[88] C. Bodnar, F. D. Giovanni, B. Chamberlain, P. Liò, and M. M. Bronstein, "Neural sheaf diffusion: A topological perspective on heterophily and oversmoothing in GNNs," in Advances Neural Inf. Process. Syst. 2022.
[89] Y. Wang, K. Yi, X. Liu, Y. G. Wang, and S. Jin, "Acmp: Allen-cahn message passing with attrac- tive and repulsive forces for graph neural networks," in Proc. Int. Conf. Learn. Representations, 2022.
[90] J. Li, H. Wang, and X. Chen, "Physics-informet neural ode for post-disaster mobility recovery." in Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2024, pp. 1587-1598.
[91] S. Hochreiter and J. Schmidhuber, "Long short-term memory," Neural computation, vol. 9, no. 8, pp. 1780, 1997.
[92] C. Zang and F. Wang. "Neural dynamics on complex networks," Proceedings of the 26th 26th ACM SIGKDD international conference knowledge discovery & data mining, 2020, pp. 892 -92-
[93] Z. Huang, Y. Sun, and W. Wang, "Coupled graph ode for learning interacting system dynamics." in Proceedings of the 27th ACM SIGKDD conference on knowledge discovery & data mining, 2021, pp. 705-715.
(94] Y. Nie, "A time series is worth 64words: Long-term forecasting with transformers," arXiv preprint arXiv:2211. 14730, 2022.
[95] I. Lazarevich, I. Prokin, B. Gutkin, and V. Kazantsev, "Spikebench: An open benchmark for spike train time-series classification," PLOS Computational Biology, vol. 19, no. 1, p. e1010792, 2023.
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The paper's contributions and scope detailed in the abstract and introduction Section I.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We have discussed the limitations in the Technical Appendices.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model sell-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't I acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
16

---

Justification:
The theorems proposed in this paper are supported by detailed proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The theorems proposed in this paper are supported by detailed proofs provided in the Technical Appendices.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
17

---

Answer: [Yes]
Justification:
We have provided the source code in the supplementary material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: /niiss wwtppbpcp/ppaactcccccccccccc /wcp///gussseeeeeeeeeeeeeeee / ssiiiooiicyyyyi for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: We have included the implementation details both the main text and the supplemental material.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification:
We report mean and standard deviation values in our main experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
18

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We report the computing cost the supplemental material.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httt:::///eurpsscc...
Answer:[Yes
Justification:
We conform, in every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We have discussed broader impacts in the supplemental material.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
19

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We cite the original paper that produced the code package dataset.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
20

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
21

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs any important, original, or non-standard components.
Guidelines:
The answer NA
that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy buttps:/nneurpp::: http///ipss.ccoooeeeeeeeeeeeeeee cc//ooofeeerccce/222//// for what should or should be described.
22

---

Introduction
This supplementary document provides extended explanations and additional results that support the claims presented in the main paper. The content is organized as follows.
1. A comprehensive related work is detailed in Appendix A.
2. Preliminaries fractional calculus presented in Appendix B.
3. All theoretical proofs in this paper are presented in Appendix C.
4. Nonsingular scenario for the kernel shown in Appendix D.
5. FADE examples are presented in Appendix E.
6. Graph differential equation models are given in Appendix
7. Datasets and more experiments the graph are available in Appendix G.
8. Datasets and more experiments for traffic forecasting are discussed in Appendix H.
9. Experiments for urban population mobility and biological neural spike train dynamics are provided in Appendix I.
10. Limitations and broader impact are discussed in Appendix J.
A Related Work
Our work builds on research in fractional differential equations, neural integer-order and fractional- order ODE models, and neural network attention mechanisms. We will present the related work from the following aspects.
A.1 Fractional Differential Equations
Fractional Differential Equations (FDEs) generalize classical differential equations by allowing the order of differentiation be non-integer, thereby providing a powerful framework for modeling systems with memory and hereditary properties. The mathematical foundations of FDEs have been rigorously studied, with seminal contributions from [67-69]. Among the most prominent formulations are the Riemann-Liouvee encode memory effects. However, such power-law kernels often impose restrictions when modeling systems with heterogeneous or deeeedee dynamics. To address these limitations, alternative definitions have been proposed, such as the Caputo-Fabrizio derivative with an exponential decay kernel [68], and the Atangana-Balean derivative with Biittg-Leefler-basee kernel [71]. These generalizations retain the core non-local structure of fractional calculus while enhancing modeling flexibility. More recently, variable-order fractional derivatives have attracted considerable attention due to their ability to reflect more flexible and complex dynamic memory mechanism in real-world phenomena [72, 43]. In these systems, the fractional order is time-dependent, denoted as a(t). enabling a more precise representation of evolving dynamic behaviors.
and Caputo derivatives [70], both of which use power-law kernels to
In addition to the rich theoretical results, FDEs have found broad applicability across wide range of fields. For instance, the authors offered a foundational overview of its practical uses in areas such as signal processing, system modeling and automatic control [73]. The authors showed that fractional calculus has emerged as a powerful mathematical framework for modeling complex systems in traffic forecasting [74]. The authors highlighted its role in viscoelasticity, illustrating how fractional-order models effectively capture the memory and hereditary characteristies inherent in such materials [75]. Mcanwhile, FDEs have also been widely utilized to improve the performance of graph neural networks [76, 19, 33]. Despite considerable progress in both theory and applications, the design of kernel functions for fractional calculus remains largely underexplored. Particularly in neural differential equations, adaptive kernel functions can learn to assign appropriate weights to historical states based on relevance, which is still in infancy.
A.2 Neural Differential Equations
Neural Differential Equations (NDEs) offer a unified framework that integrates neural networks with differential equations to model continuous time dynamic systems, bridging the gap between
23

---

deep learning and classical dynamical systems. Among the various types of NDEs, Neural Ordinary Differential Equations (NODEs) and Neural Fractional Differential Equations (NFDEs) are more closely related to our work. Introduced by [2]. NODEs model the evolution of hidden states by parameterizing the right-hand side of an ODE using neural networks. This model allows for adaptive computation in continuous time, offering memory efficiency and interpretability. To enhance the expressiveness of NODEs, several variants have been proposed. Augmented NODEs [36] expand the hidden state space with additional dimensions to improve the representational power and alleviate topological constraints that limit standard NODES. Neural Controlled Differential Equations (NCDEs) [48] further generalize the NODEs framework to handle controlled systems in modeling irregular time series data, such as in finance or healthcare. Graph NODEs [37] have demonstrated strong performance on graph traph-structured data by integrating graph convolutional operations with continuous dynamics. Spatio-temporal graph NCDE [4] achieves significant performance improvements in traffic forecasting by integrating two NCDEs for temporal and spatial processing.
Despite their flexibility, NODEs-based models are limited by integer-order calculus, restricting their ability to capture memory and long-range dependencies, thus motivating interest in fractional-order extensions. Inspired by Graph NODEs, the FROND framework introduces a generalized fractional- order continuous GNN model using Caputo derivatives to capture non-local, memory-depende dynamics, offering improved performance and mitigating oversmoothing in graph learning tasks [19]. Then, the DRAGON framework is proposed, which shows distributed-order fractional continuous GNN that learns a superposition of derivative orders, enabling flexible and non- -arrkoinnn feature updating dynamics [27]. Recently. the NvoFDE framework introduces variable-order fractional differential operators into neural networks, enabling learnable and adaptive derivative orders based on time and hidden features [33]. However, most existing fractional neural models use fixed kernels with predefined weights assigned to historical states, limiting their flexibility and adaptability. Our work addresses this gap by introducing adaptive kernel functions into the fractional differential equation framework for improved temporal representation.
A.3 Attention Mechanisms in Neural Networks
Attention mechanisms have emerged as a fundamental component of modern deep learning archi- tectures, enabling models to dynamically prioritize informative parts of the input. The transformer architecture [34] introduces self-attention, which computes pairwise interactions between elements in sequence, allowing for efficient modeling of long-range dependencies. This innovation has had transformative impacts across a variety of domains, including natural language processing, computer systems, attention mechanisms have been integrated into neural differential equations to enhance the representational power. For exam- ple, Continuous
vision and time-series forecasting [77, 78]. In continuous
Neural ODEs [39] extend Neural ODEs framework by integrating lightweight self-attention mechanism, resulting in more flexible and interpretable dynamics. Similarly, attention-based Neural ODEs have been employed in spatio-temporal prediction tasks [79].
In Graph Neural Networks (GNNs), attention mechanisms have unlocked unprecedented flexibility in neighbor weighting and hierarchical feature propagation. Graph Attention Networks (GATs) [40] use self-attention to assign adaptive weights to neighboring nodes during aggregation, improving performance in scenarios with heterogeneous node importance. Extensions of GATs, such as multi- head and hierarchical attention models [80, 81], further enhance the model's ability to capture structural nuances. Despite these advances, the integration of attention with Factional-order models remains largely underexplored. Fractional calculus, known for its inherent memory and non-local properties, offers a natural framework capture long-range dependencies. Combining it with attention mechanisms could lead to a new class of flexible and adaptive neural operators.
B Preliminaries on Fractional Calculus
This section offers additional material on fractional calculus theory, with key details presented in the main text of Section 2. Different from [38], we present results terms of v/-fractional derivative that is quite general than previous work (1²(1) l). For more detailed information, please refer to [32, 41]. We begin with the basic definitions.
Definition 4 (ı--Caputo Fractional Derivative). Let a > O, n E N, and be an interval such that -o0 < b < co. Let f. 2b e (I) be two functions such that &b is increasing and N/ (z) # for
24

---

all E I. The left 2b- Ca-Cuuto fractiona derivative of f of order a is defined by
(1)dt, D f(x) :- T(n T(n-a:) Id ('(((())))))))))))))))())))))))) (1)) (1) dt,
and the right vh- Ca-Caputo fractional derivative is given by
(-1)
[-
b
D = T(n -0- a)
(())))00)(())00-)1111 (1) dl,
where f/ (t) (1() (t) f(t). n- [a] + + & N. and n = a ifa N.
For (0,1), the left and right v D fractional derivatives reduce to
D f(x) M(1 - a) J: (0)))))(
1
and
a)) (( D f(x) 10))= T(1
-1
[[((0)) (()
respectively. For specific choices of the function tb, the v Da-Caputo fractional derivative reduces to several well-known operators [82]. Throughout this work, focus the left-sided fractional derivative. The corresponding results for the right-sided derivative can be obtained analogously with appropriate modifications.
To get some intuition, we provide specific example below.
Lemma 2. Given f E IR with B > consider the following function:
f(x) (vx(x) ((a)))-1, g(x)) = (vb(b) - (xx)))
For a > 0, we have:
r(8)
Dave f(x)
(V(x) - baa)) ---1
[(3 - a)
[(3) T(3 a)) (b(b) ((x)))
D g(x)
Now we present the relation between fractional order derivative and integer order counterparts. This can be readily seen from the following theorem that is derived mainly using integration by parts.
Theorem 2. Suppose that f, 1/ C (a, b). Then, for all a > O,
DD f(x) (v(2) - n-a))
T(n (a) r(n + at) [ (1b(z) ((1))=aa d pln) slal(t)dt (t)dt, dt
and
DD f(x) = (-1)" (wb(b) ((z))
f (b)
I(n + a)
T(n +1 a) 1 (0(1) ()x)))
1
d (t)dt.
(m)
di
From this theorem, it is found that
lim 0
Da f(x) = f(h)(())
We next present the relation between integration and differentiation of gi- ci-Caputo fractional function that is vital for the equivalent transformation between differential form and its integral form.
25

---

Theorem 3. Given a function f E Ca b] and a: > 0. we have:
[ (a)
1at (C s(x)) ((t)) = f(x) -
(v(xx) (v(x)) b(a))*
k!
1 1--0 ("Da = f(x)
(b(b) (x))))
k!
k=
Theorem 4. Given a function f E [a, b] and a > 0, we have
C D _ f(x) = fxx) and C CD f(x)= f(x).
a+
Obviously, one can simply apply l on both sides of the differential equations to get its integral form. On the other hand, we can apply C DD directly integral equations in order recover
differential equations.
Lastly, we will show the semigroup law for the d--Caputo fractional derivative. Similar to classical fractional derivative [41], semigroup law does not hold in general for fractional derivative but is indeed true for integrals. In what follows, present case that allows semigroup law.
Theorem 5. f Cmm+a(a
[a, bl for some m E N and 0, then for all i E N we have
(1o+)) (fDat) m f(x) (Daa+t) m f(c) (1b(z) ((a))-b((m) (1b(z)-b(a)) rbaa))
I(ka +
(Ta*)) ( m [(x) (Q m [(d) (vbb) (x)))
T[ka + 1)
for some E (a,x) and (x.
C All theoretical proofs
Proof of Lemma 1. Since
|| S C S/ (v((1 (T)))
xx||dr= S C (b(()) - baa))
||x||
T(a)
+(1+a)
it is immediately seen that this operator is bounded.
Proof of Theorem 1. We shall prove the uniqueness as well as stability. To show the uniqueness, we define the operator as
Tix(()) = x(a) + S' (v(t) (T))) (T() (T)(()))(())ff(()xx())))
T(aa
The Lipschitz properties of K and f lead to:
|T [x [xxe]ll S C (v1(1) - v(a)aa ||x - X2ll.
T(1 +
Again, selecting suitable > 0 and invoking the Banach fixed-point theorem ensures a unique solution. To prove the stability, from (15), we shall get
[xx(t) 22)]l2 [x((a= x(a))l2 +C L'
(vb(1) - (())) | x(T)
T(a)
Applying fractional Grönwall inequality [41, Lemma 6.19], we derive the stability result.
Observation and Motivation: The preceding review of various Caputo fractional derivatives highlights their defining feature: the use of distinct weighting kernels, which can be static or designed to vary dynamically with time t. For high-dimensional states x(t), we can extend this idea by introducing a learnable vector (t) (V)(t), ... e. (t)). where each (t) defines component- wise kernel in a ei- ci-Caputo framework, enabling adaptive, dimension-specaa However, since these kernels depend only on and 7 and not on past states x(T) or the current state x(t,, they cannot adjust their weighting based on the states correlation in the trajectory. In this paper, we propose to overcome this limitation by developing a more generalizable learnable attention kernel that extends beyond the capabilities of the ab-bused approach. Our framework will incorporate mechanisms that can adapt memory weightings based on both temporal information and the contextual relationships between past and current states.
memory modeling.
26

---

C.1
Solving FADE
The integral equation (15) nonlinear equation which can be solved using linearized technique or iterative method. Here, we adopt the latter one. Taking lj and approximating the integral in (15) using the trapezoidal rule yields
x(tz) = x(to) + Σ K(ty,
((t)).
(21)
k=
The iterative method for above nonlinear problems works as follows: taking the initial guess x(1, x(tz-1), then conduct iterations based on
x (1 x(1)) + Σ Ki, x(1( ((()))
L 1.
0=0
This procedure will lead to good approximation of x(tz) after few iterations. To address it, define
j-1
x (t,) = (x
where
① (x(()) = x (to) +
Σ
K (4i, Lu (tz)(t))))))))) (tx)) so (tx, x (tx)) h,
k=0
for the sake of simplicity. The proposed iterative solution to the discretized equation can formulated as the Basic lteration method.
Basic Iteration Method. Given an initial guess (0) (t,) = x (tz-1), we iteratively compute
(t,) (xx)))xx
L L
C.1.1 Convergence Criterion
The convergence of the above iterative method relies on the following condition: If & Rdd -> Rd satisfies a Lipschitz condition with Lipschitz constant C' < 1, namely,
|lő(x) - <ly) s C/|xx cllx
yll. Vx,y E [a, 6jd
then the iterative methods converge for any initial guess (() [a, bjdt
Given that the assumptions of Theorem are satisfied with appropriate generic constant C, can be directly verified that basic iteration method proposed here will converge. The essence is to make sure that the constant less than
C.1.2 Convergence Rate
Define the iteration error e(4) (L) x(L-1), An iterative method has order of convergence p there exists nonzero constant C' such that
[((+))]]
lim
C. |e(csl1
For the Basic Iteration method, applying the Lipschitz property of s. we obtain
[ | (xx(1-1)) (x)1--2))
<0<1,
|e(
x((1-1) x(1
indicating a linear (first-order) convergence rate.
27

---

D Nonsingular scenario for the Kernel
Suppose that the kernal K bounded, Lipschitz continuous with respect to the last two variables. We find that the operator is bounded that is immediately seen from the boundedness of K. Besides, the integral equation is well-posed. The proof is follows: Uniqueness. Define operator:
T(x((t) - x(a) + S. K(t,T.x(00)xxxr))((<(xxx)).
Using the Lipschitz assumptions, we obtain
[T T [xa]]| & C(t a) ||[| -X2
Choosing > 0 small enough that Ce 1 and applying the Banach fixed-point theorem [83] yields a unique solution.
Stability. From (Eq. (15).), using the Lipschitz assumptions again well as boundedness of K, it is readily seen that
x(1 x()]l2 (x(a) x + S. [x(T) {(T)
which gives the desired result by classical Grönwall inequality [84, Lemma B.9].
As in singular case, one can also show that the basic iteration method works well for the nonsingular case, that is, it is convergent with first order.
E
FADE Examples
Here shows multiple FADE variants based on graph learning tasks. Inspired by the models in [19], develop two variants, including Kat-GRAND and Kat-CDE, Similar to [8], Kat-GRAND includes two versions. One Kat-GRAND-nI
S: [[K(,,T K(t,T, ((O) (A(Y - -)Y (t),
(22)
where AtY(t)) (t) is given by nonlinear attention mechanism. The other version is Kat-GRAND-I:
S. K(t, Y(O) (-LY dr Y
(23)
where L is time-invariant matrix, which is linear FDE. Furthermore, based on the CDE model [10], the Kat-CDE model has the following expression:
[ K(t,T (()) (t) + div(V(t)) Y Y(t) dr= Y
(24)
where the divergence operator div(-) is introduced by [85], and stands for the element-wise product, also known as the Hadamard product. This model is crafted to handle heterophilic graphs, where connected nodes typically belong different classes exhibit distinct features.
F
Graph Differential Equation Models
To better understand baseline models, this section primarily introduces several dynamic comparison networks based graph learning tasks, namely GRAND (56), CDE [10], FROND [19], DRAGON [38] and NvoFDE [33].
GRAND [56]: The Graph Neural Diffusion (GRAND) model is graph neural network framework inspired by the heat diffusion process, where information spreads across graph nodes similarly to how heat diffuses through medium. Its governing differential equation is given by:
dx((t dt
(A(X(t)) (t)(
(25)
28

---

where (X((t) is a learnable attention brrsed adjacency matrix, and I is the identity matrix. There are two variants. The update (25) defines the GRAND-al model, where the adjacency matrix (t)(t)) is nonlinear. Let d, = _ Wij and define the diagonal matrix D with Da = d4. The random walk Laplacian is L I WD Thus in the simple case, we have the following GRAND-J model:
(x(() (WD - ()x((t = -
(26)
dt
CDE [10]: In heterophilic graph, nodes often have diverse features, posing significant challenges for graph information processing. To address this issue, the authors introduced the convection -iffussion equations (CDE) into GNds,and then proposed the Neural CDE model. This model adaptively regulates the rate of information propagation between nodes, enabling selective information sharing among dissimilar neighbors. The corresponding mathematical formulation is given by:
dx(() = (A(X(t)) - (t) + dit o X(t),,
(27)
dt
where (t) denotes the velocity field, o indicates the element-wise product, and div(-) represents the divergence operator.
FROND [19]: The FROND framework extends traditional integer-order graph neural differential equations to fractional-order dynamics using the Caputo derivative:
X() F(W, X()),, a> 0,
(28)
where F defines the graph dynamics. By leveraging the non-local nature of fractional calculus, FROND captures long-range dependencies in node features. Similar to (25), (26) and (27), FROND has the following corresponding variants:
(1) E-GROND-nl
(t) = (A(x(t)) I)x(t), 0<asl.
(29)
(2) F-GROND-I
(( = LL((() 0<asl.
(30)
(3) F-CDE
- -I)((() + div( X(I), 0 0<asl.
(31)
DRAGON [38]: Unlike conventional continuous GNNs that rely on fixed integer or single fractional- order derivatives, DRAGON adopts a learnable distribution over derivative orders:
J. D X() du(a) = F(W,X(t))
(32)
where (a, bj defines the domain of a. ll is a learnable distribution, and F denotes the graph dynamics. Similar to (25), (26) and (27), DRAGON has the following corresponding variants: (1) D-GRAND-nl
L (PX(()dp(() = (A(X()) ())
(33)
(2) D-GRAND-I
dul(a) = LX(().
(34)
(3) D-CDE
[ ('X(()dp(o) = (A(X(t)) - I)x(() + div(V(() o X()))
(35)
NyoFDE [33]: NvoFDE extends neural differential equation models by introducing learnable variable-order derivative alt, (t)) that dynamically adapts over time and feature space.
D. (0)) X(t) F(W, X()),, 0 a(t, X(t)) <
(36)
29

---

where F defines the graph dynamics. Similar to the above, there exist the following variants:
(1) Nvo-GROND-nl
D De (((1)x(1)) X(t) (A(X(t)) (t), 0< 1
(37)
(2) Nvo-GROND-I
De (() = LX(()) 0 O<a(l, 1
(38)
(3) Nvo-CDE
Dg (e(1x(()) x(t) (A(X(I)) - (()(() + div X(t),, 0 (<t(tx S
(39)
G Datasets and More Experiments on Graph for FADE Model
G.1 Datasets and Setting
The datasets used in this paper are provided separately in Table 4 and Table 5.
Table 4: Dataset statistics used in Table of the main text
Dataset
Type citation
Classes
Features
Nodes
Edges
Cora
1433
2485
5069
Citeseer
citation
3703
2120
3679
PubMed
citation
3
500
19717
44324
Coauthor CS
co-author
15
6805
18333
81894
Computers
co- perrcurcas
767
13381
245778
Photo
co-pur co-purchase
745
7487
119043
CoauthorPhy
co-author
5
8415
34493
247962
OGB-Arxiv
citation
40
128
169343
1166243
Airport
tree-like
4
4
3188
3188
Table 5: Dataset statistics used in Table 6
Dataset
Nodes
Edges
Classes
Node Features
Roman-empire
22662
32927
18
300
Wiki-cooc
10000
2243042
5
100
Minesweeper
10000
39402
2
7
Questions
48921
153540
2
301
Workers 11758
519000
2
10
Amaon-ratings
24492
93050
5
300
The authors revealed critical limitations in the commonly used benchmark datasets for evaluating models on heterophilic graphs [86]. To address this, they introduced several new datasets, such as Roman-empire Wiki-cooc, Questions, Workers and Amazon-ratinge These datasets, sourced from different fields, have low homophily scores and display a variety structural properties. We follow the experimental setup specified in the CDE model [10]. For the Workers, and Questions datasets, employ the ROC-AUC score as the evaluation metric, as these tasks involve binary classification. Using the Amazon-ratings dataset set time = 3, step size 1, learning rate = 0.001, weight decay = 0.0005, epoch = 1000 and dim = 128. The performance of our FADE-CDE model, is evaluated against several well-known baselines, including HIDE-NNN[87] GRAND [56], GraphBel [85], NSD [88], ACMP[89], CDE [10], F-CDE [19], D-CDE[38] and Nvo-CDE[33].
example to reveal the experimental parameter settings,
G.2 Node Classification on Heterophilic Graph
Performance and Analysis: In Table 6, we present experimental results on heterophilic graph datasets. It is evident that our model FADE-CDE achieves competitive or better performance, demon- strating effectiveness. performance among all compared models, outperforming the Nvo-CDE model by approximately 0.7% on both datasets. This advantage stems from the framework's flexibility in kernel function design, enabling it to capture more complex feature-updating dynamics.
Workers and Amazon-ratings datasets, FADE-CDE achieves the best
30

---

Table 6: Node classification results(%).
The best and the second-best result for each criterion highlighted in red and blue, respectively.
Model
Roman-empire
Wiki-cooc
Questions
Workers
Amazon-ratings
TDE-GNN(87]
64.29=0.58
84.95=0.77
68.94=1.69
75.13=0.81
40.33=1.37
GRAND-I[56]
0.524-0053
91.58-+0.37
68.54=1.07
7..59=0.86
48.99:10.35
GRAND-nl[56]
71.60=0.58
92.03-+:0:66
70.67=11228
75.33-0.84 0.84
45.05:10.65
[raphBel[85]
69.47=0.37
90.30+0.50
70.79=0.99
73.02=0.92
43.63=0042
NSD[88]
77.50=0.67
92.06-+-0.40
69.25
79.81
0.99
37.96-10.20
ACMP[89]
71.27+0.59
92.68-10.37
71.18 1.03
75.03 0.92
44.76:10.52
CDE[10]
9..64-0028
97.99-10.38
75.17-00..
80.70 1.04
47.63-10.43
F-CDE(19] 19]
93.06=0.55
98.73++0688
75.17=0.99
82.68 0.866
0.86
49.01+0.56
D-CDE[38]
93.87=0.41
98.58=0.12
7.95
83.02=0.86
49.43=11226
Nvo-CDE[33]
93.42-+0.22
99.32++028
74.87=0023
83.33 0.65
50.09-0.40
FADE-CDE (ours)
93.46=0.32
98.9441.12
75.10=0.11
8.302-0.38
50.75=00455
H Datasets and More Experiments for Traffic Forecasting
H.1 Datasets and Setting
We evaluate the effectiveness of STDDE using five real-world traffic datasets: PcMSD7(M), PeMSD7(L), PeMS04, PeMS07, and PeMSO8. These datasets are sourced from the Caltrans Perfor- mance Measurement System [59], which collects traffic flow data every 30 seconds. For analysis, the data is aggregated into 5-minute intervals, resulting in 288 time steps one day. summary of the dataset statistics can be found in Table 7. These datasets are pre-divided into training, validation, and testing sets using 6:2:2 ratio. The training procedure and hyperparameter settings are kept consistent with those reported in [4]. For instance, the model is trained for 200 epochs using the Adam optimizer.
Table 7: Datasets for Trafffic Forecasting
Datasets
Sensors
Edges
Time Steps
PeMS04
307
340
16992
PeMS07
883
866
28224
PeMS08
170
295
17856
PeMS07(M)
228
1132
12672
PeMS07(L)
1026
10150
12672
Table Forecasting error on PeMSD7
PeMSD7
Model
MAERMSE MAPE
HA[60]
45.12 65.64 24.51 %
VAR[60]
50.22 75.63 32.22%
TCN[61]
32.72 42.23 14.26%
DSANet162]
31.36 49.11 14.43%
AGCRN]633
22.37 36.55 9.12%
STFGNN[64]
23,46
36.60 9.21%
Z-GCNETs(65]
21.77 35.17 9.25%
STGODE[6]
22.59 37.54 10.14%
STG-NCDE[4]
20.53 33.84 8.80%
STG-FADE (ours)
20.46
33.70 8.94%
31

---

700
STG-KatFDE
STG-KatFDE
STG-NCDE
600
STG-NCDE
500
Traffic Flow
Traffic Flo
400
300
200
100
50
252
216
Time Stop
Time Step
(a) Node 211 In PeMSD4
(b) Node 111 In PeMSD4
350
300
Trffic Flow
250
Traffic Flow
200
truth
truth
STG-KatFDE
100
STG-KatFDE
STG-NCDE
36
72
108
144
180
216
252
288
36
72
108
144
180
216
252
288
Time Stop
Time Stop
(c) Node 167 in PeMSD8
Node 123 in PeMSD8
Figure 1: Traffic forecasting visualization in PeMSD4 and PeMSD8
H.2 Experimental Results
From Table 8, our model STG-FADE achieves generally the best performance on PeMSD7 dataset. Compared to NDEs-based models, demonstrates a stronger ability to capture complex dynamics. Figure illustrates the predicted traffic flow from STG-FADE in comparison with STG-NCDE and the ground truth on PeMSD4 and PeMSD8 datasets. The horizontal axis denotes the time steps (5-minute intervals), and the vertical axis represents the traffic flow. A total of 288 time steps are selected, covering an entire 24-hour period.
Each subfigure corresponds to specific node and is annotated with zoomed-in region to highlight prediction differences in more dynamic complex traffic periods. Overall, both models demonstrate a strong ability to follow the ground truth trends. However, the proposed STG-FADE consistently achieves closer alignment with the ground truth, especially in rapidly changing regions.
Particularly in Figure 1:
Node 211 and Node 111 in PeMSD4 (top row) show that STG-FADE better captures sudden increases and local peaks, maintaining smoother yet accurate transitions.
Node 167 and Node 123 in PeMSD8 (bottom row) further validate STG-FADE superiority, with visibly reduced eиror margins in congested and fluctuating segments, as shown in the zoom-in windows.
These results support the quantitative findings discussed in the main text and demonstrate the robustness and generalization capacity of STG-FADE across different traffic environments.
H.3 Parameter Analysis
Hidden Dimension Analysis: Figure presents the performance of STG-FADE on PeMSD8 dataset with varying input feature dimensions: 16, 32, 64, and 128. It can be observed that as the feature dimension increases, the model's 's performance improves consistently across all three metrics. In particular, the lowest RMSE and MAE are achieved at dimension 128, indicating that
32

---

20
24
Table Forecasting error metrics (MAE, RMSE, MAPE) for different step sizes on PeMSD4 and PeMSD8 datasets.
PeMSD4
PeMSD8
Step size
MAE RMSE MAPE MAE RMSE
MAPE
19.48 31.34
12.94%
17.45
27.55
11.07%
MAV
19.88
31.83 13.14%
17.10 26.83 10.66%
Figure 2: Performance comparison of STG- FADE with varying feature dimensions on the PeMSD8 dataset.
higher-dimensional representations help capture more complex spatiotemporal patterns traffic data. Notably, the improvement becomes more pronounced when increasing the dimension from 32 to 64, and then stabilizes between 64 and 128. These findings suggest that while increasing feature dimensionality benefits performance, the marginal gain diminishes beyond a certain point.
Step Size Analysis: Table 9 reports the forecasting error metrics STG-FADE on the PeMSD4 and PeMSDS datasets, evaluated with varying step sizes, As the step size increases, MAE increases from 19.48 19.88, and RMSE from 31.34 to 31.83 on PeMSD4; while on PeMSD8, MAE changes marginally from 17.45 to 17.10 and RMSE slightly decreases from 27.55 to 26.83. The experiments suggest that a moderately larger step size contributes improved performance on PeMSD8, while has the opposite effect on PeMSD4.
Experiments for Urban Population Mobility and Biological Neural Spike Train Dynamics
I.1 Urban Population Mobility Prediction
Post-disaster urban mobility dynamics exhibit complex patterns driven by both the disruptive disaster context and underlying habitual mobility. The original population mobility data from SafeGraph records daily movements between Census Block Groups (CBGs) during the period from August to September 10, 2019. The authors aggregated these inter-CBG flows the county level obtain daily within-county and between-county population flows, representing intra-regional and inter-regional population flows nodes and edges, respectively [90]. Based this, we evaluated FADE on Florida's s population mobility data during Hurricane Dorian. This dataset records daily inter-regional movements within Florida.
We compared FADE against baselines including LSTM [91], AGCRN [63], NDCN [92], CG- ODE [93]. STG-NCDE [4], PatchTTST [94] and CDGON [90]. Following [90], utilize multiple evaluation metrics, including Mean Absolute Error (MAE), Normalized
Poot-Mean-Squal Error (NRMSE), and Coefficient of Determination (R²). As shown in Table 10, FADE achieves state-of- the-art performance. This is quintessential use case for FADE: the fractional operator enables long memory, while the adaptive attention kernel allows dynamic weighing of recent disaster-related vs. routine historical patterns. The model can down-weight pre-disaster commuting patterns during the hurricane and re-integrate them during recovery.
1.2 Biological Neural Spike Train Dynamics
We next tested FADE on biological time-series data neural spike trains from multiple animals and brain regions. Neural systems are inherently eestoryddependemt the evolution of neuron's 's's membrane potential influenced by prior inputs and spike events, and downstream firing patterns are shaped by this accumulated temporal context. We tested the experimental results spiking datasets
33

---

Table 10: Average prediction error of different models on post-disaster urban mobility
Model
MAE
NRMSE
R²
LSTM
002884.0938
0.6689
0.5526
AGCRN
273478. 9062
0.5377
0.7109
NDCN
406064.2812
0.3987
0.8411
CG-ODE
024787.1250
0.2936
0,9138
STG-NCDE
226016.
0.6683
0.5533
PatchTST
96736.0547
0.1632
0,9734
CDGON
59767.4805
0.0724
0,9948
FADE (ours)
24040.9355
0.0506
0.9974
Allen and Retina [95]. The Allen dataset contains spike train data from various brain regions and is designed to evaluate models for temporal and spatiotemporal neural activity classification. The Retina dataset provides spike trains from salamander retinal ganglion cells under four visual stimuli for stimulus-type classification. We compared FADE against baseline models, primarily including LSTM and Neural ODE [2]. As Table 11 shows, FADE outperforms standard recurrent and neural ODE models, highlighting its advantage in modeling nonlinear and temporally dependent neural systems.
Table 11: Test Accuracy of different models neuron spike train classification
Dataset
LSTM
Neural ODE
FADE (ours)
Allen
85.05
85.05 92.25
86.03
Retina
90.36
94.79
Limitations and Broader Impact
Our key contribution is using flexible kernel-attentin fractional neural ODE that replaces fixed power-law memory kernels traditional fractional differential equations. This framework FADE effectively unifies and extends integer-order, variable-order, and ri-Caputo dynamics within single continuous-deg depth framework. the same time, we have validated its effectiveness and applicability across various real- wal-world domains. However, there are some limitations of FADE. First, FADE is based on a deterministic equation. In practice, we often need to handle stochastic data and outputs with confidence intervals, which the current framework does not support. Second, the spatial interaction (i.e., spatial partial differential components) at given time not explicitly included in the framework with direct incorporation of physical laws. To make FADE more generally applicable, future work should explicitly include spatial interactions and consider stochastic models instead of purely deterministic formulation. From a broader societal perspective, FADE also carries potential risks of misuse or unintended consequences, such as in the transportation field. Therefore, it is essential to ensure that technological advancements lead to positive outcomes while keeping the application of FADE aligned with social values and ethical standards.
34


Reviews:
[{"review_id": "6WuODceCZD", "reviewer": "Reviewer_ikK2", "summary": "NeuralODEs have shown success in integrating differential equations. However, the integer order neuralODEs do not implicitly by construction account for memory effects or historical dependencies. Motivated by Fractional calculus which is capable of accounting for memory effects and historical dependencies, the authors propose a generalised kernel-attention fractional Neural ODE framework (KatFDE). While fractional neural networks use statich memory weightings that are not learnable or adaptive, the authors replace the fixed / static kernels with neural attention kernels that adaptively weight historical dependencies better and lay better emphasis on important temporal contributors. Extensive evaluation has been carried out on graph learning problems and spatio-temporal traffic flow forecasting tasks and the KatFDE was shown to significantly outperform integer order neuralODEs and existing fractional approaches such as NvoFDE, FROND, etc.", "questions": "The only question I have is, have the authors considered testing KatFDE spatio-temporally continuous non-linear temporally evolving systems such as fluid flows / heat conduction/ black-scholes/reaction-diffusion/allen-cahn systems? Would have been a nice plethora of systems to evaluate the KatFDE framework on to know where KatFDE is most likely applicable or even establish if KatFDE is universally applicable with more concrete evidence. **Actionable:** Please provide a set of limitations, possible extensions in a separate paragraph in the conclusion.", "limitations": "The limitations if any, are not addressed by the authors towards the end of the paper. Would like the authors to highlight what other open questions remain to make KatFDE universally applicable for spatio-temporal systems (Discrete/ Continuous)", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "**Strengths:**\n1. Strong mathematical background presented for the development of the KatFDE modeling framework. Convergence criterion, the kernel attention definitions, network construction. training methodology, and the overall modelling framework. \n2. A thorough literature survey presented to identify the gaps and lacunae in the literature and the objectives have been well constructed around the gaps with sufficient motivation. \n3. The numerical experiments are well detailed and a plethora of baseline models from literature are compared against technically sound evaluation metrics. \n4. The capability of modelling temporal dependencies, and memory effects is extremely critical when modelling strongly non-linear, non-ideal and temporally evolving systems. KatFDE would be a useful framework in such scenarios.", "was_revised": true, "final_justification": "My score remains unchanged but I have recommended a strong accept of this work as most of my concerns have been addressed in the rebuttal.\n1. Additional clear test results were presented for different spatio-temporally/ temporally evolving non-linear systems which showed that KatFDE is a clear win amongst the other methods presented.\n2. As recommended, the limitations of KatFDE in terms of being able to capture spatial interactions has also been addressed. \n\nImportantly, KatFDE architecture would be a novel and valuable addition in the galore of methods to work with temporally evolving nonlinear systems.", "rebuttal": "Thank you for your careful review of our paper. We are grateful for your recognition of our novelty.\n\nFurthermore, we would be honored if you could lend your insights during the discussion phase. Your perspective would be invaluable in clarifying any potential misunderstandings about our work and in contributing to a more robust discussion.\n\n\n\n## Testing Other Spatio-temporally Continuous Non-linear Temporally Evolving Systems \n\nWe thank the reviewer for this suggestion. To further demonstrate KatFDE’s generality and capability, we have added three additional experiments on complex temporally evolving systems.\n\n__1. Fluid Flow Prediction:__\n\nWe evaluated KatFDE on turbulent boundary-layer flow [H1], with velocity fields measured using particle image velocimetry at five Reynolds numbers (Re = 600, 980, 1370, 1780, 2220), each containing approximately 6,000 snapshots. We trained on four Reynolds numbers and tested on the fifth. Models observed 6 consecutive snapshots to predict the next 6, using a CNN encoder–decoder for spatial feature extraction and LSTM, Transformer, Neural ODE, or our KatFDE for latent state temporal evolution.\nTurbulent flows exhibit strong memory effects due to the cascade of energy across different scales and the persistence of coherent structures. The complex temporal evolution of turbulence makes it an ideal test case for our attention-based fractional framework, which can adaptively weight historical flow states based on their relevance to current dynamics. The preliminary prediction results are summarized in **Table R1**, demonstrating that KatFDE's adaptive memory mechanism effectively captures the nonlinear complexity inherent in turbulent flows, outperforming other approaches.\n\n| Model       | LSTM+CNN | Transformer+CNN | ODE+CNN | Ours+CNN |\n|--------------|----------|-----------------|---------|----------|\n| RMSE     | 1.5035   | 0.4701           | 0.5962  | **0.3215** |\n| MAE      | 0.6027   | 0.3532           | 0.2799  | **0.2041** |\n\n**Table R1**: Average prediction error of different models on turbulent vector field prediction.\n\n[H1] Towne, A., Dawson, S., Brès, G. A., Lozano-Durán, A., Saxton-Fox, T., Parthasarthy, A., Biler, H., Jones, A. R., Yeh, C.-A., Patel, H., Taira, K. (2022). A database for reduced-complexity modeling of fluid flows. AIAA Journal 61(7): 2867-2892.\n\n\n__2. Urban Population Mobility Prediction:__\n\nPost-disaster urban mobility dynamics exhibit complex patterns driven by both the disruptive disaster context and underlying habitual mobility. We evaluated KatFDE on Florida’s population mobility data during Hurricane Dorian (August 1–September 10, 2019), sourced from SafeGraph and reported in [H2].\nThis dataset records daily inter-regional movements within Florida. We compared KatFDE against baselines including LSTM, Transformer, and neural ODE variants. As shown in **Table R2**, KatFDE achieves state-of-the-art performance.\nThis is a clear use case for KatFDE: the fractional operator enables long memory, while the adaptive attention kernel allows dynamic weighing of recent disaster-related vs. routine historical patterns. The model can down-weight pre-disaster commuting patterns during the hurricane and re-integrate them during recovery.\n\n| Model         | MAE         | NRMSE  | R²     |\n|--------------|-------------|--------|--------|\n| LSTM          | 302884.0938 | 0.6689 | 0.5526 |\n| AGCRN         | 273478.9062 | 0.5377 | 0.7109 |\n| NDCN          | 406064.2812 | 0.3987 | 0.8411 |\n| CG-ODE        | 224787.1250 | 0.2936 | 0.9138 |\n| STG-NCDE      | 226016.1562 | 0.6683 | 0.5533 |\n| PatchTST      |  96736.0547 | 0.1632 | 0.9734 |\n| CDGON         |  59767.4805 | 0.0724 | 0.9948 |\n| **KatFDE (ours)** | **24040.9355** | **0.0506** | **0.9974** |\n\n**Table R1**: Average prediction error of different models on post-disaster urban mobility.\n\n[H2] Li, J., etc. Physics-informed neural ode for post-disaster mobility recovery. In KDD 2024.\n\n__3. Biological Neural Spike Train Dynamics:__\n\nWe next tested KatFDE on biological time-series data of neural spike trains from multiple animals and brain regions. Neural systems are inherently history-dependent: the evolution of a neuron's membrane potential is influenced by prior inputs and spike events, and downstream firing patterns are shaped by this accumulated temporal context.\nWe tested the experimental results on spiking datasets Allen and Retina [H3]. The Allen dataset contains spike train data from various brain regions and is designed to evaluate models for temporal and spatiotemporal neural activity classification. The Retina dataset provides spike trains from salamander retinal ganglion cells under four visual stimuli for stimulus-type classification.  As **Table R3** shows, KatFDE outperforms standard recurrent and neural ODE models, highlighting its advantage in modeling nonlinear and temporally dependent neural systems.\n\n\n| Dataset | LSTM  | Neural ODE | KatFDE (ours)    |\n|--------|--------|------------------|---------|\n| Allen  | 85.05 | 85.05      | **86.03** |\n| Retina    | 90.36| 92.25    | **94.79** |\n\n**Table R3**: Test Accuracy of different models on neuron spike train classification (%)\n\n\n[H3] Lazarevich I, Prokin I, Gutkin B, et al. Spikebench: An open benchmark for spike train time-series classification[J]. PLOS Computational Biology, 2023, 19(1): e1010792.\n\n\n### Limitations Discussion \n\nWe will include the following Limitations Discussion in the paper conclusion part:\n\nThere are some limitations of KatFDE.\nFirst, KatFDE is based on a deterministic equation. In practice, we often need to handle stochastic data and outputs with confidence intervals, which the current framework does not support.\nSecond, the spatial interaction (i.e., the spatial partial differential components) at a given time is not explicitly included in the framework with direct incorporation of physical laws. In this paper, we rely on other modules, such as GNN and CNN, to handle spatial interactions. A more natural and deeper fusion of these components may be explored in future work.\nThird, KatFDE may require higher computational cost when using nonlocal kernels.\nTo make KatFDE more generally applicable, future work should explicitly include spatial interactions and consider stochastic models instead of a purely deterministic formulation. In addition, designing more efficient numerical solvers and incorporating more efficient attention mechanisms will help reduce computation and memory usage, especially for large-scale datasets."}, {"review_id": "kEIsoBdvRF", "reviewer": "Reviewer_Qt1E", "summary": "This paper introduces KatFDE, a new framework for modeling continuous-time dynamics. The problem is that normal neural ODEs and also the fractional neural ODEs cannot well model systems with memory. The proposed method, KatFDE, uses a fractional differential equation but replaces the fixed mathematical kernel with a learnable attention kernel. This lets the model decide which past states are important for the current state. The authors give theory to show the model is well-posed and stable, and they test it on graph node classification and traffic forecasting, showing it can perform better than other methods.", "questions": "1. The paper claims the framework is general for systems with memory. Can you give concrete examples of systems (e.g., in physics or finance) where older FDE models do not work well but KatFDE would be better because of the adaptive kernel? Explaining this would make the motivation stronger. If you give good answer, I will raise my score.\n2. The experimental results are not consistent. For example, on some datasets your model is not better than the baseline. Can you give an analysis why the model performs well on some datasets but not others? This is needed to understand where the model is useful. A good analysis would make me raise the score.\n3. The ablation study is missing. This is a critical experiment to understand the value of the new attention kernel. Please, can you add a study that compares your proposed kernel to simpler versions? This would justify the model's complexity. If this is added, I would raise my score.\n4. The \"Related Work\" section is in the appendix. This is not standard. Can you move this section into the main paper? This is a major issue with the structure. Fixing this is important and would make me reconsider my score.\n5. In Section 3, you mix the review of known attention with your proposed model. To make it clear, can you change Section 3.1.1 to separate the explanation of standard attention from your new fractional attention kernel? This will help me to see better your contribution.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 1, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- Originality: The idea to use a learnable neural attention inside the fractional operator is a novel contribution. It has potential for modeling complex systems.\n- Theoretical Support: The paper gives rigorous mathematical proofs for the KatFDE framework. This analysis of boundedness and well-posedness gives confidence in the method.\n- Generalization: The framework is a good generalization that connects integer-order ODEs, fractional ODEs, and attention mechanisms in one formulation.\n\nWeaknesses:\n- Clarity: The paper is written in a style that is hard to read. It is like a math textbook, not a research paper. For example, putting the definition of a derivative (Equation 1) is not necessary. Also, the \"Related Work\" section is in the appendix, which is very unorthodox.\n- Experiments: The experimental part has major flaws. I see no ablation study to show why the attention kernel is important. The improvement over other models is also not consistent across all datasets, which makes me question if the model is truly effective.\n- Presentation of Novelty: It is hard to understand what is the exact new contribution. The paper mixes the explanation of the model with review of well-known things like scaled dot-product attention, so the novelty is not clear.", "was_revised": true, "final_justification": "The authors have provided extensive experiment results on real-world datasets against a wide range of compelling baselines, and have made clarifications about the ablation studies. The authors are encouraged to update their manuscript to include related works and be specific about the novelty of their architecture. If possible, please move some of the mathematical details to the appendix for ease of reading.", "rebuttal": "## I. Novelty, Clarity, and Writing Style\n\n1. Our key contribution is using a flexible kernel-attention fractional neural ODE that replaces fixed power-law memory kernels in traditional fractional differential equations. This framework effectively **unifies and extends integer-order, variable-order, and  $\\psi$-Caputo dynamics within a single continuous-depth framework.** Furthermore, we **provide rigorous theoretical foundations** for our framework, including boundedness under both singular and nonsingular cases. We establish the well-posedness of the resulting neural integral equations using Banach fixed-point arguments, ensuring solution uniqueness under appropriate conditions.\nThe scaled dot-product attention mechanism itself is well known, and _we do not claim novelty in its formulation_, but showing its effectiveness as a learnable kernel in fractional derivatives is new and one of our contributions.\n\n2. Writing Style:  Our paper aims to bridge the mathematics and machine learning communities, where some readers may be unfamiliar with fractional definitions. We therefore **introduce the concepts progressively**, starting with the simple first-order definition, then moving to the constant-order Caputo definition, and finally to the variable- and $\\psi$-order definitions. We explicitly outline these derivatives because _their comparison naturally motivates our general KatFDE formulation_, as discussed prior to Eq. (15).\n\n3. We will bring the related work into the main paper if it is accepted, since there is allowance for one more page. Thank you for your suggestion.\n\n\n***\n\n## II. New Supporting Systems with Experimental Results\n\nWe appreciate the reviewer’s critique of the experimental evaluation and thank them for motivating us to test our framework on more complex, non-linear, temporally evolving dynamical systems. The marginal advantage reported in the paper draft may be attributable to the evaluated datasets not being sufficiently challenging to fully demonstrate the strengths of our framework.\n\nTo further demonstrate KatFDE’s generality and capability, we have added three additional experiments on complex temporally evolving systems.\n\n__1. Fluid Flow Prediction:__\n\nWe evaluated KatFDE on turbulent boundary-layer flow [H1], with velocity fields measured using particle image velocimetry at five Reynolds numbers (Re = 600, 980, 1370, 1780, 2220), each containing approximately 6,000 snapshots. We trained on four Reynolds numbers and tested on the fifth. Models observed 6 consecutive snapshots to predict the next 6, using a CNN encoder–decoder for spatial feature extraction and LSTM, Transformer, Neural ODE, or our KatFDE for latent state temporal evolution.\nTurbulent flows exhibit strong memory effects due to the cascade of energy across different scales and the persistence of coherent structures. The non-local temporal dependencies in turbulence make it an ideal testbed for our attention-based fractional framework, which can adaptively weight historical flow states based on their relevance to current dynamics. The preliminary prediction results are summarized in **Table R1**, demonstrating that KatFDE's adaptive memory mechanism effectively captures the nonlinear multi-scale temporal dependencies inherent in turbulent flows, outperforming other approaches.\n\n| Model       | LSTM+CNN | Transformer+CNN | ODE+CNN | ``Our KatFDE+CNN`` |\n|--------------|----------|-----------------|---------|----------|\n| RMSE     | 1.5035   | 0.4701           | 0.5962  | **0.3215** |\n| MAE      | 0.6027   | 0.3532           | 0.2799  | **0.2041** |\n\n**Table R1**: Average prediction error of different models on turbulent vector field prediction.\n\n[H1] Towne, A., Dawson, S., Brès, G. A., Lozano-Durán, A., Saxton-Fox, T., Parthasarthy, A., Biler, H., Jones, A. R., Yeh, C.-A., Patel, H., Taira, K. (2022). A database for reduced-complexity modeling of fluid flows. AIAA Journal 61(7): 2867-2892.\n\n\n__2. Urban Population Mobility Prediction:__\n\nPost-disaster urban mobility dynamics exhibit complex patterns driven by both the disruptive disaster context and underlying habitual mobility. We evaluated KatFDE on Florida’s population mobility data during Hurricane Dorian (August 1–September 10, 2019), sourced from SafeGraph and reported in [H2].\nThis dataset records daily inter-regional movements within Florida. We compared KatFDE against baselines including LSTM, Transformer, and neural ODE variants. As shown in **Table R2**, KatFDE achieves state-of-the-art performance.\nThis is a clear use case for KatFDE: the fractional operator enables long memory, while the adaptive attention kernel allows dynamic weighing of recent disaster-related vs. routine historical patterns. \n\n| Model         | MAE         | NRMSE  | R²     |\n|--------------|-------------|--------|--------|\n| LSTM          | 302884.0938 | 0.6689 | 0.5526 |\n| AGCRN         | 273478.9062 | 0.5377 | 0.7109 |\n| NDCN          | 406064.2812 | 0.3987 | 0.8411 |\n| CG-ODE        | 224787.1250 | 0.2936 | 0.9138 |\n| STG-NCDE      | 226016.1562 | 0.6683 | 0.5533 |\n| PatchTST      |  96736.0547 | 0.1632 | 0.9734 |\n| CDGON         |  59767.4805 | 0.0724 | 0.9948 |\n| **KatFDE (ours)** | **24040.9355** | **0.0506** | **0.9974** |\n\n**Table R1**: Average prediction error of different models on post-disaster urban mobility.\n\n[H2] Li, J., etc. Physics-informed neural ode for post-disaster mobility recovery. In KDD 2024.\n\n__3. Biological Neural Spike Train Dynamics:__\n\nWe next tested KatFDE on biological time-series data of neural spike trains from multiple animals and brain regions. Neural systems are inherently history-dependent: the evolution of a neuron's membrane potential is influenced by prior inputs and spike events, and downstream firing patterns are shaped by this accumulated temporal context.\nWe tested the experimental results on spiking datasets Allen and Retina [H3]. The Allen dataset contains spike train data from various brain regions and is designed to evaluate models for temporal and spatiotemporal neural activity classification. The Retina dataset provides spike trains from salamander retinal ganglion cells under four visual stimuli for stimulus-type classification. As **Table R3** shows, KatFDE outperforms standard recurrent and neural ODE models, highlighting its advantage in modeling nonlinear and temporally dependent neural systems. Due to limited time during the rebuttal period, we will include more baselines in the final version.\n\n\n| Dataset | LSTM  | Neural ODE | KatFDE (ours)    |\n|--------|--------|------------------|---------|\n| Allen  | 85.05 | 85.05      | **86.03** |\n| Retina    | 90.36| 92.25    | **94.79** |\n\n**Table R3**: Test Accuracy of different models on neuron spike train classification (%)\n\n\n[H3] Lazarevich I, Prokin I, Gutkin B, et al. Spikebench: An open benchmark for spike train time-series classification[J]. PLOS Computational Biology, 2023, 19(1): e1010792.\n\n***\n\n## III. Clarify Ablation Study:\n\nWe emphasize that the baselines in Table 1 in the paper already provide a clear ablation analysis. This is fully consistent with our explanation of the novelty, which is framed relative to the first-order derivative, the constant-order Caputo derivative, and the variable-order derivative definitions. **Each of these progressively introduces greater advantages and flexibility to the continuous system. Our framework fully generalizes and unifies all of these approaches, and shows clear advantages.**\n\nThe ablation study is shown in **Table R4**, reproduced from Table 1 in the paper.\n\n| Model         | Comparison                | Cora         | Citeseer     | Computer     |\n|---------------|---------------------------|--------------|--------------|--------------|\n| GRAND-l       | first-order ODE [48]      | 83.6±1.0     | 73.4±0.5     | 83.7±1.2     |\n| F-GRAND-l     | constant-order FDE [18]   | 84.8±1.1     | 74.0±1.5     | 84.4±1.5     |\n| D-GRAND-l     | constant-order FDE [51]   | 85.1±1.3     | 74.5±1.1     | 87.3±1.3     |\n| Nvo-GRAND-l   | variable-order FDE [32]   | 86.0±0.5     | 75.6±0.8     | 87.9±0.8     |\n| **KatFDE-l**      | **our unified learnable FDE** | **86.4±0.5** | **76.1±0.6** | **88.3±0.9** |\n\n\n| Model         | Comparison                | Cora         | Citeseer     | Computer     |\n|---------------|---------------------------|--------------|--------------|--------------|\n| GRAND-nl      | first-order ODE [48]      | 82.3±1.6     | 70.9±1.0     | 82.4±2.1     |\n| F-GRAND-nl    | constant-order FDE [18]   | 83.2±1.1     | 74.7±1.9     | 84.1±0.9     |\n| D-GRAND-nl    | constant-order FDE [51]   | 83.9±1.3     | 74.8±1.6     | 87.1±1.0     |\n| Nvo-GRAND-nl  | variable-order FDE [32]   | 85.4±1.0     | 75.9±0.6     | 87.2±1.4     |\n| **KatFDE-nl**     | **our unified learnable FDE** | **86.0±0.4** | **76.2±0.8** | **87.7±0.9** |\n\n**Table R4**: Ablation study among differential-equation-based continuous models. _The results show that through generalization, our model achieves an advantage._"}, {"review_id": "lpeFdHn1RV", "reviewer": "Reviewer_jEHE", "summary": "This paper presents KatFDE, a neural model that replaces fixed fractional kernels with a learnable attention kernel so it can decide which past information matters. It proves the solver is sound and reports higher accuracy than earlier integer-order and fixed-kernel fractional models on graph node-classification and traffic-forecasting benchmarks.", "questions": "What are the training and inference NFE counts? What is the wall-clock time for running KatFDE, along with the GPU model, batch size, and solver tolerances used? What is KatFDE’s peak memory consumption? Compared with vanilla Caputo solvers and other baselines, does KatFDE require fewer resources to run?\n\nCompared to other non–differential equation methods (such as DDGCRN [1] and Hierarchical‑Attention‑LSTM [2]), what advantages does KatFDE offer for tasks like traffic forecasting?\n\n[1]https://www.sciencedirect.com/science/article/pii/S0031320323003710?casa_token=vWU0ZxgyAdEAAAAA:fjEXyBsi65YNLn1MWuAUrFlvrtQouPXFEJPRKd_jzk1b5T_2qo7qgBezHmiowdNVXhodvpittQ\n\n[2] https://arxiv.org/abs/2201.05760", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strength**:\nThe idea of using a kernel-attention fractional neural ODE that replaces fixed power-law memory kernels with learnable attention mechanisms is novel. This method effectively unifying and extending integer-, variable-order, and ψ-Caputo dynamics within a single continuous-depth framework. While I did not examine every proof in detail, the theoretical foundation of the paper appears robust and well-developed.\n\n**Weakness**:\n- There is a typo on line 157 (“andmaps”). \n\n- Several important details are missing, such as the choice of step size $h$, the window size used in KatFDE, specifications of KatFDE-I and KatFDE-nl, training settings, and model architecture. \n\n- The paper also lacks an analysis of computational complexity and memory usage. Since standard self-attention scales with $O(T^2)$ time and $O(T^2)$ memory, KatFDE’s use of it raises concerns about efficiency. The absence of such information is a major concern.\n\n- Additionally, the absence of visualizations makes the paper harder to follow. Lastly, the discussion would benefit from including the following related works \\[1–3].\n\n[1] Zhao, Chunna, et al. \"FLRNN-FGA: Fractional-Order Lipschitz Recurrent Neural Network with Frequency-Domain Gated Attention Mechanism for Time Series Forecasting.\" Fractal and Fractional 8.7 (2024): 433.\n\n[2] Vellappandi, Madasamy, and Sangmoon Lee. \"Physics-informed neural fractional differential equations.\" Applied Mathematical Modelling 145 (2025): 116127.\n\n[3] Fernandez, Arran, Mehmet Ali Özarslan, and Dumitru Baleanu. \"On fractional calculus with general analytic kernels.\" Applied Mathematics and Computation 354 (2019): 248-265.", "was_revised": true, "final_justification": "The authors address most of my concerns by providing extensive experimental results on real-world tasks. KatFDE demonstrates impressive performance compared to numerous benchmarks. The inclusion of training details enhances the paper's clarity, and the promised visualizations will further improve readability. Although the time complexity is still missing, the authors report memory usage, which is comparable to Transformer + CNN models. Given the paper’s scope and KatFDE’s strong performance, I consider this a borderline accept.", "rebuttal": "## Experimental details\n\nSome experimental details are presented in the appendix, and more will be added for completeness. The codebase for reproducing our experiments with exact parameters will be fully public.\nFor brevity, we describe the setup for the node classification task on graphs. The experiments are implemented using the PyTorch framework on a single NVIDIA RTX4090 24GB GPU. For different datasets, parameters such as time steps and evolution time are slightly adjusted. Taking the Cora dataset as an example, we set t = 25, step size = 1, learning rate = 0.01, weight decay = 0.05, epoch = 800, and dim = 256. Following [48, 18], the model architectures of KatFDE-l and KatFDE-nl are provided in Appendix F. The visualization results are shown in Appendix I. If the paper is accepted, we will move the visualization results into the main text. Furthermore, we will include attention heatmaps in the revised version to provide better visualization, as suggested by Reviewer jCMz.\n\n***\n\n\n## Computational complexity and memory usage. \n\nThe main focus of this work is to propose a unified framework of FDE methods from a new perspective and to verify its effectiveness. This framework opens new possibilities, such as fusing the attention mechanism with the memory kernel of FDE, which is investigated in this work. We focus only on the standard attention mechanism. Therefore, the extra computational complexity and memory usage from the attention mechanism are inevitable and standard.\n\n\nTo further demonstrate KatFDE’s generality and capability, we have added three additional experiments on complex temporally evolving systems (we refer the reviewer to the full results presented in the top rebuttal to Reviewer ikK2). Here we present the **fluid flow prediction task with accuracy and memory consumption comparison:**\n\nWe evaluated KatFDE on turbulent boundary-layer flow [H1], with velocity fields measured using particle image velocimetry at five Reynolds numbers (Re = 600, 980, 1370, 1780, 2220), each containing approximately 6,000 snapshots. We trained on four Reynolds numbers and tested on the fifth. Models observed 6 consecutive snapshots to predict the next 6, using a CNN encoder–decoder for spatial feature extraction and LSTM, Transformer, Neural ODE, or our KatFDE for latent state temporal evolution.\nTurbulent flows exhibit strong memory effects due to the cascade of energy across different scales and the persistence of coherent structures. The non-local temporal dependencies in turbulence make it an ideal testbed for our attention-based fractional framework, which can adaptively weight historical flow states based on their relevance to current dynamics. \n\nThe preliminary prediction results are summarized in **Table R5**, demonstrating that KatFDE's adaptive memory mechanism effectively captures the nonlinear multi-scale temporal dependencies inherent in turbulent flows, outperforming other approaches. The GPU memory usage of KatFDE is slightly higher than the baseline LSTM and ODE models, but comparable to the Transformer. This is expected since the extra computational complexity and memory usage from the attention mechanism are inevitable and standard. Importantly, with nearly the same GPU memory usage, our continuous KatFDE framework performs much better than the standard Transformer.\n\n\nTo alleviate GPU memory usage, due to the flexibility of KatFDE, one can incorporate efficient attention mechanisms such as [4], which can reduce the computational cost from $O\\left(L^2\\right)$ to $O(L \\log L)$ in practice. However, investigating this type of improvement and other variants is beyond the scope of this work and will be left for future research.\n\n| Model       | LSTM+CNN | Transformer+CNN | ODE+CNN | ``Our KatFDE+CNN`` |\n|--------------|----------|-----------------|---------|----------|\n| RMSE     | 1.5035   | 0.4701           | 0.5962  | **0.3215** |\n| MAE      | 0.6027   | 0.3532           | 0.2799  | **0.2041** |\n| Training Memory | 11372MiB   | 14142MiB          | 11250MiB  | 14722MiB |\n\n**Table R5**: Average prediction error and memory consumption of different models on turbulent vector field prediction.\n\n***\n\n\n## Advantages of KatFDE compared to the references\n\nRecall that KatFDE is built from a general integral equation that explicitly incorporates the attention mechanism [3], while DDGCRN [1] and Hierarchical‑Attention‑LSTM [2] are not equation-based and rely on LSTM architectures. The advantages of KatFDE for tasks like traffic forecasting and **the new tested spatio-temporally continuous non-linear temporally evolving systems** (we refer the reviewer to the full results presented in the top rebuttal to Reviewer ikK2) are threefold:\ni) The forecasting data may follow some first principles and thus lead to an equation. Our integral equation is general enough to cover a wide range of such principles, and the kernel is set to be learnable so that KatFDE can capture them accurately. In this context, KatFDE is expected to perform better;\nii) KatFDE naturally inherits temporal relations from the integral equation and incorporates the attention mechanism for temporal modeling. Both are well suited for time series data with long-range dependencies, which are hard to model under LSTM frameworks;\niii) KatFDE is more flexible and compatible with recent efficient attention mechanisms [4].\n\n[1]Weng Wenchao, et al. A decomposition dynamic graph convolutional recurrent network for traffic forecasting[J]. Pattern Recognition, 2023, 142: 109670.\n\n[2]Zhang Tianya. Network Level Spatial Temporal Traffic Forecasting with Hierarchical Attention LSTM (HierAttnLSTM)\n\n[3]A. Vaswani, et al. Attention is all you need, NeurlPS2017. \n\n[4]Zhou Haoyi, et al. Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting, AAAI2021."}, {"review_id": "AgjKdbtmQD", "reviewer": "Reviewer_jCMz", "summary": "This paper proposes KatFDE, a generalized fractional neural ODE framework with learnable attention kernels. Instead of using fixed power-law kernels like in traditional fractional calculus, KatFDE introduces an attention-based kernel that adaptively weighs past hidden states. The framework unifies existing integer-, fractional-, and variable-order models, and provides theoretical guarantees (boundedness, convergence, etc.). Experiments on graph classification and traffic forecasting show improved or comparable performance.", "questions": "1) Can you show how far KatFDE actually looks into the past especially in time seires benchmarks? (e.g., attention heatmaps or span stats)\n2) How does KatFDE compare with Informer/FEDformer/ContiFormer?\n3) Ablation: what happens if you remove attention and just use fractional kernels?\n4) Any insights from the learned kernels? Are they interpretable?", "limitations": "Limited experiment scope and lack of behavioral analysis. No strong evidence that long-range dependencies are captured better than existing methods. Claims about generality aren’t fully supported by the benchmarks used.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "+ Conceptually clean integration of fractional dynamics and attention.\n+ Theoretical analysis is solid. Unified view is elegant.\n+ Better performance than previous FDE models like FROND and NvoFDE.\n\n– Claims about capturing long-range dependencies aren’t backed by empirical evidence (no attention weight analysis or memory span plots).\n– Evaluation is narrow: just two tasks. No irregular or noisy time series.\n– No comparison with recent Transformer-based time-series models (Informer, FEDformer, etc.).\n– Little discussion about model behavior or efficiency.\n– Conclusion is underwhelming.", "was_revised": true, "final_justification": "The authors' rebuttal, which includes new, well-designed experiments on challenging time-series tasks, has thoroughly addressed my primary concern regarding the narrow scope of the initial evaluation. The results from these new benchmarks provide compelling evidence for the KatFDE framework's superior modeling capacity and generality. However, the paper's overall contribution, while significantly improved, does not entirely warrant a change in my initial assessment of its impact and scope. Therefore, I have decided to maintain my original score.", "rebuttal": "## I. New Supporting Experimental Results\n\nWe appreciate the reviewer’s critique of the experimental evaluation and thank them for motivating us to test our framework on more complex, non-linear, temporally evolving dynamical systems. The marginal advantage reported in the original paper may be attributable to the evaluated system not being sufficiently challenging to fully demonstrate the strengths of our framework.\n\nTo further demonstrate KatFDE’s generality and capability, we have added three additional experiments on complex temporally evolving systems.\n\n__1. Fluid Flow Prediction:__\n\nWe evaluated KatFDE on turbulent boundary-layer flow [H1], with velocity fields measured using particle image velocimetry at five Reynolds numbers (Re = 600, 980, 1370, 1780, 2220), each containing approximately 6,000 snapshots. We trained on four Reynolds numbers and tested on the fifth. Models observed 6 consecutive snapshots to predict the next 6, using a CNN encoder–decoder for spatial feature extraction and LSTM, Transformer, Neural ODE, or our KatFDE for latent state temporal evolution.\nTurbulent flows exhibit strong memory effects due to the cascade of energy across different scales and the persistence of coherent structures. The non-local temporal dependencies in turbulence make it an ideal testbed for our attention-based fractional framework, which can adaptively weight historical flow states based on their relevance to current dynamics. The preliminary prediction results are summarized in **Table R1**, demonstrating that KatFDE's adaptive memory mechanism effectively captures the nonlinear multi-scale temporal dependencies inherent in turbulent flows, outperforming other approaches.\n\n| Model       | LSTM+CNN | Transformer+CNN | ODE+CNN | Ours+CNN |\n|--------------|----------|-----------------|---------|----------|\n| RMSE     | 1.5035   | 0.4701           | 0.5962  | **0.3215** |\n| MAE      | 0.6027   | 0.3532           | 0.2799  | **0.2041** |\n\n**Table R1**: Average prediction error of different models on turbulent vector field prediction.\n\n[H1] Towne, A., Dawson, S., Brès, G. A., Lozano-Durán, A., Saxton-Fox, T., Parthasarthy, A., Biler, H., Jones, A. R., Yeh, C.-A., Patel, H., Taira, K. (2022). A database for reduced-complexity modeling of fluid flows. AIAA Journal 61(7): 2867-2892.\n\n\n__2. Urban Population Mobility Prediction:__\n\nPost-disaster urban mobility dynamics exhibit complex patterns driven by both the disruptive disaster context and underlying habitual mobility. We evaluated KatFDE on Florida’s population mobility data during Hurricane Dorian (August 1–September 10, 2019), sourced from SafeGraph and reported in [H2].\nThis dataset records daily inter-regional movements within Florida. We compared KatFDE against baselines including LSTM, Transformer, and neural ODE variants. As shown in **Table R2**, KatFDE achieves state-of-the-art performance.\nThis is a quintessential use case for KatFDE: the fractional operator enables long memory, while the adaptive attention kernel allows dynamic weighing of recent disaster-related vs. routine historical patterns. The model can down-weight pre-disaster commuting patterns during the hurricane and re-integrate them during recovery.\n\n| Model         | MAE         | NRMSE  | R²     |\n|--------------|-------------|--------|--------|\n| LSTM          | 302884.0938 | 0.6689 | 0.5526 |\n| AGCRN         | 273478.9062 | 0.5377 | 0.7109 |\n| NDCN          | 406064.2812 | 0.3987 | 0.8411 |\n| CG-ODE        | 224787.1250 | 0.2936 | 0.9138 |\n| STG-NCDE      | 226016.1562 | 0.6683 | 0.5533 |\n| PatchTST      |  96736.0547 | 0.1632 | 0.9734 |\n| CDGON         |  59767.4805 | 0.0724 | 0.9948 |\n| **KatFDE (ours)** | **24040.9355** | **0.0506** | **0.9974** |\n\n**Table R2**: Average prediction error of different models on post-disaster urban mobility.\n\n[H2] Li, J., etc. Physics-informed neural ode for post-disaster mobility recovery. In KDD 2024.\n\n__3. Biological Neural Spike Train Dynamics:__\n\nWe next tested KatFDE on biological time-series data of neural spike trains from multiple animals and brain regions. Neural systems are inherently history-dependent: the evolution of a neuron's membrane potential is influenced by prior inputs and spike events, and downstream firing patterns are shaped by this accumulated temporal context.\nWe tested the experimental results on spiking datasets Allen and Retina [H3]. The Allen dataset contains spike train data from various brain regions and is designed to evaluate models for temporal and spatiotemporal neural activity classification. The Retina dataset provides spike trains from salamander retinal ganglion cells under four visual stimuli for stimulus-type classification.  As **Table R3** shows, KatFDE outperforms standard recurrent and neural ODE models, highlighting its advantage in modeling nonlinear and temporally dependent neural systems.\n\n\n\n| Dataset | LSTM  | Neural ODE | KatFDE (ours)    |\n|--------|--------|------------------|---------|\n| Allen  | 85.05 | 85.05      | **86.03** |\n| Retina   | 90.36| 92.25    | **94.79** |\n\n**Table R3**: Test Accuracy of different models on neuron spike train classification (%)\n\n\n[H3] Lazarevich I, Prokin I, Gutkin B, et al. Spikebench: An open benchmark for spike train time-series classification[J]. PLOS Computational Biology, 2023, 19(1): e1010792.\n\n***\n\n## II. Ablation Study: If Remove Learnable Attention:\n\nWe emphasize that the comparisons in Table 1 already provide a clear ablation analysis. This is fully consistent with our explanation of the novelty, which is framed relative to the first-order derivative, the constant-order Caputo derivative, and the variable-order derivative definitions. **Each of these progressively introduces greater advantages and flexibility to the continuous system. Our framework fully generalizes and unifies all of these approaches.**\n\nThe ablation study is shown in **Table R4**, reproduced from Table 1 in the paper.\n\n| Model         | Comparison                | Cora         | Citeseer     | Computer     |\n|---------------|---------------------------|--------------|--------------|--------------|\n| GRAND-l       | first-order ODE [48]      | 83.6±1.0     | 73.4±0.5     | 83.7±1.2     |\n| F-GRAND-l     | constant-order FDE [18]   | 84.8±1.1     | 74.0±1.5     | 84.4±1.5     |\n| D-GRAND-l     | constant-order FDE [51]   | 85.1±1.3     | 74.5±1.1     | 87.3±1.3     |\n| Nvo-GRAND-l   | variable-order FDE [32]   | 86.0±0.5     | 75.6±0.8     | 87.9±0.8     |\n| **KatFDE-l**      | **our unified learnable FDE** | **86.4±0.5** | **76.1±0.6** | **88.3±0.9** |\n\n\n| Model         | Comparison                | Cora         | Citeseer     | Computer     |\n|---------------|---------------------------|--------------|--------------|--------------|\n| GRAND-nl      | first-order ODE [48]      | 82.3±1.6     | 70.9±1.0     | 82.4±2.1     |\n| F-GRAND-nl    | constant-order FDE [18]   | 83.2±1.1     | 74.7±1.9     | 84.1±0.9     |\n| D-GRAND-nl    | constant-order FDE [51]   | 83.9±1.3     | 74.8±1.6     | 87.1±1.0     |\n| Nvo-GRAND-nl  | variable-order FDE [32]   | 85.4±1.0     | 75.9±0.6     | 87.2±1.4     |\n| **KatFDE-nl**     | **our unified learnable FDE** | **86.0±0.4** | **76.2±0.8** | **87.7±0.9** |\n\n**Table R4**: Ablation study among differential-equation-based continuous models. _The results show that through generalization, our model achieves an advantage._\n\n***\n\n## III. Interpretability of the Kernel\n\nThe complex fully learnable kernel is inherently difficult to interpret. We will include attention heatmaps in the revised version to provide better visualization.\nSeveral traditional fractional derivative operators (see Eq10. and Eq12. in the paper) define\n$$\nK(t, \\tau, \\mathbf{x}(t), \\mathbf{x}(\\tau))=\\frac{\\psi^{\\prime}(\\tau)(\\psi(t)-\\psi(\\tau))^{\\alpha-1}}{\\Gamma(\\alpha)} \\quad \\text { or } \\quad K(t, \\tau, \\mathbf{x}(t), \\mathbf{x}(\\tau)) = \\frac{(t-\\tau)^{\\alpha-1}}{\\Gamma(\\alpha)},\n$$\nwhich depend only on the temporal distance between states. In these cases, __closer states exert a stronger influence on each other than more distant ones__. For a general kernel $K$, however, direct interpretability is considerably more challenging.\n\n***\n\n## IV. Comparison with Informer/FEDformer/ContiFormer\n\nAs demonstrated before, this work aims to propose an innovative framework that is flexible and scalable for a wide range of applications. Within this framework, we verify the performance of the fusion of FDE model and scaled dot product attention mechanism. Informer[w1], FEDformer[w2] and ContiFormer[w3] are proposed to improve transformer either by reducing the complexity from $O(L^2)$ to $O(L\\log L)$ or even $O(L)$ with ProbSparse self-attention or frequency enhanced attention with Fourier/wavelet transform respectively, or using averaged continuous inner product instead of dot product in transformer to tackle irregular time series. These advanced strategies are essentially compatible with our framework. Specifically, one can readily use attention mechanism proposed in Informer, FEDformer and ContiFormer by specifying corresponding kernels in (20). Therefore, we will work with Informer,FEDformer and ContiFormer in the current framework, instead of comparing with them.\n\n[w1]Zhou Haoyi, et al. Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting, AAAI2021. \n\n[w2]Zhou tian, et al. FEDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecasting, ICML2022.\n\n[w3] Chen Yuqi,et al. ContiFormer: Continuous-Time Transformer for Irregular Time Series Modeling, NeurlPS2023."}]

Author rebuttal:

Meta-review: This paper introduces KatFDE, a generalised kernel-attention fractional Neural ODE framework motivated by fractional calculus. The key idea is to replace fixed power-law memory kernels in traditional fractional differential equations with a flexible, learnable kernel based on attention mechanisms. This design unifies integer-order, variable-order, and Caputo dynamics within a single continuous-depth framework, supported by rigorous theoretical analysis. The authors establish boundedness under both singular and nonsingular cases and prove well-posedness of the neural integral equations using Banach fixed-point arguments, ensuring uniqueness of solutions. While the scaled dot-product attention mechanism itself is not novel, its use as a learnable kernel within fractional derivatives is a fresh and meaningful contribution.

The empirical evaluation is strong. The numerical experiments are comprehensive and compare KatFDE against a wide range of baselines using appropriate metrics. The results demonstrate KatFDE’s ability to effectively model long-term temporal dependencies and memory effects, which is particularly important in non-linear, non-ideal, and evolving systems.

The presentation is carefully crafted to bridge mathematics and machine learning audiences. The gradual introduction of fractional calculus concepts—from first-order to variable-order definitions—makes the paper accessible without sacrificing rigor. This pedagogical approach strengthens the work’s potential impact across communities.
Decision: Accept (poster)