KoNODE: Koopman-Driven Neural Ordinary Differential Equations with Evolving Parameters for Time Series Analysis
Authors: Hanru Bai
Weiyang Ding
Venue: ICML 2025 poster (2025)
Keywords: Neural ODE; Koopman Operators; Parameter Modeling; Time Series Analysis; Dynamics Modeling;
Primary area: general_machine_learning->sequential_network_and_time_series_modeling

Abstract: Neural ordinary differential equations (NODEs) have demonstrated strong capabilities in modeling time series. However, existing NODE- based methods often focus solely on the surface-level dynamics derived from observed states, which limits their ability to capture more complex underlying behaviors. To overcome this challenge, we propose KoNODE, a Koopman-driven NODE framework that explicitly models the evolution of ODE parameters over time to encode deep-level information. KoNODE captures the essential yet simple intrinsic linear dynamics that govern the surface dynamics by employing Koopman operators. Our framework operates at three hierarchical levels: the observed state dynamics, the parameter dynamics, and the Koopman linear dynamics, representing the fundamental driving rules of the state dynamics. The proposed approach offers significant improvements in two critical time series tasks: long-term prediction (enabled by the simple linear dynamics) and generalization to new data (driven by the evolving ODE parameters). We validate KoNODE through experiments on synthetic data from complex dynamic systems and real-world datasets, demonstrating its effectiveness in practical scenarios.

KoNODE: Koopman-Driven Neural Ordinary Differential Equations with Evolving Parameters for Time Series Analysis
Hanru Bai Weiyang Ding
Abstract
ory networks (LSTMs) (Hochreiter & Schmidhuber, 1997), primarily focus on learning discrete-time transition patterns from these observations. However, these methods do not explicitly model the continuous dynamics that govern the system. Since the available observations are often partial and noisy representations of the true dynamics, overlook- ing the intrinsic temporal continuity can lead to suboptimal performance in time-series analysis (Abeliuk et al., 2020),
Neural ordinary differential equations (NODEs) have demonstrated strong capabilities in model- ing time series. However, existing NODE-based methods often focus solely on the surface-level dynamies derived from observed states, which limits their ability to capture more complex un- derlying behaviors. To overcome this challenge, we propose KoNODE, a Koopman-drives NODE framework that explicitly models the evolution of ODE parameters over time to encode deep-level information. KoNODE captures the essential yet simple intrinsic linear dynamics that govern the surface dynamics by employing Koopman opera- tors. Our framework operates at three hierarchical levels: the observed state dynamics, the parame- ter dynamics, and the Koopman linear dynamics, representing the fundamental driving rules of the state dynamics. The proposed approach offers sig- nificant improvements in two critical time series tasks: long-term prediction (enabled by the simple linear dynamics) and generalization to new data (driven by the evolving ODE parameters). We validate KoNODE through experiments on syn- thetic data from complex dynamic systems and world datasets, demonstrating its effective- ness practical scenarios. Our code is available at https//giihbb.comm/BaaiieKOOooO
Ordinary Differential Equations (ODEs) provide a contin- uous framework for modeling time-series data. When the explicit form of the ODEs is unknown and only discrete observations are available, Neural ODEs (NODEs) (Chen et al., 2018), denoted by dt fxx(t) , t; 0), where is neural network parameterized by 0, offer data-driven al- ternative to learn the continuous dynamics. Various NODE- based methods, such as HBNODEs (Xia et al., 2021) and MoNODEs (Auzina et al., 2024), have been proposed to im- prove efficiency and predictive performance. However, most existing NODE-based approaches assume a static parameter 0, limiting their adapting ability to constantly changing envi- ronments. To overcome this limitation, ANODEV2 (Zhang et al., 2019) introduced time-evolving parameters, enabling the model to adapt to different temporal evolution patterns and enhance its generalization ability. However, ANODEV2 primarily focused on the improvement of model generaliza- tion by evolving parameters, without fully exploring their potential to model more complex dynamical behaviors.
dz(())
In contrast, we argue that @(l), which directly drives the state dynamics, inherently encodes the underlying dynam- ics of the evolution (namely deep-level information). By explicitly modeling the evolution of 0(1), we not only en- hance the model's generalization ability but also capture the intrinsic dynamics beneath the surface-level behaviors, of- fering a more comprehensive understanding of the complex dynamic system. This perspective is further supported by the following observations. In systems like robotic arms, internal physical properties such as frictional torque funda- mentally shape the system's 's evolution over time. As time progresses, factors like joint lubrication changes and fric- tional wear cause gradual variations in the frictional torque, potentially leading to trajectory deviations, motion delays, and other dynamic effects. These changes, in turn, modify
1. Introduction
Time-series data from various domains are typically sam- pled from underlying continuous dynamic systems (Voss al., 2004). Conventional deep learning approaches, such as Recurrent Neural Networks (RNNs) (Rumelhart al., 1986; Medsker & Jain, 1999), and Long Short-Term Mem-
'Institute of Science and Technology for Brain-Inspired Intel- ligence, Fudan University, Shanghai, China. Correspondence to: Weiyang Ding <dingwy @ condan.cdu.cn2...
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 20255 by the author(s).

---

KoNODE: Ceopmann-Drveen Neural ODEs with Evolving Parameters for Time Series Analysis
the system's underlying dynamical behavior and influence its overall performance and adaptability. Thus, this point of view expands the NODE models beyond the application of data-driven state predictors, positioning them as frame- work for uncovering the fundamental principles that govern system evolution.
observed state x(t) A A000
x(t) 000
ODE parameters o())
0
Koopman linear wr(t)
none
Three system levels
KoNODE
NODE
Figure I. The roadmap for the proposed framework and its com- parison to NODE.
Based on this insight, we further observe that the most fun- damental evolution of the system is governed by simple yet essential intrinsic linear dynamics. This is inspired by observations in robotic arms, where complex robotic trajectories are inherently driven by linear dynamics (see App. 1.1 for the theoretical analysis). This suggests that although 8, the neural network parameters, may exhibit nonlinear dynamical behavior, its true underlying dynamics can be fundamentally linear. In other words, the apparent nonlinearity in parameter dynamics arises from observation limitations, which is consistent with the principles of the Koopman theory (Kurdila& & Bobade, 2018). The Koopman operators transform complex nonlinear dynamics into lin- car representations by expanding the system dimensions. To this end, we use the Koopman operators as the dom- inating parameters for the dynamics of 0, capturing the intrinsic linear space and revealing dynamics beneath the surface. In summary, conceptualize complex dynamics as three-level hierarchical structure as illustrated in Fig. 1: the trajectory from the observed state dynamics, influenced by the parameter dynamics, and ultimately governed by the underlying Koopman linear dynamics.
thus propose a three-level hierarchical structure-spanning the observed state dynamics, ODE parameter evolution, and Koopman linear dynamics. The Koopman operators, which govern the deepest dynamics, could help identify the system's 's dominant driving modes and frequencies and offer insights into stability, periodicity, or other inherent characteristics, thus gaining mathematically interpretable understanding of the system's 's evolution and governing laws.
(3) Our framework excels in two critical time series tasks: long- long-term prediction and generalization to new data. Experi- ments on two synthetic datasets from complex dynamic sys- tems demonstrated its superior performance. Additionally, we evaluated the framework's is applicability in real-world sce- narios through robot motion prediction across three datasets and multivariate forecasting on six datasets.
2. Preliminaries
2.1. Neural ODEs
Neural ODEs (NODEs) (Chen et al., 2018) extend discrete traditional neural network architectures by modeling hidden states as continuous trajectories governed by an ODE
Specifically, we introduce KoNODE, a Goopman-drives NODE framework that captures the intrinsic linear dynam- ics by applying the Koopman model to the time-evolving ODE parameters. Our framework enhances two key tasks in time series analysis: generalization to new data, by explic- itly modeling parameter evolution, and long-term prediction, which is guaranteed by effectively learning global structures through capturing the underlying simple linear dynamics. Notably, our approach diverges from ANODEV2 (Zhang et al., 2019) in several ways. While ANODEV2 treats pa- rameter dynamics as an ODE and uses Green's 's functions for time convolution, primarily focusing on image data, we model parameter dynamics as nonlinear observations of intrinsic linear dynamics that uncover the deeper driving rules of the system, providing a richer understanding of the dynamic systems beneath time series data. Our main contributions are as follows:
dan(t) dt
f fx)()),1;0),
(1)
where f is neural network parameterized by 0, and ae(1) E R= is the system state at E [Lo, 41]. Given the initial state xto)), the solution at any time t E (to, t1 is obtained by integrating the differential equation: ant) =
ar(4)) J f (x((1) (; 0) dl,, rather than propagating acti- vations through discrete layers as in recurrent or deep net- works. Neural ODEs employ numerical solvers, such as the Euler method (Gear & Petzold, 1984), Runge- Kutta methods (Runge, 1895; Kutta, 1901), or solvers with adap- tive step size such as Dormand- Porinand-Prnnce (Prince & Dormand, 1981), to calculate this integration efficiently.
Since the trajectory of az(1) is governed by a neural network parameterized by 0, learning the system dynamics requires optimizing these parameters. This is formulated as an ODE- constrained optimization :
(1) We introduce novel approach to encoding deep- level, time-dependent information from ((), transforming NODEs from simple data-driven state predictors into a pow- erful framework that enables uncovering the fundamental principles driving system evolution.
mine L(x), s.t.
dæ di = f(x(t), t; 0), x(to) = Zo. (2)
(2) Applying Koopman theory on (t), we capture the in- trinsic linear dynamics that underlie complex systems. We
where L(x) A + Erer 8 (()) is the average loss func- tion over the sampled time points and E is the error term at
2

---

KoNODE: Kcopman-Driven Neural ODEs with Evolving Parameters for Time Series Analysis
specific time that quantifies the discrepancy between the predicted and the actual system states.
3.1. Problem Formulation
Our framework, KoNODE, is proposed within the NODE setups. As outlined in the introduction, the core idea of KoNODE is to uncover the simple intrinsic linear dynam- ics underlying the complex observed systems. We denote the observed system as da(t) f (()), where xx(t) E R" is the trajectory of the system, and ((t) are the parameters of the neural network f. We let the intrinsic linear dynamics be represented as w((). They reveal the underlying driving rules of x(t). Next, we proceed with the detailed formulation of KoNODE.
To optimize the parameters, the adjoint method (Chen et al., 2018) is commonly used to compute VeC in NODE for- mulation. It efficiently computes gradients by solving a backward ODE, avoiding storing intermediate states during the forward pass. This memory- NODEs particularly attractive for long- oongtiime-series modeling. Please refer to App. A.3 for the details about the adjoint method utilized in NODEs. Related work about Neural ODEs can be found in App. A.I.
dt
approach makes
We start by refining the structure of KoNODE, which is built three hierarchical levels:
2.2. Koopman Operator
The Koopman theory is powerful tool for nonlinear dy- namical system analysis. It acts on the observable functions of the system state, which evolve linearly, rather than the nonlinear ones in the state space itself (Koopman, 1931).
(1) The observed dynamics: The trajectory xxf) evolves according to das()) f (x) t; (t)), where O(t) encodes de the deep-level driving rules of the observed dynamic.
Definition 2.1 (Koopman Operator, (Mezić, 2005)). For a fnite-dimensional state space X' C I= in which the state evolves according ar क (x,,), I [Lo (1] where + X X' is the operator from the state the next time. The Koopman operator K. is a linear operator that reveals such evolution by acting on an infiniteddimessional function space of observable functions g R² -> TR such that
The parameter O(t), mod- eled by Koopman operators, evolves as an observation of the trajectory w(()) of an intrinsic linear system.
(2) The parameter dynamics:
(3) The Koopman linear dynamics: At the deepest level, the dynamics of w(t),, which underlies @(t), is simplified to dur(t)
a linear form, i.e., di = Att(t), where AA is a linear op- erator representing the essential structure of the system. In Koopman Theory, w((( is inherently defined in an infinite- dimensional space. For practical implementation, we ap- proximate wtt) in dimee-dimensional subspace, assuming w() E RT. Under this approximation, the dynamics of w=)) become dw(() = Atr(t, where A E Rm is a finite-dimensional matrix. Thm. 3.2 in Sec. 3.3 provides theoretical bounds for this approximation to confirm its validity.
(Kg)(() = (x)(
In numerical practice, the infinite- dimensioadl Koop- man space is commonly approximated by modeling the observable function space through a finite number of
bases u [u1, U2,, "2, =mm]" (Brunton et al., 2021). The Koopman operators are estimated by ux)()))
[Ku1, Kug, Kun]] = Ku(x) where K E mmxm
is an approximated Koopman matrix. Hence, for an observ- able function g. let g A Kg un); (g, u2)..... (g, um) then (((())) R & (u((r)) for the m-dimensional sub- space. Specifically, for a vector of linearly uncorrelated observable functions g = Pu and P is a nonsingular ma- trix, we have g()(x)) = PKP-1 g(z) for the evolution of the observables. Related work about Koopman Operators can be found in App. A.I.
Together, these three levels form the basic model of our KoNODE framework, which is expressed as the following formulation:
min A,
L(x)
(3a)
s.t.
da(() dt
f (x((), (; (;)))) x(to) x0.
(3b)
((t) = h(w(();; $),
(3c)
dwi(t)
Aw((), w = wo L E (1.442. (3d)
3. KONODE Framework
dt
Here, h : R + θ is a neural network parameterized by eb to estimate the inverse mapping of the observable function vector g. where θ the parameter space with ((t) in. We model h using neural network due to its flexibility.
In this section, we present the theoretical framework of KoNODE. We begin by formalizing the proposed model (Sec. 3.1) into an optimization problem and solving using the Lagrangian approach. The succeeding adjoint method is used to perform backpropagation (Sec. 3.2). Finally, we provide theoretical results to present both the guidance for choosing the Koopman space dimension and the analy- sis of the error introduced by approximating the Koopman operators (Sec. 3.3).
Next, we give further modeling of A. Numerically, we perform an eigen-decomposition on A, i.e., A = PDP-1 to a product of a non-singular matrix P and a block diagonal matrix D in a format of D = diag (A1, '2,"" Ap. A1, A2, ... Ag}, where p + 2g
3

---

By leveraging Koopman operators, our framework can use spectral tools for system analysis. Fundamental evolution principles are revealed by the spectrum of the Koopman operators, which correspond element-wise to matrix A and are given by Λy(K) = e A 3,At + i sin 3,) in our framework. Specifically, the real part of the eigenvalues indicates the evolution speed, while the imaginary part cor- responds to intrinsic frequencies for the periodicity. Analyz- ing the elements of A helps identify the system's 's dominant driving modes and frequencies, thus offering insights into stability and periodicity, respectively. Furthermore, KoN- ODE enables possible physics-informet constraints over O via regularization on A, zb. or Wo. We derive the adjoint (6) results for stable systems by constraining A's eigenvalues (Mezic & Banaszuk, 2000). See App. B.5 for details on KoNODE: Koopman-Drin Driven Neural ODEs with Evolving Parameters for Time Series Analysis
m and {A.)?=] are real eigenvalues of A and the matrix 1, ................................ , are real matrices that represent the imaginary eigenvalues of A. To simplify the problem, we unify the treatment of all eigenvalues by al- lowing redundancy in the dimensions of wt).. Specifically, we assume that all eigenvalues are conjugate complex eigen- values, that is, D diag (A,,A2,". Amm22) where m is even. Let ( - P.p.. g(f) be a vector of alternative ob- servable functions, the resulting trajectory wr(() also follows a linear dynamic driven by another approximated Koopman operator K eDi.D_ Hence, without loss of generality, assume A = D using the observable function g. See App. B.2 for the rigorous theoretical analysis of the above formulation of matrix A.
where V,C - a ((tr)(((((()))))) (ET) is the direct partial
blocks A, 0i 8:
derivative of L. Among them, Eq. (5) is a backward-in
-B; ai
time ODE for the adjoint function (t),, Eq. (6) reveals the algebratic relation between ax(1) and ((t), and Eq. (7) is the backward-in-tin ODE for the adjoint function ((t) with our unknown parameters A and zb.
The gradients of & regarding ub, A, and wo are,
02 @h(w:b) 84p
ff( dt,
(8)
Oup
= Bo. = S B. w T dt, 8A
00
8 Jwa
(9)
Note that if optimality conditions are achieved with respect to 1b A, and wp. then 8y detailed derivation of conditions is provided in App. B.3.
02 0, ab 0, Buro a
ab 0. The
3.2. Adjoint Model for the Optimization Problem
However, the unknown parameters A only appear in Eq. (7), which represents an ODE for the adjoint function (()), mak- ing them unable to solve directly through the KKT condi- tions. Consequently, we turn to gradient- based optimization approaches to iteratively update these parameters.
In this part, we apply the Lagrangian method to solve the constrained optimization problem defined in Eq. (3). Sub- sequently, the Karush- Crrrush-Kuhn-Tiee Kuhn-Tucker (KKT) conditions ob- tained from the Lagrangian function in Eq. (4) imply the restrictions in the gradient descent method. The resulting adjoint method was used perform backpropagation.
Theorem 3.1 (Equivalence of Gradient and Adjoint Trajec- tories). the partial Jacobian matrices are Lipschitz contin- uous. The gradient trajectories of xxt) and w(t) are a(t)
3.2. 1. LAGR ANGIAN FUNCTION
Oc and (t)) respectively, i.e., (x(() (x)), (t) (t).
ac
The Lagrangian function of Eq. (3) is
Thm. 3.1 proves that the adjoint trajectories (t)) and ((t) are actually continuous equivalents of the gradient trajec- tory during backpropagation through the KoNODE model. Consequently, the adjoint trajectory substitutes the back- propagation process for network training. This conclusion hence aligns with the adjoint method in NODE-based meth- ods. Therefore, we utilize the adjoint method to compute the derivatives of the loss function C with respect to s A, and wo to perform optimization. The overall adjoint model is summarized in App. B.4, while the algorithm is detailed in Alg. 1, App. B.4.
L =(())- - au (z((4) - xo) - B (w(to) - wo)
1 a())T
( dd((t)
f(x((t), 1;;(;;)(())) ((1))) de
di
(4)
- Sh B(t)
(tartt) (
- Aw((t) dt
dt
- - r(t) (((t) - h(w((t); xb) )at
where (tt), ((t), and (tt) are the corresponding adjoint variables for the constraints in Eq. (3). We give the KKT conditions of Eq. (4) below.
3.2.2. OPTIMALITY CONDITIONS
The KKT conditions consist of (1) the given conditions in (3b), (3c) and (3d); (2) the gradients of & regarding A, sb, and wo equaling to zeros; and (3) the variants of & concerning (t), (t). wtt) being zeros, which result in relations for the three adjoints (x(1), (()) 7(). Specifically, taking the variants of & with respect to x (t)., (t) and w() yield Eq. (5), Eq. (6), and Eq. (7) as follows (where the argument t is omitted for simplicity).
dæ dt +
(f(m):;0) dw
a + V.C = O, V, -- (((())) = 0, (5)
Of(m.(:)) 80
a - 7 = 0,
48 di
+ A
B +
ah( (
7 = 0, ((1)) 0,
(7) these constraints and the corresponding adjoint results.
Bar
4

---

KoNODE: Koopman-Driven Neural ODEs with Evolving Parameters for Time Series Analysis
3.3. Theoritical Results
Theorem 3.4 (Probabilistic Error for Data-Driven Koop- man Operator). For N data drawn independently and identically from Hilbert space at single time point and any probabilistie tolerance 5 E (0,1) we have P[[] 2/3 om 2 [K]]/ (1, ||]]] 108888--- - //3 om²) where K & e^ is the theoretical Koopman matrix, KN is the matrix estimated by N data and a is the isotropic standard deviation in the space of w(t)..
This section presents a theoretical analysis of the error caused by the proposed framework. Sec. 3.3.1 aims at guiding the choice of dimension m, and See. 3.3.2 proves that the additional Koopman module causes only minor er- ror compared to conventional ODE models, apart from the potential improvement due to the advanced fitting ability.
KN|FF I 5E 2 - 1 E
3. THE FINITE-RANK APPROXIMATION OF THE KOOPMAN MODEL
The result of Thm. 3.4 shows that the error bound E ~ ((/VV)) as N + cc. Thm. C.15 in App. C.4 shows that the regression error is also controlled by ((I/VN) which ensures the feasibility of the batch loss. Furthermore, Thms. 3.2 and 3.4 together reveal the theoretical error bound of the proposed model in real-world data, as shown in Thm. 3.5.
We first provide the theoretical upper bound of the approxi- mation error of the Koopman model regarding w(l) with di- mension under the guidance of Kurdila & Bobade (2018). g(0)= = 5 uO) is the vector of observable functions and K é e E R mmmw m is the order-m Koopman matrix.
Theorem 3.2 (Regression Error for the Order-m Koop- man Operator). The estimation of the Koopman space by m-dimensional observables has a relative error of, So & (u(0)-Kg(())
Theorem 3.5. For N data drawn independently and iden- tically from Hilbert space at a single time point and any observable function g and its corresponding & with any probabilistic tolerance 8 E (0,1) we have
d0 [e]Kg(0))3 S m mm= where& & A ((9, u2) ... (g. um)) and r & | maxmeigN log, (g, us) - where N is the number of data and { are the bases of true Koopman space.
P
LE (xxxx())KK(((0))) EEl [o(]]]
(x( (-m]a(1.r()))- + 2 1 6,
where KN is the order-m Koopman matrix estimated by the N data, p /N 2m 1)/(v33 om²) for the data standard deviation a, e = o(m 8). ((.) is the spectral radius of operator K.
The conclusion ensures that the relative error of the regres- sion results is inversely bounded by the approximated di- mension of the Koopman space.
Proposition 3.3. If the data trajectories are in a space of rank s and function f satisfies the Lipschitz condition and is differentiable, we have D
The presented error provides guidance for the choice of di- mension m, and a proof for the minority of errors caused by the additional Koopman module compared to conven- tional ODE models, apart from the potential improvement due to the advanced fitting ability. Thm. C.22, App. C.6 further presents the conclusion by comparing the proposed and ODE models. The proofs of all theorems can be found in App. C.
S m S s.
D=
If we take the data rank into account, can further provide the possible range of the Koopman dimension Prop. 3.3 provides the result, as reference for the choice of hidden dimension. The ideal dimension in practice can be various due to (1) the auxiliary dimension design of the Koopman matrix, (2) the multiple choice of dynamics for (l), and (3) more accurate dynamic regression by multiple frontier sets. In subsequent sections, we further address the error caused by the proposed method with respect to the chosen
4. Practical Implementation
This section outlines the practical implementation of our framework, which consists of three levels marked æ. O, and w in Fig. 2. The forward propagation in Fig. 2 requires the given input states (=(t) shown by the red line in Fig. 2 and the initial Koopman observables obtained by random initialization (if given initial point) or IKOE (if given sequence). Then we deduce the trajectory w, by the trainable matrix A. The observables w() are then passed through the neural network h obtain the parameters O(t). Subsequently, we utilize the ODE solver to predict x(t) any time after T imput shown by the green dashed line in Fig. 2. Using the true trajectory and the predicted trajectory, we can calculate the loss. For the backward propagation, we use the adjoint model results in Sec. 3.2 to update the trainable parameters, including &b in h. A, and network parameters
3.3.2. THE ESTIMATION ACCURACY OF THE DATA-DRIVEN METHOD
Noted that the error bound regarding wtt) is better con- trolled by the number of data N, we further refer to the influence of data amount in the accuracy of estimation. In the data-driven scenario, we provide bounds for the errors in the estimation of the Koopman operator and Koopman regression method (Bevanda et al., 2024) regarding N ran- domly drawn data. If the dataset was drawn independently and identically, the theorem for the error bound of the ap- proximated Koopman operator, as a corollary of Thm. 12, Nüske et al. (2023) claims as follows.
5

---

KoNODE: Driven Neural ODEs with Evolving Parameters for Time Series Analysis
Random Setting
Grussian
(to der(()) dt
neural network f with purameter p
so(o)
Learnable Setting: IKOE
trainable modules or parameters addition
RNN
mothod
A2
Pi
M
-Rt
a,
xm
Anyz
predicted trajectory
40
8
(0)(t))
40 (tn Ac)
40 Cinput)
(t)
40 (t,
At)
Koopman
h:
hr:
h
Parameters
h(wr(:
f:0(ss
1:0((inp))
1:0
A: Ac)
xiTI
x
x(aa)
(to + ( x (upput)
(t))
dx(t)/dt dt ((0
(x,,)
Tingut
Trajectory
Time
Observations
fipput
Figure 2. Our proposed framework of system estimation. The overall structure is separated into three levels, i.c., the Koopman linear dynamics, parameter dynamics, and observed state dynamics, which directly leads observations. The initial Koopman observable wo6o serves as encoding of the system over N time points which is either obtained randomly for simple systems (method I) predicted by the Initial Koopman Observable Estimator (IKOE) module from the first Naqur time points (method IT).
in IKOE. Our framework is trained in an end-to-end manner, allowing for simultaneous optimization of all the levels in model.
prove the ability to model complex systems, particularly in terms of long-term prediction and generalization (Sec. 5.2), and what empirical insights into the governing rules of sys- tem evolution does our framework reveal through spectral analysis? (Sec. 5.3)
We now discuss the design of wo. wo is an encoding specific system identifies the trajectory of w(t) under determined matrix A and thus influences the estimated ODE, There are two settings for the initial observables. (I) Learnable setting when given sequence as input: We estimate Wy in a data-driven fashion using the Initial Koop- man Observable Estimator (IKOE) E, which is based on an RNN architecture (Hochreiter & Schmidhuber, 1997: Paszke al., 2017), i.e., wa =[{=(()) (II) Ran- dom setting when given initial point input: We draw we ~ (0,00 a I) from centered normal distribution. In the latter part of the article, we experimentally demonstrate that when the initial point is provided as input, the random initialization setting remains effective.
(3) Real- World Applications: How does the model perform in real-world applications? (Sec. 5.4 and Sec. 5.5)
What is the contribution of each com- ponent in the framework, specifically regarding parameter evolution and Koopman operators (App. D.1.1)?
(4) Ablation Study:
5.1. Spiral Curve Fitting: Validating the Global Representation Capacity of KoNODE
In this experiment, we adapted the single trajectory fitting task proposed in Chen et al. (2018) vali- date the global representation capacity of KoNODE. We added a time dependency to the matrix A using sines and cosines to produce the new time-dependent matrix - 1 + 0.5sin(t)
5. Experiments
2.0 + (t)
A(t) - 2.0 - 0.5cos(21) -0.1 0.5sin The tra-
We set up the experiments to answer the following problems regarding our proposed framework:
jectory data were then generated from ODE
dz
di
(1) Global Structure Representation: Does our method effec- tively capture the global structure, thereby ensuring robust long-term prediction performance? (Sec. 5.1)
[No. yol T [2, O We extend the matrix A from Chen et al. (2018) by making it time-dependent, us- ing sine and cosine functions to define its temporal variation. The results of the KoNODE framework were compared with
A(t) [a3 y²
(2) Validation of Major Capability: Does our method im-
6

---

KoNODE: Cnopman-Driver Neural ODEs with Evolving Parameters for Time Series Analysis
Table Test MSES for the Oscillator and Robot Motion datasets in Secs. 5.2 and 5.4. The experiments for the Oscillator dataset include the Imitation cube pick, and Imitation Note that the MSE values for the Damped Oscillator Family are rescaled by factor 10² Results are presented std across 5 runs with different seeds. The best-performing results are highlighted in bold, while the second-best results are underlined.
Oscillator and Damped Oscillator Family. The experiments for the robot motion dataset include Imitation C.
(SEC. 5.2) OSCILLATOR
(SEC. 5.4) ROBOT MOTION
NON-LINEAR OSCILLATOOR
DAMPED OSCILLATOR
IMITATION CUBE PICK
METHOD
IMITATION C
IMITATION S
FAMILY (x10
LATENT ODE
0.711
12.522 1.533
0.13410.000
0.156 0.047
0.20410.075
NEURAL PROCESS
0.460
13.844-0.00
0.09410.00 004
0.08711.000 0006
0.142 0.26
NODE
0.497
14.906-0.204
0.13610.055 0.666
0.0841100035
0.171
HYPERNET
0.381 104
1.52+1.505
0.031 007
0.041#0.008
2.063+0.022
ANODEV2
0.419#0.100 109
0.895+0. 126
0.046+0.000
0.071#0.058
0.035+0.045
HBNODE
0.447=0.019
0 004
0.021=0.003
0.030=#00012
0.015
LEADS
0.12011.0044 0.049
0.907 0.33
0.050110.010
0.060
0.018
CODA
0.12310.017
0.91410.0336
0.03911.004
0.06910.016
0.0101100000
OINNS
0.006
23.512=6.425
0.022+10.010
0.042-100017
0.0161100015
MONODE
0.449
0.906
0.166=#0.052
0.039#0.022
0.127#0.049
KONODE
0.085=100031
0.810=0.0099
0.017=100003
0.022:10.001
0.005#1.0002
the standard Neural ODE approach (Chen et al., 2018). App. D.2 details the implementation, architecture, and hyperpa- rameters in the experiment.
to evaluate the model generalization from one another (Sec. 5.2.1); (II) System family modeling, where trajecto- ries have different parameterizations to assess generalization to unseen systems (Sec. 5.2.2). As the training data cov- ers a shorter period than the test data, both experiments simultaneously evaluate the long-term prediction capability.
In Fig. 3, KoNODE effectively fits the dynamic systems, whereas NODE fails due to the complexity of the dynam- ics. The results demonstrate that the proposed framework effectively captures the underlying global structure of com- plex dynamics by leveraging the linear dynamics modeled through the Koopman operator. The NODE model, only focusing on local non-linearities, struggles to represent this global pattern. The presented superiority suggests that KoN- ODE allows for improved long-term prediction on complex trajectories.
Dynamics systems. The two chosen dynamic systems are both governed by the Duffing oscillator (Kovacic & Bren- nan, 2011). 1). generalized framework for oscillatory systems. The Duffing equation is dq प P. ar dp -aq .q - >P,
where a, 3, and 7 are scalar parameters that determine the linear stiffness, non-linear stiffness, and damping, respec- tively. The characteristics of the Duffing oscillator can be changed by adjusting parameters. Complete implementation details on the training setup, model architecture, and hyper- parameters can be found in App. D.3.3 and App. D.3.4.
0.59
GT
PT
Comparison methods. We compared our framework with the following models: (1) baselines: vanilla NODE (Chen et al., 2018), Latent ODE (ODE enc) (Rubanova et al., 2019), and Neural Process (Garnelo et al., 2018), (2) Hy- pernet (Ha al., 2016), (3) ANODEV2 (Zhang al., 2019), (4) HBNODE (Xia et al., 2021), (5) LEADS (Yin et al., 2021). (6) CoDA (Kirchmeyer et al., 2022), (7) OINNs (Zhi et al., 2022), and (8) MoNODE (Auzina et al., 2024). Among these methods, the ODE parameters in Hy- pernet and ANODEV2 vary over time. The other methods are achieved by their pubic codes. Refer to App. D.3.1 for details of these comparison methods.
0.03
NODE
X
MAE
KoNODE (ours) X
Figure 3. The test MAE and the phase portrait of the test tra- jectory obtained from NODE and KoNODE, respectively. The green lines represent the ground truth (GT) trajectories and the blue dashed lines represent the predicted trajectories (PT).
5.2. Dynamics Systems Learning: Validating Benefits for Long-Term Prediction and Generalization
Performance metric. Following prior work, such Auzina et al. (2024), we evaluated model performance using the mean squared error (MSE) between the predicted phase space trajectories and the ground truth from the test set.
In this section, we conducted two experiments on different dynamic systems: (I) System modeling, where all trajecto- ries share the same differential equation parameterization

---

KoNODE: Ceopmann-Driven Neural ODEs with Evolving Parameters for Time Series Analysis
Table Test MSES for long-term prediction. Results are presented as mean=std across runs with different seeds. The best- perfforming results are highlighted in bold, while the second-best results are underlined.
METHOD
CUP
WEATHER
ETTH1
ETTH2
ETTMl
ETTM2
NODE
0.993+00000
0.553+0.017
1.064=0.067
0.597+0.033
1.08440.036
0.545=0.023
HBNODE
0.879#0.069
0.513+0.031
0,905=0.039
0.592+0.041
0.914+0.017
0,493=0.068
DEEPVAR
0.962=00.00
0.561#0.011
0.947=0.223
0.592#0.033
0,905=0.017
0.398=0.055
AUTOFORMER
1.163:10.005
0.449:1.0016
0.868-0.034
0.37510.003
1.071
0.22710.002
PATCHTST
1.066:10.000
0.42110.001
0.72210.007
0.366:10.000
1.008:1..002
0.238
KOOPA
1.139:0.0077
0.42741.0044
0.605
0.357-10.002
0.759+10007
0.229
SST
1,085#0.001
0.417=0.000
0.759=0.018
0.370=10.00
0.960=0.002
0.239=0.000
KONODE
0.864=1.0099
0.386:10.009
0.678=0.005
0.289-10.001
0.664=0.015
0.211
5.2. I. SYSTEM MODELING: NON-LINEAR OSCILLATOR
9
We first performed system modeling on the nonlinear os- cillator system, setting a = - = 1, = 0, to test the ability of long-term prediction and generalization to new tra- jectories. The training set includes 600 trajectories with 30 time steps and was perturbed by Gaussian noise, while the test set consists of 200 trajectories with 300 time steps from the same system. See App. D.3.3 for more details about data generation. Note that in this part, wo is randomly initialized to evaluate the effectiveness of our random initialization strategy.
(a) Non-linear Oscillator
(b) Damped Oscillator Family
error
10
20
30
40
50
0.125
GT
0.100
0.075
dimension
PT
2
10
12
16
18
(c) error W.T.t. 40 dimension
Figure 4. Visual results of KoNODE predictions. (a) Non-linear oscillator trajectories, where different initial conditions produce four distinct patterns, shown in different colors. (b) Damped os- cillator family, where different parameters result in varying tra- jectories, with predictions shown in purple and red. Gray lines represent ground truth (GT) trajectories in (a) and (b). (c) Test MSE across different dimensions, where the blue polygonal line corresponds to the blue scale at the top, and the green line corresponds to the green scale the bottom.
The results in the Non-linear oscillator column, Tab. 1 show that our framework improves the test accuracy across all comparison methods. Notably, Hypernet and ANODEV2, which allow time-evolving parameters, perform better than baseline methods, highlighting the importance of model- ing parameters evolving over time for complex systems. However, they still lag behind ours, highlighting the advan- tage of using the Koopman model for parameter dynamics, which thereby enhances long-term prediction accuracy. We showed the predicted results of our method in Fig. 4 (a).
epochs. This concludes that the proposed method converges faster, though it spends longer for each iteration. Regard- ing the memory assumption, the proposed method uses O (max (n.m, n, D) while NODE uses (nn)) where n, m, h are the dimensions of se(t). ((t), and the hidden layer. D is the model size for the differential function f. The the- oretical conclusion indicates that the memory assumption of the proposed method is comparable to the ODE-based models, which are superior to other methods. In practice, the auxiliary memory assumption would further narrow the gap between them. See Tab. 5, App. D.1.2 for more results.
w dimension. We also presented the test MSE across dif- ferent w dimensions in Fig. 4 (c). Fig. 4 (c) shows the corresponding test MSE when w dimension increases from 10 to 50. Fig. (c) shows curve dropping [2, 10] and rising if m >> 10 with a tolerable relative perturbation. Experimentally, more weights are introduced in networks, which may add extra uncertainty and instability during train- ing when m is large. Theoretically, the error bound in Thm. 3,5 is majorly dominated by two terms, one with factor of m² in the coefficient p and the other with the order of m 5 which does indicate the instability when m too large and may infer overfits. Additionally, we reported the results of method using other ODE solvers in Tab. 4, App. 1.1.2.
VN
5.2.2. SYSTEM FAMILY MODELING: DAMPED OSCILLATOR
In this section, we focus on system family modeling on the damped oscillator family, setting a E [0.9, 1.1], e [0.99, 1.01], B = 0, to test the ability of long-term predic- tion and generalization to unseen systems. See App. D.3.4 for details on data generation and implementation. The re- sults in the Damped oscillator family column, Tab. 1, and
Time and memory analysis. We analyze the running time and memory assumption in this experiment. During the experiment, KoNODE converges in 19 seconds using 10 epochs, while NODE spends 215.50 seconds using 48
8

---

KoNODE: Koopman- Driven Neural ODEs with Evolving Parameters for Time Series Analysis
Fig. (b) demonstrate the effectiveness of our framework in generalizing to new systems.
details about data description and generation.
The results in the Robot Motion column of Tab. highlight our method's superiority in accuracy and generalization for real-world applications. This demonstrates its effec- tiveness in modeling complex robot dynamics, with poten- tial for robotic control and trajectory prediction. Addition- ally, HBNODE, which captures high-order information, and OINNs, which enhance linearity, ranked second and third, showcasing their ability to extract underlying dynamics from real-world data.
5.3. Koopman Spectral Analysis: Showing How to Uncover the System's Evolution Laws Empirically
In this section, provide more detailed analysis and interpretation of the learned dynamics in Sec. 5.2 by visual- izing the spectrum of the approximated Koopman operator for the Oscillator dataset. The results are shown in Fig. 5. First, the magnitudes of the spectrum are approximately one for two dynamic systems, indicating boundary stability, where the state remains on a stable trajectory without diverg- ing or converging. Second, the dominant spectrum of the nonlinear oscillator (on the left) has similar imaginary parts (i.e.. the frequency components are the same), indicating that the system exhibits clear periodicity. Modes with the same imaginary part have the same periodic components.
5.5. Application Multivariate Forecasting: Validating Long-term Prediction Ability in Real-world Scenarios
In this experiment, we evaluated the long-term prediction capability of our framework on six real-world datasets: CUP, WEATHER, ETTHI, ETTH2, ETTMI, and ETTM2. We followed the preprocessing methods of datasets outlined in Liu et al. (2023) and split the datasets into training, vali- dation, and test sets with ratio of 7 1 2. For further details on the datasets and implementation, please refer to App. D.5.1 and App. D.5.2 respectively.
A Unit Cicle
X, Unit Circle
Comparison methods. In this experiment, we com- pared the proposed method with (1) NODE-based methods:
NODE and HBNODE (performs well in Sec. 5.4); (2) Statistical-based method: Deep (Salinas al., 2019); (3) Transformer baasd methods: Autoformer (Wu et al.,
Figure 5. Spectral Analysis the Approximated Koopman Op- erator for Oscillatory Systems. The plots illustrate the eigenvalue spectra of the approximated Koopman operator for two classes of oscillators: the nonlinear oscillator (left) and the damped oscillator family (right). The real and imaginary parts of the cigenvalues are plotted along the horizontal and vertical axes, respectively. The green unit circle serves as a reference for spectral stability. where eigenvalues inside the circle indicate dissipative behavior. Blue dots represent the computed spoctrum from the approximation of our framework.
2021) and PStchTST (Nie et al., 2022); (4) Koopman-based method: Koopa (Liu et al., 2023); and (5) 3SM-based method: SST (Xu et al., 2024). See App. D.5.3 for more details about these comparison methods.
As shown in Tab. 2. KoNODE achieved the best perfor- mance on five out of six datasets, demonstrating its strong capability in long-tern prediction in real-worke applications. While Koopa performed best on E KoNODE remained competitive. Both methods leveraged Koopman operators to capture global system dynamics, making them particu- larly effective for datasets with periodic or quasi-periodic structures such as ETT datasets.
5.4. Application on Robot Motion Prediction: Validating Generalization in Real- World Scenarios
We evaluated our framework's generalization in real-world scenarios through robot motion prediction on three datasets. Unlike synthetic data from precise dynamics, real-world data comes from unstable systems in complex environments. The experiment demonstrates the method's ability to gen- eralize from training to unseen test data. We compared the results with the same models as See. 5.2, with implemen- tation details in App. D.4.2.
The ablation study can be found in App. D.1.1 whose results confirm the effectiveness of parameter evolvement and the Koopman modeling.
6. Conclusions and Future Work
We propose a Koopman-drives hierarchical NODE frame- work, with theoretical and experimental results demonstrat- ing its ability provide deep understanding of time series. In the future, we will explore accelerating the KoNODE algorithm to improve the overall modeling efficiency and adapting the model for irregularly sampled data.
Data setup. The trajectory datasets were obtained under three different robot tasks proposed by SeaansariZZadhh & Billard (2011), i.e., drawing "S" shapes (Imitation S), plac- ing a cube on a shelf (Imitation cube pick), and drawing out large "C" shapes (Imitation C). See App. 1.4.1 for more

---

KoNODE: Xoopman-Driven Neural ODEs with Evolving Parameters for Time Series Analysis
Acknowledgements
Choromanski, K. M., Davis, J. Q., Likhosherstov, V., Song, X., Slotine, J.-J., Varley, J., Lee, H., Weller, A., and Sindhwani, V. Ode to an ODE. Advances in Neural Information Processing Systems, 33:3338- 3350, 2020.
This work was partially supported by the National Natural Science Foundation of China (No. 12471481, U24A2001), the Science and Technology Commission of Shanghai Mu- nicipality (No. 232R1400 (430000,,, and the Open Foundation of Key Laboratory Advanced Manufacturing for Optical Systems, CAS (No. KLMSKF202403).
Garnelo, M.. Schwarz, J., Rosenbaum, D., Viola, F., Rezende, D. J., Eslami, S., and Teh, Y. W. Neural pro- cesses. arXiv preprint (rXiv:/807.01222, 2018.
Impact Statement
Gear, C. W. and Petzold, L. R. ODE methods for the solu- tion of differential/algeerric systems. SIAM Journal on Numerical analysis, 21(4):716-728, 1984.
This paper presents work whose goal advance the field of Machine Learning. There are many potential societal consequences of our work, particularly in domains that rely on accurate and interpretable time-series forecasting. By en- hancing the ability to model continuous dynamics through Koopman-driven Neural ODEs, our approach has implica- tions in fields such as robotics, environmental sciences, and so on. By improving prediction accuracy and interpretability, framework enhances decision-making in these critical areas.
Guo, D., Chen, B., Zhang, H., and Zhou, M. Deep poisson gamma dynamical systems. Advances in Neural Informa- tion Processing Systems, 31, 2018.
Ha, D., Dai, A., and Le, Q. V. Hypernetworks. arXiv preprint 1609:0900990 2016.
Haslbeck, J. M., Bringmann, L. F., and Waldorp. L. J. A tutorial on estimating time-varying vector autoregressive models. Multivariate behavioral research, 56(1):120- 149, 2021.
References
Abeliuk, A., Huang, Z., Ferrara, E., and Lerman, K. Pre- dictability limit of partially observed systems. Scientific reports, 10(1):20427, 2020.
Hochreiter, S. and Schmidhuber, J. Long short-term memory. Neural Computation, 185):1335-770,, 1997.
Auzina, A., Yıldız, Ç., Magliacane, S., Bethge, M., and Gavves, E. Modulated neural ODEs. Advances in Neural Information Processing Systems, 36, 2024.
Khansari-Zadeh, S. M. and Billard, A. Learning stable non- linear dynamical systems with gaussian mixture models. IEEE Transactions on Robotics, 27(5):943-957, 2011.
Azencot, O., Yin, W., and Bertozzi, A. Consistent dynamic mode decomposition. SIAM Journal on Applied Dynami- cal Systems, 18(3):1565-1588) 2019.
Kirchmeyer, M., Yin, Y., Donå, J., Baskiotis, N., Rako- tomamonjy, A., and Gallinari, P. Generalizing to new physical systems via context-informed dynamics model. International Conference on Machine Learning, pp. 1183-11301. PMLR, 2022.
Bevanda, P., Beier, M.. Lederer, A., Sosnowski, S., Hüllermeier, E., and Hirche, S. Koopman kernel re- gression. Advances in Neural Information Processing Systems, 36: 12207-1222,, 2024.
Koopman, B. O. Hamiltonian systems and transformation Hilbert space. Proceedings of the National Academy of Sciences, 17(5):315-3) 1931.
Brenner, M. Weber, E., Koppe, G., and Durstewitz, D. Learning interpretable hierarchical dynamical sys- tems models from time series data. arXiv preprint arXiv:2410.04814, 2024.
Kovacic, I. and Brennan, M. J. The Duffing equation: non- linear oscillators and their behaviour. John Wiley & Sons, 2011.
Brunton, S. L., Budišié, M., Kaiser, E., and Kutz, J. N. Modern Koopman theory for dynamical systems. arXiv preprint arXiv:2 /2086 2021.
Krauss, P., Schuster, M., Dietrich, V., Schilling, A., Schulze, H., and Metzner, C. Weight statisties controls dynamics in recurrent neural networks. PloS one, 14(4):e0214541, 2019.
Carvalho, A. N., Langa, A., and Robinson, J. Non- autonomous dynamical systems. Discrete & Continuous Dynamical Systems-Series B, 20(3), 2015.
Kurdila, A. and Bobade, P. Koopman theory and línear approximation spaces, 2018.
Chen, R. T., Rubanova, Y., Bettencourt, J., and Duvenaud, D. K. Neural ordinary differential equations. Advances in neural information processing systems, 31, 2018.
Kutta, W. Beitrag zur näherungsweisen Integration totaler Differentialgleichunggr Teubner, 1901.
10

---

KoNODE: Koopman-Driven Neural ODEs with Evolving Parameters for Time Series Analysis
Liu, Y., Li, C., Wang, J., and Long, M. Koopa: Learning non- stationary time series dynamics with Koopman predictors. Advances in neural information processing systems, 36: 12271-12290, 2023.
Runge, C. Über die numerische auflösung von differential- gleichungen. Mathematische Annalen, 46(2):167-178, 1895.
Salinas, D., Bohlke-Schneidee M., Callot, L., Medico, R., and Gasthaus, J. High-dimensional multivariate forecast- ing with low-rank gaussian copula processes. Advances in neural information processing systems, 32, 2019.
Lusch, B., Kutz, J. N., and Brunton, S. L. Deep learning for universal linear embeddings of nonlinear dynamics. Nature communications, 9(1):4950, 2018.
Medsker, L. and Jain, L. C. Recurrent neural networks: design and applications. CRC press, 1999.
Tong, X., Wang, J., Liu, W., Samah, H.-A., Zhang, Q., and Zhang, L. A time-varying state-space model for real-time temperature predictions in rack-based cooling data centers. Applied Thermal Engineering, 230:120737, 2023.
Mezić, I. Spectral properties of dynamical systems, model reduction and decompositions. Nonlinear Dynamics, 41: 309-325, 2005.
Voss, H. U., Timmer, J., and Kurths, J. Nonlinear dynamical system identification from uncertain and indirect measure- ments. International Journal of Bifurcation and Chaos, 1905-93995 2004.
Mezic, I. and Banaszuk, A. Comparison of systems with complex behavior: spectral methods. In Proceedings of the 39th IEEE Conference on Decision and Control (Cat.
No. 00CH37187), volume 2. pp. 1224-1231. IEEE, 2000.
Wu, H., Xu, J., Wang. J., and Long, M. Autoformer: Decom- position transformers with auto-correlation for long-term series forecasting. Advances in neural information pro- cessing systems, 14422449-224435 2021.
Narasingam, A., Son, S. H., and Kwon, J. S.-I. Data-driven feedback stabilisation of nonlinear systems: Koopman- based model predictive control. International Journal of Control, 77(3):770-781, 2023.
Xia, H.,, Suliafu, V., Ji, H., Nguyen, T., Bertozzi, A., Osher, S., and Wang. B. Heavy ball neural ordinary differential equations. Advances in Neural Information Processing Systems, 14:11664-1866 2021.
Nie, Y., Nguyen, N. H., Sinthong, P., and Kalagnanam, J. A time series is worth 64 words: Long-term forecasting with transformers. XrXiv preprint arXiv:2211.14730, 2022.
Xu, X., Chen, C., Liang, Y., Huang, B., Bai, G., Zhao, L., and Shu, K. SST: Multi-scale hybrid tramba-transformer experts for long-short range time series forecasting, 2024. URL https: : arxiv org/ /as/////////// /et.///44..5533
Nüske, F., Peitz, S., Philipp, F., Schaller, M., and Worth- mann, K. Finite-data error bounds for Koopman-basee based prediction and control. Journal of Nonlinear Science, 33 (1):14, 2023.
Yin, Y., Ayed, I., de Bézenac, E., Baskiotis, N., and Galli- nari, P. Leads: Learning dynamical systems that general- ize across environments. Advances in Neural Information Processing Systems, 34:7561-7573, 2021.
Paszke, A., Gross, S., Chintala, S., Chanan, G., Yang, E., DeVito, Z., Lin, Z., Desmaison, A., Antiga, L., and Lerer, A. Bech.nnnRNNN.
https://pytorch orgg/rooooooo.. org/docs/ /ocs/ stable htabbl//generatee/eeeeee
nn. RNN. html,
2017. Accessed: 2025-01-19.
Zhang, T., Yao, 7.,, Gholami, A., Gonzalez, E.,, Keutzer, K., Mahoney, M. W., and Biros, G. ANODEV2: coupled neural ODE framework. Advances in Neural Information Processing Systems, 2019.
Prince, P. J. and Dormand, J. R. High order embedded Runge-Kutte Kutta formulae. Journal of computational and applied mathematics, 7(1):67-75, 1981.
Zheng, R., Bai, H., and Ding, W. Koonpro: variance- aware koopman probabilistic model enhanced by neural process for time series forecasting. In The Thirteenth International Conference on Learning Representations.
Redman, W. T., Fonoberova, M., Mohr, R., Kevrekidis, I. G., and Mezic, An operator theoretic view on pruning deep neural networks. arXiv preprint arXiv:2110.11 14856.,.... 2021.
Rubanova, Y., Chen, R. T., and Duvenaud, D. K. Latent ordinary differential equations for seguularly-samplle time series. Advances in neural information processing systems, 32, 2019.
Zhi, W., Lai, T., Out, L., Bonilla, V., and Ramos, Learn- ing efficient and robust ordinary differential equations via invertible neural networks. In International Conference Machine Learning, pp.
PMLR, 2022.
Rumelhart, D. E., Hinton, G. E., and Williams, R. J. Lear- ing representations by pack-propagatte errors. nature, (2388)))::3333336 1986.

---

KoNODE: Koopman-Driven Neural ODEs with Evolving Parameters for Time Series Analysis
Appendix Overview
The following lists the structure the appendix, with links to the respective sections.
A Background Knowledge Supplement:
A.1 Related Work
A.2 Notation Convention For Gradients
A.3 Adjoint model for NODE
Framework Supplement:
B.I The Inspiration of KoNODE
B.2 Analysis for the Formulation of Matrix A
B.3 The Derivation of Optimality Conditions
B.4 The Overall Adjoint Model and the Detailed Algorithm
B.5 Details about Our Framework Regularization Term Regarding Physical-informedd Constraints
C Theoretical Supplement:
C.I The Proof Equivalence Between Gradient and Adjoint Trajectories
C.2 The Proof of Error Bound in the Finite-Rank Koopman Model
C.4 The Proof for Estimation Accuracy of the Data-Driven Method
C.5 The Error Caused by Both Finite-Order Koopman Under Data-Driven Scenario
D Experimental Supplement:
D.I Additional Experimental Results
D.2 Details about Experiment in Sec. 5.1
D.3 Details about Experiment in Sec. 5.2
D.4 Details about Experiment in Sec. 5.4
D.5 Details about Experiment in Sec. 5.5
12

---

KoNODE: Koopman-Driven Neural ODEs with Evolving Parameters for Time Series Analysis
A. Background Knowledge Supplement
A.1. Related Work
A.1.I. NEURAL ODES
Neural ODEs (Chen et al., 2018) have been widely studied for modeling continuous time dynamics, with various extensions improving their expressiveness and adaptability. Since their introduction, various NODE-based methods have been extensively developed to enhance efficiency, stability, and predictive performance modeling continuous-time dynamies. To address stability and long-term prediction, HBNODES (Xia et al., 2021) use second-order heavy ball dynamics, mitigating numerical instabilities and improving trajectory accuracy over extended time horizons. Modulated Neural ODEs (MoNODEs) (Auzina et al., 2024) enhance predictive adaptability by decoupling dynamic and static factors. ODEs via Invertible Neural Networks (Zhi et al., 2022) transform the dynamics to promote linearity, improving learning speed and robustness. However, the aforementioned methods assume that the ODE parameters remain static, which limits the generalization ability of the NODE framework. ANODEV2 (Zhang et al., 2019) was the first to propose modeling parameters to evolve over time and introduced a coupled neural ODE framework. Subsequently, ODEtoODE (Choromanski et al., 2020) proposed a nested NODE framework, where the parameter weights are constrained to be orthogonal, primarily focusing on the stability benefits of this approach. Our framework continues to focus on modeling the parameter evolution to encode the deep-level information that reveals the driving rules of the systems.
1.1.2. KOOPMAN OPERATORS
Over the past two decades, Koopman operator theory has gained significant attention, facilitating advancements in various domains, including system analysis (Lusch et al., 2018; Azencot et al., 2019), control (Narasingam et al., 2023), optimization (Redman et al., 2021), and forecasting (Liu et al., 2023; Zheng et al.). These approaches leverage Koopman- based representations to extract linearizable global structures from nonlinear dynamics, enhancing interpretability and control.
Our method builds upon Koopman-driven frameworks, capturing underlying linear dynamical properties. Unlike traditional Koopman approaches that rely on predefined observables, our framework uses neural networks to improve its flexibility.
A.1.3. NON-AUTONOMOUS DYNAMICAL SYSTEMS AND TIME- VARYING MODELS
In addition to Neural ODE-based paradigms, which have provided powerful tools for modeling continuous- dynamics, another well-established line of work addresses -on-stationarity by explicitly modeling time-dependent parameters. This approach closely related to the theory of non-autonomous dynamical systems, where the evolution rules themselves vary over time (Carvalho et al., 2015). Such models have been applied to diverse domains, including climate modeling, neuroscience, and economics, where the assumption of stationarity is often violated. More recently, in machine learning and statistics, various time-varying parameter models have been proposed, such as time-varying autoregressive models (Haslbeck et al., 2021) and time-varying state-space models (Tong et al., 2023). In deep learning, architectures like recurrent neural networks with dynamic weights (Krauss et al., 2019). In addition, there has been a growing body of recent work exploring hierarchical latent variable models with time-varying structure, particularly in the context of learning representations that generalize across domains and tasks, such as Guo et al. (2018), Brenner et al. (2024). These approaches illustrate the importance of incorporating temporal variability directly into the model structure, aligning with approach.
A.2. Notation Convention For Gradients
We first define the notations for the derivatives in the appendices as follows.
Definition A.1 (Notation Convention For Gradients). We denote the loss gradient w.r.t. a parameter (or activation) tensor æ de which has the same shape, i.e., [dC] 8c for any index in the index set. as dz
de
dz.
Definition A.2 (Notation Convention For The Jacobian Matrix). We denote the Jacobian matrix of vector y E R w.r.t. dy another æ E RT as Za E _
The differentiations the above conventions can replaced by partial derivatives in certain occasions where the response also considered& a function of other fixed parameters.
Note that under conventions A.1 and A.2, scalar-to-veete gradient w.r.t. x E Rm results in vector dc de E R while the
vector-to-vecte vector Jacobian matrix is dC E RR even for 1-dimensional response (n = 1). dm
13

---

KoNODE: Koopman-Driven Neural ODEs with Evolving Parameters for Time Series Analysis
A.3. Additional Prelininaries Neural ODEs: Ajoint Method
For the adjoint method of the Neural ODE structure, auxiliary variable, ((t) The evolution of (() is governed by
ac introduced to help compute dc
Bm(t))
50
dA(() (f(x(1),1;0) T dt dat(t)
X(1),
(10)
with the terminal condition
ac
A(ti) (z((t))
(11)
Once the adjoint variable (t) is computed, the gradient of the loss with respect to @ can be efficiently obtained by
ac
(f(z(t).t;0)
A(t)dt.
(12)
30
00
This approach avoids storing intermediate states, reducing memory requirements and enabling efficient training of Neural ODEs in large-scale applications.
B. Framework Supplement
B.1. The Inspiration of KONODE
If we refer to simple mechanism in robotics that consists of two nodes, A and B, connected by rod where both nodes have motors to control the mechanism. Assume the motor at node A is fastened to the basement. Let æ., n, and 6, be the location, the axis of rotation, and the phase of rotation of the motor at the node respectively. If the two motors rotate at angular velocities шл and wB, we have the evolution formula,
a
I
x A
nA
nA
O A
0
BB
(t At)
=
R(At)
DB
(t)
R(At)
(13)
næ
R(A))
ThB
0B.
GB
WBAt
where R(At) is the rotation matrix caused by motor A which is,
cos WAAt
sin WASt 0
WA
R(At) PA
sin wAAt
COS WA 0
P.P.
PA
WA
1 0
PT
At.
(14)
1
0
1
where PA is unit orthogonal matrix with the last column being nAA Consequently,
I
I
A
n
WA
A
0
W A
pl
0
+
W
(15)
æB
W A
ng
1
WA
ng
0
0n
1
00
0
WA
0
where P = diag 11,1, PA. 1A. and W
WA
0
0
0
0
14

---

KoNODE: Koopman-Drive Neural ODEs with Evolving Parameters for Time Series Analysis
B.2. Analysis for the Formulation of Matrix A
We formulate the real matrix A E R as A lim P.D.P.' - where P.D.P.' is the real eigen decomposition
of the matrix + &3 to the product of non-singular matrix Pa and block diagonal matrix De format of,
D2 = diag (2,,2.. .. Ap. A1. Aqt.
where p+ 2q n;; the matrix blocks A;
are real matrices in replace of the conjugated imaginary eigenvalues. The perturbation E is a diagonal matrix that ensures the uniqueness of the eigenvalues of the matrix A +++ +.
04 3
B
, ..2,. 1,2,...,
03
Note that we can select the perturbation as Σ diag (1, 2, ... , n so that the matrix &E has no eigenvalue whose order is greater than 1 when < min MISA.JSmii
|p= Pjl for the eigenvalues {p:)== for matrix A.
Under this formulation, we assume D lim D. and define the parameter vector for the matrix A [A1,A2;" "M AAp aq, "1, ,,,]
Note that the eigenvalues of A, or + B, thus A; Q diag far B: i, -B -B=ii As result, eA: e TuQQQ= = e which leads to,
K ADI lim P. At) = lim Ps e c-00+
c-0+
where e diag fehrsr ,e e a I2, ,e and I. o(1), - 0+. The equality holds as the limit Ki is known to exist.
Let (0) & P g0) be an alternative observable function, the resulting sequence w(t) follows a linear dynamic with an finite-order operator K- e Here, P an approximation of lim,--+ P Hence, without loss of generality, we assume D. That to say, our formulation, matrix A takes the following form:
B1
-1
01
Q22
B2
A
- B8
0r2
amn/
Bmi/2
Bmm/2
am /2
B.3. The Derivation of Optimality Conditions
To start with, define the one-order variations tool so that applying it to the Lagrangian function shown in Eq. 4, manuscript results in the KKT conditions.
Definition B.1 (One-order variations). Consider a functional L(x) F R where F is the for functions. For function =(t), denoted and its variation dx, the one-order variation 52(x:) is
00 dæ.
82(x) dx
due to the Taylor's expansion L(x + 5x) = 2xx) +
8x + 1()))=113)
de
Some arguments of the functions are omitted for simplicity under the assumption of being time when they not specified.
Variations with respect to xt))
In order to satisfy the first optimality condition on 2, we have 52(x))
ag
fa. 0 for any variation dæ in space and time,
Fm
15

---

KoNODE: Cropman-Drivee Neural ODEs with Evolving Parameters for Time Series Analysis
hence
00 òæ
dæ
L 520(1) - -.100 a dæ + S/m (ff(z,t;))
a
dx dl ag 8ar(1o)
to
dæ
ET
V L (z) - [a(ti) x(t1)
(a(t))<x(t)) - acto) xo(to) - (r
x do + f (f((xtt0))
dx a) óæ dt - aj dx
1E
da 0f(x,t;0)
V. L (t) +
a
óx dt + (V - aa(i)] dx((t)) + [a((t)) - aol dzx(to),
to
dt
dx
EE
do
dæ dl (,,f-((1))) a<(4)) dar(1)) V.L.
(a((4)) oxo) dar((o) 0,
dt
dæ
holds for any variations dæ. Therefore, the first set of adjoint equations are as follows,
da
af(x,t;0) a V++ -, acti) 0.
(16)
dt dr
Note that simultaneously, we have orp atto) for the solution of oxp-
Variations with respect to (()
With 80
80 we have
(")
5) (f((ztt
T
50 dt + f
1
11
0f(x,
Of
t: (x,t;
(xx
T
50
a
TT
50 dt
7
Cx
50 dt
(17)
00
00
00
t0
Gradient with respect to wo The gradient w.r.t. Wo shows that
00 Owp
30 0.
(18)
Variations with respect to wt)
With 8w 0 we have
ag
11
dB
ow dt B(tt)) ((tt))Tw(tti) + T(to owo(to) + 1/ TT -hh(w; +b)
dw
+
A
β
8w dt BJ0 dw((t))
Ow
dt
Sw
d3
Öh(w; bb)
7 ow di B(()) Sw)(k) + (B((6) Bo) swo(o) = 0,
+
A
3 +
di
Ow
(19)
holds for any variation Sw. Together with Eqs. (17) and (18), we obtain the second set of adjoint equations.
d3 di + A 3
Sh(w; +b)
0f a 0;
a 0,3((1) 0.
Ow
a0
Simultaneously, we have (tp) = 3, = which provides a second boundary condition for the ODE ,3 satisfies which adds up to the complications of directly solving the KKT conditions. practice, we omit the condition that Bo = to use the gradient-based methods.
16

---

KoNODE: Koopman-Driven Neural ODEs with Evolving Parameters for Time Series Analysis
Algorithm 1 Training of the KoNODE Model
Input: trajectory æ=t), start time fo. stop time i,, network parameters ab, matrix A, IKSE E Initialize = / by setting a; = and B3 0.
Estimate w((o)) = E( ( (eTTw))
Draw w(to) (0, repeat
(Forward process}
in aaallaaugsstte state = (to). u utto) attgstatteeeasassseaaaaaaaaeeeet
def aug- dynamic(( [z, w), 1, ub, A)
return If (z;; h w: +) v)), Aw]
[z) (nreSolver(miit.nuu.iii aug to (4b, A))
(Backward process)
Compute ((x) and retrieve V,C, Vt [to, $1] init_aug. statesttee = (xx(t)), wtt1), V, C. Ow Og, 0Al def aug aug.dgmamic(((, w, ot, 3, +) 1b, A) of
return (f; h(w; ))), Aw. ãe a V,C,
8h 85 60 a
a-
AT-ABB.
8h
T T8f 00 ar, B w
[. " Ow((o)' @w((o)' B 8A
Oc ac 0C
ODESolver wwwgussatttee aug mug.dmimmic, t to. |/b A))
Backpropagate from (to) by BC )a(to)
Backpropagate from w(lo) by aw(ts) Update &b and A by 0C and Ac respectively until Convergence
Dc
Th JA
Gradient with respect to A The gradient w.r.t. A shows that
0c
G w
dl.
(20)
0A
B.4. The Overall Adjoint Model and the Detailed Algorithm
The summarization the formulas from App. B.3 leads to the overall model for the adjoint method which expressed as integrations in Eq. (21) and algorithm in Alg. 1.
The integration is formulated as
(1)
(141)
f(x(s), 8((ss)
wt)
w((1)
Aw((s)
(xt)
Ve,K
0f(x(s), s: (s)) T
+ l
a(()) - -r V.C
Ox(ss
ds,
(21)
(t))
0 Ow(())
ah(w(>);;
8f(a()),s;0(})) <((s) AT A((s)
(w)(s
00(s)
ac
0+
0h (h(w(s):
Tf (z(s), ((s)
0
Onp
00(s)
82
0
(()) w (s)
0A
where Oz represents zero vector with the same shape of tensor and 57(s) Eier 8(8 - 1) is a comb function based on the Dirac function 5. The detailed algorithm is listed in Alg. 1.
17t
17

---

KoNODE: Koopman- Driven Neural ODEs with Evolving Parameters for Time Series Analysis
B.5. Details about Our Framework Regularization Term Regarding Physical-informed Constraints
By modeling the intrinsic simple system, our framework not only improves the accuracy and the generality of the prediction tasks but also allows leveraging spectral tools to analyze system behavior. Furthermore, physics-informe informed constraints be imposed by adding regularization terms in the objective function in Eq. (3), manuscript. In our formulation, the regularization to can transform to A, wo or sp. So, we reformulate Eq. (3) as,
min [(x) + R(a, wo 2b), 10
(22)
subject (3b), (3c), and (3d), where the vector composed of all the non-zero elements of the matrix A R mm i.e. = [11 /2m/22 ... ...
..m.. 92m/2] T
In this part, we derive the regularization inherited from the spectral method to constrain the eigenvalues of A as example (Mezic & Banaszuk, 2000). Especially, stable dynamic systems require the Koopman operator K. to have eigenvalues on unit circle, which leads the r diagonal elements for matrix A being 0. We utilize a quadratic loss function penalize for the constraint, i.e., R(a) X S,À where S, diag ii=1 ex; is the selector of eigenvalues and A11,"",, Amml = [a1, 01, /2 /2 T are the diagonal elements the matrix A.
Theorem B.2. The gradient of R(a) w.r. the vector is,
@R(a) 2 vec (diag [S, Al)
da
where vec Rmxm + R.n is mapping that maps matrix A + la a.l where = =A111 + A22, ... Am -m1mm--1 + dL [6L
Ammll and ag |A1 A2,1, An- An Note that under such definition, have
da 0A
According to Thm. B.2, the gradient w.r.t. A in Eq. (9) converted into the gradients regarding a as
vec diag [2S, A1+ + I/ 3(1) w T(() dt
(23)
da
C. Theoretical Supplement
C.1. The Proof of Equivalence Between Gradient and Adjoint Trajectories
We prove the two conclusions in Thm. 3.1 in the manuscript separately by Prop. C.I and Thm. C.2 respectively.
Proposition C.1. If the partial Jacobian matrix (f) is Lipschitz continuous, the gradient trajectory for x(t)
dx((t
OC
equals (t),, that is dx(() = (t).
Proof. As [(x) + Eier {(x(()) is auto independent regarding time, we denote (x, 3 / Everntent)) Elx(t))
18

---

KoNODE: Koopman-Driver Driven Neural ODEs with Evolving Parameters for Time Series Analysis
dC
Subsequently, the gradient trajectory (1)(1)
follows
d dc
1
0C
0c
lim
dt 0x((t
At-01
At
0x + At)
@xx(t)
1
lim
L1+At:o
(xx(t) L1:00
-
A
@xi(t + At)
1
dxolt + At)
2 dzt(t + At)
0
lim
Le+At:00
Li+
Lix+
-+ A
@ar(t + At)
dar(t)
dx((t
(0(,++4)==))
lim
1
θ (x(()
xx(t)
r-
f(x(s), s; (s)) ds
0c
0E[x(t)]
- At
0x(( + At)
Ox(())
lim
1
++
Tff(x(t), t; (()))
+ols t1} ds
Oc
V.C.
- Al
dar(I)
dar(1 At)
+ At o(A))
OC dx(t At)
lim
V.C,
dx(()
(((1))
V.C, OC
@x((t) (x(()
where the differentiation is converted into partial derivatives in because 0(s) is irrelevant to a=(1) for 2 I and the infinitesimal quantity in - is ensured by the finite values of Jacobian and Hessian matrices due the Lipschitz continuity of partial Jacobian matrices.
(t (f(x(t),(00()) alI) + V.C and ((1)) = V,, C, which has same differential @xx(t)
Note that, (()) satisfies
dt equation and the boundary condition the gradient trajectory OL/Ox((). As ((()) is finite due the existence of the derivate, the differential function of the ODE for (xtt) Lipschitz continuous. Due the Picard-Lindelöf
Theorem, we have ai()) = OL/Ox(().
then prove the latter part of Thm. 3.1.
Theorem C.2 (Thm. 3.1, manuscript). If the partial Jacobian matrices t(( dx(()) and 0f(x(t), ((t) are Lipschitz continuous, the gradient trajectories of sequence a=)) and wl) are a(l) and B(I) respectively. i.e.
00(1)
ac dx(()
a(t). Oc
dw(() B(t).
19

---

KoNODE: Koopman-Driven Neural ODEs with Evolving Parameters for Time Series Analysis
Proof. According Prop. C.I, the first conclusion is proven. On the other hand, have
d
dC
1
dC
dc
lim A
dvr(l)
AI
@ur(1 + At)
der(1)
1
Dc
dx(( + At)
Oc
dw(( At)
ac Eeo (1 + A1)
lim
Ar-
AI
@ro(1 + AI)
dun(1)
(a + At)
dun(1)
1
I dwlll + AQ)
0C
I+AI
OC
lim At-
A ds
At
dw((t
Sur + At)
Owi( + At)
++D Tff (()) d0(t)
UC
+ o(|s -tl)
ds
a0(t)
dwo(t)
dx(( + At)
1
(f(())) (())) (1) 80(1) dw(l)
Oc
0C Sw + At)
lim
At
AtA
At- At
Oxlt + At)
dh(w((); xb)
Of(x(t), ((t)) Oc
dc
(a((1) A
dun(l)
80(1)
Orv(1)
dd((w(!);W) ab) (t)
((t) AT OL
@w(())
where _ holds due to the other condition -(t) 0f (f(())) 1; (()) (t
xx(t) and Prop. C.I. As, on the other hand,
dB(t) + p(t) @h(w(();; 4b)
y(t) and ((t)) O. which has the same differential equation and the boundary
(w((
condition as the gradient trajectory @C/ Using the Picard-Lindelöf Theorem again allows to obtain 3(t) aL/Ow(().
C.2. The Proof of Error Bound in the Finite-Rank Koopman Model
This section proves Thm. 3.2 in the manuscript under the guidence of Kurdila & Bobade (2018). The major conclusions in the article starts from definition of spectral approximation space. Definition C.3 (Spectral Approximation Space). spectral approximation space [H) is defined for a Hilbert space H as
)
(T = f E H 3 bases (uxlie s.l.
Σ (FU < {
Kurdila & Bobade (2018) estimated the bound of low-rank estimations in Example 17 which leads to our result. The major conclusion of Example 17 is summarized as follows.
Proposition C.4 (Example 17, Kurdila & Bobade (2018)). For any observable function f E (L)) where A is the spectral approximation space of the D2 space where is the measure space Q. If represents the projection of the Koopman operator V to the j-dimensional subspace we have
11(2 - -Uj)fllegg(0) S n [lf (|l = - |[(
where nj is the dimension of the subspace spanned by the j basis functions which are closely related to the approximation method, w is the evolving function for the argument of function f, and is the approximation rate that reveals the smoothness of the Koopman operator.
Then we prove Thm. 3.2.
Theorem C.5 (Thm, 3.2, manuscript). The estimation of Koopman space by m-dimensional observables results relative error of
| Ku Kgllla(
S m
20

---

KoNODE: Koopman- Drriven Neural ODEs with Evolving Parameters for Time Series Analysis
where max log, (g, un) 2 mSiSN
and & A Kgg ui)
(g, u2)
(g. um)l with N being the number of data.
Proof. Note that the argument of the observable function is (t) in the proposed setting. If the parameters in manifold θ of order m, the observable function should be estimated by g(0) = Σ & u;(0). As u (u)
Umm [0)] and the order-m Koopman matrix is K - eA. The dimension of the spanned space Bm = m for a common optimization method. Therefore, we can reformulate the error bound in Prop. C.4 under our formulation of KoNODE, Thm. 3.2 the manuseript.
U2( (f)
(g,u.) < <ir = -m m, N which leads to the result when p = 2. Note that we only consider i = m + 1, 1. .N because the first m (finitely many) bases do not influence the convergence of the progression and the order is less than the number of training data.
Note that according to Def. C.3. 9 E ( >
C.3. The Guidance the Koopman Dimension
In this section, we refer to the rank of the data s to provide guidance for the choice of hyper-dimension m, which the Koopman dimension. Prop. 3.3 of the manuscript exhibits theoretical upper and lower bounds for the number of dimensions. As discussed in the manuscript. the best dimension is larger in practice, and we chose 10 for simple systems and 100 for complex ones in experiments. The proof of the proposition is provided as follows using Def. C.6 and Lms. C.7 and C.8.
Definition C.6 (Frontier Manifold), For a dynamic in Euclidean space, a fixed time to and origin 0*, the frontier manifold 3: the quotient space of the equivalent class decided by the trajectories, i.e., To satisfies (1) 0" E 3o. and (2) the normal vector E 3o (00, to). Then, we define 3. the of evolved (t) with ((t)) E 30-
Lemma C.7 (The Theoretical Upper-Bound Bound of m)) Inequality S s holds if: (1) The data trajectories are approximately in an s-dimensional manifold M. (2) The function I satisfies the Lipschitz condition and is differentiable.
Proof. Since M, being an s-dimensional manifold, admits a bijective chart v :M + R. For the projection w- A argmin|ly - x=I, the mapping yEM
Φ : R
db (t)) O(1) + dx -1)
is bijective. This, together with the uniqueness of xtf) (by Picard-Lindelöf theorem), implies that (()) lies dimensional manifold. Hence, m S
Lemma C.8 (The Theoretical Observable Function). For Og E 3., if the trajectory O(1) follows the dynamic
d0(t) = +((((), Ott)) = Oo.
dt
a scalar observable function g0)) & C- exp( (0,(((0, d0) maps it into a Koopman space.
Proof. Note that for (to) = 0, E 3.. ((t) is uniquely identified by the Picard-Lindelöf Theorem. Moreover, all elements in frontier manifold 3. are on different trajectories, which creates bijection between (0g. t) and @()). Let t, (0) be the mapping from time under the condition that Og E S..
Then, let g0) be scalar observable function,
(- A ( 2 C where a exp
(- inv (nv()(0,,,,
21

---

KoNODE: Gpopman-Driven Neural ODEs with Evolving Parameters for Time Series Analysis
where À and C constants and inv[:] is the element-wise inverse of a vector. Therefore.
dg(O) k. g = -Ve(())T(())) (00, k- ( o (0.t) inv[@(0,0)//) di
Φ (0,t)) (Veg(0) A/D ( inv
Velg(0) =0
Hence, g(0) lies in the Koopman space.
Proposition C.9 (Prop. 3.3, manuscript). the data trajectories are in a space of rank and function f satisfies the Lipschitz condition and differentiable, we have 1 m
<
Proof. Under the assumption of differentiability of f. the 0 dynamic where
d0(t)
()) best models the trajectory x(t)
0f
d<x(()) as o [(<(())
(0(),
(24)
(t(t)
dt
Ōt
(x)))
and A being the pseudo-inverse of matrix A which is the Jacobian matrix in the formula.
Lm. C.8 infers that for 0g 3r,, Vl", only one dimension of w is needed. Lm. C.7 indicates that the gradient (p(0()),
lies in a space with the highest rank of s, thus the frontier manifold 3.- covers dimension of at least D - To ensure full cover of O-space in the initialization, the lowest dimension of w is D Together with Lm. C.7, we may obtain the result.
Please pay attention that, without loss of generality, we choose the function bases u as the observable functions g in the following part of App. C for simplicity.
C.4. The Proof for Estimation Accuracy of the Data-Driven Method
We prove Thm. 3.4 of the manuscript and propose additionally C.15 in this section. Nüske et al. (2023) proposed an error bound for the operator in Thm. 12. The major results were summarized by Def. C.10 and Thm. C.I1.
Definition C.10. For N bases { in the Koopman space, define the matrices A, C' E NNN to satisfy
Ciş a (1) Au & (1e, Lıby).
Consequently, the Koopman operator in the full Koopman space is
Ly C-1 A, where V span { (1
Theorem C.11 (Thm. 12, Nüske et al. (2023)). Assume that the observable functions are drawn independently and identically from (2)() with the variance matrix of samples Ea where % is compact and forward invariant space and μ the normalized Lebesgue measure. For any error bound e and probabilistic tolerance å E (0,1) we hav
P(I[Cv E. 2- 5.
for any amount m E 14 of data points such that
2
min {1 [AJlIC-
|A 2| 2||||
Eo 12
where
and 8
822
+
We can obtain a corollary of the theorem which is Thm. 3.4 in the manuscript where we need to highlight that the notations m and N are switched in places. The proof is given below.
22

---

KoNODE: Koopman-Driven Neural ODEs with Evolving Parameters for Time Series Analysis
Theorem C.12 (Thm. 3.4, manuseript). For N data drawn independently and identically from Hilbert space single time point and any probabilistic tolerance 8 E (0, 1), have
P(K K 1/l 56) 56)21-6
2v3 om3
s.t. E min [1,]]K[] VēN v3 om
where is the theoretical Koopman matrix, KN is the matrix estimated by N data and a is the isotropic standard deviation in the space of wt)..
Proof. Note that C I in Thm. C.11 for orthonormal bases hence
|<||
||C_| "AcI ||ACII
||AC|]
|
max
max
max
||A|].
||
||A||
|||||
||<||
Therefore & min (1,
K
We assume on the other hand, that the data sampled isotropically with
|
2 K +ē
standard deviation o, and the variance matrix satisfies
|E ||
om. For the number of data points N we thus have
m²
om?
E
2
o²m² =
8N
V8N
Consequently,
2v3 a033mm1
o K|
min
(I]) VON
/ om3
Removing the tilde notations leads to the final expression.
On the other hand, the bound estimation of w(() is obtained in a KKR model proposed Def. C.13.
Definition C.13 (Koopman Kernel Regression (KKR)). Consider discrete-time dataset D2 {a// UT N modeled by linear time-invariant (LTT) predictor such that
(3)
i = 1 ZT, 21+1 Azt, @(xx).
which is trained by the optimization problem,
M argmin
1lu
M(x M1 + 71 M|2
MEH
where 7 E + and
|| ||x is the reproducing kernel Hilbert space (RKHS) norm.
On the other hand, Thm. 3, Bevanda et al. (2024) finds the gap between the regression error of the true model and that by N data Rx (M)
N
Iu (x ) The conclusion is summarized as Thm. C.15.
N UT Timl be a dataset consistent with a Lipschitz system on non-recurrent domain. Then the generalization gap of a KKR model M is, with probability - 8, upper bounded by
Theorem C.14 (Thm. 3, Bevanda et al. (2024)). Let D = {a
(s)
KT
8log 3
R(M) - RNN( ARB
+
N
N
where R is an upper bound on the loss in the domain, the supremum of the base kernel.
IR + is the upper bound of model M such that || S B and k
23

---

KoNODE: Koopman-Driven Neural ODEs with Evolving Parameters for Time Series Analysis
Formulating vt) by the KKR model and applying the theorem lead Thm. C.15.
Theorem C.15 (Probabilistic Gap Regression Error for Data-Driven Koopman Model). For N data trajectories, the theoretical difference between the mean residual and true residual safisfies,
F ( RNl 56) = 5,
8 log
s.t. 4RBT
+ N
where T is the number of time points, R is an upper bound on the loss in the domain, B the bound of model parameters [|M]] Il and the supremun of the base kernel. Here,
Ro A 1
Eoo()) ((ol()) g[)o(t)]])
77
RN
(t,,(t) (t]lI,
T
1ET
where wt,t + 1) = R (t), w.( = holds for any i E {0, 1,2,......
Notably, Thm. C.15 shows the fact that the error bound of the regression errors the Koopman model also follows O(I/VN) as N + oc. The theorems in this section conclude that the errors of data-driven estimations are inversely controlled by the square root of data amount N.
C.5. The Error Caused by Both Finite-Order Koopman Under Data-Driven Scenario
further combine Thm. C.5 and Thm. C.11 into Thm. C.19 to obtain error caused by both the finite-order estimation of the Koopman operator and the finite-dtta-drive model. We first introduce two lemmas.
Lemma C.16 (Probabilistic Error of 1,2 Norm). Let f E (?²()) where O has a metric defined by the normal density. If |f ( 828, then [=)) S $) 2 8-
Proof. We prove the lemma by contradiction. We provide the hypothesis that P([f(z)) > > 8, then
J ((0))()) (( / (²)(())= = = E A) > 28
[f(x) > }]. The result contradicts with the condition [[f]] <8, hence the hypothesis is incorrect, i.e., (())()) > E) S 8, which leads directly to the conclusion.
where a x
Corollary C.17 (Probabilistic Error of Relative L Error). If the relative L² error of function Af. L2) satisfies
| <elIf(l<a(e),
have
P (IAVI ) 2 e eb.
Proof. Setting f Af/lf/l in Lm. C.16 would lead to the result.
Lemma C.18 (Probabilistic Error of Koopman Estimation). For the Koopman operator K. and its order-m approximation matrix K E Rmxmm we have for any 0,
(|I r(K)) se 2 2m
28,
where s = 2m 2mmmmm (m--1)-5 1
(.r with T being the bound satisfying F ||0||| < <--10(0)) 2 1 - -8, and r is the
T-((K)
convergence rate characterizing the finiteness of the Koopman space.
24

---

KoNODE: Koopman- -rriven Neural ODEs with Evolving Parameters for Time Series Analysis
Proof. (f)) denotes the spectral radius of the operator K., 3g", s.LKKg" =r(K)g" since K (Cr((K)) is not invertible. Let <* - [("", g1) (g)
(g", 9m.) and و _ {" g the m-dimensional estimation of function g and و = g" -
- the
residual.
Note that as K is the estimation of K. the vector E representing the projection of the eigenfunction corresponding to the spectral radius, is the same for K, i.e., 5" K = | E"
Thm. C.5 implies that IP (ET _ (0( - Kg(()) > m h || -
< m á for random O's. The conclusion satisfies
for any g E (+) and g _ (9.91) (9, 92)
(g. (m.)
We can then obtain by setting g = =
| - -(K)) 10(0)) E +r(()) -x())))((0))) s IP (|II]I- g(0) - r(C) _ g(0) r(K) ((0) > 5*) < m
where 5² = m s. |Kigll
1 hence
P |K]] ( r(K))
- [ m 5. |Kgg |Kg ( + +r(k)mmm - -1)= |10 l10l2
[|IK] - r(x3)/
[m |Kg IK + +r(() -111))) P (tāl s (m* - -1) -100l 11)1)
>P
- 1o(01]
11 m $). [1 (m" - 1) 1)- 2m 2m-
where is because (m" 1)-3 - -m= 5.. (m² - 1)-3 3 S mm= 5 and
holds due to the fact that
| (1lg* || |0/1/||9* |[
m -
m for the (²) norm]l - |l,
m
3
where < holds due to the triangle inequality in the denominator and S holds because of Prop. C.4.
Under (0) norm, if |g@|| < x² (0), we have ||K''|| S ||A||| ||g || = r(K) |/ + āll S r(K) (19) + lllI) <
r(k) 72, (()))). As result, for satisfying P((l@ < 72 92 (0)) 2 1 - 5, P(] < --m- 72 (0)) 2 1 - 8. I-m-r Consequently,
r(K)
T. rrk(
P
K
r(k)
s 2m 2m²mmm - -1)-
min(1,r(K3)) )
|||||| r(x)) S <m- 3m2 (K)) .T m (m² - 1)- + r(K) (m" - 1)-3 .F))
2P
21 2m-5mm))-- (1 - 8)3 > 2m 2m -25
The result directly leads to the conclusion.
Theorem C.19. For N data drawn independently and identically from a Hilbert space at a single time point and any observable function g and its corresponding &. with any probabilistic tolerance 8 E E0,,1) we have
(t) - Kg((((t)l [
<0)= 2 1
IP
Se)
8,
IEll
(t)]l
v3 am2 r(K)
s.t. E = 2||E|| 1|0((())
VN(6 2m -) min (1,) - v3 om3 + +o(m=b) 8),
where Ky is the order-m Koopman matrix estimated by the N data.
Proof. We find the following inequality of the mapped observable functions at argument (t)).
- T kx (g(((t)) S|s K Kyw(()= $ (w((t)) + I Kw(t) (g((()))),
I || KK N - Klee- [|w + 15 - Kw((t)Kg(((()).
25

---

KoNODE: Koopman-Driven Neural ODEs with Evolving Parameters for Time Series Analysis
where is true due to the Cauchy-Schwand inequality indicating that |Ax ||| ||=||-
Note that Cor. C.17 indicates P (15 (gg(0)-Kgp) S m -I()oll
2 1 - mm= for random O's. Consequently,
P((5 T K (l) Kg(((()) S E)
PP ||E|| || N K|IF ||(((1)] E m 5. (K))
P (1E Kw(() -Kg(O())) S 3 3-
2(1 -). IP |KKN K|FF S
E- m i |Kgll
|| (t)ll
8-m 8 Kall 2
I [ Letting IP [KKN K|FF S
be 1 5 and applying Thm. C.11 obtains
2v3 om K
È |El|| ||w(())l
+ + 3 |K ia
min [K
N88-
v3 om
Let B(k) A 2||E|| [[)((t)] max
{
m 5. (IKg91)200) forp= om! pt-
No 3
then E B([] (I).
Note that Lm. C.18 shows that
sEt) 2 1- 2m - -28",
where s* 2m (m" 1)-3
T-r(KC
min(1, (K,]
Let be the interval (r((CC - r(C)) + =*1 we have
P K|] B(r(K3)) s AB) 2 P
[P||K]] K|| -r(3)/ ( SE*) 2 1 2m 2m
- -20",
E" sup (8((k) and supker (B'())
1/(pImin - 1)) if Imin = r(k) - < V4-1+1
where AB 2
p
KEl
El [[]|l
1/(p 1/(p-1), 11,
otherwise.
Consequently,
P [E T K (t)(t) (g(((t))) + AB).
2P [5' K ww)( (o(()())) s B(| (([ II) P ([B([] K() B(r(K)) S AB),
211 - 8) (1-2m-= - 28) > 1 - 2m 1 - -88- - 8,
r(K)
where B(r(K)) 2|| ||
pmin (1,r(KC))- m |Kgllt
Note that ((((1)) = E- (1) we have
B(r(KC) + AB -|| [(]l pmin [1,r(K)} - + m r(K)
r(K)
1-m--r Td
T r-r(K))
1
1
+ +| [t]l m 8 (m" -1)-
V min(1,
max
- 5*) (((r(K)-s*) 1)2 p
where d is the bound of the observable function. Let the RHS be 1 8, then = 8 2m 35 and p = /N 2m : - //v3 om?) which leads directly to the conclusion by converting the last two terms into the infinitesimal term and omitting 8*.
26

---

KoNODE: Koopman-Driven Neural ODEs with Evolving Parameters for Time Series Analysis
C.6. The Tightness of the Overall Estimation Compared to the ODE Models
For the overall error of the proposed model, we dig deeper find the theoretical reasoning for its superiority, Thm. C.22 provides the theoretical advantage of the proposed method in estimating complex systems compared to conventional ones. The theorem will not be included in the manuseript for three reasons, i.c., (1) the incquality of theory is not significantly tight, (2) the numerical experiment better proves the effectiveness of the proposed method, and (3) better estimations of the error may require methods similar to those for Neural ODE evaluation such as boundings by the eigenvalues of the feedback gain matrix.
We propose two definitions for the theorem.
Definition C.20 (The Static Dynamic Model Condition). The dynamic satisfies the condition if and only if for the function 9 and any E, 30 0,
P(lg(0") g(O(t)]l < 2
where 0* A argmin fa ((((()))(0) ((=(((, ((1) is the LSE estimation of static 0 and (() is randomly drawn
from the ideal trajectory defined in Eq. (24). The condition holds when the true dynamic behind the trajectory is static, which ensures a good of the NODE.
Definition C.21 (Estimation Bound of Trajectory). We define an upper bound for the trajectory estimation as,
0f (f())
B(0(()] _ sup 0
[I0(() A())l.
(25)
00
where (()) is the ideal trajectory in Eq. (24). It is clear that the overall estimation error of the trajectory using O()) can be bounded by 3 (t)..
Consequently, We prove the theorem as Thm. C.22.
Theorem C.22 (The Superiority of the Proposed Model). During the modeling of trajectory x(tt). the estimation error upper bound of the proposed model is smaller than that of a conventional ODE method if the observable function g does NOT satisfy the static dynamic model condition, i.e.,
P (1) < 2
where 0 (t) is the trajectory predicted by the proposed method while 0" consists of the static parameters in the conventional ODE method.
Proof. We use the notations .", (t), 5 and defined in Def. C.20. Apparently, Eq. (25) gives a bound of the estimation error where the "sup" exists as f is Lipschitz continuous, and the bound is reached for a linear mapping f.
Note that ((t) 0* in NODE and (t) of (t) & h(ū((t); bb) g gg' (t)) in the proposed Koopman model where (I) satisfies the Koopman model of matrix K x- Thm. 3.4, Manuseript shows that
P(C wt + At) g(ō(t + At)] < e" - (t)ll) 2 1 1- 85
with ex representing the RHS. As the base u, uI) ()) for observables ú Lemma C. Appendices implies (P)) < v A 1-- ((z)) 1 and max |,, < r(X)),
mym -r(K) 2
P([||| [|)((t)] <
V8
Consequently, we let
2/30m (())
+ + 8),
/ No66 - 2m 2m-3)) 3) min (1, (1,r(K)) V3d0m²
then PP(I5 (t + At) - 0 At)] <) 2 1 1-28.
27

---

KoNODE: Gbopman-Driven Neural ODEs with Evolving Parameters for Time Series Analysis
Table Test MSEs of ablation study across Hypemet, CONDE, OURS, and OURSSPARSE on NON-LINEAR OSCILLATTOR (01) and DAMPED OSCILLATOR FAMILY (02). Results are mean across 5 runs with different seeds. The best results are highlighted in bold. while the second- recond-best is underlined.
DATA
HYPERNET
CNODE
OURSSPARSE
OURS
01
0.381
0,614 0,446
0.114 0.009
0.085
02
0.023
0.008
Table Results of different ODE solvers on non-linear oscillator dataset.
METHODS
ECLER
RK4
DOPRI5
NODE
0.333
0.361
0.366
KONODE
0.053
0.098
0.099
On the other hand, for g failed satisfy the static dynamic model condition at error é we have
PI(I00 (()] < ||0 (()))).
=P((g]0 g(0(1)] s lo(0)) g(O(t)]))
-Pl 0 (t) - g(((t)]] S 1g(0") - gg0(()) and (t) eQ,, 2F((g] 0 (t)] -g(0(()] & lg(0") gg(0())])) P(O(t) E Q) 2 1 1"
where Q 2 100) e(0) < à and 8 - 28 - 8+ E Q).
As result, with a probability of 8", the error bound in Eq. (25) for the proposed method is lower than that of the conventional method.
D. Experimental Supplement
D.1. Additional Experimental Results
1.1.1. THE ABLATION STUDY
To analyze the impact of each part in our framework, we conduct an ablation study on the non-linear oscillator and damped oscillator family data comparing four framework settings: (1) 0 are modeled as @t). as Hypernet; (2) 0 are further constrained as the solution of another ODE parameterized by a neural network, i.e., d0(t) de = q(((t), t; T). as in a normal coupled neural ODE framework (we refer this framework as CNODE); and (3) ours, which further models parameter dynamics using the Koopman model. To further highlight the advantage of intrinsic linear modeling, we also impose sparsity by setting 80% of the parameters in h to zero. This ensures that the model's behavior is predominantly driven by the intrinsic linear modeling component.
From Tab. 3, the poor performance of CNODE compared to Hypernet suggests that strictly constraining O(t) as an ODE process may be overly restrictive and increase the training burden of the network, limiting flexibility and making the network more challenging to train. In contrast, using the Koopman model for (t) enables the excellent performance, even with a sparse h, demonstrating the effectiveness of our approach in capturing underlying dynamics efficiently.
1.1.2. ADDITIONAL RESULTS ABOUT NON-LINEAR OSCILLATOR
Tab. presents the performance of various ODE snlvers-euled rk4, and dopri5- on the non-linear oscillator dataset, comparing our method with NODE. As is well known, RK4 and dopri5 are higher-order solvers, with dopri5 using adaptive step sizes. Our approach demonstrates strong performance even with the simplest euler method.
We compare the computational efficiency of our framework and NODE by reporting both the average forward and backward propagation times over 200 training epochs in Tab. 5. Additionally, we report the total cumulative time spent on forward and backward propagation for each method from the beginning of training until convergence. The convergence criterion
28

---

KoNODE: Ceopman-Driven Neural ODEs with Evolving Parameters for Time Series Analysis
Table Results of running time regarding average forward propagation time (01), average hackward propagation time (02), total forward propagation time from the beginning of training until convergence (03), total backward propagation time from the beginning of training until convergence (04), and convergence epoch between KoNODEKours) and NODE on the non-linear oscillator dataset.
METHODS (S)
02(5)
03(s)
04(S)
CONVERGENCE EPOCH
NODE 1.0147=0.0022
0.0588#0.000 00073
43.5089
171.9932
48
KONODE 0.0382+0.000 00023
0.1656=0.0190
20.6565
88.5394
10
based on monitoring the stability of the training loss over consecutive batches. Specifically, after each batch, the absolute difference between the current loss and the previous loss is computed. If this difference remains below a predefined threshold (5 x 40=4) for consecutive batches, the training is considered converged. The training loss curves for our framework and NODE are also presented in Fig. showing that our method converges faster than NODE.
observe that although method requires propagation compared to NODE, its superior expressive power enables it to converge faster. As result, the total forward and backward propagation times at convergence are lower for our method than for NODE. In addition, we note that in our formulation, w(t) has an explicit solution, has an explicit solution, which is also one of its advantages.
From Tab.
time per iteration for both forward and backward
Comparison of Smoothed Training Loss Curves Between KoNODE and NODE
0.7
KoNODE
NODE
0.5
Training Loss
0.4
0.3
0.1
0.0
1000
2000
3000
4000
5000
Iteration
Figure 6. Comparison of smoothed training loss curves between KoNODE and NODE.
In Fig. 7, we visualize trajectory results on the nonlinear oscillator data, comparing with the top cop-performing Hypernet and ANODEV2.
29

---

KoNODE: Koopman- Driven Neural ODEs with Evolving Parameters for Time Series Analysis
(a) Hypernet
(b) ANODEV2
(C) KoNODE (ours)
Figure Trajectories on nonlinear oscillator data across Hypernet, ANODEV2, KoNODE (ours).
D.2. Details About Experiment in Sec. 5.1
D.2.1. IMPLEMENTATION DETAILS
All the experiments throughout the article were trained single GPU of model GTX 3090 Ti.
Data Generation. The dataset is created by solving ODE
2
Zo
2
dt
A(t)
(26)
yn
0
- 0.1 + 0.5sin(t) 2.0 + (.5c
2.0 + (t)
where A(t)
0.1 0.5sin
Following (Chen et al., 2018), the initial state æp = [2, 0] used as the starting condition. The time range for trajectory generation is set E [0, 25). and the sampling resolution is set to 1000 time steps. The reference trajectory is computed using dopri5 5 from torchdi feq.
Training Procedure. In this experiment, aim single trajectory fitting task. During training, each training batch consists of randomly sampled set of initial conditions from the reference trajectory. A batch contains 10 consecutive time steps, that is, the model predicts the remaining 9 time points regarding initial points during training. We set the batch size to 5. The optimizer is Adam, with an initial learning rate of 0.01. Learning rate scheduling following an exponential decay. Decay occurs at iterations 500, 4000, with scaling factors 1.0, 0.1, and 0.01. We train runs for 5000 iterations. The loss function is MAE (Mean Absolute Error) between predicted and ground truth trajectories. After training, the trained model evaluated on the full test trajectory measure the trajectory modeling performance.
D.2.2. ARCHITECTURE AND HYPERPARAMETER DETAILS
We adopt dopri5 as ODE solver, with ato1=10~"_tool=10 in this experiment to learn the ODE system. Gradient computation is performed using the adjoint sensitivity method. The network architecture for is MLP. The dimension of w 2. We is randomly sampled from Gaussian distribution. The differential function network f architecture is MLP, consisting three layers with 300 neurons. We adopt the same architecture, training procedure, hyperparameters, and ODE solver for KoNODE and NODE.
D.3. Details about Experiment in Sec. 5.2
1.3.1. COMPARISION METHODS
To evaluate the effectiveness our framework, we compare against several existing neural ODE-based models, each representing different design choices and extensions of the standard neural ODE (NODE), Below, we provide brief overview of each comparison model.
Vanilla NODE (Chen al., 2018). Standard Neural ODE dze(t)
(xx(t), 0), where f is chosen to be a feedforward
30

---

KoNODE: Shopman--riiven Neural ODEs with Evolving Parameters for Time Series Analysis
fully connected network with four hidden layers of size 80 just as our framework.
Hypernet (Ha et al., 2016). Hypernet extends the vanilla NODE by allowing the parameters of the ODE function to evolve over time, introducing additional flexibility in modeling complex dynamics. Instead of using static parameter set 0, the model employs hyper network that generates time dime-dependent parameters:
dar(t) f(=(t),;;(())),
(27)
where (t) is dynamically generated based auxiliary network, namely, hypernetwork, which is implemented as multilayer perceptron (MLP). The function governing the ODE is chosen to be fully connected feedforward network, consistent with our framework.
ANODEV2 (Zhang et al., 2019). ANODEV2 introduces novel coupled neural ODE framework in which the parameters of the ODE function, (t),, are not fixed but instead evolve over time. This makes ANODEV2 the first method to integrate parameter dynamics directly into the Neural ODE formulation. Additionally, ANODEV2 employs the Gershgorin function to model (t) further to ensure numerical stability and interpretability. In our experiments, the function f governing the ODE is chosen to be fully connected feedforward network as our framework.
Heavy Ball Neural ODE (HBNODE) (Xia et al., 2021). HBNODE incorporates second-order dynamics inspired by momentum-basse optimization. Instead of modeling first-order differential equations, HBNODE considers a second-order formulation:
d'x((t) + dxx(t) dt
+ ( =
(28)
dt
where y represents damping term. This design mimics heavy-ball acceleration techniques in optimization and enhances the stability and efficiency of learning in NODE-based systems.
LEADS (Yin et al., 2021). LEADS is a framework for learning dynamical systems that generalize across different environments by capturing both shared and environment-specifte dynamics. Instead of training a single global model or separate models per environment, LEADS learns a common model for shared dynamics and augments it with components tailored to each environment. This approach reduces sample complexity and improves generalization to both seen and unseen environments, with theoretical guarantees and strong empirical performance on linear and nonlinear systems.
CoDA (Kirchmeyer et al., 2022). CoDA is framework designed to enable fast and efficient generalization of dynamical system models to new, unseen environments. It addresses distributional shifts by conditioning a shared dynamics model on environment--peccfic context vectors, inferred from data using a hypernetwork. This approach constrains the hypothesis space, allowing rapid adaptation and improved generalization with limited data. CoDA is theoretically grounded and achieves state-of-the art results across diverse nonlinear dynamical systems, with the ability to infer system-specifie parameters from context with minimal supervision.
ODE via Invertible Neural Networks (OINNs) (Zhi et al., 2022). OINNs leverage the invertibility property of normalizing flows to construct a NODE formulation that preserves bijective mappings between states. The system is governed by:
dxx(t) (z(()))))
(29)
dt
where g(-) is invertible transformation. This approach allows for more stable training and enables efficient computation of the inverse mapping.
Modulated Neural ODE (MoNODE) (Auzina et al., 2024). MoNODE introduces external control mechanisms into the NODE framework, allowing for dynamic modulation of the ODE function. This design is particularly useful in scenarios where external factors influence system dynamics. The governing equation takes the form:
dxx(t) dt
f(x((),
(30)
where ul() is an external modulation signal.
As described above, Hypemnet and ANODEV2 explicitly incorporate time-dependent ODE parameters, enhancing adaptability in dynamic environments. We implement NODE, Hypernet, and ANODEV2 using the torchdi fee library, with their detailed architectures described above. For HBNODE, OINNs, and MoNODE, we leverage publicly available code to implement them.
31

---

KoNODE: Koopman-Driven Neural ODEs with Evolving Parameters for Time Series Analysis
D.3.2. TRAINING SETUP
The following reports the training setup of two experiments in Sec. 5.2. All models are implemented using PyTorch. We employ the Adam optimizer for training with an initial learning rate of 1 10-4, which follows exponential decay schedule with decay rate of 0.95. The batch size is set to 10, and the model is trained for 200 epochs. For solving the ordinary differential equation (ODE), we utilize the euler solver with atol = 10-?, and rtol 10-9. Gradient computation is performed using the adjoint sensitivity method, which allows for memory-efficient backpropagation through ODE solvers. The loss function during training is MSE between the predicted trajectory and the ground truth.
D.3.3. IMPLEMENTATION DETAILS ABOUT NON-LINEAR OSCILLATOR
System details. In this section, we consider an undamped and unforced nonlinear oscillator. We set a -1, 1 O.
The system equations are as follows:
dq
dp di
q - 9²:
= p.
dt
where q is the position, p is the momentum, and the parameters are a -1, 3 = and y 0. We give the characteristics analysis of this system in the following.
(1) No Damping Term (y = 0): The damping term typically causes the system's 's energy to dissipate gradually, leading to a decrease oscillation amplitude over time. However, in this system, due to the absence a damping term, the system will continue to oscillate without any energy dissipation.
(2) Nonlinear Restoring Force: In this system, the restoring force consists of two parts: linear term -aq and a nonlinear term 3
Linear Term (q): This is the standard spring restoring force term, describing the basic elastic behavior of the system.
Nonlinear Term (- This is the key term in the Duffing equation, reflecting the nonlinear characteristies of the system. Typically, this term causes the restoring force to depend not only on the linear displacement but also on the cubic power of the displacement.
(3) System's Motion Behavior: Due to the nonlinear restoring force, the system's motion can exhibit various complex behaviors, such as periodic motion, chaotic motion, or quasiperiodic oscillations. The nonlinear term usually causes the system to deviate from simple stable trajectory and may lead to different phase trajectories and periods. In particular, when B 3=1 0, the system exhibits sensitivity initial conditions.
Data generation. We first consider an undamped and unforced nonlinear oscillator, setting the parameters as -1, 1, 0. The training set includes 600 trajectories, each starting from an initial state uniformly sampled from an annular region in [0.2, 1]. Each trajectory spans 30 time steps with a fixed step size of 0.1, 1, and Gaussian noise 0.01n, n ~ WN 1) is added to simulate real-world data. The test set consists of 200 trajectories from the same system, with identical initial state sampling but extending to 300 time steps. This setup enables us to assess both the ability of long-term prediction and generalization. All trajectories are generated using dopri5. We visualize the trajectory data of four types under our parameter settings as shown in Fig. 8.
Model architecture and hyperparameter. The neural ODE function is parameterized as a fully connected network with four hidden layers, each containing 80 neurons with ReLU activation. wo is randomly sampled from Gaussian distribution.
The dimension of w is 10.
D.3.4. IMPLEMENTATION DETAILS ABOUT THE DUFFING OSCILLATORS
System details. The system family represents a weakly damped linear oscillator, governed by the equations:
dq dp
P. dt
oq 7P
(31)
dt
which can be rewritten as a second-order differential equation:
32

---

KoNODE: Koopman- Koopman-Driven Driven Neural ODEs with Evolving Parameters for Time Series Analysis
(a)
(b)
(c)
(d)
Figure Four types trajectory data generated from non-linear oscillator system.
dq 7
aq 0.
(32)
dt
Key characteristics:
Underdamped oscillation: The system exhibits oscillatory motion with gradual amplitude decay due to small damping.
Frequency dependence on a: The oscillation frequency is approximately the range [0.7, 1.0].
Exponential decay controlled by y: Weak damping leads to slow energy dissipation.
Parameter sensitivity: Small variations in a and y induce minor shifts in oscillation frequency and decay rate, making the system useful for studying stability under small perturbations.
Data generation. We consider weakly damped linear oscillator, setting the parameters as a E [.19, 3 = 0, and e [0.09, 0.11]. The training set consists 300 trajectories, each initialized from a random state uniformly sampled within an annular region r e [0.5, 1]. Each trajectory spans 100 time steps with fixed step size of 0.1, and Gaussian noise of magnitude 0.01n,
(O, 1) is added simulate real-world variability. The test contain 100 trajectories, following the same initial state sampling process the training set but extending to 200 time steps, allowing for the evaluation of long-term predictive capabilities. All trajectories are numerically integrated using the Dopri5 solver (a fourth-order Runge- Kutta adaptive method). To visualize the system's behavior, we randomly sample and plot several trajectory examples in the phase space (momentum p vs. position q), as shown in Fig. 9. This dataset serves as a benchmark for evaluating the ability of learning models to capture damped oscillatory dynamics and generalize across different time horizons.
Figure 9, Several trajectory examples in the phase space of duffing oscillators family data.
Model architecture and hyperparameter: The Neural ODE function is parameterized as a fully connected network with four hidden layers, each containing 500 neurons with ReLU activation. wo initialized using method II, that is, using a
33

---

KoNODE: Coopman-Driven Neural ODEs with Evolving Parameters for Time Series Analysis
Table Datails about robot motion datasets regarding "S", "CUBE", and "C".
DATA
NUMBER TRAJECTORIES
TIME POINTS
DATA DIMENSION
"C" "CUBE" "S"
12
1000
3
14
1000
1000
RNN, which processed the first 5 observed trajectory points to infer an initial latent state. Note that all of the comparison methods in this experiment use five observed trajectory points as inputs. The dimension of the w is set to 100. h is also the fully connected network.
D.4. Details about Experiment in Sec. 5.4
Training setup. The training setup of Experiment IV is the same to Experiment App. D.3.2.
D.4.1. DATASET
Dataset Description The dataset, proposed by Khansari-Zadeh & Billard (2011), consists of three types of trajectory data, all derived from real-world motion demonstrations, including:
(1) Drawing "S" shapes. This trajectory represents object or a robot executing an "S""shaped motion, which may be used for handwritten character recognition, path planning, or robot trajectory imitation learning.
(2) Placing a cube on shelf. This task involves manipulating objects (c.g., a robotic arm grasping and placing an object), which is commonly used in robotic grasping and placement tasks evaluate the precision and stability of robotic operations.
(3) Drawing out large "C", This trajectory involves performing large-scale "C"'shaped motion, which may be used for handwritten character recognition or path planning. It can also serve test case for trajectory generation models.
Data generation For preprocessing, we follow the approach of Zhi et al. (2022), where B-spline interpolation is applied to smooth the trajectories and standardize the number of time points to 1000. Detailed information about these three datasets provided in Tab. and the visual results are presented in Fig. 10.
Dataset:
Dataset: CUBE
Dataset:
60
S
80
55
50
45
60
40
10
-60
-60
Xeanis
40
X-ris
45
40
Faag
-50
-40
-20
Figure 10. Trajectory example of robot motion datasets.
For the "S" dataset, trajectories are used for training, while trajectory is allocated for validation and for testing, For the "CUBE" dataset, 10 trajectories are selected for training, with 1 trajectory for validation and for testing. For the "C"
34

---

KoNODE: Koopman-Drive Neural ODEs with Evolving Parameters for Time Series Analysis
Table Datails about datasets Experiment V regarding Type and Data Characteristics.
DATASET
TYPE
DATA CHARACTERISTICS
CUP
AIR QUALITY
LONG-TERM DEPENDENCIES, NONLINEAR POLLUTANT DIFFUSION
WEATHER
WEATHER VARIABLE
SHORT-TERM DYNAMICS DOMINANT, NON-PERIODIC FLUCTUATIONS
ETTHI
TRANSFORMER TEMPERATURE
LONG-TERM DEPENDENCIES, PERIODIC TRENDS
ETTH2
TRANSFORMER TEMPERATURE
LONG-TERM DEPENDENCIES, COMPLEX PERIODICITY
ETTM1
TRANSFORMER TEMPE, (MINUTE)
STRONG SHORT-TERM DYNAMICS, FEEEEEREE DATA
ETTM2
TRANSFORMER TEMPE. (MINUTE)
HIGH-FREQUENC DYNAMICS, STRONG SHORT-TERM TERM FLUCTUATIONS
Table Datails about datasets Experiment V regarding Frequeney and Dimension.
DATASET
FREQUENCY
DIMENSION
CUP
HOUR
270
WEATHER
10 MIN
ETTHI
HOUR
7
ETTH2
1 HOUR
7
ETTMl
15 MIN
ETTM2
15 MIN
dataset, 10 trajectories are designated for training, while I trajectory is assigned for validation and for testing. During training. random starting point is selected from the trajectories in the training set, and the subsequent 100 data points are extracted training samples. The model's performance then evaluated across the entire trajectory.
D.4.2. IMPLEMENTATION DETAILS ABOUT ROBOT MOTIONS: MODEL ARCHITECTURE AND HYPERPARAMETERS.
For dataset "S" and dataset "CUBE", the Neural ODE function is parameterized as a fully connected network with four hidden layers, each containing 200 neurons with ReLU activation, while for dataset "C" it contains 350 neurons. The initial observable wy is initialized using method II, that is, using a RNN, which processed the first 50 observed trajectory points to infer an initial latent state. Note that all of the comparison methods in this experiment use 50 observed trajectory points as inputs. The dimension of the w is both to 10. h is also the fully connected network.
D.5. Details about Experiment in Sec. 5.5
1. DATA DESCRIPTION
In this study, we use the following six datasets to evaluate the predictive performance different models. These datasets cover multiple application domains, including air quality forecasting, weather prediction, and electricity load forecasting, each exhibiting distinct time series characteristics. The details of the datasets summarized in Tabs. and 8.
D.5.2. IMPLEMENTATION DETAILS
We give the details about model architecture and hyperparameters across different datasets in this experiment in Tab. 9, Both our method and the baseline models take 10 time steps as input and predict the subsequent 100 time steps.
D.5.3. DETAILS ABOUT COMPARISION METHODS
All the results of the comparison methods we reproduced are implemented based on the original paper or official code.
deep vector autoregression (VAR) model with variational inference, suitable for multivariate time series forecasting and probabilistic modeling.
Deep
Autoformer:
Utilizes auto-comelation attention and trend-seasonality decomposition for improved long-term time series forecasting.
A Transformer- based model with patching mechanisms, effectively capturing local patterns and long-range dependencies.
PatchTST:
35

---

KoNODE: Goopman-Driven Neural ODEs with Evolving Parameters for Time Series Analysis
Table Details about model architecture and hyperparametens.
MODEL SETTINGS
CUP
WEATHER
ETTH1
ETTH2
ETTMI
ETTM2
ODE NETWORK
NETWORK TYPE
MLP
MLP
MLP
MLP 300
MLP
MLP
HIDDEN STATE DIMENSION
300 4
300
250 5
150
300
HIDDEN LAYER
4
4
5
ACTIVATION
RELU
RELU
RELU
RELU
RELU
RELU
SETTINGS
II(NNN) 100
(I(NN)
(IRNN)
(I(NN)) 100
II(NNN) 10
(I(NN))
DIMENSION
100
100
100
TRAINING SETTINGS
OPTIMIZER
ADAM EXP DECAY
LEARNING RATE
1e
SCHEDULER
DECAY RATE
0.95
BATCH SIZE
32 EULER
EPOCH BACKPROPAGATION
10
NUMERICAL SOLVER
ADIOINT METHOD
Koopa: Employs Koopman operator theory for nonlinear dynamic system modeling, ideal for periodic and quasi time series.
SST: Combines state-space modeling (SSM) with Transformer architecture, excelling at both short-term dynamics and long-term dependencies.
36


Reviews:
[{"review_id": "3QrrPMhZPb", "reviewer": "Reviewer_iauo", "summary": "This paper proposes KoNODE, a hierarchical framework that integrates Koopman operators into Neural Ordinary Differential Equations (NODEs) to learn time-evolving parameters.The authors provide theoretical error bounds for the finite-dimensional approximation of the Koopman operator and show how the proposed method improves long-horizon prediction and generalization on both synthetic oscillators and real-world time series tasks.\n\n## update after rebuttal\nGiven the fact, that authors addresed all the raised question and provided a new theorem with the proof, I raise the recommendation to Accept (4).", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback.\n\n**[W1. changes in memory or runtime]**\n\nWe will move the runtime analysis from the appendix to the main text to show our advantage in convergence rate and add a discussion on memory complexity. Regarding memory, we clarify that the memory of the proposed framework is $O(\\max\\\\{n, m, h\\\\}⋅D)$ compared to $O(nD)$ of NODE for $n$, $m$, $h$ being the dimensions of $x(t)$, $θ(t)$, and the hidden layer respectively and $D$ the model size for the differential function $f$.  \n\n---\n\n**[W2 & Q2. selection of Koopman dimension]**\n\nThank you for the concern about the choice of the Koopman dimension $m$. We provide theoretical lower and upper bounds for $m$. Although other factors may influence the best picks, the bounds provide references for the systematic criterion. \nSpecifically, Thm. R.1 shows that $m\\le r$ for the rank of trajectory space $r$ and Thm. R.4 shows that $m$ has a tight lower bound $⌈\\frac{D}{D-r}⌉$ for second-order derivable $x(t)$, where $D$ is the dimension of $Θ$ (the model complexity of $f$) and $r$ is the rank of data. Note that the lower bound is commonly small for a network that is strong enough, yet the hyper-parameter may not be static in practice due to (1) the auxiliary dimensions caused by the designed form of the Koopman matrix, (2) the disturbance caused by the multiple choice of the dynamic $φ(θ(t), t)$ as $D\\gg n$, and (3) the attempt to more accurate dynamic regression by multiple $\\mathfrak F_t$ sets. The theorems are given as follows. \n\n**Theorem R.1 (The Theoretical Upper-Bound of $m$)**\n\nInequality $m\\le r$ holds if:\n\n(1) The data trajectories are approximately in an $r$-dimensional manifold $\\mathcal M$, i.e., \n$$\n∀ε>0,∃~δ> 0,ℙ_{x(t)\\sim\\text{data}}[\\Vert x(t)-x^\\perp(t)\\Vert <ε] > 1-δ,\n$$\nwhere $x^\\perp≜\\underset{y\\in\\mathcal M}{\\text{argmin}}\\Vert x-y\\Vert $ is the projection of $x$ to the manifold. \n\n(2) Function $f$ satisfies the Lipchitz condition and is differentiable.  \n\n*Proof:* Since $\\mathcal M$ admits a bijective chart $ψ:\\mathcal M\\toℝ^r$, the map $\\phi_t(θ(t)) = \\frac{dψ(x^\\perp(t))}{dx^\\perp(t)}f(x^\\perp(t), t;θ(t))$ is bijective. This, together with the uniqueness of $x(t)$ (by Picard–Lindelöf), implies that $θ(t)$ lies on an $r$-dimensional manifold. Hence, $m\\le r$. \n\n**Definition R.2 (Frontier Manifold)**\n\nFor a dynamic in Euclidean space, a fixed time $t_0$ and origin $θ^*$, the frontier manifold $\\mathfrak F_t$ is the quotient space of the equivalent class decided by the trajectories, i.e., $\\mathfrak F_0$ satisfies (1) $θ^*\\in\\mathfrak F_0$, and (2) the normal vector at $θ\\in\\mathfrak F_0$ is $φ(θ, t_0)$. Then, we define $\\mathfrak F_t$ as the set of evolved $θ(t)$ with $θ(t_0)\\in\\mathfrak F_0$. \n\n**Lemma R.3**\n\nFor $θ_0\\in\\mathfrak F_s$, if the trajectory $θ(t)$ follows the dynamic \n$$\n\\frac{dθ(t)}{dt} =φ(θ(t), t),θ(t_0) =θ_0,\n$$\na scalar observable function $g(θ)≜𝓒⋅\\exp(\\frac{λ}{D}\\int\\text{inv}[φ(θ, t_s(θ))]^⊤dθ)$ maps it into a Koopman space. \n\n*Proof:* Note that for $θ(t_0) =θ_0\\in\\mathfrak F_s$, $θ(t)$ is uniquely identified by the Picard-Lindelöf Theorem. Moreover, all elements in a frontier manifold $\\mathfrak F_s$ are on different trajectories, which creates a bijection between $(θ_0, t)$ and $θ(t)$. Let $t_s(θ)$ be the mapping from $θ$ to time under the condition that $θ_0\\in\\mathfrak F_s$. \n\nThen, let $g(θ)$ be a scalar observable function, \n$$\ng(θ)≜𝓒⋅σ^{-1}\\text{ where }σ=\\exp\\left(-\\frac{λ}{D}\\int\\text{inv}[φ(θ, t_s(θ))]^⊤dθ\\right),\n$$\nwhere $λ$ and $𝓒$ are constants and $\\text{inv}[⋅]$ is the element-wise inverse of a vector. Therefore, \n$$\n\\begin{aligned}\n\\frac{dg(θ)}{dt} - k⋅g(θ) =&∇_{θ}g(θ)^⊤φ(θ, t) - k⋅g(θ)⋅φ^⊤(θ, t)⋅\\text{inv}[φ(θ, t)]/D,\\\\\\\\\n=&φ^⊤(θ, t)⋅[∇_{θ}g(θ) -λ/D⋅g(θ)⋅\\text{inv}[φ(θ, t)]], \\\\\\\\\n=&\\frac{1}{σ}∇_{θ}[g(θ)⋅σ]= 0. \n\\end{aligned}\n$$\nHence, $g(θ)$ lies in the Koopman space. \n\n**Theorem R.4 (The Theoretical Lower-Bound of $m$)**\n\nTo model the trajectory of $x(t)$ if it is second-order derivable, the dimension of $w(t)$ is ideally $⌈\\frac{D}{D-r}⌉$ where $r$ is the rank of data. \n\n*Proof:* For a differentiable $f$, the dynamic $\\frac{dθ(t)}{dt} =φ(θ(t), t)$ best models the trajectory $x(t)$ where\n$$\nφ(θ(t), t)≜\\left[\\frac{∂f}{∂θ(t)}\\right]^{†}\\left[\\frac{d^2x(t)}{dt^2} -\\frac{∂f}{∂t} -\\frac{∂f}{∂x(t)}f(x(t), t;θ(t))\\right],\\tag{1}\n$$\nand $A^†$ being the pseudo-inverse of the Jacobian matrix. \n\nLamma R.3 infers that for $θ_0\\in\\mathfrak F_s$, only one dimension of $w$ is needed. Thm. R.1 indicates that the gradient $φ(θ(t), t)$ lies in a space with the highest rank of $r$, thus the frontier manifold $\\mathfrak F_s$ covers a dimension of at least $D-r$. To ensure a full cover of $Θ$-space in the initialization, an ideal dimension of $w$ is $⌈\\frac{D}{D-r}⌉$. \n\n---\n\n**[Q1. irregularly sampled data]**\n\nNODE-based methods model the system continuously, allowing them to handle irregularly sampled data by interpolating for intermediate time points.", "claims_and_evidence": "The paper states that modeling time-evolving ODE parameters via Koopman operators enables a deeper representation of underlying system dynamics, improves long-horizon forecasting, and generalizes better to unseen conditions. It has both theoretical evidence with finite-dimensional Koopman approximation error bound and experimental evidence with superior performance on the synthetic and real world data.", "methods_and_evaluation_criteria": "The authors conducted several experiments on synthetic and real-world data. They selected a wide range of methods including classic Neural ODEs as well as the models with evolving parameters (ANODEV2), that makes the comparison reasonable.", "theoretical_claims": "There are several theoretical claims on the error bounds. Proofs, that are provided in the appendix, look reasonable, however, they may be a bit hard to follow, specifically with these equal signs with stars and asterisks, it is better to avoid such “nonlinearities” and be more consequent.", "experimental_designs_or_analyses": "I reviewed both the synthetic and real-world experiments in detail, and the overall design and selection of baselines are sound. There are two  minor concerns. The first one relates to the potential computational overhead introduced by the added Koopman operator layer. While the appendix does include some runtime analysis, it would be helpful to include a concise overview in the main text to clarify any changes in memory or runtime requirements. The second one is that while the authors provide some insight into Koopman operator dimensionality in synthetic settings, it remains unclear how these dimensional choices generalize to real-world scenarios or whether any systematic criterion (e.g., data-driven rank selection) could be used. Addressing these points would further strengthen the paper’s clarity and applicability. Aside from that, the experimental setup appears valid, with no major issues that would detract from the paper’s conclusions.", "relation_to_broader_scientific_literature": "The paper builds on Neural ODE literature (Chen et al. 2018) and prior Koopman approaches (Lusch et al. 2018, among others). It extends ANODEV2 by modeling theta through a linear Koopman system rather than a general ODE, thus adding interpretability and improved long-horizon stability.", "essential_references_not_discussed": "n/a", "other_strengths_and_weaknesses": "The paper’s primary strengths lie in its novel integration of Koopman operators with Neural ODEs, supported by comprehensive theoretical analysis and strong empirical results across both synthetic and real-world datasets. Meanwhile, potential weaknesses include limited discussion of computational overhead in the main text (despite some runtime data in the appendix), and a need for deeper examination of how the chosen Koopman dimensionality affects performance in real-world scenarios and whether a systematic selection method could be applied.", "other_comments_or_suggestions": "n/a", "questions_for_authors": "1. How does KoNODE handle irregularly sampled data?\n2. Is there a systematic way to select the dimension of the Koopman space m? Can any data-driven rank estimation be integrated?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "A0XoP3FD7Q", "reviewer": "Reviewer_Snqd", "summary": "This paper explores the challenge of modeling time series with NODEs. The authors propose a Koopman-driven framework named KoNODE that hierarchically encodes system dynamics through evolving ODE parameters and Koopman linear operators. Specifically, they introduce a three-level architecture—spanning observed state dynamics, parameter dynamics, and intrinsic Koopman linear dynamics—to disentangle surface-level behaviors from fundamental governing rules. Extensive experiments on synthetic and real-world datasets validate the effectiveness of the proposed approach.\n\n## update after rebuttal\nAll my concerns have now been adequately addressed. Overall, this is a good work. However, considering the limited application scenarios, I will maintain my original score of \"weak accept.\"", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive feedback.\n\n**[W1. lack sufficient interpretability]**\n\nWe apologize for the unclear description of \"revealing the fundamental driving forces of system evolution\" and will clarify it in the Introduction and Experiment sections in the revision.\n\nIn our framework, the deepest dynamics, i.e., Koopman linear dynamics $dw/dt = Aw$, are obtained through Koopman operators. Fundamental evolution principles are revealed by the spectrum of the operators, which correspond element-wise to matrix $A$ and are given by $λ_j(K)=e^{α_jΔt}(\\cosβ_jΔt±\\rm i\\it\\sinβ_jΔt)$ in our framework. Specifically, the real part of the eigenvalues indicates the evolution speed, while the imaginary part corresponds to intrinsic frequencies and periodicity. Analyzing the elements of $A$ could help identify the system’s dominant driving modes and frequencies and offer insights into stability, periodicity, or other inherent characteristics, thus gaining a mathematically interpretable understanding of the system's evolution.  \n\nTo further demonstrate this, we provide a more detailed analysis and interpretation of the learned dynamics in Sec. 5.2 by visualizing the spectrum of approximate Koopman operator for the Oscillator dataset in `spectrum_reviewers_Snqd_1pn6.pdf`, https://anonymous.4open.science/r/KoNODE-D8F2/. First, the magnitudes of spectrum are approximately one for both two dynamic systems, indicating boundary stability, where the state remains on a stable trajectory without diverging or converging. Second, the dominant spectrum of the nonlinear oscillator (on the left in the figure) has similar imaginary parts (i.e., the frequency components are the same), indicating that the system exhibits clear periodicity. Modes with the same imaginary part have the same periodic components.\n\n---\n\n**[W2. comparison to generalizable NODEs]**\n\nWe have added the comparison between our framework with other generalizable NODEs, LEADS [1], and CoDA [2] in `supplementary_table_reviewers_Snqd_waH1.pdf`, https://anonymous.4open.science/r/KoNODE-D8F2/. Due to the lack of publicly available code for [3], we did not compare with it.\n\n---\n\n**[Q1. A≡D]**\n\nAs stated in lines 174-178 on the left, $\\frac{dg(θ)}{dt}=Ag(θ)\\iff\\frac{d\\tilde{g}(θ)}{dt}=PAP^{-1}\\tilde{g}(θ)=D\\tilde{g}(θ)$ hence we can assume $A\\equiv D$ by modeling $\\tilde{g}$ instead of $g$. We will clarify the definition of $P$ to avoid confusion. The assumption $A≡D$ is not only mathematically rigorous but, as addressed in [W1. lack sufficient interpretability], allows us to analyze the system's behavior directly by observing the elements of matrix $A$. \n\n---\n\n**[Q2. Hilbert space]**\n\nThe theorems are formulated in Hilbert spaces for generality, and this is a very mild assumption. The observation space of the data naturally satisfies this, as we only require the data space to be complete and have an inner product. The two functions are not necessarily characteristic functions. As clarified in line 156 on the right, function h is maintained as the inverse of the characteristic function, while f is the differential function in the NODE framework shown in Eq. (2).", "claims_and_evidence": "N/A", "methods_and_evaluation_criteria": "N/A", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "N/A", "relation_to_broader_scientific_literature": "This work is beneficial for the development of AI for Science research.", "essential_references_not_discussed": "Pls refer to Weaknesses", "other_strengths_and_weaknesses": "**Advantages:**\n\n1. The proposed KoNODE framework is both theoretically grounded and empirically effective in capturing intrinsic system dynamics.\n2. The experiments are very detailed and thorough, clearly demonstrating the effectiveness and efficiency of the proposed method.\n3. The manuscript is well-structured and clearly written, facilitating easy comprehension for readers.\n\n**Weakness:**\n\n1. From my opinion, the first contribution which is claimed as “uncovers the fundamental principles driving system evolution” appears somewhat overstated, as the identified deepest dynamics still lack sufficient interpretability. \n2. This framework is very generic, but its primary focus on time-evolving ODE parameters is not yet a common paradigm. Could the author compare their settings with other generalizable NODEs [1][2][3]?\n\n---\n**Reference:**\n\n[1] LEADS: Learning Dynamical Systems that Generalize Across Environments. NeurIPS, 2021.\n\n[2] Generalizing to New Physical Systems via Context-Informed Dynamics Model. ICML, 2022.\n\n[3] Generalizing Graph ODE for Learning Complex System Dynamics across Environments. KDD, 2023.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "1. Line 178-179, the left half of the page. Could the author clarify why the assumption $A \\equiv D$ makes sense?\n\n2. In Section 3.3.2, Theorems 3.3 and 3.4 are formulated in Hilbert space rather than in the observation space of the data. Should $f$ and $h$ be maintained as characteristic functions to ensure these two theorems valid?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "60XmsJ0An1", "reviewer": "Reviewer_1pn6", "summary": "An architecture based on neural ODE with time-varying parameters is proposed. The idea is to model the dynamics of the parameters with latent linear dynamics, which the authors motivate by referring to the Koopman operators. The superior prediction performance of the proposed method is empirically demonstrated.\n\n### Update after rebuttal\n\nThanks for the clarification, the additional discussion will make the paper more convincing. I keep my originally positive score.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful comments. \n\n**[1. empirical observation supports]**\n\nWe have added empirical evidence and analysis to support the two quoted claims. We will include concrete empirical observations in the revision. Please refer to [W1. lack sufficient interpretability], *Response to Reviewer Snqd* for further details, and `spectrum_reviewers_Snqd_1pn6.pdf`, https://anonymous.4open.science/r/KoNODE-D8F2/ for the visualization. \n\n---\n\n**[2. theoretical claims]**\n\nThank you for the suggestion and we are sorry for the confusion. We will clarify two motivations in Section 3.3 in the revised manuscript, i.e., the presented error provides (1) a theoretical reference for the choice of dimension $m$, and (2) proof for the minority of errors caused by the additional Koopman module compared to conventional ODE models, apart from the potential improvement due to the advanced fitting ability. \nThe theoretical claims are given under the assumption that the parameters $θ(t)$ indicate the running dynamics, which are intrinsically determined for a given trajectory $x(t)$. Consequently, an accurate model of $θ(t)$ directly leads to the accurate estimation of $x(t)$. \nThm. R.6 below conducts a theoretical comparison of the overall prediction error between the proposed method and a conventional ODE based on the given theories to indicate the superiority of our method. \n\n**Definition R.5 (The Static Dynamic Model Condition)**\n\nThe dynamic satisfies the condition if and only if for the function $g$ and any $\\tildeε$, $∃\\tildeδ>0$, \n$$\nℙ(|g(θ^*) - g[\\tildeθ(t)]|<\\tildeε)\\ge 1-\\tildeδ,\n$$\nwhere $\\tildeθ(t)$ is randomly drawn from the ideal trajectory defined in Eq. (1), Theorem R.4, [W2 & Q2. selection of Koopman dimension], *Response to Reviewer iauo* and $θ^*≜\\underset{θ}{\\text{argmin}}\\int_t\\left\\Vert f(x(t), t;θ)-f(x(t), t;\\tildeθ(t))\\right\\Vert$ is the LSE estimation of a static $θ$. The condition holds when the true dynamic behind the trajectory is static which ensures a good fit of the NODE. \n\n**Theorem R.6**\n\nDuring the modeling of trajectory $x(t)$, the estimation error upper bound of the proposed model is smaller than that of a conventional ODE method if the observable function $g$ does NOT satisfy *the static dynamic model condition*, i.e., for the bound\n$$\n\\mathcal B[\\hat{θ}(t)]≜\\sup_{θ}\\left\\Vert\\frac{∂f(x(t), t;θ)}{∂θ}\\right\\Vert⋅\\Vert\\hat{θ}(t) -\\tilde{θ} (t)\\Vert,\\tag{2}\n$$\nthere exists,\n$$\nℙ(\\mathcal B[{θ}^†(t)]\\le\\mathcal B[θ^*])\\ge1-δ^*,\n$$\nwhere $θ^†(t)$ is the trajectory predicted by the proposed method while $θ^*$ consists of the static parameters in the conventional ODE method. \n\n*Proof:* We use the notations in Def. R.5. Apparently, Eq. (2) gives a bound of the estimation error where the \"sup\" exists as $f$ is Lipchitz continuous, and the bound is reached for linear mapping $f$. \n\nNote that $\\hat{θ}(t)=θ^*$ in NODE and $\\hat{θ}(t)=θ^†(t)≜h(\\hat{w}(t);ψ)≈g^{-1}(\\xi ^⊤\\hat{w}(t))$ in the proposed Koopman model where $\\hat{w}(t)$ satisfies the Koopman model of matrix $\\hat{K}_N$. Thm. 3.4, Manuscript shows that \n$$\nℙ(|\\xi ^⊤\\hat{w}(t+Δt) - g[\\tilde{θ}(t+Δt)]|<ε^*⋅\\Vert\\xi\\Vert⋅\\Vert\\hat{w}(t)\\Vert )\\ge 1-δ,\n$$\n\nwith $ε^*$ representing the RHS . As the base $u_i(t)\\in L^2(Θ)$ for observables $\\hat{w}$, Lemma C.12, Appendices implies $ℙ(|\\hat{w}_i(t)|\\le\\frac{1}{\\sqrtδ} )\\ge 1-δ$ if $\\Vert u_i(t)\\Vert _{L^2(Θ)} = 1$ and $\\max|\\xi _i|\\le r(\\mathcal K)$, \n$$\n\\textstyleℙ(\\Vert\\xi\\Vert⋅\\Vert\\hat{w}(t)\\Vert\\le\\frac{m\\sqrt m⋅r(\\mathcal K)}{\\sqrtδ})\\ge 1-δ.\n$$\nConsequently, if we let\n$$\n\\tildeε=\\frac{2\\sqrt3σm^{\\frac{7}{2}}r^2(\\mathcal K)}{\\sqrt{Nδ(δ-2m^{-\\frac{r}{3}})}⋅\\min\\\\{1, r(\\mathcal K)\\\\}-1}+o(m^{-\\frac{r}{3}}),\n$$\nthen $ℙ(|\\xi ^⊤\\hat{w}(t+Δt) - g[\\tilde{θ}(t+Δt)]|<\\tildeε)\\ge 1-2δ$.\n\nOn the other hand, for $g$ failed to satisfy the static dynamic model condition at error $\\tildeε$ we have\n$$\n\\begin{aligned}\n&ℙ(\\Vertθ^†(t) -\\tilde{θ}(t)\\Vert\\le\\Vertθ^*-\\tilde{θ}(t)\\Vert ),\\\\\\\\\n= &ℙ(|g[θ^†(t)] - g[\\tilde{θ}(t)] |\\le|g(θ^*)-g[\\tilde{θ}(t)]|)\\\\\\\\\n&-ℙ(|g[θ^†(t)] - g[\\tilde{θ}(t)] |\\le|g(θ^*)-g[\\tilde{θ}(t)]|\\text{ and }\\tilde{θ}(t)\\in Q),\\\\\\\\\n\\ge &ℙ(|g[θ^†(t)] - g[\\tilde{θ}(t)] |\\le|g(θ^*)-g[\\tilde{θ}(t)]|)-ℙ(\\tilde{θ}(t)\\in Q)\\ge 1-δ^*,\n\\end{aligned}\n$$\nwhere $Q≜\\\\{θ|~|g(θ^*)-g(θ)|<\\tildeε\\\\}$ and $δ^*≜2δ+\\tildeδ+ℙ(\\tilde{θ}(t)\\in Q)$. \n\nAs a result, with a probability of $1-δ^*$, the error bound in Eq. (2) for the proposed method is lower than that of the conventional method.", "claims_and_evidence": "I found two points in the claims. One is about its practical utility, i.e., better performance in prediction. I think this aspect is nicely demonstrated with the empirical results.\n\nThe other point of the claims I found is about \"understanding\" of data or models. For example, the authors claim:\n\n> We propose a three-level hierarchical architecture ... that deepens our understanding of system behavior and its governing laws.\n\n> By leveraging Koopman operators, our framework could use spectral tools for system analysis ...\n\nAlthough I do not deny such a possible utility of the method in general, these aspects do not seem supported by concrete empirical observations.", "methods_and_evaluation_criteria": "The time-series prediction experiments seem valid.", "theoretical_claims": "A bunch of theoretical claims are presented in Section 3.3 and the appendices. They may be okay, but I do not see how relevant they are to the proposed method. The presented theories are on the estimation error of the Koopman operators, but the proposed method consists of not only the Koopman part but also other components such as the map from $w$ to $\\theta$. It would be nice if the authors could elaborate more on the motivation of the theoretical arguments: for example, the overall picture (purpose) of the analyses, key assumptions, and remaining gap to fully analyze the prediction errors of the proposed method, if any.", "experimental_designs_or_analyses": "As mentioned in the \"Methods And Evaluation Criteria\" section, the experiments make sense.", "relation_to_broader_scientific_literature": "Time-series prediction is widespread in any domain of science and engineering.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "N/A", "questions_for_authors": "I do not have major questions that strongly affect my evaluation.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xFBDJ4dWmv", "reviewer": "Reviewer_waH1", "summary": "**Edit**: Having read through the rebuttals and the other reviews, I am satisfied with the paper and have increased my score from 3 (weak accept) to 4 (accept). I am grateful to the authors for taking the time to respond, and have left remaining specific thoughts in the Rebuttal Comment.\n\n**Original Review**:\n\nThis paper introduces Koopman-Driven Neural Ordinary Differential Equations (KoNODEs). The motivation is that Neural ODEs given by:\n\n$\\frac{d\\mathbf{x}}{dt} = f(\\mathbf{x}, t, \\theta)$\n\nCan benefit from making the parameters time-dependent:\n\n$\\frac{d\\mathbf{x}}{dt} = f(\\mathbf{x}, t, \\theta(t))$.\n\nOne example being that the underlying dynamics might remain unchanged, but the parameters of the dynamics do, for example in engineering the wear and tear of instruments can change the observed dynamics, even though the underlying physics remains unchanged.\n\nAn existing solution is ANODE-V2 that models $\\frac{d\\mathbf{x}}{dt} = f(\\mathbf{x}, t, \\theta(t))$ using a neural network and $\\frac{d\\theta}{dt} = g(\\theta, t, \\phi)$, so the network parameters are given by another ODE.\n\nThis work adds further modeling restrictions to the ODE with the aim of modeling more complex behaviors, for longer time series. This is achieved by using Koopman theory, which at a high level says that non-linear dynamics can be achieved by using linear dynamics in a different space. This paper explores how this can be achieved in practice for Neural ODEs, provides an implementation, theoretical analysis and experimental evidence showing modeling Neural ODEs using Koopman operators is an effective way to restrict the dynamics of the parameters to actually improve the model.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your deep understanding of our method and insightful suggestions.\n\n**[1. comparison to baselines]**\n\nWe have included the baselines mentioned, Latent ODE (ODE enc) and Neural Processes, in `supplementary_table_reviewers_Snqd_waH1.pdf`, https://anonymous.4open.science/r/KoNODE-D8F2/. We didn't compare ODE-RNN due to its official code not supporting prediction tasks.\n\n---\n\n**[2. failure on spiral synthetic dataset]**\n\nLines 283-285 to the right clarified that the spiral dataset we were using is obtained by $[\\frac{dx}{dt}\\ \\frac{dy}{dt}]^⊤= A^⊤(t)[x^3\\ y^3]^⊤$ where $A(t)=\\begin{bmatrix}-0.1+0.5\\sin t&2.0+\\cos t\\\\\\\\-2.0+0.5\\cos 2t&-0.1-0.5\\sin t\\end{bmatrix}$, which is more complicated than the one used in vanilla NODE, i.e., $ A(t) = \\begin{bmatrix}-0.1&2.0\\\\\\\\-2.0&-0.1\\end{bmatrix}$. We did successfully reproduce fittings for the spiral proposed in both ours and NODE yet the modified data is more complicated so would demonstrate the limitation of the conventional model. \n\n---\n\n**[3. $\\xi_i$ in line 254]**\n\nSorry for the negligence in the notation. We will clarify in the revision that $\\xi=[\\xi_1\\ \\xi_2\\ \\cdots\\ \\xi_m]^⊤$ while the coordinates $\\xi_i≜\\langle g, u_i\\rangle$ even for $i>m$ as the true Koopman space has an infinite number of bases $\\left\\\\{u_i\\right\\\\}_{i=1}^\\infty$ (though no more than $N$-dimensions due to the data-driven scenario). \n\n---\n\n**[W1. writing]**\n\nThank you for the valuable suggestions. We will modify accordingly (including the minor fixes and the simplifications of the KoNODE description and Section 3.2.2). \n\n---\n\n**[Q1. larger ODE expression]**\n\nIt is a promising insight that will definitely be easier to write. However, this pipeline sacrifices the consistency of function $h$ and does not significantly reduce the computational cost (per solve). The reasons are: (1) In the forward integration, both pipelines should go through function $f$, compute $Aw$, and another network being either $h$ or $h_1$; (2) In the adjoint method, both pipelines require the gradient through network $h$ (or $h_1$) which requires a backpropagation through them; (3) The two networks, $h$ and $h_1$, both require the potential to map from the $w$-space to the $θ$-space, hence the sizes of the networks do not differ much. \n\n---\n\n**[Q2. Fig. 4C]**\n\nThank you for your deep understanding of our method. Fig. 4(c) shows a curve dropping in $[2, 10]$ and rising if $m > 10$ with a tolerable relative perturbation. Experimentally, more weights are introduced in networks, which may add extra uncertainty and instability during training when $m$ is large and [W2 & Q2. selection of Koopman dimension], *Response to Reviewer iauo* explains the reasons why the practical dimension is not that stable. Theoretically, the error bound in Thm. 3.4, Manuscript is majorly dominated by two terms, one with a factor of $\\frac{m^2}{\\sqrt N}$ in coefficient $p$ and the other with the order of $m^{-\\frac r3}$, which does indicate the instability when $m$ is too large and may infer overfits. On the other hand, the order $r$ is negative only when $m$ is too small that the first $m$ bases $\\\\{u_i\\\\}_{i=1}^m$ fail to successfully model the true observable function $g$, hence this scenario only influences the left-hand side of the graph when $m$ is below the theoretical lower bound given in Thm. R.4, [W2 & Q2. selection of Koopman dimension], *Response to Reviewer iauo*.\n\n---\n\n**[Q3. Tab. 4]**\n\nSorry for the negligence of the phenomenon as the table aimed at showing the more significant improvement from the KoNODE framework using the Euler method. We will rephrase the interpretation in the revision. Regarding the lateral comparison, the integration by RK4 and DOPRI5 is undoubtedly more accurate yet the superiority of Eurler is caused by the different intrinsic objectives under the regularly sampled trajectory, i.e., the network for Euler estimates $\\int_t^{t+Δt}f(x(t), t)\\ dt$ for a fixed $Δt$ but the others for a mutable $Δt$. The fixed $Δt$ reduces the complexity of the target function and thus simplifies the task using Euler.", "claims_and_evidence": "The claims presented are supported by experimental evidence. However, there are some relevant baselines that would improve the strength of the evidence. For example ODE-RNN or Latent ODE (https://arxiv.org/abs/1907.03907), or Neural Processes (https://arxiv.org/abs/1807.01622) as another method. These are common baselines that would improve the experimental results.\n\nI am also concerned with Figure 3, which shows vanilla NODE not performing on the spiral synthetic dataset. Why is this the case since it does perform in the original paper?", "methods_and_evaluation_criteria": "The experiments carried out make sense for the application. The datasets are a good mix of synthetic to demonstrate key points and real to show that the method works on real data.", "theoretical_claims": "The theoretical claims as far as I can tell are correct. I have not checked the proofs in the appendix in significant detail. However, I am confused by Line 254 on the left:\n\n$-\\text{max}\\_{m < i \\leq N}\\log\\_i \\xi\\_i - 1$\n\nWhat does it mean for a logarithm to be subscripted? Previously it is given that $\\xi = [\\langle g, u\\_1 \\rangle, \\langle g, u\\_2 \\rangle ... \\langle g, u\\_m \\rangle]$, so how can $i$ be larger than $m$ and $\\leq N$? Especially given that $n$ is the amount of data, but $\\xi$ seems to be the dimension $m$ of the Koopman space?", "experimental_designs_or_analyses": "The experiments and analyses associated are sound. But as mentioned would be improved with stronger ODE baselines (e.g. ODE-RNN, Latent ODE) and stronger baselines for generalising to new dynamics (e.g. Neural Processes).", "relation_to_broader_scientific_literature": "The key contributions are framed appropriately in relation to the scientific literature.", "essential_references_not_discussed": "There are none as far as I can tell.", "other_strengths_and_weaknesses": "Stengths:\n\n- The idea is robust and theoretically motivated\n- The datasets are diverse and relevant\n- The experiments support the claims\n\nWeaknesses:\n\n- The writing needs some work, see the Other Comments or Suggestions section of the review for specific cases. The description of KoNODE at points is overly complicated, or unnecessary. For example, at the beginning of page 4, one possible $D$ matrix is described in detail, before describing a second $D$ matrix with $2\\times2$ block diagonals, the first version is never used and therefore is not necessary to describe. Section 3.2.2 is another section that overly complicates the description, where the point is to say that the adjoint sensitivity method is used to calculate gradients, which is already described in detail in the original Neural ODE paper", "other_comments_or_suggestions": "This is a list of contained writing changes I would make. They are suggestions so are not required:\n\n- Line 17 Right: \"The Ordinary Differential Equations\" should be just \"Ordinary Differential Equations\"\n- Line 19 Right: \"of ODEs\" should be \"of the ODE\"\n- Line 38-39 Right: \"encodes the deeper underlying dynamical principles of the evolution (namely deep-level information) inherently\" should be \"inherently encodes the underlying dynamics of the evolution (namely deep-level information)\"\n- Line 59 Left: \"Based on the insight\" should be \"Base on this insight\"\n- Line 80-81 Left: \"and ultimately governed by a deeper-leveled, intrinsic linear core - the Koopman linear dynamics\" should be \"and ultimately governed by the underlying Koopman linear dynamics\"\n- Line 66 Right: \"Applying the Koopman Theory\" should be just \"Applying Koopman Theory\"\n- Line 94-96 Right: \", rather than propagating activations through discrete layers as in recurrent or deep networks. Neural ODEs employ numerical solvers\" should be \". Rather than propagating activations through discrete layers as in recurrent or deep networks, Neural ODEs employ numerical solvers\"\n- Line 124 Left: \"The Koopman Theory\" should be \"Koopman Theory\"\n- Line 124-125 Left \"dynamic system analysis\" should be \"dynamical system analysis\"\n- Line 141 Left: \"commonly finitely approximated\" can be just \"commonly approximated\" since the phrase \"finite number\" is used later in the sentence\n- Line 144 Left \"The Koopman operator are\" should be \"The Koopman operator is\" or \"The Koopman operators are\"\n- Line 112 Right: \"within the NODE setups\" should be \"within the NODE setup\"\n- Line 117-120 Right: \"Besides, we assume the trajectories of the intrinsic linear dyanmics are represented as w(t)\" should be \"We let the intrinsic linear dynamics be represented as w(t)\"\n- Line 126-144 Right: Describing the three hierarchical levels should refer to the \"dynamics\" not the \"dynamic\"\n- Line 158 Right: The sentence \"We model the h by the neural network due to its flexibility\" is not needed, but should be \"We model h using a neural network due to its flexibility\"\n- Line 169 Left: \"in replace of the conjugated imaginary eigenvalues of A\" should be \"that represent the imaginary eigenvalues of A\"\n- Figure 2 caption: Should refer to the \"dynamics\" not the \"dynamic\"\n- Figure 2 caption: \"over altogether N time points\" should be just \"over N time points\"\n- Section 4 description: \"just as the red line\" and \"just as the green line\" should be \"shown by the red line\" and \"shown by the green line\"\n- Line 287 Left: \"At last, we get the loss between the predicted trajectory and the true trajectory\", should be \"Using the true trajectory and the predicted trajectory, we can calculate the loss\"\n- Line 295-300 Left: \"We serve w0 as an encoding\" should be \"w0 is an encoding\" and \"Learnable setting when given sequence as input\" should be \"Learnable setting when given a sequence as input\"\n- Line 375 Left: \"Preformance metric\" should be \"Performance metric\"\n- Line 405 Left: \"should inherently low-dimensional\" should be \"should inherently be low-dimensional\"\n- Lines 433-439 Left: When using quotation marks in Latex they can be properly displayed by changing \"S\" to ``S''\n- Line 420 Right: \"ETTh1\" should be \"ETTH1\"\n- Line 427 Right: \"and the Koopman modeling\" should be \"and Koopman modeling\"", "questions_for_authors": "1. Is it possible to represent KoNODE as another larger first order ODE:\n\n$z = [x, \\theta, w]$, $\\frac{dz}{dt} = [f(x, t, \\theta), \\frac{\\partial h}{\\partial w}Aw, Aw]$, $z\\_0 = [x\\_0, h(w\\_0), w\\_0]$\n\nIf this is the case, it might be easier to write $\\frac{\\partial h}{\\partial w}Aw$ as $h\\_1(w)$ and $h(w\\_0)$ as $h\\_2(w\\_0)$. That way the whole system can be solved together in one ODE solve, which will likely reduce the computation required per solve. This also makes the adjoint sensitivity derivation trivial since the standard adjoint sensitivity can be applied to the extended ODE system.\n\n2. In Figure 4C, why does error increase as the dimension of $w$ increases. Is it hard for KoNODE to learn redundancy? Or is it encouraged to use all of its dimensions to learn the underlying Koopman dynamics? The theoretical bound presented says that the error decreases with more dimensions, why is this not the case? Does the model overfit when it has more capacity, i.e. $N$ is not large enough, so $r$ is negative in Theorem 3.2 so the error bound does not decrease with $m$?\n\n3. Can you rephrase your argument around Table 4, the argument is that the Euler method can benefit from KoNODEs, with more accurate dynamics. However this does not make sense to me. Firstly Table 4 shows that the Euler method is also better using NODE, so it does not empirically hold. Additionally, RK4 and DOPRI5 should be more accurate than the Euler method during the ODE solve, so should also benefit from improved modeling of the dynamics.\n\nBased on the answers to these questions and the others mentioned I will increase my evaluation.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper combines Neural ODEs with Koopman operators in a hierarchical approach to model non-stationary dynamics. It provides theoretical error bounds and demonstrates long-term prediction and generalization on various synthetic and real-world examples. All referees agreed that this is a solid and novel contribution, with thorough evaluation and theoretical analysis, and all four recommended acceptance, with which I concur. 

Besides what was stated in the reviews and related discussion, the authors may want to take the following additional remarks on board when drafting their revision:

- One other point that came up and should be clarified in the revision is that Theorem 3.2 is considering the infinite basis case. Also, one referee felt that the presentation of the theory could be further improved.

- Modeling non-stationary dynamics by assuming parameters to be time-dependent is a quite common approach in time series and dynamical systems modeling. I would like to encourage the authors to more thoroughly check the related lit. (maybe perplexity or just google), especially papers that discuss non-autonomous dynamical systems, and extend the coverage of related lit. which in my mind is a bit thin.

- Likewise, LEADS and CoDA are somewhat older frameworks. There is much more recent work for hierarchical modeling with time-dependent parameters and testing generalization across domains. I would recommend checking the most recent NeurIPS and ICLR publications on this.
Decision: Accept (poster)