MOCK: an Algorithm for Learning Nonparametric Differential Equations via Multivariate Occupation Kernel Functions
Authors: Victor William Rielly
Kamel Lahouel
Ethan Lew
Nicholas Fisher
Vicky Geneva Haney
Michael Lee Wells
Bruno Michel Jedynak
Venue: Accepted by TMLR ()

Abstract: Learning a nonparametric system of ordinary differential equations from trajectories in a $d$-dimensional state space requires learning $d$ functions of $d$ variables. Explicit formulations often scale quadratically in $d$ unless additional knowledge about system properties, such as sparsity and symmetries, is available. In this work, we propose a linear approach, the multivariate occupation kernel method (MOCK), using the implicit formulation provided by vector-valued reproducing kernel Hilbert spaces. The solution for the vector field relies on multivariate occupation kernel functions associated with the trajectories and scales linearly with the dimension of the state space. We validate through experiments on a variety of simulated and real datasets ranging from 2 to 1024 dimensions, and provide an example with a divergence-free vector field. MOCK outperforms all other comparators on 3 of the 9 datasets on full trajectory prediction and 4 out of the 9 datasets on next-point prediction.

Published in Transactions on Machine Learning Research (07/2025)
MOCK: an Algorithm for Learning Nonparametric Differential Equations via Multivariate Occupation Kernel Functions
Victor Rielly Deportment of Mathematics + Stafistics Portland State Universily
victor230pdz edu
Kamel Lahouel
KlahouelStgen.org
Translational Genomics Reseorch Institute
Ethan Lew
ethanlew www.eeelw.6mmmm...oooo com
Nicholas Fisher
nicholfi@pdz.c edu
Department of Mathematics + Stafistics Portland State University
Vicky Haney
ohaney&pddz.ed edu
Deportment of Mathematics + Statistics Portland State University
Michael Wells
mbwells Opde. eduu
Department of Mathematics Stafistics Portland State University
Bruno Jedynak
bjedyna2@pddz edu
Department of Mathematics + Stafistics
Portland State University
Reviewed on OpenReview: / net/ forum? id- [ XKGp:/ooooteeeodooooooooo o5Gpw125d7
Abstract
Learning a nonparametric system of ordinary differential equations from trajectories in d-dimensional state space requires learning d functions of variables. Explicit formulations often scale quadratically in unless additional knowledge about system properties, such as sparsity and symmetries, available. In this work, we propose linear approach, the multivariate occupation kornel method (MOCK), using the implicit formulation provided by veclor-valued reproducing kernel Hilbert spaces. The solution for the vector field relies multivariate occupation kernel functions associated with the trajectories and scales linearly with the dimension of the state space. We validate through experiments on varicty of simulated and real datasets ranging from 2 to 1024 dimensions, and provide an example with a divergence-free vector field. MOCK outperforms all other comparators on 3 of the 9 datasets on full trajectory prediction and out the 9 datasets on next-point prediction.
1 Introduction
1.1
Description of the problem
The task of learning dynamical systems derived from ordinary differential equations (ODEs) has garnered a lot of interest in the past couple of decades. In this framework, we are often provided with noisy data from trajectories representative of the dynamics wish to learn, and we propose candidate model to describe

---

Published in Transactions on Machine Learning Research (07 (07/2025)
these dynamics. We present learning algorithm for the vector field guiding the dynamical system that scales linearly with the dimensionality of the dynamical system's 's state space, allowing us to efficiently learn high-dimensional systems.
1.2 High Dimensional Dynamical Systems
Large systems of ODEs may arise from the numerical discretization of a PDE. They also arise in modeling the temporal evolution of complex systems composed of many interacting components. These systems appear across many disciplines. We provide some examples.
1. The Method of Lines: Consider PDE of the form
i u(z,t) Lu(x,t)
(1)
Ōt
where L is differential operator acting on the spatial variable(s). Examples include Burgers' equation, the heat equation, Fisher's equation, and other reaction-diffusion systems. The method of lines (Schiesser 2012) is technique for solving PDEs of the form (T) by discretizing the equation in all but the time variable, reducing it to system of coupled ODEs (or, possibly, differential algebraic equations). The method proceeds as follows: upon discretizing this system by U₁ ~ u(X,, t),i = 1... ...dd we obtain the family of ODEs dU dU/dt- = G(U) where U is a vector dimension and is Rd + Rd (possibly nonlinear) vector field. If the discretization has, say, 100 points along each of spatial dimensions, this discretization leads a system of ODEs of dimension one million.
Note that the specific form G depends on the underlying dynamics and on the spatial discretization scheme. In this paper, we only assume that belongs to vrRKHS, but we do not assume specific parametric form, which may be important if G not fully specified.
2. Chemical Reaction Networks and Agent-Baser Models: In biochemical systems, each species (a protein, ion, or small molecule) may follow a distinct concentration trajectory governed by mass-action kinetics or Michaelis- Miehhenls-Meenenn dynamics. The average behavior often results in a high-dimensional ODE system representing the time evolution of all species in the network. More broadly, agent-based based models that describe individual entities (cells, molecules, agents) interacting via prescribed rules can often be approximated by digh-dimensional ODEs (Anderson & Kurtz 2015 Murray (2007), Chapter 6).
3. Networks of Neurons: Neural populations can be modeled as networks of coupled nonlinear oscillators, such as the FitzHugh- -aggmmo FHN model for individual neuron dynamics. When modcling large number of neurons with electrical or chemical coupling, one obtains high-dimensional ODE system MMuray
(2077)
Chapter 7).
Protein Accumulation in the Brain and Neurodegenerative Dynamics: Models of protein aggregation and propagation discases like Alzheimer's or Parkinson's involve spatial-temporal accumulation and diffusion processes. When discretized in space (e.g., across brain regions in Positron Emission Tomography imaging studies), these systems lead to high-dimensional ODFs for tracking the progression of misfolded proteins (tau, beta-amyloid).
Mechanical Systems with Multiple Degrees of Freedom (Lagrangian and Hamiltonian Formulations): In robotics and biomochanics, systems often have many degrees of froedom. Each joint or actuator contributes to the state space. The dynamics of such systems are derived from Newtonian, Lagrangian, or Hamiltonian mechanics, yielding structured high-dimensional ODEs (Featherstone (2014), Chapter 7, Ivancevic & Ivancevic (2010)).
State of the art 1.3
We compare MOCK various dynamical systems learning methods, including sparse identification of nonlinear dynamics, reduced order models, deep learning methods, and neural ODFs. These methods learn
2

---

Published in Transactions on Machine Learning Research (07/2025)
system dynamics from trajectories in the state space with good accuracy, scale to tens, hundreds, or thousands of state dimensions, and are robust to noise. A detailed description of the comparators is presented in Section
4.4 and in Appendix G
In Heinonen et al. 2018 and (Hegde et al. 2022 the method presented parametrizes the estimated vector field for the dynamical system with a zero-mean Gaussian Process (GP) whose covariance given by a positive-definite kernel. A grid of inducing points is formed where the vector field learned directly, and the full vector field is then interpolated from these inducing points. The use of the grid of inducing points makes this model prohibitively expensive in large dimensions as the number of grid points required increases exponentially with dimension, and thus we have excluded it from our comparison.
1.4 Main contributions
We propose to use MOCK to learn dynamical systems. This method scales linearly with the number of dimensions of the state space. Building upon (Rosenfeld et al. 2019a), (Rosenfeld ct al. 2019b), (Russo et al. 2021), we provide derivation with the help of vector-valuee Reproducing Kernel Hilbert spaces (vvRKHSs) which allows for a reduction in computational complexity in special cases and extensions to physics informed kernels. We also emphasize the simplicity of the algorithm and demonstrate competitive performance on high-dimensional data, as well as noisy data, Finally, in Section 5 we craft vvRKHS for learning divergence-free vector fields, demonstrating the versatility of the MOCK method.
Rosenfeld and Russo extend the sctting of ridge regression to the case of learning vector field us- ing snapshots of trajectories, as presented in the paper (Rosenfeld et al. 2019a) and the follow-up paper (Rosenfeld et al.) 2024). In both papers, a finite basis set of vector-valued functions was used to learn the unknown slope field. The provided solution is equivalent to optimizing in a vyRKHS with an explicit kernel. Using an I-separable kernel, a special type of matrix-valued kernel to be defined below, we allow our parameter count to increase linearly with the state space dimension (d) while maintaining computational complexity that increases only linearly in d. In contrast, the model proposed by Rosenfeld and Russo requires computational complexity to increase cubically with the number of parameters of the learned model.
In addition, we generalize this work to arbitrary matrix-valued kernels. Note that we keep the least square setting of ridge regression while in (Rosenfeld et al. 2024), Rosenfeld and Russo used a more general weak formulation with Liouville operators. We create a benchmark and compare MOCK against the state-of-the-arl algorithms. We also benchmark several kernels, including Gaussian, Laplace, Matérn, and random Fourier features. Lastly, in Scction 5 we present an application where the vvRKHS is made of divergenco-free kernels. Using simulated data, we present an experiment in which the use of divergence-free kernel improved the recovery of the vector field compared to ordinary kernel.
Without any restriction on the vvRKHS, learning a vector field with MOCK requires solving an (Nd, Nd) system, where N is the total number of snapshots and d is the dimension of the state space. Improvements can be obtained when the kernel is a tensor product of univariate kernels. In this case, which includes I-separable kernels, one needs to solve (N, linear systems. When the univariate kernel is explicit, and made of q feature functions, one can instead solve d (q.g) linear systems. By contrast, Russo and Rosenfeld considered a loss equivalent to the case of arbitrary explicit vvRKHS without the tensor product structure. This has the drawback of requiring runtime that scales cubically with the total number of parameters in the model (requires solving dg x dg linear system). This is also shared by SINDy and the explicit forms of DMD.
2 Background
2.1 wRKHH for modeling vector fields
Scalar RKHSs are spaces of real-valued functions made popular for their use constructing the kernelized support vector machine classifier (Schölkopf & Smola (202)) chapter III. Scalar RKHSs generalize to vvRKHSs and provide simple nonparametric models for vector fields. A matrix-valued kernel fully characterizes
3

---

Published in Transactions on Machine Learning Research (07 2225))
vvRKHS. The choice of this kernel is left to the user and allows for encoding various properties of the functions in the corresponding vvRKHS. Choosing kernel with Lipschitz continuous diagonal elements guarantees that all functions in the uiiquely associated vvRKHS, denoted Hi,, are Lipschitz contimuous. This, in turn, guarantees the local existence and uniqueness of the solutions to the associated ordinary differential equation á [(x) where ʃ E H,, see (Lahouel et al.) 2022). Furthermore, one may allow the inclusion of prrsics-inspired constraints into the function space by appropriately choosing the kernel, and an example of divergence-free vvRKHS provided in Section 5 Kernels come in two forms: implicit and explicit. The former implicitly characterizes mapping from R to Hilbert space, which be of infinite dimension. Examples include the family of Matérn kernels, which contain the familiar Gaussian and Laplace kernels. The latter explicitly characterizes a mapping from IR to Re for some Examples include polynomial kernels and random Fourier features. Both kernel types will be used in the experiments in Section g We now provide a mathematical presentation kernels and vvRKHSs.
2.2 Pactor-Valueds RKHS (wvRKHSs)
Definition 1. Let M(a,a) be the set of (d,d) matrices, d 2 1. A positive definite matrix-valued kernel K is mapping: X x X + M(d,a), such that
symmetric: K(x, x') = ((x',))² for any a, E X; M denotes the transpose of the matrix M.
positive definite: for any 5n X, and for any w1..... wn in Rd
Σ W K( 20
(2)
The simplest kernels are the separable kernels
K(x,x') x'A,
(3)
where & is positive definite scalar kernel and A is positive semi-definite (PSD) matrix. When A = I, H is made of d copies of the RKHS of k. We call such kernel I-separable. If the kernel k(x,,') is an explicit kernel (can be written as o(z) 6(x') for some function 6 IRd + R*) it has the additional properties:
K(x,x') = =<x) O(x') = (o(x)" oz')) (1'1)
(4)
(5)
(Ø(x) @ 1)" (Ø(x') I)
(6)
where & denotes the Kronecker product. Defining (z) Rd + Redxd by
- =(x)=(()) 1
(7)
we see that the matrix-valued explicit kernel is
K(r,z')=vo
(8)
All kernels used in our experiments in Section 4 are of the I-separable form, while the divergence-free kernel in Section 5 is not an I-separable kernel. There are three ways to characterize a vvRKHS. They are presented in Appendix D for completeness. Here, we briefly present the construction using the Riesz representation theorem, which critical in developing the MOCK algorithm.
Definition 2. Let HI be Hilbert space of functions X' Fd. We denote by (.-) the inner product in H and || or simply ||-|| the associated norm. Critically, assumee that for any X,, the evaluation functional f + fxx) is continuous, that is, there is constant M, R, such that
M, l1
4

---

Published in Transactions on Machine Learning Research (07 (07/2025)
for all f E H. Then, using the Riesz representation theorem, for any direction E RR, there exists unique function in H. denoted oz such that
a- v²((x)) = (2,0)
(10)
Define the matrix- matrix-valuct valued kernel K such that
Kiy(z (2 Serees) .1, = 1...d
(11)
where .1,..., is the natural basis of TR.. Then K is kernel, HH is the vvRKHS of K and (.)0))) k(, The reproducing properly of the kernel is, for any ʃ E Hi, X, and v E R&,
J f(x) (f,K()
(12)
2.3 Occupation kernel functions for vRRKHS
The notion of occupation kernel functions, introduced in (Rosenfeld al. 2019a), is central to this work. Let H be vvRKHS of functions Rd + Fa with kernel K. Consider a parametric curve æ: [a, b] Fd + x(t) and define the operator Lg from H to Rd, Lz(f) = f f(x(t))dt. Lg is linear. If + (xx x) is continuous for each = = 1...... d, then Lg is also continuous] For each E Rd, the occupation kernel function of the curve x in the direction v, denoted Ln is then defined as the element in H such that v (f) = (f. Ly.,) for all I E H. By the Riesz representation theorem, this element exists and is unique.
Let us now provide a useful characterization of Lip terms of the curve H, the vector v and the kernel matrix K. Note that for any Rd
w L (v) (I K(,,K(())))) (())))) (K(, y)w, Linn)
L (K((K))))
K K(x(t), wdtt TT L"
| K(y,
T
(13)
thus
(-) /// K(, (((:a(())a))
(14)
Here, we used the reproducing property 12 the symmetry of the inner product, the definition of the occupation kernel and the functional Lz, and the symmetry of the kernel K. It is convenient to notate the matrix of functions
Ma(+) Ma()=far ff" (((()x(t))d
(15)
so that the (-) = Mz(-)v. When K an I-separable kernel, this reduces to L. = (I. (())
Using the same arguments (see Appendix D.2 we find
(Li = T 1 S' ((<(s),, u(t))dsdt w
(16)
IA derivation provided Appendix D.1
5

---

Published in Transactions on Machine Learning Research (07 (07/2025)
It is also convenient to use the notation
Mx. S/"² [[" ((x(s)
(17)
that
(L Ligo) = 2² Mayw..
(18)
Note that when K is I-separable,
(io (fo M(x(s), u(t) vww.
(19)
3 Methods
Consider the ODE
a = fo(x), Hd where fo Rd Rd fixed, unknown vector field. Consider also curves
(20)
xn
b.] + RR il..... n}
(21)
3.1 The multivariate occupation kernel (MOCK) algorithm
We describe the MOCK algorithm in two steps. In the first step, we assume that we observe the curves in 21 We aim to recover the vector field fo driving the ODE in 20 Assuming that this vector field belongs to a vvRKHS, and under a penalized least square loss, provide a solution expressed in terms of occupation kernel functions. In the second step, we relax the hypothesis and assume that snapshots along these trajectories are provided in place of the trajectories. Rearranging these trajectories and replacing the integrals with numerical quadratures makes the problem tractable.
3.2 The occupation kernel algorithm from curves
Let us design an optimization setting, an inverse problem, for recovering fo from these trajectories. Let be vvRKHS of continuous vector-valued functions IRd Rdd A constant, and let us define the functional,
1 Σ [aa(()) x: (tsbs) x (a;())
J(
+ +*IE
(22)
Rd
The first term minimized when = fo- This is consequence of the fundamental theorem of calculus. However, aside from degenerate cases, this minimizer is not unique. Thus, the second term is regularization term. The problem of minimizing (22) over IH is well-posed. It has unique solution that can be expressed using the occupation kernel functions as follows:
Theorem 1. The unique minimizer of J over the woRKHS H with kernel K is
--( Σ on (:) Σ
Ma. (a,,, e Rd
(23)
1=1
1=1
where Li the occupation kernel function of the curve x; along the interval Jai, b.] in the direction ass that. is
[1- K K(sIx;(1)dd os = Mn,
Li (-)
(24)

---

Published in Transactions on Machine Learning Research (07 (07/2025)
and where the vector (a......, ....., ..," is the solution of
(M + AnT) a x(b) x(a),
s(a) (xI(a1)" ..... T (a))))))
x(b) = (x1(01)" ..... In (b(b))))
(25)
and M is the (nd, nd) matrir made of (d, d) blocks, each defined by
Miy- = Ma,.; = -
bb
b K (z,( dsdi
(26)
t=ay s=a;
To build some intuition about how the ridge penalty term ensures well-posedness of the minimization problem, suppose that many vector fields can generate trajectories perfectly fitting the data, making the first term in the cost function equal to zero. Occam's razor diclates choosing the simplest such vector field. Here, simplicity corresponds to the smallest vvRKHS norm as measured by the second term. The strict convexity of the squared vvRKHS norm ensures that no two solutions can have the same minimal norm. Indeed, any convex combination of such solutions would yield a smaller norm, contradicting the minimality assumptionn. Therefore, the minimizer is unique.
The proof of Theorem is natural generalization of the representer theorem in RKHSs. It is presented in Appendix E Figure illustrates the result of the theorem. In the case of an implicit I-separable kernel, the linear system in (25) decouples into d linear systems, each of size
where is the number of trajectories. This allows for an algorithm linear in d. If the I-separable kernel derives from an explicit scalar-valued kernel, the linear systems are X q where is the number dimensions of the scalar-valued kernel. The experiments in Section | exploit this remarkable situation. The experiments in Section use non-separable kernels. This is a tractable design when not too large.
3.3 The occupation kernel algorithm from data
Let us now assume that instoad of observing n curves that are solutions of 20 we observe m + snapshots,
sometimes noisy, 2 = (2(0) 2 (())) at time-points to,..., fm coming from true trajectory 24, I...n. = mn trajectories, each made of two samples. This is done by viewing every couple of consecutive observations as two observations associated with single trajectory. In other words, we assume that we observe (possibly with noise) the initial and final points of trajectorios æ4, i 1... N, that we denote by 2; (2) (0) {()),, the times which 200) and 2, are denoted by (10) and (("). 2, is a matrix of dimensions d by 2.
Firstly, we reshape this data into observat ions coming from =
(1) are sampled
Secondly, we replace the integral in 24 by an integral quadrature, noted $. and the double integral in 26 by a double integral quadrature, noted $ Observations 2 are used to compute the quadrature involving the trajectory Fz. In both cases, the trapezoidal rule, providing
(100) (0))
$ (K(,
(K(52) (0)) -(() )
(27)
2
and
dadt -1-- )()
(0))
HK (zI(S), dadt
(K(z))) .2 (0)
+ K(2 (0) (2))) +K(2 (1) ,
4
(00) + k(z x(=)),.")))
(28)
The resulting algorithms for learning the vector field and predicting a trajectory given an initial condition are presented Algorithm I and Algorithm respectively. Algorithm [ and Algorithm P are simplified and do not contain the optimizations implemented reduce the computational complexity (see Scction 3.4). The linear system in line 5: of Algorithm T is solved using numpy www.llnnll....m solve from NumPy v1.26.4
7

---

Published in Transactions on Machine Learning Research (07/202) /025)
Figure 1: Illustration of the MOCK algorithm with the Gaussian separable kernel. We observe the red and blue trajectories, respectively Iı and z2 (running counterclockwise). The horizontal and vertical red vector fields correspond respectively to the occupation kernel functions L and L and similarly for the horizontal and vertical blue vector fields z2- The influence of each trajectory is local. The grey vector field is the MOCK algorithm solution, linear combination of the red and blue vector fields.
Algorithm MOCK learning: Estimate the parameters defining the vector field. Require: Training data. 2;, 1...N N (N matrices of dimension (d, 2))
Compute å = (2(²) 11 ...., (1) 200) Nxd
2: for = 1 1... 1...N.J-- N.j=1 1...N.. N do
M,j + f (s,(s), (1) dsdt
end for
Solve the linear system (M AN a for Return:
Algorithm MOCK inference: Generate trajectories given initial conditions.
Require: Training data. 2, 1... N. (N matrices of dimension (d, 2))
Require: Output of Algorithm 1 (a,,...,a,)""
Require: Initial conditions. p vectors :(vP.-.. (.P.....1p))
1: for jl.... 1...p
do
2: Using a numerical integrator, generate the solution of:
i = (2))
(,) 0i
( I K 2z (t)) a
end for

---

Published in Transactions on Machine Learning Research (07 (07/2025)
3.4
Computational complexity
We specialize to the separable kernels case with A = I, see Section 2.2 In such case, the lincar system in 25 becomes equivalent to solving an N x N linear system where the right-hand side is N x d. Therefore, this requires O (dN2) to construct the kernel matrix, (N2) to estimate the integrals, and (dN3) to solve the linear system. Overall, the MOCK algorithm is thus linear in d. The space complexity is O (N²).
On the other hand, we let (x)) E pdxq, then (xx,y) = (x))**/)))()))" E Rddd an explicit kernel. In such a case, a q q linear system solved find f, requiring N, x q by q x matrix-matrix multiplications to construct the system. This provides computational complexity of O(dN q² q²). Further details are provided in Appendix F
4 Experiments
(a) FHN oscillator
(b) Lorenz63
(c) Lorenz96 128 dimensions
Figure 2: The grey points are the training data. The black curves are the test trajectories. The rod curves the predictions the test set. For data in dimensions higher than two, only the first two dimensions shown.
4.1 Datasets
We test the methods on diverse set of synthetic and real-world datasets that reflect the challenges of learning systems of ODEs. Table provides the dimension of the state space, the number of trajectories, and the average number of snapshots per trajectory for each of the datasets considered. each data set, the trajectories were partitioned into training (60%), validation (20%), and lesting (20%) sets.
Name
Type
NFHN
Synthetic
150
201
Lorenz63
Synthetic
150
201
Lorenz96-16
Synthetic
16
100
243
Lorenz96-32
Synthetic
32
100
243
Lorenz96
Synthetic
128
100
243
Lorenz96-100
Synthetic
1024
100
100
CMU
Real
50
75
106.7
Plasma
Real
425
2.95
Imaging
Real
117
231
2.26
Table 1: For each dataset, d is the number of dimensions of the system, which the dimension of the state-space, is the total number of trajectories, and m is the average number of samples per trajectory. Tn each data set, the trajectories were partitioned into training (60%), validation (20%), and testing (20%) sets.

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Noisy FitzHugh-Nagumo (NFHN) The FitzHugh-Nagumo oscillator FitzHuugh 1961 is nonlinear 2D dynamical system that models the basic behavior of excitable cells, such as neurons and cardiac celle We
add considerable noise to this otherwise simple dataset (see Figure 3).
Noisy Lorenz63 (Lorenz63) The Lorenz63 system model of atmospheric convection.
[orrenz 1963
3D system provided as a simplified
Lorenz96 The Lorenz96 data arises from Lorenz 1995 in which system of equations is proposed that may be chosen to have any dimension greater than 3. (Lorenz96-16) has 16 dimensions, etc.
CMU Walking (CMU) The Carnegic Mellon University (CMU) Walking data is a repository of publicly available data. It can be found at
Plasma This dataset includes imaging and plasma biomarkers from the WRAP study, see (Johnson et al., 2018
Imaging This dataset includes regional imaging biomarkers from the WRAP study, see (Johnson al. 2018
4.2
Data format
Each dataset is stored as rectangular matrix. Each row corresponds to a data point. The columns include the id of the trajectory, the time, the features, and three binary columns indicating whether the row is to be used for training, validation, or testing.
4.3 Kernels, regularization, and hyperparameter tuning for the MOCK algorithm
We train the MOCK models using the scalar-valued Matérn kernels, including the Gaussian, Laplace, and C kernels (Fuselier et al. 2016). The analytic expressions are provided in Appendix C.2 We also train an explicit kernel with 200 Fourier Random features (Rahimi & Recht, 2007). We use Bayesian optimization for hyperparameter tuning of the regularization parameter À and bandwidth parameter a. À controls the smoothness of the solution, while controls the relevant scale for the imput data. Specifically, we use the gP minimize function from the Seikit-Optimiz (skopt) package, version 0.10.2, which implements Gaussian Process optimization.
4.4 Comparable methods
We select competitive methods covering various categories. In addition, we provide a brief ablation study comparing our method standard regression approach for fitting the slopefield (see Appendix H..
Sparse Identification of Nonlincar Dynamics (SINDy) SINDy is a well-developed class of data-driven methods for identifying dynamical systems models from trajectories. (Brunton et al., 2016) These mot rely sparse regression techniques to isolate the most relevant terms in the governing equations from set of candidate functions. SINDy demonstrates robustness for sparse and limited data (Kaheman et al. 2020
Fasel et al. 2022
Reduced Order Models (ROMs) ROMs are class of methods for simplifying high-dimensional dynamical systems by projecting them onto a lower-dimensional space. Dynamic mode decomposition (DMD) is a data-driven method for extracting spatiotemporal patterns and coherent structures from high-dimensional data generated by dynamical systems [Tu 2013). eDMD extends DMD to handle nonlinear dynamics by
working in a nigher-dimensioall feature space. (Williams al. 2015). We benchmark with eDMD-RFF (Lew
al. 2023
cDMD-Poly (Williams al. 2015 and eDMDD-Dep (Yeung et al. 2019h
Deep Learning Methods Deep learning methods can be used in conjunction with ROMs to learn transformations to lower-dimensional spaces. These methods use deep learning to construct an efficient.
her details the all datasets provided Appendix
10

---

Published in Transactions on Machine Learning Research (07/2025)
representation of the dynamical system and to capture nonlinear dynamics Lusch al. 2018 ; Yeung
et al. 2019). (Li et al. 2019). Deep learning methods can also be incorporated into the SINDy framework to
identify sparse, interpretable, and predictive models from data Champion et al. (2019), (Bakarji et al. 2022)
We benchmark with ResNet (He et al. 2016) [Lu et al. 2021
Time Series The Latent ODE method (also a deep learning method (Rubanova et al., 2019 is an update of the Neural ODEs model introduced in (Chen et al. 2018
Latent ODEs for Irregularly-Samplle
The core idea is to represent the hidden dynamics time series data continuous latent space using ODFs. Given the latent trajectory, the observations of the time series are assumed be independent. The latent trajectory dymanics where the encoder maps observed data to a latent initial value via an RNN architecture. The hidden states are then carried through neural ODEs between times of observations. Finally, the decoder generates the latent trajectory forward in time and predicts future observations
governed by neural ODE model. The model uses encoder-decoder framework
The hyperparameters for each method are shown in Table ! in Appendix G All experiments are run in Google Colab using the default scttings. The GPU was only enabled (A100, High-RAM) for the doep learning techniques. Lastly, to get an idea of the difficully of each problem, we compare all methods against the null model, which predicts no change, that is z(t) Zn for all t [a, bj. Datasets for which models are able to do much better than this null model are difficult datasets to learn and perhaps less useful for benchmarking learning methods.
4.5 Evaluation Method
For each dataset, the trajectories were partitioned into training, validation, and testing. At test time, integrated the predicted trajectories starting the given initial conditions and compared to the true trajectories. To measure the performance of each method, we define two types of errors for each trajectory of the test set. We define:
T74
Err-
1 2:112
(29)
where y/ and D: are the observed and predicted trajectory samples at time 4, respectively. Additionally, we define 1-Err by setting m = 2 in 229 All the methods were trained, validated, and tested on the same data. Table 2 provides the average Err and 1-Err errors across the trajectories of the test set. To test whether the MOCK methods are (statistically) significantly better than comparable methods and vice versa, we first generated a Wilcoxon test p-value for every MOCK method (four different kernels) against the other comparable methods. The pairs of observations used in the test are the errors of the two compared methods for every trajectory in the test set. Finally, to generate p-value comparing all MOCK methods together against another comparable method, we used Fisher's method (Mosteller & Fisher 1948) to combine the four p-values of the MOCK methods. We report a * in Table if the group of MOCK methods is significantly better (p < 0.01) and i if the compared method outperforms MOCK with statistical significance (p < 0.01). Otherwise, no method is significantly better than the other.
4.6 Evaluation Performance
Examples of the output of the MOCK algorithm are presented visually in Figures and 4 Table summarizes the prediction errors for each experiment. While no single method outperforms other methods over all datasets, MOCK outperforms all other methods in 3 of the 9 datasets (on the task of full trajectory prediction, denoted Err), and outperforms all other methods on of the datasets on the task of predicting the next sample (denoted 1-Err). It also performs competitively on the remaining datasets on both tasks. Notably, eDMD-Deep yielded the best results for the CMU experiments. Deep learning models and DMD methods learn the dynamics in a feature space and therefore have less stringent constraints on the dynamics. We believe this is why they outperform with datascts such the CMU datasct our model assumes the
Latent ODEs was implemented using the Github repository: https:/ tenes//giituull http//ttttttbbm///o/ /tggg//uuuu.o//ulllaaiiiii Rubanowa /latent ode. The subsam- pling was disabled for the Plasma and Imaging datasets because the training trajectories were short these datasets.
11

---

Published in Transactions on Machine Learning Research (07/202) 2025)
(a) 25 randomly sampled trajectories in the NFHN (Noisy FHN) training set.
Tue
c3ND dess
c3ND paly
cOND-il
SNDypwyy
Reclar
latert-O2E
(b) 25 randomly sampled trajectories in the test set for NFHN. Black: true trajectory, red: eDMD-deep, green: eDMD-poly, purple: eDMD-rff, yellow: SINDy- poly, orange: ResNet, Prown:latent-ODE. and blue:L
Figure 3: NFHN dataset
12

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Table 2: Dynamical System Estimation Results
Lorenz63
NFHN
Lorenz96-16
Err
1-Err
Err
1-Err
Err
1-Err
Res
2.07*
.014'
ResNet
5.91
.064"
ResNet
6.42
.010"
eDMD-Deep
2.91*
.09'
eDMD-Deep
4.54"
.036*
eDMD-Deep
5.93"
.040
eDMD-Poly
2.22"
.012*
eDMD-Poly
4.10"
.037"
eDMD-Poly
6.91
.036"
:DMD-RFF
1.93"
.011"
eDMD-RFF
4.01*
.037*
eDMD-RFF
6.22
.029"
mil
2.56'
.01''
null
9.69*
.043*
null
7.50"
.039"
SINDy-Poly
.99"
.002
SINDy-Poly
1.68*
.022
SINDy-Poly
.40"
.0003"
MOCK-G
.66
.002
MOCK-G
1.40
.029
MOCK-G
.69
.0007
MOCK-L
.65
.002
MOCK-L
1.11
.030
MOCK-L
.22
.0001
MOCK-M
2.52
.012
MOCK-M
1.41
.029
MOCK-M
.22
.0001
MOCK-F
2.52
.012
MOCK-F
1.38
.028
MOCK-F
.22
.0001
Lode
1.49"
.092'
Lode
.64
.131"
Lode
3.82
.138"
Lorenz96-32
Lorenz96-
128
Lorenz96
-1024
Err
1-Err
Err
1-Err
1-Err
ResNet
11.56
.020*
ResNet
21"
2.61"
ResNet
66.1
6.48
eDMD-Deey
8.79"
.057'
eDMD-Deep
17"
.089"
eDMD-Deep
26.9"
.41"
eDMD-Poly
9.67
.051*
eDMD-Poly
19"
.109"
eDMD-Poly
61.5"
.4""
cDMD-RFF
8.10"
.028*
cDMD-RFF
16T
.201*
eDMD-RFF
54.2"
.38"
mull
10.49
.056"
null
21
.112"
null
67.0"
.16*
SINDy-Poly
8.56"
.037"
SINDy-Poly
19"
.066"
SINDy-Poly
NA
MOCK-G
.0001
MOCK-G
17
.064
MOCK-G
18.0
.013
MOCK-L
.48
.0001
MOCK-L
16
.035
MOCK-L
17.8
.014
MOCK-M
.49
.0001
MOCK-M
16
.040
MOCK-M
17.9
.013
MOCK-F
.47
.0001
MOCK-F
16
.036
MOCK-F
18.0
.015
Lode
6.06"
.217
Lode
15'
.342"
Lode
15.8
1.04
Plasma
Imaging
CMU
Err
1-Err
Err
1-Err
Err
1-Err
ResNet
4.93"
2.89'
ResNet
27.4"
19.4
ResNet
22.6*
.86'
eDMD-Deep
4.09
2.42
eDep
15.6*
12.5
eDMD-Deep
15.7t
2.03*
eDMD-Poly
4.01
2.39
eDMD-Poly
15.7
12.5
eDMD-Poly
14.9
.78f
eDMD-RFF
4.07
2.43
eDMD-RFF
27.9"
19.3
eDMD-RFF
14.9
.76!
mull
4.00"
2.41
null
16.2*
12.6
null
21.6"
1.01*
SINDy-Poly
3.95
2.40
SINDy-Pole
96.1*
53.7'
SINDy-Poly
33.8*
.91*
MOCK-G
3.96
2.41
MOCK-G
15.6
12.3
MOCK-G
21
.89
MOCK-L
3.96
2.41
MOCK-L
15.6
12.3
MOCK-L
19.6
.93
MOCK-M
3.96
2.41
MOCK-M
15.7
12.3
MOCK-M
21.5
1.01
MOCK-F
3.96
2.41
MOCK-F
15.8
12.4
MOCK-F
21.6
1.01
Lode
4.28*
2.68*
Lode
14.8
11.7
Lode
14.9
1.88*
Description: Minimum values are in bold. A indicates a significant (Fisher's method) p-value (< 0.01) (01) in favor of the MOCK methods the comparison. indicates significant p-value favor of the method compared with all MOCK methods. There is no statistically significant difference otherwise. Our models are labelled as MOCK-G occupation kernel method with Gaussian kernel, MOCK-M occupation kernel method with Matérn kernel, MOCK-F occupation kernel method with Random Fourier Features (RFF), and MOCK-L- occupation kernel with Laplace kernel (See section 1.2). We compare against SINDy-Poly, 1DMD-Deep eDMD-Poly, eDMD-RFF, ResNet and Latent Ode (Lode) (See section 4.3). No result could be obtained for SINDy-Poly for this dataset due to computational complexity issues.
13

---

Published in Transactions on Machine Learning Research (07 (07/202) 2025)
2.7
R
R
2
0.68
(a) (0 years) Measurements
(b) (0 years) MOCK Initial State
2.7
2.7
R
2
1.4
1.4
0.68
0.68
0
(c) (+7.7 years) Measurements
(d) (+7.7 years) MOCK Prediction
Figure 4: MOCK Predicted Biomarker Values from the imaging dataset.
dynamics arises from a homogencous system, and this not the case with the CMU datasct. See Appendix B.2
The differences observed in Table 2 are not always significant. Recall that use the sign when method is better than all the MOCK methods. We count 5 models out of 9 with at least one t for Err and only 2 out of 9 for the 1-Err. Note that the MOCK algorithm is optimized for the 1-Err since the trajectories of the training set are systematically reshaped into trajectories of two observations. Thus it is not surprising that it performs better for this metric.
When comparing the kernels used for the MOCK method, the Laplace kernel performs the best overall, consistent with other analyses of this kernel (Geifman et al. 2020). Note also that the random Fourier features kernel, which approximates the Gaussian kernel, performs well for all datasets, allowing for linear implementations in N, which is the number of examples in the training set. Moreover, we use the same number of features (200) in all cases, leading to models with parameter sizes of 200 d for Fourier random feature implementations. Importantly, because our model size scales linearly with d we do not need to increase the number Fourier random features for higher dimensional problems.
The trajectories predicted by MOCK-L for NFHN (in blue in Figure 3b) are visually close to the ground truth (in black) in all but the last case. SINDy-poly (in yellow) is competitive. ResNet provides very irregular trajectories. Latent ODE provides good long-term predictions. However, the initial points do not coincide with the initial points of the ground truth provided for all the algorithms. In other words, while Latent- ODE performs very well on the task of prodicting full trajectorics, it performs very poorly on the next sample prediction task which is an inherent limitation of some deep learning models like Latent-ODE and Resnet
We tested MOCK with a dataset of 1024 dimensions to show how well it scales. The implementation of SINDy-Poly that we used did not provide a result for this data due to runtime issues; see NA in the Table 2
4.7 Comparative computational complexity
The computational complexities are compared in Table 3 The MOCK algorithm with implicit and explicit kernel is linear in d, the dimension of the state space. Resnet and Lode are quadratic in d. In principle, SINDy-Poly, cDMD-Poly, cDMD-RFF, and cDMD-Deep are lincar in d. However, q, the number of parameters in the learned model, needs to increase with to obtain acceptable performances. This means the complexity is, in practice, more than linear in d. In the case of SINDy-Poly with quadratic basis functions, q is ((d).
14

---

Published in Transactions on Machine Learning Research (07/2025)
MOCK implicit
MOCK explicit.
ResNet
eDMD-Deep O(TN + q²) + ddq qD)
(dNNa)
O(dNq² + dq²)
0 (kdPTN)
eDMD-Poly
eDMD-RFF
Lode
SINDy-Poly
O(NN + d)g² + N2 +2) O((N + d)g² + +²) O (nT (Ryde df + k2n²))
O q2 + d.Ng)))
Table 3: Runtimes of training for each method, excluding syper-parametest validation. d is the dimension of the state-space. N is the total number of samples in all the trajectories. is the number of features. T is the number of epochs. h and 22 are constant.
5 Learning vector fields with constraints
Divergence-free (V 0) and curl-free (V x v = 0) vector fields appear in a diverse set of applications
including fluid dynamics [Wendland 2009): Fuselier et al. 2016), magnetohyyrrddyamiis (McNally) 2011 and modeling magnetic fields (Wahlström et al. 2013), image processing (Polthier Preuß 2003). and surface reconstruction (Drake et al.) 2022). Accordingly, significant amount of research has been devoted to the dovelopment of curl-frec and divergenco-free kernel methods (Narcowich & Ward 1994. (Lowitzsch
2005 Narcowich et al.) 2007), (Fuselier 2008). (Fuselier & Wright, 2009). (Gao et al. 2022), (Scheuerer & Schlather 2012), as well as to the development of Gaussian processes constrained by PDFs (Harkonen et al. 2023), [Henderson et al. (2023) in recent years. We will demonstrate how the occupation kernel method can be adapted to ensure that the recovered vector field analytically satisfies the divergence- ftre/curl-free raistraint for an appropriate choice matrix kernel K. Then, for ease of presentation, we will apply just the divergence-free kernels to both real and synthetic datasets.
5.1 Divergence-fres free kernels
Let IRd and 2.3, and let <[[x=]) be a radial basis function. Then the standard construction 2008 for divergence-free and curl-free value radial basis functions are
(Fuselier
VV (l(==))) and VV ((1x=1))
(30)
respectively, where A -E 8 0xg and [V 0 10x,00, 1 i,j I d. For example, if d == 2, then
8'S Ox 82' T22U2T
8² 0
{-AI+ V
8z 021 dzz
(|=|l).
(31)
828 do7
Thus, the columns of the matrix-valued radial basis function Similarly, we have that the columns of ([|x]) are curl-free.
{-A1 + VV Ho[[<=]] are divergence-free.
We select the divergence-free kernel K(x,y) v) (-AI+ "0")0 y|)) where our choice of scalar radial basis function the 1 Matérn kernel (uuselier et al. 2016).
5.2 Hamiltonian systems
A system
= ((01) do g(x1,22))
(32)
is called a Hamiltonian system if there exists a function (21, (called the Hamiltonian) for which f 7= aH/Ong and g = - /r.. This implics that the vector field (f.g) divergence-free. Furthermore, along every orbit we have (11, x22) = constant and any conservative dynamical equation = (() leads to Hamiltonian system where the Hamiltonian coincides with the total energy. For example, every arbit of the conservative system corresponding to the equation describing the motion of a pendulum
9 sin(x1)) E
$1 = =22,
$2 =
(33)
15

---

Published in Transactions on Machine Learning Research (07 (225))
satisfies the conservation law for some constant
2 9 H()) = cos(21) = E
(34)
5.3 Experiments
We test the divergence-free MOCK method on the following real and synthetic datasets.
Pendulum Problem We generate data using 33 with e and g = 9.8. This 2-dimensional synthetic dataset consists of 100 trajectories of 50 samples each. This dataset allows to test whether the MOCK method can, simultaneously, learn known Hamiltonian system while analytically enforcing the divergence free constraint.
Buoy Data We obtained a buoy dataset from htlps://oceanoofhing..ee. wwappiiggg.aapamamiii. /at//daaaaaaabaa which contains two-dimensional trajectories of buoys submerged the ocean. We sampled every tenth observation of the raw data and converted the time measurements into years. Trajectories with only one sample were dropped.
5.4 Evaluation
The error was computed using 22 for both datasets. For the pendulum problem, the scalar Matérn kernel provided an error of 1.5 10 while the divergenco-free kernel provided an error of 1.9 10-2, 2 an improvement by order of magnitude. Not only does the divergence-free kernel allow for physically constrained solution, but it is also more accurate. For the buoy data, the scalar Matérn kernel provided error of 3.5 x 10 while the divergence-firee kernel provided an error of 3.1 x 10a Thus, despite the presence of noise (which is not necessarily divergence- free,, we still achieve a reduction in the error of about 10%.
Figure 5 shows the error in the computed vector fields of the pendulum problem. A plot of the results for the buoy data problem using the divergence-free kernel, along with the training data, is shown in Figure
10
10
C
10
10"1
-3
10
10"1
-1
-10
-15
-30
-15
-10
10
15
(a) Scalar Matérn kernel
(b) Divergence-free kernel
Figure 5: The magnitude of the error in vector field approximation of the pendulum problem obtained using different choices of kernel along with the training set (gray).
6 Summary and discussion
The implicit matrix-valued kernel formulation has enabled to demonstrate that the MOCK algorithm, originally proposed by J. Rosenfeld and collaborators, effectively learns the vector field of an ODE in multivariate settings or under predefined constraints, while maintaining linear scalability with the dimension of the state space. We have benchmarked the algorithm against a representative selection of competitive methods and provided compelling experimental evidence showcasing its superior performance on many datasets, along with consistently competitive results across all cases tested.
A rigorous analysis of the algorithm's convergence and error, including precise assumptions regarding noise, is deferred to future work. This analysis will incorporate the quadrature error and the effects of noise,
16

---

Published in Transactions on Machine Learning Research (07 (07/202) 2025)
-30
-35
-40
-45
-50
-55
-60
25
30
35
40
45
50
Figure 6: A detail of the vector field learned using the divergence- free kernel from the buoy data together with the training set (red).
well as provide concentration inequalities to quantify the error between predicted and true trajectories as functions of quadrature precision, observation noise levels, and data granularity, offering insights into the algorithm's generalization capabilities.
The surprising simplicity of the MOCK technique, combined with its strong performance, opens the door to numerous opportunities for optimization, generalization, and further exploration. Future work is expected to advance the state-of-the-art in high-dimensional dynamics learning, extend applications to PDEs, and analyze generalization properties in detail.
References
David Anderson and Thomas G Kurtz. Stochastic analysis of biochemical systems, volume 674. Springer, 2015.
Joseph Bakarji, Kathleen Champion, Nathan Kutz, and Steven L Brunton. Discovering governing equations from partial measurements with deep delay autoencoders. arXie preprint arX::22001.05336 2022.
Steven Brunton, Joshua Proctor, and J Nathan Kutz. Discovering governing equations from data by sparse identification of nonlinear dynamical systems. Proceedings of the national academy of sciences, 113 (15):3932-3033) 2016.
Kathleen Champion, Bethany Lusch, J Nathan Kutz, and Steven L Brunton. Data-driven discovery of coordinates and governing equations. Proceedings of the National Academy of Sciences, (16(45):22445-2224 2019.
Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. Advances in neural information processing systems, 31, 2018.
Anthony M DeGennaro and Nathan M Urban Scalable extended dynamic mode decomposition using random kernel approximation. STAM Journal Scientific Computing, 41 (1):AA 14)221482-00 A1499, 2019.
17

---

Published in Transactions on Machine Learning Research (07/2025)
Kathryn P. Drake, Edward J. Fuselier, and Grady B. Wright. Implicit surface reconstruction with curl-free radial basis function partition of unity method. SIAM Journal on Scienlific Compuling, 2022. doi: 10.1137/22M1474488.
Urban Fasel, J Nathan Kutz, Bingni W Brunton, and Steven L Brunton. Ensemblo-sindy: Robust sparse model discovery in the low-data, high-noise limit, with active learning and control. Proceedings of the Royal Society A, 478(2260))202109044 2022.
Gregory F. Fasshauer and Michael J. McCourt. Kernel-based Approsimation Methods using MATLAB. World Scientific, New Jersey, 2015.
Roy Featherstone. Rigid body dynamics algorithms. Springer, 2014.
Richard FitzHugh. Impulses and physiological states in theoretical models of nerve membrane. Biophysical journal, 166))445-445 1961.
E. Fuselier, V. Shankar, and G. Wright. A high-order radial basis function (rbf) leray projection method for the solution of the incompressible unsteady stokes equations. Computers and Pluids, 128:41 52, 2016.
E. J. Fuselier. Sobolcv-type approximation rates for divergence-freee and curl-free rbf interpolants. Math. Comput., 77:1407-1423, 2008.
E. J. Fuselier and G. B. Wright. Stability and error estimates for vector field interpolation and decomposition on the sphere with rbfs. SIAM J. Num. Anal. 47:3213-3239, 2009.
W. W. Gao, G. E. Fasshaner, and N. Fisher. Divergence-free quasi-interpolatio.. Appl. Comp. Harmon. Anal., 60:471 488, 2022.
Amnon Geifman, Abhay Yadav, Yoni Kasten, Meirav Galun, David Jacobs, and Basri Ronen. On the similarity between the laplace and neural tangent kernels. Advances in Neural Information Processing Systems, 33:1451 1461, 2020.
Mare Harkonen, Markus Linear Partial Differential Equations with Constant Coefficients. In Proceedings of the 40th International Conference on Machine Learning, pp. 126111 PMLR, July 2023.
and Bogdan Raita. Gaussian Process Priors for Systems of
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. Tn Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770 778, 2016.
Pashupati Hegde, Çagatay Yildiz, Harri Lähdesmäki, Samuel Kaski, and Markus Heinonen. Variational multiple shooting for bayesian odes with gaussian processes. In Uncerlainly in Arlificial Intelligence, pp- 790-799. PMLR, 2022.
Markus Heinonen, Cagatay Yildiz, Henrik Mannerström, Jukka Intosalmi, and Harri Lähdesmäki. Learning unknown ode models with gaussian processos. In International Conference on Machine Learning, pp. 1959 1968. PMLR, 2018.
Jain Henderson, Pascal Noble, and Olivier Roustant. Characterization of the second order random fields subject to linear distributional PDE constraints. Bernoulli, (0(4):339633422 November 2023. doi: 10.3150/23 228-18
Vladimir G Ivancevic and Tijana T Ivancevic. Dynamics and control of humanoid robots: A geometrical
approach. Paladyn, Journal of Behavioral Robolics, (44):004 218, 2010.
Sterling Johnson, Rebecca L Koscik, Erin M Jonaitis, Lindsay R Clark, Kimberly Mueller, Sara E Berman, Barbara B Bendlin, Corinne Engelman, Ozioma C Okonkwo, Kirk J Hogan, et al. The wisconsin registry for alzheimer's prevention: review of findings and current directions. Alzheimer's & Dementia: Diagnosis, Assessment & Disease Monitoring, 10:130-142. 142, 2018.
18

---

Published in Transactions on Machine Learning Research (07/2025)
Kadierdan Kaheman, J Nathan Kutz, and Steven L Brunton. Sindy-pi: robust algorithm for parallel implicit sparse identification of nonlinear dynamics. Proceedings of the Royal Society A, 476((2242)200202777 2020.
Kamel Lahouel, Michael Wells, David Lovitz, Victor Rielly, Ethan Lew, and Bruno Jedynak. Learning nonparametric ordinary differential equations: Application to sparse and noisy data. arXiv preprint WrXiu2220.112111 2022.
Ethan Lew, Abdelrahman Hekal, Kostiantyn Potomkin, Niklas Kochdumper, Brandon Hencey, Stanley Bak, and Sergiy Bogomolov. Autokoopman: A toolbox for automated system identification via koopman operator linearization. In International Symposium on Aulomated Technology for Verification and Analysis (ATVA), 2023. Under review.
Yunzhu Li, Hao He, Jiajun Wu, Dina Katabi, and Antonio Torralba. Learning compositional koopman operators for model-ban based control. arXiv preprint NKK::1910.0826: 2019.
Edward N Lorenz. Deterministic nonperiodic flow. Journal of atmospheric sciences, 20(2):130- 1400, 1963.
E.N. Lorenz. Predictability: a problem partly solved. In Seminar on Predictabilityy volume 1, pp. 1-18, Shinfield Park, Reading, 1995. ECMWF.
S. Lowitzsch. Error estimates for matrix-valued radial basis function interpolation. Journal of Ayprusimation Theory, 137:238-24 229, 2005.
Lu Lu, Xuhui Meng, Zhiping Mao, and George Em Karniadakis. Deepxde: A deep learning library for solving differential equations. SIAM review, 63(1):208 228, 2021.
Bethany Lusch, J Nathan Kutz, and Steven L Brunton. Deep learning for universal linear embeddings of nonlinear dynamics. Nature communications, ((1):4950, 2018.
Colin P McNally, Divergence-fres free interpolation of vector fields from point values exact V- B == in numerical simulations. Monthly Notices of the Royal Astronomical Society: Letters, 413(1):L76-L800 2011.
Frederick Mosteller and R. A. Fisher. Questions and answers. The American Statisticion, 2(5):30-33 1948. ISSN 00031305. URL http:// /www///wwwww//o./////. jstor org//stable/2611650
James D Murray, Mathematical biology: An introduction, volume 17. Springer Science & Business Media, 2007.
F. Narcowich and J. Ward. Generalized hermite interpolation via matrix-valued conditionally positive definite functions. Mathematics of Computation, 63:661-688, 1994.
F. Narcowich, J. Ward, and G. B. Wright. Divergence-free rbfs on surfaces. J. Fourier Anal. Appl., 13: 643 663, 2007.
Konrad Polthier and Eike Preuß. Identifying vector field singularities using discrete hodge decomposition. In ChrssChristian Hege and Konrad Polthier (eds.), Visualization and Mathemalics III, pp. 113 134, Berlin, Heidelberg, 2003. Springer Berlin Heidelberg.
Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. In Neural Information Processing Systems, 2007. URL
Joel A Rosenfeld, Rushikesh Kamalapurkar, Benjamin Russo, and Taylor T Johnson. Occupation kernels and densely defined liouville operators for system identification. In 2019 IEEE 58th Conference on Decision and Control (CDC), pp. 6455 6460. IEEE, 2019a.
Joel A Rosenfeld, Benjamin Russo, Rushikesh Kamalapurkar, and Taylor T Johnson. The occupation kernel method for nonlinear system identification. arXiv preprint arXiv:1909.11 11792, 2019b.
Joel A. Rosenfeld, Benjamin P. Russo, Rushikesh Kamalapurkar, and Taylor T. Johnson. The occupation kernel method for nonlinear system identification. SIAM Journal on Control and Optimization, 62(3): 1643-1663 2024. doi: 10.1137/19M12700 URL http:://boi.org/11..13333
19

---

Published in Transactions on Machine Learning Research (07/2025)
Yulia Rubanova, Ricky TQ Chen, and David K Duvenaud. Latent ordinary differential equations for irregularly-sampled time series. Advances in neural information processing systems, 32, 2019.
Benjamin P Russo, Rushikesh Kamalapurkar, Dongsik Chang, and Joel A Rosenfeld. Motion tomography via occupation kernels. arXiv preprint arXiv:2101.080 2021.
Michael Scheuerer and Martin Schlather. Covariance models for divergence-free and curl-free random vector fields. Stochaslic Models, 28(3):433 451, 2012. doi:
URL https
//doi.org/10 1080/15326349 2012. 699756
William E Schiesser. The numerical method of lines: integration of partial differential equations. Elsevier,
2012.
Bernhard Schölkopf and Alexander Smola. Learning with kernels: support vector machines, regularization,
optimization, and beyond. Adaptive computation and machine learning. MIT Press, 2002. URL http:
www..worldcat. wwt.///aaatttogoleeeeeeeeeoo orggooccc/487////
Jonathan H Tu. Dynamic mode decomposilion: Theory and applicalions. PhD thesis, Princeton Universily, 2013.
Niklas Wahlström, Manon Kok, Thomas B. Schön, and Fredrik Gustafsson, Modeling magnetic fields using gaussian processes. In 2013 TEFE International Conference on Acoustics, Speech and Signal Processing, pp. 3522 3526, 2013. doi: 10.1 1111109/SASSPP0. 2013.663831..
H. Wendland. Divergence-free free kernel methods for approximating the stokes problem. SIAM J. Numer. Anal., 113158-3179) 2009.
Matthew O Williams, Toannis G Kevrekidis, and Clarence W Rowley. A data driven approximation of the koopman operator: Extending dynamic mode decomposition. Journal of Nonlinear Science, 35:1307-1346 2015.
Enoch Yeung, Soumya Kundu, and Nathan Hodas. Learning deep neural network representations for koopman operators of nonlinear dynamical systems. In 2019 Americon Control Conference (ACC), pp. 4832 4839. IEEE, 2019.
A Experiment Code
We created code capsule on Code Ocean that hosts all the methods used in our experiments for system identification of ordinary differential equations. The code repository allows for the repeatability of our results for a single experiment Lorinnenn--orenz633 and facilitates the evaluation of the performance of different models on synthetic and real-world datasels. The real world plasma and imaging experiment are restricted data and will not be available for the evaluation.
We provide an anonymized zipfile (6.6 MB) export of the code capsule for review at httpss//drriee. .tt.g/gglivee The code can be run locally in a Docker container with instructions in the README. and REPRODUCING.md.
To ensure the reproducibility of the experiment, we have tuned the hyperparameters for each method, except for ResNet, which is known require substantial tuning time. The code repository provides detailed instructions for running each method experiment, including the necessary input data and the corrosponding hyperparameter settings. Please note that due to limitations on Code Ocean, have set. the TensorFlow library to use the CPU rather than the GPU, as using the GPU resulted in mumerous errors. This may increase the runtime of the experiments. For the paper, we ran some experiments in google colab on A100 GPUs where the training time was much lower.
Upon running the code repository, the experiments will take approximately 1.5 hours to complete. The results of each experiment are provided in the form of trajectory CSV files, capturing the predicted trajectories of the identified systems. These trajectory files can be further analyzed or visualized as desired. Additionally, a summary of the performance of each method is available in the code repository.
20

---

Published in Transactions on Machine Learning Research (07 (025)
B
Datasets
B.1
Synthetic data
Noisy FitzHugh-Nagumo
The FitzHuug-Naguuno oscillator FitzHugh 1961 is nonlinear 2D dynamical
system that models the basic behavior of excitable cells, such as neurons and cardiac cells (See Figure 3) The system popular in the analysis of dynamical systems for its rich behavior, including relaxation oscillations and bifurcations. The FHN oscillator consists of two coupled ODEs that describe the membranc potential v and recovery variable w,
2 w + RI 3
(35)
T bw
(36)
Here, / is the external current input, a and positive constants that affect the shape and duration of the action potential, and is the time constant that determines the speed of the recovery variable's response. We added random Gaussian noise of standard deviation 0.12 to this dataset.
Noisy Lorenz63 The 3D Lorenz 63 system (orenz 1963 is a simplified model of atmospheric convection, but has since become a canonical example of chaos in dynamical systems. The Lorenz 63 system exhibits sensitive dependence on initial conditions and parameters, which gives rise to its characteristic butterfy-shaped chaotic attractor. The equations are
dr = oly- - x)
dt
dy = x(p - 2)
(37)
df
dz
dt ay Bz
Here, z, y, and state variables representing the fluid's convective intensity, and Ø, P, and B are positive parameters representing the Prandtl number, the Rayleigh number, and a geometric factor, respectively. We added random Gaussian noise with standard deviation 0.5 to this dataset.
Lorenz96 The Lorenz96 data arises from (oorenz 1995) in which a system of equations is proposed that may chosen to have any dimension greater than 3. The chaotic system is defined by:
dxb = I2
2641 24 F
(38)
dt
where we take F= 8 and consider 16, 32, 128 and 1024 dimensional systems. Indices are assumed to wrap so that for an n dimensional system Φ-2 = Tn-2-
B.2 Real Data
CMU Walking The Carnegie Mellon University (CMU) Walking data is repository of publicly available data. It can be found at http://mocap.cs.
can.cap.cs.cm...du... cmw..du// It was generated by placing sensors on a number of subjects and recording the position of the sensors as the subjects walked forward by a fixed amount and then walkod back. There were 50 spatial dimensions of data as recorded by the sensors, which were recorded regular times. Since the observation times were not provided, we separated each observation in time by 0.1 units and began each trajectory at time zero. The CMU Walking data which used for experiments consists of 75 trajectories with average of 106.7 observations per trajectory. We could not use the full amount of data provided because some subjects did not walk in the same manner as other subjects.
It should be noted that it may not be appropriate to consider the CMU walking data as being generated by a single dynamical system. Since there were multiple subjects in the datasot, it reasonable conclude that there are multiple dynamical systems responsible for the motion of these different subjects. However, we fit model to this dataset assuming that it generated by single dynamical system.
21

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Plasma This dataset includes imaging and plasma biomarkers from the WRAP study, (Johnson et al. 2018). The imaging biomarker is a distribution volume ratio obtained from Pittsburgh Compound-B positron emission tomography. The plasma biomarkers include A that reflects specific amyloid beta (A3) proteoforms; ptau217, which has a high correlation with amyloid PET positivity, GFAP, which measures the levels of the astrocytic intermediate filament glial fibrillary acidic prolein, and NFI, a recognized biomarker of subcortical large-caliber axonal degeneration, for total of 6 biomarkers. There are a total = 425 trajectories, one per subject, and, on average, 2.95 time points per trajectory, The data was split (70%, 10%, 20%) corresponding resp. to training, validation, and testing.
Imaging This dataset includes regional imaging biomarkers from the WRAP study, see (Johnson et al, 2018). The imaging biomarker is atlas-based distribution volume ratio obtained from Pittsburgh Compound-B positron emission tomography. There are total of 117 biomarkers. There are total of n 231 trajectories, one per subject, and, on average, 2.26 time points per trajectory. The data was split (70%, 10%, 20%) corresponding resp. to training, validation, and testing.
C Reproducing Kernel Hilbert Spaces (RKHS)
Basic notions and notations associated with RKHSs are important for understanding the algorithms presented in this paper. We thus provide a short presentation. We limit ourselves to RKHSs over the field of real numbers. RKHSs are Hilbert spaces for which the evaluation functional is continuous. The evaluation functional at R is mapping from RKHS H to R, which associates to function its evaluation at x, that is f + (x). Riesz representation theorem allows to interpret evaluations of function in an RKHS as geometric operation consisting of computing an inner product. That is, there is unique vector kg E H. such that fxx = (f, kz). Morcover, let us define, for any æ, E R, the so-called kernel k(x, y) = (ka ky)- Let us use this to characterize the function kg- Evaluating kg at y and using Riesz representation provides ki (y) (ka, ky) = (ky: ki)) = k(y, x). Thus the function (-)(-) the function k(,,z). Moreover, for any f E H, f(x) (f) This is the reproducing property of the kernel. Applying this property ky implies that k(z, = (k(; y))
C.1 Linear functionals and occupation kernels
Let be an RKHS of functions from I to with kernel k. Assume that x + k(x,,)) 3) is continuous. Consider a continuous parametric curve [0, - R. t + x(t) and define the functional from an RKHS H to R. La Lz( = So (((())) Lz is clearly linear. The Cauchy-Schwarz inequality implies Ly is bounded. Indeed,
S/ (o(O))
(39)
1." (f, (f,k(, dt
(40)
s L. (ff,((, |dt
(41)
< I [1SIII((,
(42)
llfll l- (k(a(t)
(43)
x) are where we have used the Sanchy-Schwartz incquality in 42 Now, since t xxt) and x + k(x,
continuous, the integral in (3)) is upper-bounded. Thus the functional (Tx( is continuous.
22

---

Published in Transactions on Machine Learning Research (07 (07/2025)
We use Riesz representation theorem to define the occupation kernel function as the unique element Lg in H that verifies (f) = (J, [ig) Note that
(2(1) (L2, k(.v)) (k(., L2) Lo(k(;,))) L/ k(x(t), v)
(44)
Furthermore,
Sf [( 1200)) [ k(x(s),
(Lg b.y) (T((ta) =
(45)
C.2 Kernels
standard scalar-valued positive definite kernels, references to which may be found, see, for example, Table 3.1 page 42 of (Fasshauer & McCourt 2015). Additionally, in section 5, we demonstrate how a divergence-free kernel allows us to incorporate physical constraints in our model. The specific divergence-free kernel we use is given in (Fuselier et al. 2016). For the sake of clarity, we include the analytical forms the kernels used throughout the paper below:
For all experiments (except for Section
Guassian kernel:
(- || y v|? 20
g (0), exp
(46)
Laplace kernel:
(s7) = exp (- || bl1 v11)
(-16
(47)
Fourier Feature kernel:
1 [cos(<Tax/o +3),, co.("////0////0000000 + Ba)]"
[cos(=] u/o + B1). .... cos(2, u/a + By)] (48)
Vq
Matérn kernel C
1
exp( exp(-r/oo ((2)² 15 (5())
r
420 (-)
(2) 945
m(z, v. o)
+ 105
+ 945
(49)
945
where in 48
NO0, I) and Bg ~ U(0 2T), and in 49 r ||
y||-
We now introduce the divergenco-free
kernel of Section 5. Let
(1, = 9450² expp (-rp- ((2) + 10
1
+ 45
+
105
+ 105
0
and
(()
2
d2(x,y,0)
945 exp expp-rr////
+6
15 -
15
and define
(1, -d2(1, U, (x2 y2)² - di v,
(22(1,V.0) -d2(2,,) v1)² 2- di
and
012(2, = Φ21 (z, 0) = dz( v, (I) y1) 42),
where x = [2),"
121. and r = || -ll- Then,
Jnn1 $12
Maii (z., v, =
(z, (2,y.01 o).
(50)
Φ21
222
23

---

Published in Transactions on Machine Learning Research (07 (07/2025)
D
Vector-Valued Reproducing Kernel Hilbert Spaces
There are three ways to characterize a vvRKHS.
Firstly, we can construct vyRKHS with linear functions the kernel: Definition 3. Let
Ho =
(( - E X, wa E IR,, 1
(51)
then, consider the encoding in Hn of the functions and 9,
Ti...., In and ,,..... Yme
(52)
W1 Wn
1 mm
Define the inner product
(f.9) =
ii K (II:
(53)
then the closure of Hg for (..-) is the vvRKHS
The second construction starts from Hilbert space and uses the Riesz representation theorem.
Definitionn 4. Let IHI be Hilbert space of functions Rd + R2 such that for any E Rd the evaluation functional f + fz) is continuous: there is a constant M, E R, such that
llf())) & M=Ilf
(54)
for all f E H... Then, using the Riesz representation, for any RRd there exists a unique Kz H such that v² f²² = (f, K
(55)
This equation is the reproducing property. Define the matrix-valued kernel
Ky)(x, = (Krees K2 ,,5 = 1...d
(56)
where ,I,...,ed ed is the natural basis of IRd, Then K a kernel and H is the vvRKHS of K.
The third formulation allows for checking that Hilbert space is vrRKHS. Definition 5. Let be a Hilbert space of functions Fd + Rd. Let K be (d, d) matrix-valued kernel. H is the vvRKHS of K when
1. For any x' E X, E ERd 2 K(z,a')0 E TH
2. The reproducing property holds: for any E H. E X, E Rd. 55
holds
D.1 Verifying the continuity for the occupation kernel in multiple dimensions
Let(ei,...,.e) ex) be the standard basis of Rd. Then
La = 5. of M(()))a
(57)
J." 10 (x((()()))
(58)
= I"" KRS, (,,
(59)
s |151| 1 ())(((())
(60)
24

---

Published in Transactions on Machine Learning Research (07 (07/2025)
where we have used the Cauchy-Schwartz inequality. If x: + (z) x:) is contimuous, then since t + z(t) is continuous, of Ty is bounded for each J s d, which implies that ba is bounded and thus continuous.
D.2 Occupation Kernel inner product
Consider
12,00)= = [-" k((
()) [/" (())(())dto
we wish to evaluate
(1 (yn,w = w Ly (L)
(61)
= w f { ((u(s), ds
(62)
= w" (%/) (-" ((y(s), x(t))
633)
E
Proof of Theorem 1
Consider the linear span of the occupation kernel functions Ly = 1...n .1
=={EEH H /
F
Σ Ii a (on...., an) Rdxx
(64)
Note that F is linear and inite-dimensional. thus, it is closed linear subspace of H. We can then project any function in H orthogonally onto it
f f=
(65)
Now, for each i I...n, n, and e e Rd,
o" / (z:(1))tt=d) L.) = ()) = (Le JF f5=) = (L Sr)
(66)
where the previous to last equality comes from the fact that for is perpendicular to L Thus,
| d(x:(t)ddt= (fxr
(67)
Next, using the Pythagorcan equality
=11==11611 -|15==11 ||
(68)
Thus, for any f E H, J(fz) s J(f) which proves that the minimum of J actually belongs to F. Moreover, note that
||
Σ
L Loannr ( Issnss)
i==1
n
[m
pbi by
((zs(s) aj = Σ
I Miaa = aTMa (69)
a
i-==1
1-=1
25

---

Published in Transactions on Machine Learning Research (07 (07/202) 2025)
Also,
(fr) = Lxs
Σ Li Tingas
(70)
Σ
T 1o (2r
(71)
as
j=1
K (1zi()) oj
(72)
Jag
Σ Mijaj
(73)
[Ma],
(74)
The minimization problem in f is thus equivalent to minimizing a
1 Σ J(a)
[Maa], r2;(b) + (a, da TaTMa Ma
(75)
1=1
equivalently,
((a)
|M x(b) + =(a) + >a Ma
(76)
n
since À > 0 and M is PSD, this is solved by
(M+ a= x<(a)
(77)
F
Computational Complexity
We detail below an analysis of the complexity for MOCK in terms of d, the dimension of the state space, and N, the total number of observations. We present the implicit and explicit kernels successively.
F.1 Implicit kernel
In several of our experiments, an implicit Gaussian kernel was used. The Gram matrix for the kernel is N x m where N is the total number of samples in the datasct X and m is the total mumber of samples in the dataset Y. If X e BdddN and Y E Rdxm, we may compute any radial basis function kernel defined by:
Gi,j = f(1a= 3)
(78)
by observing:
D outer(sum(X @ 1m) - y + outer(1x, sum Y))
(79)
Where Di, = -X, - 2 is the matrix of square distances, @ is the Hadamard product, sum is column sum, outer is an outer product, and In is the ones vector in N dimensions. G E NxN is then calculated by applying f component by component to for the training set X with X. The computational bottleneck of this Gram matrix computation is the N x by d m matrix matrix multiplication, with a worst case run time that is O(dNm) Odd?2) when Y X (with naive implementation of matrix matrix multiplication).
We integrate intervals of single pair of samples, and these integrals estimated using the trapezoid rule quadrature. Therefore, for instance, if single trajectory is given, we would get our estimate of all our integrals by simply taking the matrix
h
1
(80)
4
1
26

---

Published in Transactions on Machine Learning Research (07 (275)
and convolving it with the Gram matrix G. If multiple trajectories are given, we apply the same convolution to G' and ignore terms involving sums that mix samples from different trajectories. notation, we may apply the above convolution with the expression:
In NumPy indexing
1²2 M
Gl: - G[[ ;: -1] + Gl: -1, -1,
(81)
4
This adds an additional O(N3) computational complexity (for fixed E NNN).
Finally, the linear system we solve is:
(M + A
(82)
Where E NNN and A, Rxd, which adds the dominating computational complexity term O(dN3).
F.2 Explicit Kernel Assuming the vvRKHS has a matrix-valued kernel of the form
k(x,y) (x))
(83)
where
+b Rd + Rdd
(84)
we may recall:
f(x) b (()))((((T)) dro; =b(x) = (olo)// J[" (a:s(T)) [
(85)
where B E R9 is defined by B B= ==Ex lat ((T) Letting
E Rdxq
(86)
be defined by:
V E Rx I b:
(z)((T))
(87)
Let
1, = VHT
(88)
then
L BT
(89)
Thus, we reduce the minimization problem
22
N
1
min (()) =
|| (1(b)) + +x((a))) + A8T 3
(90)
N
1=1
This simplifies to
(V' v + ANI) 3 4" 3
(91)
where y Nd with yi Rd and y 2;(b:) - (a,). Thus, in the explicit kernel case, require evaluating I which is O(Ndg), x Nd by Nd x matrix-matrix multiplication (Nq))Nd) and x q lincar system solve which is O(²²). Notice, in the explicit kernel case, no assumptions on the overall structure of the kernel matrix (( y) are needed. For instance, the matrices no longer need to be diagonal or proportional to the identity matrix. However, in practice, for best results, q will typically be dependent on the dimension d. If the kernel is Iseparable however, the runtime may be improved. Indeed, supposing v0(x) = =(xx) & I E Red where Φ ddd - RPP The runtime reduces from Og Nd + q²) ((dd)² Nd + (pd)²) to Opp2nd p²d)
27

---

Published in Transactions on Machine Learning Research (07 (025)
Method
Hyperparameters N/A
Null
Sparse Identification of Nonlinear Dynamics
SINDy (Brunton al. 2016)
polynomial degree, sparsity threshold
Reduced Order Models
eDMD-RFF (DeGennaro & Urban 2019
number of features, lengthscale polynomial degree
cDMD-Poly (Williams et 2015
Deep Learning
ResNet (He et al. 2016) Lu et al.
2021
network depth/breadts /breadth latent space dimension, autoencoder depth/brit /breadth latent space dimension, autoencoder and decoder Depth/breadee
eDMD--Deey (Yeung al. 2019)
Latent Ode (Rubanova et al.
2019
Table 4: Hyperparameters for the comparators
G Comparators:
The methods we benchmark against are presented in the Table ! References and hyperparameters are specified. The Null model is used as a baseline to determine that something was learned of the dataset. Our null model has no parameters and the trivial dynamical system for which the slope-field is zero everywhere. A model that fails to outperform the mull model likely did learn any useful information about the dynamical system. We present now the computational complexity for each method.
G.1 SINDY
The computational complexity for the SINDy algorithm is based on the Sequential Threshold Least Squares (STLS) process, and influenced by two main factors:
1. Number of library functions (q) This refers to the number of candidate terms in the library, which is formed based on the polynomial degree and dimensionality of the system. In our case, we used polynomials at most degree 3 and we used 16, 32, and 128 dimensions the Lorenz system.
Number of iterations (T) - The number of times the STLS algorithm iterates to threshold the coef- ficients. Each iteration performs regression on a progressively smaller set of terms after thresholding.
The complexity is derived by:
Sequential Thresholding: In each iteration, the algorithm identifies and zeroes out small coefficients (based on the threshold), which has complexity O(gd), where q is the number of terms and d is the number of variables (or dimensions).
Least Squares Regression: After thresholding, the algorithm re-solves the least squares problem for the remaining large terms. The complexity of this operation depends on the size of the remaining set p. and it scales O(p²), where p S 9. In addition, as with our technique constructing the linear system is an Oddp²?) operation.
Hence, after each iteration, the algorithm refines the set of non-zero coefficients in the library, progressively reducing the number of terms included in the least squares regression.
Iterations
The number of iterations, T, can vary depending on the data and thresholding behavior. In general, is relatively small compared to the size of the library, but it still affects the total complexity.
Overall complexity:
28

---

Published in Transactions on Machine Learning Research (07 (025))
Given that the STLS algorithm iterates T times, the total computational complexity is O(T (gd + 9² + d. The cubic and quadratic terms Ogg² + dNo')) dominate when solving the least squares problem, especially when the number of terms is large, which is typical when higher-degree polynomials or large systems are considered.
G.2 Resnet
The Resnet algorithm residual network that learns the vector field. The input is a vector of dimension d. The output is also vector of dimension d. There are g fully connected layers. The Resnet is trained during T epochs. The computational complexity is O(TNgd)).
G.3 Latent ODEs
The Latent ODE model consists of two primary components: an encoder and a decoder. In the encoder, set of ODEs is solved backward in time, where the ODEs operate the observed state space of dimension d. In the decoder, the ODEs are solved forward in time, but they operate latent space of dimension q.
To compute the overall complexity of the algorithm, we define several key variables:
T: the number of epochs used to train the neural network,
2. by: the total number of function evaluations (time points) used in the numerical solver for the encoder ODEs,
3. k2: the number of function evaluations for the decoder ODE,
n: the number of observed trajectories.
Given these notations, the per-epoch time complexity of the encoder is O(nkid") the ODEs in the encoder are evaluated on state space of dimension d. The per-epoch complexity of the decoder is O(nk24²), as the ODEs in the decoder are evaluated on latent space of dimension Thus, the total per-epoch complexity of the model is:
O (n (kid + k2g²))
Finally, considering that the training runs for T epochs, the overall complexity of the algorithm becomes:
O (nil (kid² k2q²))
G.4 eDMD-Poly and eDMD-RFF
The complexity of the eDMD method can be expressed in terms of N: the number of training examples, q: the number of library functions, and d: the dimension the data. We used a polynomial library for eDMD-Poly and random Fourier features for eDMD-RFF. The underlying algorithm was the same.
The first step is evaluate the library functions on the training data, obtaining an (N, q) matrix V. Next, we must numerically approximate the time derivatives of these quantities, yielding matrix W. The complexity of these operations is O(Nq).
In the next step, we compute an approximation of the Koopman operator, given by K. K= t+ The pseudoinverse has a complexity which depends on whether < q or N 2 q. In the former, the complexity is O(N2g), while in the latter, it O(N2²). The matrix-vector product t/ costs O(Ng²) to compute.
Then we compute an eigendecomposttion of K, costing O) The eigenvectors correspond to approximate eigenfunctions of the true Koopman operator. Let E be the matrix of eigenvectors.
29

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Let B be the matrix of coefficients of the full-state observable in terms of the library functions. This is (d, matrix. The eigemmodes are thus given by BF-1 The matrix E-1 can be expressed as the left eigenvectors of K;; thus, it is computed during the eigndeconposition. Therefore, the eigenmodes only require multiplying (d, q) and (q.9) 9) matrices together, costing (dd")).
The matrix B, if not computed analytically, may be computed by inverting (q,9) matrix and performing d matrix-vector products, resulting in a complexity of (dqq² )²).
Thus, the total complexity is given by O((N d)q² + Nºq + 9²) if < and O((N d)y²² if if
The pscudoinverse step may be impractical in situations where there are many training examples. Thus, it may be substituted with a low-rank approximation using SVD. The rank of this approximation is a hyperparameter and will affect the complexity of the algorithm. We omit this consideration for the sake of simplicity.
G.5 eDMD-Deep
The eDMD-Deep method attempts to learn the library functions and the Koopman operator simultaneously. It does so by defining the library functions as the output of a neural network and minimizing the error incurred from a single Koopman operator update. Thus, the complexity is dependent on the architecture of the neural network used and the choice of optimizer.
Assume that optimization is done with stochastic gradient descent and the neural network outputs q library functions. Assume also that the data consists of observations of d-dimensional trajectories. Suppose the neural network has L layers with g neurons cach. Then a single iteration of SGD for the neural network has a cost of (LLg). Thus, an epoch as a cost. of (dV Training for T epochs then costs O(TLNd4).
One must simultaneously optimize the cost function for the Koopman operator K, which is a (g. q) -immenniia matrix, with the neural network. Computing the gradicnt the cost with respect to requires multiplying a (1,9) matrix and g-dimensional vector, costing (gg²). We compute this product for each item in the training set, costing ((Nq²). Thus, the total cost of optimizing K over T epochs is O(TNg²). Therefore, the cost of training the Koopman operator together with the library functions is O(TN (Ldg + q²)).
Once one has obtained the library functions and the matrix K, one proceeds as in eDMD and computes an eigendecomposition of K, followed by computing the eigenmodes, costing Ogg + dg Thus, the complexity of the entirety of the eDMD-Deep algortihm given by O(TN (Ldq + q²) + dg² + +²).
G.6 Runtime Results for the models
The theoretical runtimes of each benchmarked method is provided in Figure 7 The experimental runtimes are also provided. The training and validation procedures differed for each of the methods making runtime comparisons difficult to interpret. Moreover, the hardware on which the models were run differed as well. For instance, all deep models were trained and run on GPUs while all other methods were not. In Figure we show the experimental runtimes for Torenz96-16, Lorenz96-32 and Lorenz96-12 to give some suggestion to how the runtimes scale the dimension of the problem increases. The runtime of MOCK is not greatly impacted by the increase in the dimension.
The runtime for ResNet and Lode is quadratic in (as can be seen in Figure eDMD-Poly and SINDy-Poly suffer from the problem that the number of polynomial features scales at least linearly with d. Thus these models scale at least cubically in d. q is determined by the user for eDMD-RFF and eDMD-Deep. If the user fixes q. then these methods will be linear in d. See Figure and the discussion of Section G.6
In addition,
H Regression Ablation Study:
One simple way to learn a vector field for a dynamical system to solve a regression problem for the slopes of the trajectories. This regression problem introduces the same regularization the same RKHSs; however,
30

---

Published in Transactions on Machine Learning Research (07/2025)
MOCK
ResNet
eDMD-Deep O(T (Ldq + q²) + dg² + q")
(d
(KdTTN)
Data
val (h)
nol-val (8)
Data
val (h)
non-val (8)
Data
val (h)
non-val (8)
L96-16
2.27
9.0
T96-16
.02
86.1
T96-16
5.63
62.34
L96-32
2.5
9.1
L96-32
.03
107.8
L96-32
4.58
81.3
1.96-128
3.7
9.5
L96-128
.07
232.5
L96-128
6.8
123.9
cDMy-Poly
cDMD-RFF
Lode
O((N + d)d² + N + 4
O((N + d)d² + g²)
O (nT (kid + k2q²))
Data
val (h)
nol-val (8)
Data
val (h)
non-val (s)
Data
val (h)
non-val (s)
L96-16
.31
7.0
L96-16
.08
1.56
L96-16
.23
829
L96-32
.43
8.1
L96-32
.08
1.6
L96-32
.44
1568
L96-128 128
.5
9.4
L96-128
.15
2.58
L96-128
4.7
16891
SINDy-Poly
( + 1² + d(N
Data
val (h)
non-val (s)
L96-16
.1
1.4
L96-32
.0025
1.5
L96-122
128
.03
Figure 7: val(h) is the training time including the training of the hyperparameters, hours. non-val(s) the training time excluding the training of the ayper-paramters, seconds. For non-val, The MOCK algorithm scales most favorably with the dimension of the dataset, as increasing the dimension from 16 to 128 increases training time by less than 10%.
solves a different loss. We have performed an ablation study, replacing the functional in 22 by
1 n J(J) =
Σ -().,())) AILSII
(92)
and estimating (t) using finite differences. Specifically, considering only trajectories of two data points, as in Section 3.3 this becomes,
2 (1) (0)
J( n-1
f(z;
AIlfII
(93)
We used the Gaussian kernel and the same validation procedure as for the MOCK algorithm. We obtain an error equal to 1.9, where the error defined in (29). This is to be contrasted with the error for the MOCK algorithms presented in Table 2 which was 1.40 for the same kernel. This experimentally shows the advantages of using the MOCK algorithm compared to regression. Note that this result not surprising. The use of finite difference for estimating derivative is known to be problematic for noisy data. Estimating an integral is preferred, and this is what the MOCK algorithm does.
We choose to run the experiment on the noisy FHN data, denoted NFHN; see Section 4.1
31


Reviews:
[{"review_id": "biMzPCEvPM", "reviewer": "Reviewer_WSpE", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros: \nThe paper has a detailed literature review, propose an easy-to-follow algorithm. The proposed method also has a convincing performance and low computation complexity than other baselines. \n\nCons:\nThe method is well-proposed, but the motivation of proposing the method seems not to be adequately discussed.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Compared to the previous quadratic methods, the author proposed a linear approach to learn the dynamic systems derived from ODEs. The proposed MOCK is easy to implemented and high-performed on the high-dimensional data.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "When solving a system of ODEs, it is necessary to discuss the necessity of high-dimensional data or why high-dimensional data often exists. From the experiment, the extremely high dimensions come from the synthetic data set. If the existence of high-dimensional data is explained at the beginning, it can make the paper more well-motivated.", "broader_impact_concerns": "No.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "4PSorzRsui", "reviewer": "Reviewer_TGRp", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This paper is well-written and easy to follow, detailed theoretical derivations are provided. The proposed methods are thoroughly evaluated over various benchmarks against comparable methods. Sufficient discussions and details about these methods are reported, including a complexity comparison reflecting the computational advantage of MOCK. From my perspective, this paper's claim is well supported. \n\nThe weakness is mainly from the debatable assumption of I-separable itself, which is crucial for the computational convenience that has been elaborated in this paper, but might at the same time restrictive itself.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work proposes MOCK, a multivariate occupation kernel algorithm, to learn the systems of ODEs from trajectories or their snapshots. Comparing to existing quadratic methods, MOCK makes use of the implicit formulation of the vector-valued reproducing kernel Hilbert spaces (vvRKHS) and scales linearly w.r.t. the dimension of state space. Experimental results over synthetic and real-world datasets show promising performance of MOCK. A divergence-free version of MOCK is also proposed and evaluated to show its ability to learn constrained vector fields.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I would like to suggest the authors improve the formatting, for example:\n\n- There are 90 equations in total (which are highly appreciated), but just a few of them are referred in the main paper. Is it better to only show labels for important ones so that readers can capture main results?\n\n- References to equations, figures, and tables are inconsistent, just name a few, equation xx/(xx), f/Figure xx, t/Table XXX, Alg/Algorithm XXX. Could the authors please check the paper again to make those references consistent?\n\n- For referred papers, please use citep and citet properly, e.g., ResNet (He et al. 2016) instead of ResNet He et al. (2016).\n\n- MOCK is a nice name and is used to represent the proposed methods in the whole paper. However, in Table 2, the MOCK methods are replaced by ockG, ockL, etc. to the specific scalar kernel, which makes sense but such inconsistency is a little bit confusing. I suggest the authors either (i) explain ockG etc. in the main context, e.g., Section 4.3 or 4.4, or (ii) use MOCK-G, MOCK-L, etc.\n\n- Since the paper has mainly compared with some of the featured papers for comparison, having a discussion of more recent related works would be benificial for a better position of this paper, more specifically:\n    - Using kernel methods within Gaussian Processes for ODE have also been investigated ([1, 2])\n    - There are also more recent works in latent ODE approaches ([3, 4, 5])\n   Such aspects could be simply discussed in 4.4 for simplicity.\n\n- [1] https://proceedings.mlr.press/v80/heinonen18a/heinonen18a.pdf \n- [2] https://proceedings.mlr.press/v180/hegde22a/hegde22a.pdf\n- [3] https://arxiv.org/abs/1905.10994\n- [4] https://arxiv.org/abs/2103.12413\n- [5] https://arxiv.org/abs/2406.02352", "broader_impact_concerns": "No broader impact concerns", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "a9TZNUsyUM", "reviewer": "Reviewer_wyKJ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "S: The method is principled and well-derived contribution to ODE learning\n\nS: The results are competive in a large array of experiments and comparisons. The divergence-free experiment is a useful addition.\n\nW: The method's real-world applicability is not really demonstrated. The paper should discuss in which usecases a practitioners should choose the method, and why", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes learning kernelised ODE vector field from observed trajectories using the occupation kernel. The authors derive a version of the representer theorem, and present a practical training algorithm.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Authors should discuss what kind of inductive bias the occupation kernel gives to the learned vector field, and how does it differ from a vanilla kernel regression (without the occupation). This would also be useful to show empirically as an ablation or comparison.\n\n- The figure 1 is not trivial to interpret. The L’s seem to go in a different direction from the learned vector field. It also seems that the L fields have constant angle: all the red and blue arrows point in similar direction with just the magnitude changing. What does this mean? Is this a desired effect? Contrasting fig1 with standard kernel regression (without occ) would be useful.\n\n- Authors should discuss the real-world applicability of the method: when and why should a practitioner choose this method? Authors should also discuss the limitations of the method. \n\n- The buoy experiment needs to be revised or justified better. Fig 6 shows strong stochasticity in the data, and I'm not convinced that modelling this as div-free or Hamiltonian makes sense. I'm not sure what kind of conclusion the authors are after here.", "broader_impact_concerns": "No issues.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)