Learning Efficient and Robust Ordinary Differential Equations via Diffeomorphisms
Authors: Weiming Zhi
Tin Lai
Lionel Ott
Edwin V Bonilla
Fabio Ramos
Venue: ICLR 2022 Submitted (2022)
Keywords: Differentiable ODE integrator
Neural ODEs
diffeomorphisms
differential geometry

Abstract: Advances in differentiable numerical integrators have enabled the use of gradient descent techniques to learn ordinary differential equations (ODEs), where a flexible function approximator (often a neural network) is used to estimate the system dynamics, given as a time derivative. However, these integrators can be unsatisfactorily slow and unstable when learning systems of ODEs from long sequences. We propose to learn an ODE of interest from data by viewing its dynamics as a vector field related to another \emph{base} vector field via a diffeomorphism (i.e., a differentiable bijection). By learning both the diffeomorphism and the dynamics of the base ODE, we provide an avenue to offload some of the complexity in modelling the dynamics directly on to learning the diffeomorphism. Consequently, by restricting the base ODE to be amenable to integration, we can speed up and improve the robustness of integrating trajectories from the learned system. We demonstrate the efficacy of our method in training and evaluating benchmark ODE systems, as well as within continuous-depth neural networks models. We show that our approach attains speed-ups of up to two orders of magnitude when integrating learned ODEs.

Under review as a conference paper at ICLR 2022
LEARNING EFFICIENT AND ROBUST ORDINARY DIFFERENTIAL EQUATIONS VIA DIFFEOMORPHISMS
Anonymous authors Paper under double-blind review
ABSTRACT
Advances in differentiable numerical integrators have enabled the use of gradient descent techniques learn ordinary differential equations (ODEs), where a flexible function approximator (often neural network) is used to estimate the system dynamics, given as a time derivative. However, these integrators can be unsatisfac- torily slow and unstable when learning systems of ODEs from long sequences. We propose to learn an ODE of interest from data by viewing its dynamics as a vector field related to another base vector field via diffeommrphism (i.e., differentiable bijection). By learning both the diffeomorphism and the dynamics of the base ODE, provide avenue to offload some of the complexity in modelling the dynamics directly on to learning the diffeomorphism. Consequently, by restricting the base ODE to be amenable to integration, we can speed up and improve the robustness of integrating trajectories from the learned system. We demonstrate the efficacy of our method training and evaluating benchmark ODE systems, as well as within continuous- depth neural networks models. We show that our approach attains speed-ups of up to two orders of magnitude when integrating learned ODEs.
INTRODUCTION
The problem of fitting an ordinary differential equation (ODE) observed data is ubiquitous throughout many disciplines of the natural sciences and engineering (Perko) 1997). Although traditional approaches have focused on fixed-form system dynamics and inferring their parameters, here we consider the more general problem of learning ODEs, when their dynamics are completely unknown. This problem arises, for example, in robot learning where ODEs are often used to parameterize learned motion (Sindhwani et al.] [018 [Singh et al.] 2020). In deep learning, this problem appears within the context of Neural ODEs (Chen al. 2018). 8) family of continuous-depth models where the evolution of hidden states is an ODE. Recent developments in learning ODEs have allowed the use of differentiable adaptive step-size numerical integrators to train neural network dynamics via the adjoint method (Chen et al. 2018).
The learned ODE system allows us to integrate contin- uous trajectories at different initial conditions. To roll out long and complicated trajectories with a numer- ical integrator, the neural network dynamics model is queried sequentially at each step. This can be un- satisfactorily slow for time critical applications, such as those in robot control, and is known to suffer from numerical instabilities (Gholami et al.) 2019 Choro manski et al. 2020). With these challenges in mind, we propose an alternative approach learning ODEs: we view the desired target ODE dynamics as a vec- tor field that is a "morphed" version of an alternative base vector field via diffeomorphism, i.e., bijective mapping where both the mapping and its inverse are differentiable. Thus, instead of directly modelling the time derivative of the desired ODE with a neural network, use invertible neural network learn the diffeomorphism that relates the target ODE
Figure 1: Related vector fields can be thought of as a vector field "morphed" into another. (Left) Five integral curves (red) of vector field of a Linear ODE, overlaid on grid points (blue): (Right) Corresponding "morphed" in- tegrals of the related vector field and grid.

---

Under review as a conference paper at ICLR 2022
to base ODE. Crucially, by restricting the base ODE to be less complex and more amenable to integration, we can obtain a solution to the (more complex) target ODE by integrating the simpler base ODE and passing its solution through the bijection. Figure I shows an example of related vector fields: the simple integral curves in the left figure are passed through a diffeomorphism to give corresponding complicated curves in the right. Unlike rolling out trajectory, evaluating the diffeomorphism needs not to be sequential, allowing for efficient GPU computation.
We investigate restricting the flexibility of the base ODE improve integration efficiency, and offloading more of the representation burden to the diffeomorphism. Specifically, we assume the dynamics of the base ODE be: (1) linear or (2) modelled by a neural network. Restricting the base ODE to be linear allows the computation of closed-forma solutions, providing major speed benefits. In this set-up we can achieve significant speed-ups of up to two orders of magnitude when employing GPUs, as compared against differentiable integrators with standard settings. We can also restrict the learned ODE to be provably asymptotically stable by adding simple constraints to the linear base ODE, Alternatively, when additional flexibility is required, we remove the restrictive assumption of a linear base ODE and model the dynamics using a neural network. We show that in this setting we can improve the performance of learning challenging ODEs compared to existing differentiable integrators, even when we use a simpler neural network for the base ODE.
In summary, main contributions are:
1. a novel paradigm to learn ODEs from data: invertible neural networks are trained to "morph" the target ODE to an alternative related base ODE, which can be more tractably integrated;
2. analysis of the base as (i) a linear ODE and (ii) non-linear ODE with neural network dynamics. In the linear case, we demonstrate how by restricting the flexibility of the base ODE, we can obtain closed- form integrals, providing significant speed-ups to integration. In the non-linear case, we demonstrate that by learning ODEs as related vector fields, we can flexibly learn challenging ODEs with simpler networks;
3. a principled method to enforce asymptotic stability of learned ODEs, by adding restrictions
to the base ODE.
Proofs and additional details can be found in the appendices.
2 RELATED WORK
Learning of ODEs and Neural ODEs: Dynamical systems governed by ODEs can be found throughout many disciplines of science and engineering. Earlier work on approximating free-form dynamics of ODEs include gradient matching (Ramsay et al. 2006) and using Gaussian processes (Heinonen et al. 2018). Most recent work on this problem model the unknown dynamics with a neural network and leverage differentiable numerical integrators, which use the adjoint method
(Pontryagin al. 1962 to train in memory- tractable manner (see, e.g.. CChen et al.) 2018).
A particular usage of ODE learning is within Neural ODEs, which are neural network models that model the hidden state continuous ODEs rather than discrete layers (Chen et al.) 20188 Massaroli ct al. 2020). Other continuous neural network models which incorporate an ODE, such as latent ODEs Rubanovaa al. 2019) have found application in time-series tasks. Subsequent strategies have been introduced to improve the training of these models, including augmenting the ODE state-space (Dupont et al. 2019) heper-network extensions (Choromanski al. 2020), regularisation techniques (Finlay et al. 2020 Pal et al. 2021). and investigating integrator step-sizes (Ott et al., 2021). At the core of all neural ODE models the differentiable integrator used to learn the underlying ODE. Our proposed approach improves the learning of the underlying ODE, and is compatible with models that incorporate learnable ODEs. We note that the term "neural ODE" has typically been used in the literature to refer to neural networks that incorporate ODEs, including the original work in Chen ctt al. (2018). However, "neural ODE" has occasionally been used to refer to an ODE with dynamics parameterized by neural network (Norcliffe et al. 2021). To disambiguate, throughout our paper, we refrained from referring to the latter model as "neural ODEs", but rather as "ODEs with dynamics parameterized by neural network".
Invertible neural networks and Normalizing Flows: Invertible neural networks (INNs) are a class of function approximators that learn bijections where the forward and inverse mapping and their
2

---

Under review as a conference paper at ICLR 2022
Jacobians can be efficiently computed (Ardizzone et al. 2019). INNs are typically constructed by invertible building blocks, such as those introduced in Kingma et al. (011)) Dinh et al. (2017): Durkan
et al. 2019). Advances in INNs are largely motivated by normalizing flows (Rezende & Mohamed
2015 Papamakarios et al. 2021), an approach to construct a flexible probability distribution by finding differentiable bijection between the target distribution and a base distribution. Our approach is similar in spirit to normalizing flows, as we analogously aim to learn a diffcomorphism that relates the vector fields of the target ODE and some base ODE. However, unlike normalizing flows, we do not require the burdensome computation of Jacobian determinants (Karami et al., (019) to obtain trajectories. separate line of work, broadly characterized as continuous normalizing flows, use ODEs to build invertible approximators (Grathwohl et al. 2019; Chen et al. 2018). Our work proposes the opposite where invertible approximators are used to learn ODEs.
PRELIMINARIES
In this section we introduce the problem of learning ODE dynamies with neural networks. We then describe the notions of tangent spaces and pushforwards, which will be used in section 4 to develop our method.
3.1 LEARNING ODES WITH NEURAL NETWORKS
Consider dynamical system given by ordinary differential equations of the form:
y())) f()(),,),
y()) Yo.
(I)
where t is time, y(t) are the states at time f, and f provides the dynamics. Unlike traditional approaches where f assumed known with only few parameters to estimate from data, here we consider the more general problem where the dynamics are completely unknown. Thus, we can use flexible mapping fo as given by neural network with parameters w. Henceforth, we will drop the explicit dependence on time, and consider the autonomous ODEs given by y() (t)))). Non- autonomous ODEs, which explicitly depend on time, can be equivalently expressed as autonomous ODEs by adding dimension to the states y (Davis et al., 2020). For an initial condition yı, at start to. and some end time te, a solution of the ODE can be evaluated by a numerical integrator (ODESolve), such as Runge- Kutta methods (Butcher 1987):
ODESolve(fe. y(t))= = y + 1. -. to).
The learning problem involves estimating, with fur the dynamics of the ODE, provided n, ob- servations yt ... yooe at specified times. We can learn the ODE by optimising the parame- ters to minimize loss between the observations at the given times and the integrated ODE, ((w) = Loss( (y y : (y(t+))T=i) where [1 are obtained by solving eq. 2 Advances in the neural ODE literature have introduced differentiable numerical integrators, which allow gradient-based techniques to be used to optimize I(w). Furthermore, by using the adjoint sensitivity method outlined in Chen al. (2018). the gradients of adaptive integrators be obtained in memory tractable manner, without differentiating through the integrator operations.
Nevertheless, the flexibility of neural network dynamics for ODE learning comes at the expense of a nigh-coomutatiinnll cost and potential numerical instabilities, especially when considering long trajectories. We will develop in section 4 an alternative method that transforms this problem into that of learning simpler ODE along with a diffeomorphism, by treating the dynamics of the original (target) ODE and simpler (base) ODE as related vector fields.
3.2 TANGENT SPACES AND PUSHFORWARDS
As mentioned above, shall be analysing the system dynamics of ODEs vector fields. Here briefly introduce the differential geometry notions of tangent spaces and pushforwards, which will be used to define related vector fields, core concept underpinning methodology.
Tangent Spaces: A manifold is a space that locally resembles Euclidean space. Throughout this paper, all manifolds will be assumed to be differentiable, with defined tangent spaces. For
3

---

Under review as a conference paper at ICLR 2022
TM
TN
DF
P.M
X
Y
p E Dr (Xp) Tew
Ip Tp
F
M
N
D2F
F-1
(a) If F' maps points p M F(p) E N, . a single tangent vector at p. Xp E Tp.M, can be mapped to TF(,)-V. However, an entire vector field X on .M cannor in general he mapped to valid vector field on N. The pushforward by diffcomorphism is special case where a valid vector field can be obtained.
(b) If vector fields X and Y on manifolds M and N respectively are related by diffcomorphism F, then they are related via pushforward unknown, we have another path to evaluate by F X TM DF, T.N
is
Figure 2: Vector fields can be related by diffcomorphisms
n-dimensional manifold M, at a point p MM. the tangent space Tp.M an n-dimensional real vector space, where each element passes tangentially and is referred to as a tangent vector. The tangent space provides a higher-dimensional analogue
tangent plane at point on a surface. The collection of tangent spaces for all points on M is known the tangeni bundle denoted by TM.
For mapping F M + V between two manifolds, M and N, the pushforward by F is linear mapping between the tangent spaces of the manifolds, DpF TpM + I (p) Tangent vectors at p in the domain M can be mapped to tangent vectors at the corresponding point F(p) in the codomain A/ via the pushforward. This is computed by the matrix product of the Jacobian of F at p. (P), and a tangent vector at p.
Pushfoward:
METHODOLOGY
We study the dynamics of ODEs, f. as vector fields, and solutions as their integral curves. We model the desired ODE dynamics target vector field that is related to another base vector field. First, we introduce the concept of related vector fields, outline how they can be learned, and elaborate on the benefits of learning them. Then, describe possible choices of base vector field models.
4.1 RELATED VECTOR FIELDS FOR ODE LEARNING
A vector field X defined on manifold M is a function that assigns tangent vector Xp TpM each point p M. Intuitively, our aim is construct a mapping F' which shapes the manifold where a base vector field X is defined, such that the pushforward of X by F extrinsically appears "morphed" to match the data.
What are the requirements of these mappings, for the pushforward of vector fields to be valid?
Provided mapping between manifolds F M + N. we can push a single vector, X, E TpM, to the tangent space of N at F(p). TF(p) via the pushforward, D ((p)). Figure 2a sketches out an example of the pushforward of tangent vector between tangent spaces. However, this notion does not extend in general to vector fields. If F is injective and non-surjective, the pushforward of X outside the image of not defined. If F is surjective and non-injective, there may be multiple differing pushforwards for point. In special cases when the pushforward by F' defines valid vector field on the codomain. N, the vector field and its pushforward are known to be F-related. Latent ODEs. described in (Rubanova et al., 2019). which
auto-encoder to map input data to latent space, where an ODE is learned is similar in spirit to our method. Latent ODEs, however, do not define valid vector field in the input space.
Definition 4.1 (Related vector fields), Let F M N be a smooth mapping of manifolds. vector field X on M and a vector field Y on N are related by F, or F-related, for all M,
Dp F(Xp) YF(p)-
(3)
4

---

Under review as a conference paper at ICLR 2022
Related vector fields arise in particular when F is a diffeomorphism, i... bijective mapping, where both the mapping itself and its inverse are differentiable.
Proposition 4.1 (Propasition 8.19 in Lee (012)). Suppose F M + N is a diffeomorphism between smooth manifolds M, N.. For every vector field X on M, there a unique vector field Y on N that is F-related to X.
By considering the F-rrelted properties of vector fields, we have pathway define unknown vector fields using the pushforward of F', as shown in fig. 26 If vector field X on M is F-related to some vector field Y on N,, instead of directly evaluating the vector field Y, we can instead obtain tangent
values for any E NN, via N - M X TM DF TN. Therefore, the vector attached for each V is. Y4 Dr (q) (XF- Jp (F (q)) p-)(()) where J.p is the Jacobian of F.
4.2 DIFFEOMORPIISM LEARNING VIA INVERTIBLE NEURAL NETWORKS
The machinery to learn invertible mappings has seen extensive development with the progress of normalizing flows for estimating probability distributions. Invertible neural networks (INNs, Ardizzone et al. 2019) are function approximators which learn differentiable bijections. INNs can be trained on forward mapping, and get the inverse with no additional work, by the definition of their architecture. In this paper, we use INNs of the type described in Dinh et al. (2017). The basic unit is a reversible block, where inputs are split into two halves, u1 and ug. The outputs V1 and V2 are: U2 @ exp(s1(v1)) - (v(1)), where @ indicates element wise multiplication, and f1, t2 and 81, 82 are functions modelled by conne-con neural networks with non- non-linear activations. These expressions are clearly invertible:
@ exp((s) (2 (2)) + to (u2),
(4)
11 (V1 t2(22)) @
U2 = (V2 - 1 (v1)) O (x)(-s1(v1)). (5)
Note that the functions 4, 12 and 81, 82 themselves are not required to be invertible.
4.3 ODE SOLUTIONS AS INTEGRAL CURVES OF RELATED VECTOR FIELDS
Why would be beneficial to construct a desired vector field Y indirectly, by way of a related X?
We shall answer this by considering integral curves on Y, which represent solutions to the ODE associated with Y. An integral curve of on A a differentiable curve y : - N, whose velocity at each point is equal to the value of Y at that point, i.c. y'(t) Yy(t) Tyy() for all t E RR The integral curves of Frrrelaed vector fields are also linked by F: integral curves on one vector field are uniquely mapped to the other via single pass through F, and the Jacobian Jp is not required.
Proposition 4.2 (Proposition 9.6 in Lee (2012)) Suppose X and are vector fields on manifolds M and N respectively.
X and Y are related by mapping F : M + N and only for each integral curve x IR- + M, y = Fx) is an integral curve of Y.
In the ODE learning problem outlined in section 3.1 we denote to be the vector field associated with the target ODE, During both training and inference, we need obtain integral curves y of Y either by numerical integration, or by y = Fxx), where x denotes the corresponding integral curve of X, related Y via the diffeomorphism when we are working with the integral curves.
Critically, the Jacobian of F does not need to be evaluated
integral curves of X can be found in more efficient, or less error- error-pront prone manner, than by numerically
integrating curves of Y, we can leverage the relationship y = F(x) for ODE learning. This can be done by an INN, Fo. with parameters O. We denote the base ODE as x' x'(1) = I (x)),,, where y are parameters. We can then use the target ODE within some learning problem, minimising loss over target ODE trajectories and observations:
+ 1/' (x()))))))
E(0, up) = Loss (fygh " M
(6)
where there are n, data time points, yo is an initial condition for the system, and yi are ob- served data points at times t.. After training, the dynamics of the target ODE is given by y'(t) = Jr. (F, (y(t))) (F, (y())) and the ODE solutions (integrals) are obtained with:
y(t) Fo Fo (yo) + (x(())))
5

---

Under review as a conference paper at ICLR 2022
In practice, we often required to evaluate entire trajectory, i.e., y(t) multiple times fi...., tend with one initial yo. as outlined in Algorithm I This allows us to batch up the pass through Fo. which makes this highly efficient when executed on a GPU. The benefits of our method are apparent when it is advantageous to integrate the base ODE and then pass the solution through the diffeomorphism, Fo, rather than numerically integrate the target ODE.
Next, we investigate restricting the flexi- bility of the base ODE, so that it is more amenable to integration, offloading the complexity of learning to the diffeomor- phism. We consider two choices of base ODEs: (1) Linear ODE; (2) ODE with neu- ral network dynamics.
Algorithm 1: Efficient integration learned ODEs
Input :oo 9p- No. fi....,tend
Output: ......... (((e)rd)))
Xo + Fo (yu)
x(t,) t Xo + (x(())) = 1..i end;
// The integral is easier to solve.
4.4 LINEAR ODE AS BASE: FAST INTEGRATION AND STRAIGHTFORWARD ASYMPTOTIC STABILITY
y(h)................... ((len)).))(l))))) +- Fa(x(t))...., x(tend)) // Batched pass through INN can be efficiently computed on GPUS.
We can speed-up integration significantly by modelling the base as Linear ODE, of the form (t) Ax, where x(t) E R are n--dimensional variables, and A E BR*x. Linear ODEs can be solved very efficiently as they admit closed- form solutions. Provided an initial solution Xo, the solution of x(I) and that of the target ODE yt) are: x(t) = Σm=1 ro exp(A+t), and yt) = (x())))
where lg. r& and A are the corresponding left, right eigenvectors and eigenvalues of matrix A. respectively. We learn the eigenvalues and eigenvectors of matrix A jointly with diffcomorphism F'g
Linear ODEs are also interesting because their long-term behavior, which is determined by their eigenvalues, is easy to analyse. We shall see how this property allows to craft the long-term behavior of the target ODE, In particular, in many applications, consideration is given properties of ODEs, namely what happens to the solutions after a long period of time. Will they converge to equilibrium points, periodic orbits, or diverge and fly off? Our method provides a straightforward way to restrict the learned ODE to be asymptotically stable. In robot motion generation problems, such as that in SSnddhwani et al. (2018), we aim to learn an asymptotically stable ODE. We begin by defining equilibrium points and asymptotic stability of first order ODEs.
the asymptotic
Definition 4.2 (Equilibrium point). An equilibrium point y" of an ODE y' f(y(t), is point where f(y') O.
Definition 4.3 (Asymptotic stability). An ODE y'() = (t)(t),t) is asymptotically stable if for every solution yt), there exists a 6 >>000 O, such that whenever |1y(l)) y*lk 8, then y(t) + y" as t + 00, where y" is some equilibrium point.
Intuitively, asymptotically stable systems of ODEs will always settle at some equilibrium points after long period of time. In the context of vector fields related by diffeomorphism, the asymptotic stability properties of the ODEs are shared.
Theorem 4.1. Suppose two ODEs x x1) = g(x(1)), where F is a diffeomorphism. If the former ODE is asymptotically stable with ne equilibrium points xi x then the latter is also asymptotically stable, with equilibrium points F(xi)....,4 F(x,,.)
are related via y() F()(t)),
Therefore, if we can restrict the base ODE to be asymptotically stable, then the target ODE learned by our method is also asymptotically stable. When the base is an dimensional linear ODE, we can restrict it to be asymptotically stable by directly learning the eigenvalues, X; for = 1,... n, and constraining them to be negative. This can be done by setting X, -sxx,²² E, where is a small positive constant, and learning sx, instead of learning the eigenvalues.
4.5 NEURAL NETWORK ODE AS BASE: IMPROVED ROBUSTNESS FOR 'DIFFICULT' ODES
Using linear systems as base ODEs provides a dramatic increase in speed at the cost of flexibility. We observe that the computation overhead of single backward pass Fg and batched single

---

Under review as a conference paper at ICLR 2022
forward pass Fy is minimal when compared with numerical integration which requires sequential querying. When the ODE is difficult to learn, we can also parameterize the dynamics of the base ODE using neural network. This is particularly appealing for ODEs that are considered stiff, with rapid varying of the solution in time in one dimension, while the other dimensions remain largely unchanged. Existing differentiable integrators often struggle to directly learn these ODEs. Directly learning the target ODE will require exceedingly small step-sizes (Hairer et al. 1993). Although by simply scaling the data before training, we can lessen the stiffness of the ODE to learn, the relatively rapid changes isolated in single dimension can still result in the ODE being hard to learn. To tackle this, we learn a neural network dynamics of the base ODE jointly with the diffeomorphism. The burden of accurately representing the stiff dynamics is shared by Fe. providing additional flexibility, The diffeomorphism can learn to relate the target ODE to base that is amenable to integration. This allows us to more accurately learn challenging ODEs, with much smaller neural network model of the base ODE, than that used to directly learn the target ODE, As passes of the INN are incxpensive, we gain a speed-up during integration.
EXPERIMENTAL RESULTS
We empirically evaluate the ability of our method to speed up the integration of learned ODEs, along with the robustness of integration when learning difficult ODE systems. Throughout this section, we compare the error and integration times of our method against a variety of solvers. We include fixed step-size solvers: Euler's, s, midpoint, and Kutte-Kutta 45 (RK4), and the adaptive step-size solvers Dormand-Prince 5 (DOPRI5) and Dormand-Prince 8 (DOPRI8). Unless specified otherwise, for fixed step-size solvers, we set the step-size equal to the smallest time increment of outputs. For adaptive step-size solvers, we set absolute and relative tolerances to 10 5, We augment the ODE states for all the systems in accordance to Dupont et al. (2019) in all of the ODEs trained during the experiments, except when recreating latent ODE results from Rubanova et (2119), where use the original implementation provided by the authors. The differentiable solvers are implemented in the torchdiffeq library. Details on experiments available in the appendices.
5.1 INTEGRATION SPEED-UPS BY LEARNING WITH A LINEAR ODE BASE
We test our hypothesis that the availability of a closed-form expression for the integral, when using a linear base ODE, can provide significant integration speed-ups. We evaluate on learning synthetic ODE systems, real-world robot demonstrations, and within continuous deep learning models. Here, report test error/accuracy and integration times. Training times can be found in the appendices.
We train and evaluate models on data from the 3D Lokta- Volterra system, which models the dynamics of predator-prey populations. The data is corrupted by white noise with standard deviation of 0)055. We train our model using a linear base ODE, and assess the capability of our model in interpolating the data points 10x the data resolution, and generalizing to 16 hidden test initial conditions to integrate trajectories, also at 10x the data resolution. We report the integration time for generalization. Figure 3 shows interpolation results and newly generated trajectories, where we see that our model is able to capture the dynamics of the system. Furthermore, table [ provides a quantitative evaluation, where we see that our method significantly faster
Learning 3D Lotka-Volterree Volterra:
Lotka-Volterr
Imitation
Imitation cube pick
Imitation C
MSE
MSB (G)
Time (ms)
MS)
Time (ms)
MSE (G)
Time (ms)
MSE
Time (ms)
Ours (Lin)
0.1
1.1
9.3
6.1a1.2
6.62 0.2
18.616.2
1.6
8.12
0.5
Euler
0.3
4.62 0.1
385.62 14.4
10.3+2.9
724.72 8.3
14:92 1.4
728.42
2.0
753.9z 1.4
Midpoint
0.38# 0.05
0.1
670.4t 31.3
10.9+3.3
581.6t 13.3
12.9 1.3
1267
13.6
6.9 2.2
1305. 4s 14.7
RK4
0.35# 0.005
0.62
1316.12 30.8
10.343.0
2501.74 18.9
0.992
2522.82233
7.62
1292.3+ 22.0
DOPRIS
0.93x 0.05
0.1
264.7s 17.0
10.8+288
1277.7 14.3
14.9*
504.0t 12.3
7.1
623.4t 15.6
Table I: The mean squared error for interpolation, MSE (I), and generalization, MSE (G), and mean execution times (in ms, + standard deviation) on the 3D Lotka-Volterre Volterra system and the time critical motion generation for our method with linear base ODE and competing augmented ODE models, with neural network dynamics, with various numerical integrators. Our method, with a linear base ODE, provides comparable or better accuracy, with significant integration speed-ups.
7

---

Under review as a conference paper at ICLR 2022
than competing approaches using numerical integrators with speed-ups of more than two orders of magnitude, while achieving comparable or better accuracy.
A Time Critical Application-Rooot Motion Genera- tion as Stable ODE Learning: The ability to quickly roll-out trajectories is erucial in motion robotics settings. We consider the application of generating robot manipula- tor motion trajectories from provided demonstrations. In particular, (Sindwaani et al. (2018) showed that modelling the motion as trajectories of stable ODE critical for generalizing and being robust to perturbations in imitial conditions. The goal is learn an ODE system where trajectories integrated at different starting points mimic the shown demonstrations. We use three sets of real- world data from [Khansari-Zades & Billard 2011 of trajectories: drawing "S" shapes; placing a cube on a shelf, drawing out large "C" shapes. We use 70% of the data for training, and test our generalization on the remaining demonstrations. In these datasets, the motions are known converge to equilibrium points. Hence, we constrain the learned ODE to be asymptotically stable. We report the performance and run-times of generalizing to new starting points in table [
Prg.
Learning the 3D Volterra. (Left) Interpolat- ing (red) data (blue): (Right) Generating trajectories (red) at hidden test initial conditions and the ground truth (blue).
Figure 3:
Lotka-Volterra.
We see that our approach is competitive in the quality of generalized trajectories, while achieving speed-ups of more than two-orders of magnitude.
Learning for Continuous Deep Learning Models: We evaluate our method as a component of Latent ODEs (Rubanova et al. (019), con- tinuous deep learning model. Latent ODEs embed the time series observa- tions as hidden states via an encoder- decoder. An ODE, with dynamics pa- rameterized by a neural network, is fit on the hidden states which allows for irregularly sampled series. In these ex- periments, our method is applied with a linear ODE base to learn the dynamics governing the hidden states. We report results for the periodic curves and the human activity classification problem used in the original latent ODE paper (Rubanoval et al. 2019), as well as for ECG classification and image sequence generation. The periodic curves problem requires us to reconstruct trajectories at different resolutions, with 100 and 1000 time-steps. The human activity classification dataset contains readings over the body of subjects over time, and the problem has a sequence-to-sequenee setup, requiring us to predict the human activity at each time point. The ECG problem is sequence classification problem, where the sequences are ECG signals. In the image sequence generation problem, we train on 100 sequences rotating "3" characters, similar to that in Yildiz et al. 2019). learning nigh-dimenssonal ODE which governs a latent representation of the image rotation over time. We test on 100 unseen "3" characters and report results at 10x training resolution. The performance and times spent on integrating the hidden state dynamics are reported in table We observe that by leveraging the closed-for expression of integrals of the base ODEs, we achieve ODE integration times that are hundreds of times faster, while achieving competitive performances against compared differentiable integrators. We note that the main cost of the integral in our method the pass through the invertible neural network. GPUs allow us to batch the pass at practically constant cost, whereas the sequential nature of integrators give linear increases in run-time. A qualitative evaluation of our method on the image sequence problem is shown in fig. 4 We observe the structure of the rotated "3" character remains consistent with the test initial image. Our approach, when restricted to a linear base ODF, able to learn high-dimensional ODEs which can be integrated significantly faster, without compromising performance.
ODE
3
3
3
3
3
3
3
3
3
3
3
Figure We model latent dynamics of rotating MNIST "3" characters, as an ODE. We learn the ODE via diffeo- morphism and a linear base ODE. Given an unseen test "3" character, we efficiently generate image sequence of rotat- ing "3"s (illustrated here, from upper left to lower right).
5.2 ROBUST INTEGRATION BY LEARNING WITH A NON-LINEAR NEURAL NETWORK BASE
We test our hypothesis that using a neural network base ODE allows us to learn ODEs that are difficult to integrate. We learn and evaluate models trained the chaotic Lorenz system, and the stiff
8

---

Under review as a conference paper at ICLR 2022
Periodic 100
Periodic 1000
Human Activity
ECG
Image Seq.
MSE
Int. time
MSE
Int. time
Acc.
Int. time
Acc.
Int. time
MSE
Inc. time
Ours (Lin)
0.030
2.74 0.6
0.008
2.8+ 0.8
0.864
4.2+ 1.8
0.966
7.62 2.5
0.028
5.7 1.3
Euler
0.040
2.6
0.043
326.61
0.815
2.9
0.963
100.0#
0.028
103.342.5
Midpoint
0.032
54.5=
0.074
510.1# 15.5
0.865
114.2+
0.963
3.8
0.026
187.9#6.8
RK4
0.039
95.6# 1.6
0,052
1020.0# 60.0
0.857
221.24 4.2
0.963
425.5+4.8
0.027
401,2+9.1
DOPRIS
0.045
83.42 2.2
0.050
264.7=
0.869
67.92
0.963
123.3 2.6
0.025
194.5# 9.2
DOPRIS
0.041
99.62 2.3
0.049
282.7# 6.4
0.724
94.82 1.6
0.963
171.6# 3.6
0.026
399428.4
Table 2: The mean squared error, accuracy and mean integration times (in ms, + standard deviation)
when using latent ODEs on the tasks of periodic curve reconstruction using 100 and 1000 time-steps, the classification problems of human activity and ECG, and the image sequence generation with our method using a linear base ODE and competing numerical integrators.
ROBER system yat!
cats Dopris
Dur
MAB
Time(ms)
MAE
(immo(as))
Ours
0.20
230.99
0.01
157.8
Huler
10.99
456x 13
0.22
201
Midpoint
8052
3/0.11
RK4
6.81
1761a 206
0.041
660118
DOPRIS
7.55
632z 83
0.039
18967
16:4 36-4
16-2
102
105
Figure 5: Results chaotic (Lorenz) and stiff (ROBER) systems. Left: A trajectory from the learned
Lorentz system in red, with data in blue. Center: The second dimension of the ROBER system, against time in logscale. Our method is much capable capturing the sudden variation. Right: Quantitative evaluation of our method with a non-linear base ODE and competing integrators on augmented ODEs with neural network dynamics.
Robertson's ' system (ROBER, Robertson 1966). We use a DOPRI5 adaptive step-size integrator to learn our base ODE, generate trajectories 10x data resolution. Figure (left) illustrates generated trajectory from the Lorenz system, which closely resembles the data points (blue). We can see that both the initial large and small variations, indicated by the small dense spiral at the end, captured. Figure 5 (center) (center) illustrates the particular rapidly changing second dimension of the ROBER system. We see that the added flexibility of the diffeomorphism allows us to better capture the rapid variations over time, while the directly learned ODE struggles to handle the sudden increase. Following the equation rescaling described in Kim et al. (021) before we train on our method and comparisons, we rescale our data by the maximum training value in each dimension and operate in logscale time. Figure [ (right) provides the performance and integration times of learning with our method and baseline integrators, where we that our method is more accurate than competing approaches while also requiring shorter integration times, due to having a simpler network modelling the dynamics.
CONCLUSIONS
We have proposed a novel approach learning ODEs with unknown dynamics, which uses invertible neural networks to learn a diffeomorphism relating a desired target ODE to a base ODE that is often easier to integrate. We have investigated using base ODE that linear or parameterized by a neural network. By leveraging the closed form solution of linear ODEs, our method provides significant speed-ups and allows for asymptotic property constraints on the learned ODEs. Alternatively, by using base ODE parameterized by neural network, our approach can learn "difficult" ODEs, with simpler networks modeling their dynamics. We have validated our method by learning ODEs on synthetic and real-world data, robotic learning problems and within continuous-deny neural network models. Future work could explore more on how to balance offloading the burden of learning the diffeomorphism and the base ODE.
REPRODUCIBILITY STATEMENT
It is extremely important that the work published in ICLR is reproducible. To this end, have included source code for our experiments, including code to generate the benchmark dynamical systems, supplementary materials to the submission. Furthermore, additional details of experiments

---

Under review as a conference paper at ICLR 2022
and data, along with clear explanations of any assumptions and complete proof of the theoretical claims are included the appendix.
REFERENCES
Lynton Ardizzone, Jakob Kruse, Sebastian Wirkert, Daniel Rahner, Eric W. Pellegrini, Ralf S. Klessen, Lena Maier-Hein, Carsten Rother, and Ullrich Köthe. Analyzing inverse problems with invertible neural networks. In International Conference on Learning Representations (ICLR), 2019.
J. C. Butcher. The Numerical Analysis of Ordinary Differential Equations: Runge-Kutte and General Linear Methods. Viley-Interscciene,, 1987.
Ricky T. Q. Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural ordinary differential equations. Advances in Neural Information Processing Systems, 2018.
Krzysztof M Choromanski, Jared Quincy Davis, Valerii Likhosherstov, Xingyou Song, Jean-Jacques Slotine, Jacob Varley, Honglak Lee, Adrian Weller, and Vikas Sindhwani. Ode to ode. In Advances in Neural Information Processing Systems, 2020.
Kaggle Dataset. ECG heartbeat categorization dataset. 2018.
Jared Quincy Davis, Krzysztof Choromanski, Jake Varley, Honglak Lee, Jean-Jacques Slotine, Valerii Likhosterov, Adrian Weller, Ameesh Makadia, and Vikas Sindhwani. Time dependence in non-autonomous neural odes. ICLR 2020 Workshop DeepDiffEg, 2020.
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real NVP. In International Conference on Learning Representations (ICLR), 2017.
Asen Dontchev and R Rockafellar. Implicit Functions and Solution Mappings: A View from Varia- tional Analysis. 2009. ISBN 978-0-387-87820-11 doi: 10. 10.10077/988---33--888811
Emilien Dupont, Arnaud Doucet, and Yee Whye Teh. Augmented neural odes. In Advances in Neural Information Processing Systems, 2019.
Conor Durkan, Artur Bekasov, lain Murray, and George Papamakarios. Neural spline flows. In Advances in Neural Information Processing Systems, 2019.
Chris Finlay, Joern-Henrik Jacobsen, Levon Nurbekyan, and Adam Oberman. How to train your neural ODE: the world of Jacobian and kinetic regularization. In International Conference on Machine Learning, 2020.
Amir Gholami, K. Keutzer, and G. Biros. Anode: Unconditionally accurate memory-efficien gradients for neural odes. In IJCAI, 2019.
Will Grathwohl, Ricky Q. Chen, J. Bettencourt, Ilya Sutskever, and D. Duvenaud. Ffjord: Free- form continuous dynamics for scalable reversible generative models. International Conference on Learning Representations (ICLR), 2019.
E. Hairer, P. Nørsett, and Wanner. Solving Ordinary Differential Equations (2nd Revised. Ed.): Nonstiff Problems. Vprilger-Verlag, Berlin, Heidelberg, 1993.
Charles R. Harris, K. Jarrod Millman, Stéfan J. van der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel J. Smith, Robert Kern, Matti Picus, Stephan Hoyer, Marten H. van Kerkwijk, Matthew Brett, Allan Haldane, Jaime Fernández del Río, Mark Wiebe, Pearu Peterson, Pierre Marchan,, Kevin Sheppard, Tyler Reddy, Warren Weckesser, Hameer Abbasi, Christoph Gohlke, and Travis E. Oliphant. Array programming with NumPy. Nature, 585(7825). 2020.
Markus Heinonen, Çagatay Yildiz, Henrik Mannerström, Jukka Intosalmi, and H. Lähdesmäki. Learning unknown ode models with gaussian processes. In International Conference on Machine Learning, 2018
Roger A. Horn and Charles R. Johnson. Matrix Analysis. Cambridge University Press, 2nd edition, 2012.
10

---

Under review as a conference paper at ICLR 2022
J. D. Hunter. Matplotlib: A 2d graphies environment. Computing In Science & Engineering, 2007.
Mahdi Karami, Dale Schuurmans, Jascha Sohl-Dickstein, Laurent Dinh, and Daniel Duckworth. Invertible convolutional flow. In Advances in Neural Information Processing Systems, 2019.
S. M. Khansari-Zadeh and A. Billard. Learning stable nonlinear dynamical systems with gaussian mixture models. IEEE Transactions Robotics, 2011.
Suyong Kim, Weiqi Ji, Sili Deng, Yingbo Ma, and Christopher Rackauckas. Stiff neural ordinary differential equations. Chaos, 2021.
Durk P Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Ilya Sutskever, and Max Welling. Improved variational inference with inverse autoregressive flow. In Advances in Neural Information Processing Systems, 2016.
John Lee. Introduction to smooth manifolds. 2nd revised ed. Springer, 2012.
S. Lefschetz and R. Alverson. Stability by liapunov's direct method with applications. Physics Today, 1962.
Stefano Massaroli, Michael Poli, Jinkyoo Park, Atsushi Yamashita, and Hajime Asama. Dissecting neural odes. In Advances in Neural Information Processing Systems, 2020.
Alexander Noreliffe, Cristian Bodnar, Ben Day, Jacob Moss, and Pictro Liò. Neural ode processes. In International Conference on Learning Representations (ICI.), 2021.
K. Ott, Katiyar, Hennig, and M. Tiemann. Resnet after all: Neural odes and their numerical solution. In International Conference on Learning Representations (ICLR), 2021.
Avik Pal, Yingbo Ma, Viral B. Shah, and Chris Rackauckas. Opening the blackbox: Accelerating neural differential equations by regularizing internal solver heuristics. In International Conference on Machine Learning, 2021.
George Papamakarios, Eric T. Nalisnick, Danilo Jimenez Rezende, S. Mohamed, and Balaji Laksh- minarayanan. Normalizing flows for probabilistic modeling and inference. Journal of Machine Learning Research, 2021.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Devito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, sege--effrrmncce deep learning library. In Advances in Neural Information Processing Systems 32. 2019.
Lawrence Perko. Differential Equations and Dynamical Systems. Springer-Verlag. Verlag, 1991.
Lev Semenovich Pontryagin, EF Mishchenko, VG Boltyanskii, and RV Gamkrelidze. The mathemat- ical theory of optimal processes. 1962.
Ramsay, G. Hooker, D. Campbell, and Jiguo Cao. Parameter estimation for differential equations: generalized smoothing approach. Journal of The Royal Statistical Society Series B-statistical Methodology. 2006.
Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. In International Conference on Machine Learning, pp. 1530-1538, 2015.
H.H. Robertson. The solution of a set of reaction rate equations. 1966.
Yulia Rubanova, Ricky T. Q. Chen, and D. Duvenaud. Latent odes for iregularly time series. Advances in Neural Information Processing Systems, 2019.
Vikas Sindhwani, Stephen Tu, and Mohi Khansari. Learning contracting vector fields for stable imitation learning, 2018.
11

---

Under review as a conference paper at ICLR 2022
Sumeet Singh, Vikas Sindhwani, Jean-Jacques E. Slotine, and Marco Pavone. Learning stabilizable dynamical systems via control contraction metrics. In Algorithmic Foundations of Robotics XIII, 2020.
Cagatay Yildiz, Markus Heinonen, and Harri Lahdesmaki. Ode2vae: Deep generative second order odes with bayesian neural networks, In Advances in Neural Information Processing Systems, 2019.
12

---

Under review as a conference paper at ICLR 2022
The following sections contain the supplementary text which gives additional results, additional details about the experiments, and proofs.
A ADDITIONAL RESULTS: REPRESENTATION LOAD OF THE INN
In the setup of using a non-linear neural network base ODE, both the INN and neural network dynamics are trained jointly. We empirically investigate the representation burden on the invertible neural network, when using a non-linear neural network base ODE. We train, for 5000 epochs, augmented ODEs with neural network dynamies models of different parameter sizes, without the diffeomorphism, on the Lorentz system and report the MAE values of integrating with the dopri5 integrator with the same setup as the Lorentz experiments in Section 5.2. We then compare take the neural network dynamics model with the smallest number of parameters (with parameter count of 2256), and add INNs with an increasing number of invertible layers. The results of the ODE with neural network parameters only, with increasing parameter counts, are tabulated in table 3 while the results of the neural networks with an increasing number of invertible layers are tabulated in table
Each invertible layer contains 15090 parameters. We note here that the querying the INN (and thereby having a large INN) adds negligible integration time, as during the entire integration we only need one forward and one inverse pass of the INN, while the system dynamics needs to be queried sequentially. We see that, in general, adding parameters to the system dynamics of a learned ODE and to the diffeomorphism adds to the representation power of our ODE models. However, see that the addition of an INN improves performance, and additional layers to the INN, up to around 5 invertible layers, improves performance. We see that the model with 5 invertible layers and base ODE of 2256 parameters greatly outperforms the ODE model with only neural network dynamics of 83006, while the two models are similar in parameter count. This indicates that the invertible neural network plays a large role in representing the learned ODE.
clearly
NN-dynamics paramcter count
2256
8106
17556
30606
47256
67506
83006
MAE
10.95
10.94
10.94
8.04
7.35
7.06
7.08
Table 3: The MAE of interpolating trajectories from the Lorentz system, with ODEs using neural networks of various sizes.
Number of INN Layers MAE
0
1
2
3
4
10.95
2.9
2.53
0.76
0.45
0.21
0.38
0.29
Table 4: The MAE of interpolating trajectories from the Lorentz system, with a small base ODE with 2256 parameters and increasing number of invertible layers.
B ADDITIONAL RESULTS: TRAINING TIMES
We present the training times for directly learning ODEs with our method, using linear base ODE, These include the training times on the 3D Lotka- Volterra, and the robot imitation datasets, outlined in Sections 1.1.1 and 5.1.2 of the main paper. We run training for 1000 iterations, where in each iteration the batch includes the entire training set. We see that, by leveraging the closed-form solution of linear ODEs, method is able to also drastically speed up training. Additionally the parallel nature of passing through the invertible neural network allows more consistent training times across datasets.
13

---

Under review as a conference paper at ICLR 2022
3D Voltera
Imitation S
Imitation cube pick
Imitation
Per fier (S)
Total (8)
Per iter (8)
Total (S)
Per fier (S)
Total (8)
Per iter (8)
Total (S)
Ours
0.030= 0.002
29.57
0.029# 0.002
29.35
0.031 0.007
31.34
0.030s 0.004
29.78
Euler
0.13140.003
130.94
1.740z 0.017
1740.43
1.713# 0.016
1712.53
1.704z 0.010
1703.85
Midpoint
0.235+0.006
235.07
3.228# 0.027
3227.51
3.1774 0.028
3177.37
3.2074 0.025
3206.92
RK4
0.46940.005
468.52
6.6714 0.048
6671.44
6,3884 0.046
6388.10
6,4414 0.057
6440.61
Dopris
0.408+0.037
408.36
1.413# 0.034
1413.34
1.2464 0.023
1245.65
1.247s 0.022
1246.67
Table 5: The training times in seconds with standard deviations, for 1000 iterations. By leveraging the closed-form solution of linear ODEs, training time with our method is consistently orders of magnitude faster than by using differentiable numerical integrator.
C ADDITIONAL RESULTS: ABLATION STUDY
We study the effects the number of layers in the invertible neural network and number of parameters in the sub-network, which are the main nyper-parameters of the invertible neural networks used. To this end, we conduct ablation studies of the speed and performance of our method on the real-world datasets outlined in section 5.1.2 of the paper. Our basic model uses invertible neural network with 5 layers and sub-networks in the invertible network had 1500 hidden dimension size. We alter the number of layers to be: 2, 4, 5, 6, 7, and hidden dimensions of the sub-networks within the invertible network to be: 500, 1000, 1500, 2000, 2500. The results are presented below:
Imitation S
Imitation cube pick
Imitation
No. Layers
Sub-Net Hid. Dim. Size
Int. time (ms)
MSE
Int. time (ms)
MSE
Int. time (ms)
MSE
2
1500
0.585
122.40
2.993=0.103
41.51
2.91420.049
20.69
3
1500
4.589# 1.193
130.49
4.026=0.12
15.00
5.150=1.407
26.20
4
1500
5,418# 0.746
24.54
6.294-0.939
26.18
5.374+0.382
10.33
5
1500
6.461 0.686
4,40
7.401=1.531
26.56
7.463+1.929
13.16
6
1500
7,529= 0.698
8.17
8.993=2.212
17.16
8.994+2.781
18.27
7
1500
9,426 1.664
4.91
9.858+1 1888
20.39
9.669+1.244
25.76
8
1500
10.636+
5.62
10.73242.111
14.56
10.958=2.518
6.57
5
500
7.159# 1.475
5.62
8.018=2. 109
19.37
7.315+14483
6,22
5
1000
6.972* 1.247
6.04
6.376+0.203
11.02
7.297&1.311
6.37
5
1500
7.031# 1.289
4.40
7.321=1.236
26.56
6.901#0.802
13.16
5
2000
7.787+ 1.363
1.443#1457
14.05
7.77642.514
9.48
5
2500
7.208 1.628
10.92
6.521=0.082
11.66
1.67
5.59
Table 6: Ablation study results of different configurations for the invertible neural network model.
We see that as we increase the number of invertible network layers, the integration times increase, while the hidden dimension size of the sub-networks within the invertible network does not visibly affect the integration times. Overall, the generalisation performance improves as the number of invertible layers are used, up to some number layers. Beyond this number of layers, adding layers does not vary performance significantly. Additionally, the hidden dimension sizes, for the values tested do not greatly vary the generalisation performance.
D PROOFS
Proofs for Propositions 4.1 and 4.2 can be found in Lee 2012)
Propositions 8.19 and 9.6.
Theorem 4.1. Suppose ODEs x' g(x(t)), (t) f(y(t)) related via y(t) F(x(t)),
where F is a diffeomorphism. If the former ODE is asymptotically stable with ne equilibrium points x!...., x then the latter ODE is also asymptotically stable, with equilibrium points F(xt), F(x)..)
Proof. First we show F(x").... F(x".) are equilibrium points of ODE (t) = f(y(t). By y() F(x(()), can write the time derivatives y' F(x)
y'(t) - )(F((()) = dF(((()) = Jr J=(x(t))g(x(()
(8)
dt
14

---

Under review as a conference paper at ICLR 2022
where (x(t)) is the Jacobian of F. F is diffcomorphism and hence invertible over its domain. By the inverse function theorem (Dontchev & Rockafellar 2009), the Jacobian Jr(x(()) is invertible, and furthermore, by the invertible matrix theorem (Horn & Johnson 2012). it has a null-space
containing only the zero vector. Therefore, y'(t) f(F (x())) = (x) (x(((((()) = 0 if and only
g(x(t)) = As g(x (t)) for x E {xi ... x,, J. then we also have f(F(x* (t)) = 0, hence
y" E (F(x)), ..... F(x".) ) gives equilibrium points for y² (t)()))
We now show asymptotically stability of y' - f(y())), by the existence of a Lyapunov function [Lffscettz & Alverson 1962). Vy _ + R. where n is the dimension of y, such that av,(y)
< 0
8t
for all y E IR" {x(x))..... F(x.,) )} and 8t
"v,y") 0 for y E {F(xi)..... ((xx,))) The
existence of such Lyapunov function is necessary and sufficient condition for stability. We assume the candidate function to be By (y)), where Vx is a valid Lyapunov function of the asymptotically stable (t) g(x())), with (xx(x) 8t < for x E R xx. and
(v+(x*) 0 for x {xi,.... x², J. Consider the time derivative of the candidate function:
(y) vV, dy 0Vy fly) 0t 0y ōt dy
(9)
@V OF OF
8x dy 0x g(x)
(10)
x=F X=F- i(y)
@VVx JF (x)()))
(11)
0x
x=F-
@VVx g(x))
(QVx(x)
-xxxxx
0x
x=F (y))
01
x=F (yy
(12)
Equation 11 by the inverse function theorem (Dontchev & Rockafellar 2009). Therefore, our
candidate y is valid Lyapunov function for y' (t) [t(()). Thus, the system y' (t) (t)(t) is asymptotically stable.
E ADDITIONAL IMPLEMENTATION DETAILS
We run all of our experiments on machine with an Intel i7-3770k 3.50GHz processor, 32GB RAM and an NVIDIA GTXI080 GPU, with 8GB vRAM. For all of our experiments, we use the optimizer ADAM with step-size 10-4, except for the experiments in the Latent ODE, which where we use the standard set-up from the Latent-ODE repository (Rubanova et al., 2019). The dynamics models of compared ODEs have the architecture: Input=>dense(Ipput dimen- sions. tan->ttannn>>dese((( 50, 15001550000hhh---ea (aanh)))denne(150,11 (ta00))ttnnn--dessett 150) >dense(1500 10,, (50))=xtann())ddeeee(((( (150, dimensionns--outppu. Except for the Latent-ODE com- parisons where settings from the original repository (Rubanova et al.] 2019) is used. For all of the experiments, except latent ODE experiments where we follow the original set-up, we train for 5000 iterations in total.
For all the experiments where we directly learn a dynamical system, we use an invertible neu- ral network with 5 invertible layers, and sub-networks with one hidden layer of 1500 units. For non-linear base ODEs parameterized with a simple neural networks, we use the architec- ture: Input->deeseeInput >dense(30,Oung) dimensionns)>Outpuus. Additionally, all learned dynamics, both with ours and compared methods, excepted when adhering the original Latent-ODE set-up, were augmented with the same number of additional zeros as original state dimensions, for example 3 dimensional systems were augmented to dimensions. An exception to this is the high-dimensional image rotation problem, where we found that adding half as many augmented states as the original state dimensions was sufficient.
The Lotka- Volterra system used has the dynamics:
x' z'(1) (0.75 0.75y(t))
(13)
/ (t) v(t)5 0.75x(t) - 0.752(t))
(14)
z' 2(t) =(1)(-0.75 0.755(())
(15)
15

---

Under review as a conference paper at ICLR 2022
fort E [0, with initial conditions (2.6,,), (..,
(3, 3, (2, (4, 4,3), (3, 3, 4), (1,1,5)}.
The Lorenz system used has the dynamics:
z' (1) = (0()()) x(1)
(16)
y (1) - =2)) - v()) x(1)
(17)
21(1) =())(() 3 =0(()
8
(18)
fort E [0, 2] with the initial conditions (0.15, 0.15, 0.15).
The Robertson's system used has the dynamics:
(t) 00.4x(()) + v()=(())
(19)
y (t) = 0.04r(t) - 3 x y(t) - (t)((t)(tt)
(20)
=1) = 3 x 10²)(1)²
(21)
for E [0, 120] with the initial conditions 01,0, 0). During training and evaluation, we rescale the data dimensions, and roll out the ODE in log-space.
In the latent- EDe problem setup, an observable time-series is assumed to have latent variables which follow some ODE dynamics, and uses Encoder -> ODE -> Decoder architecture where an ODE is used model the hidden latent dimensions between the Encoder and Decoder. Note that valid ODE is not guaranteed in the space of observable data, but only in the latent dimensions. Our set-up follows the repository given by Rubanova et al. (2019), with the training settings for the Encoder and Decoder architecture below:
Periodic 100: We train the entire model for 500 epochs with Adamax optimizer and an initial learning rate of 10-2 We sub- sample 5% of the original time points and the size of the latent state is 10. The noise weight is set as 0.01 and the total number time points is 100. For the Neural ODE architectures, there is layer in the recognition ODE and layer in the generative ODE, and 100 unit per layers. For the GRU unit there exists 100 units per layer for the GRU update network. All the above settings are exactly the same as the configuration given repository (Rubanova al. 2019).
Periodic 1000:
Settings are the same as Periodic 100, except that the total number of time points is as 1000 to predict for finer time steps.
Human Activity: The model is trained for 200 epochs, with dimensionality of 15 in the latent state. There are 4 layers in the recognition ODE and 2 layers in the generative ODE, and 500 units per layer. The GRU unit has exists 50 units per layer.
ECG: Settings are the same as the classification task of Human Activity, except that we use the ECG Heartbeat data available at Dataset (2018), removing the '0" class.
Rotating Image Sequence: We the first 100 MNIST "3" characters as training and the next "3" characters as test. We create sequence of 44 images for each initial character until we rotate the initial image by 180 degrees. During testing, we integrate to obtain sequence of 440, at 10x data resolution. We obtain latent representations of each image by training convolutional autoencoder to obtain 64 dimension latent vector. We fit the ODE on these latent dimensions, with an ADAM optimizer with learning rate 5 x 10
ODE dimensions and sequence lengths: The following table contains details on the dimensions of the ODE models. For latent ODE models, these are the dimensions of the latent state. We also provide the lengths of the integrated trajectories of the ODEs during inference.
F ADDITIONAL FIGURES
We provide figures for learning additional Lorenz system for t [0, 5), with trajectory at initial condition (-3.1, - 1., 8.15). We see that our method, with a base ODE parameterized by a neural network, can generate trajectories that closely match the ground truth:
16

---

Under review as a conference paper at ICLR 2022
Dimensions ODE Learn
Time points in Trajectory
3d-LV
3
700
Imitate S
2
1000
Pick eube
3
1000
Imitate C
3
1000
Periodic 100
10
100
Periodic 1000
10
1000
Human Activity
15
157
ECG
15
188
Rotating MNIST
64
440
Lorenz
3
800
Robertson
3
500
Figure 6: A learned Lorenz system with the generated trajectory, 10x data resolution, and ground truth.
20
xyz-coordinates
-20
2
time
Figure 7: A learned Lorenz system with the generated trajectory, at 10x data resolution, and ground truth, rolled in time

---

Under review as a conference paper at ICLR 2022
We provide the change in coordinates time, for the trajectory shown in figure 5(a) in the paper:
50
40
30
20
10
-10
0.50
0.75
1.00
1.25
1.50
1.75
2.00
time
Figure 8: The corresponding plot showing the coordinates the time interval of learned Lorenz system over t E [0, 2]. at 10x data resolution, which corresponds to the 3d figure shown as fig 5 (left).
We provide additional plots of trajectories, at different start points, from a learned Lotka- Volterra system. The ground truth data is in blue, while generated trajectory, 10x data resolution, is in red.
Figure 9: We see that trajectories from the learned Volteraolttrra system, in red, closely matches the ground truth, in blue.
We provide an additional figure for trajectories generated at unseen starting points after being trained on the "imitation C" training data. The four generated trajectories are in red, while the ground truths are in blue. Our generated trajectories match the ground truth, and accurately capture the motion of drawing "C" character.
18

---

Under review as a conference paper at ICLR 2022
65
60
45
40
35
100
75
25
-80
-70
25
-60
-50
50
-40
-5
33
110
-2
Figure 10: Robot motion trajectories in red, that imitate drawing "C" character. The ground truth is given blue.
G LICENSES FOR PACKAGES
Common scientific packages used our code include: (i) Numpy (Harris et al. 2020) (BSD license), for general linear algebra and miscellaneous math operations (ii) Matplotlib (Huntrr 2007 (BSD compatible custom license), for plotting figures.
More specialized packages used include (i) FrEIA (Ariizoone et al. 2019) (MIT license), for invert- ible neural networks; (ii) TorchDiffEq (Chen et al. (2018) license,, for differentiable numerical integrators; (iii) Pytorch (Paszke al. (019) (BSD license), for optimisation and automatic differen- tiation; (iv) Latent-ODE (Rubanova et al. 2019) (MIT license), for latent ODE implementation.
19


Reviews:
[{"review_id": "jvY4vFj1TWL", "reviewer": "Reviewer_HKa2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I don’t agree with the dichotomy presented in paragraph 1 of the related work: \n\n“Our proposed approach improves the learning of the underlying ODE, and is compatible with models that incorporate learnable ODEs. We note that the term “neural ODE” has typically been used in the literature to refer to neural networks that incorporate ODEs, including the original work in Chen et al. (2018). However, “neural ODE” has occasionally been used to refer to an ODE with dynamics parameterized by a neural network (Norcliffe et al., 2021). We follow the former convention.”\n\nMy understanding is “Neural ODE” refers to ODEs parameterized by neural networks. This includes the case where such an ODE is incorporated into a deep learning model. See the second paragraph of [1]\n\nRelated work: I think [1], [2], [3], [4] should be included in the related work section, as they all propose methods to improve the efficiency of neural ODEs. [4] is very recent, so maybe it’s not necessary to cite it, but it seems very related to this paper.\n\nCan you more thoroughly explain the benefits of asymptotically stable Neural ODEs? How do these models maintain expressivity while being asymptotically stable? Are there example of Neural ODEs without diffeomorphisms that can be empirically demonstrated to be asymptotically unstable?\n\nIt doesn’t seem that this definition of asymptotic stability in Definition 4.3 permits periodic solutions, yet there are experiments with these dynamics. Can you explain this?\n\nA principal advantage of composing a diffeomorphism with an ODE, as outlined in Section 4.3, rather than an arbitrary function, is that solution curves can be obtained without evaluating the jacobian of the diffeomorphism. I’m not sure why this is a big advantage. \n- Multiplying by the Jacobian of the diffeomorphism is relatively cheap (2x more expensive than evaluating F) via vector-jacobian products. Do quantities besides a vector product need to be computed (e.g. the trace of the jacobian?)\n- If we replaced F with some non-invertible neural network, we would also not require evaluating its jacobian to compute predictions from the model. We would need two separate neural networks (for pre- and post-processing, respectively) to match the expressivity, but otherwise I don’t see why we need to maintain the invertibility beyond the fact that we’d like the entire function to be an ODE.\n\nHow do we make use of the fact that the entire model is one well-defined vector field / ODE? The principal advantage seems to be for some time series problems, where the whole function is now an ODE.\n\nWhy is there a requirement that the model be entirely composed of one ODE? Many neural ODE models incorporate preprocessing steps, and the pre- and post- processing steps are not inverses of each other (e.g. one might be downsampling). Training continuous normalizing flows might be one application where a diffeomorphism is required, and experiments on this task would be interesting.\n\nIt isn’t well-characterized when it is suitable to use a Linear Base ODE vs. a neural network ODE. When is the linear base ODE expressive enough? Are there tractable components that can be composed with sufficient expressivity, instead of having to resort to a neural network ODE?\n\nAre there more fine-grained measures of time-cost of different models? For example, function evaluations from the numerical integrator can be measured as a more robust measure than the total seconds elapsed. Does Table 2 include results from models in Rubanova et al. 2019? Or is it just Linear ODE models integrated with different solvers?\n\nThe experiments are not extensive enough. Perhaps it might be worth trying some of the tasks investigated in [4]?\n\nTypos:\nSection 2:\n- “normlizing flows” (second paragraph)\nSection 4.1: \n- “If mapping F” (second paragraph)\nSection 4.2:\n- “and get the inverse with no further labor” (first paragraph)\n\n[1]: “Learning Differential Equations that are Easy to Solve”, Kelly et al. (arxiv.org/abs/2007.04504)\n[2]: “How to train your Neural ODE”, Finlay et al. (​​arxiv.org/abs/2002.02798)\n[3]: “Opening the Black Box: Accelerating Neural Differential Equations by Regularizing Internal Solver Heuristics”, Pal et al. (arxiv.org/abs/2105.03918)\n[4]: “Neural Flows: Efficient Alternative to Neural ODEs”, Biloš et al. (arxiv.org/abs/2110.13040)\n", "summary_of_the_paper": "ODEs parameterized by neural networks can be slow when learning systems from long sequences. This paper proposes learning an ODE with a fast diffeomorphism, so that the ODE can be a simpler function that’s more efficient to numerically integrate, while preserving expressivity. ", "summary_of_the_review": "The paper does not explain clearly enough the benefits of the proposed approach over regular Neural ODE models, nor thoroughly validate their claims empirically.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "CLPk5Mi9WjH", "reviewer": "Reviewer_9Lrk", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper addresses a very timely topic and shows a very interesting connection between differential geometry and ODE systems. I find this theoretical contribution (or rather perspective) highly significant. Considering the convincing experimental results, I believe the paper should be accepted. For the first round of reviewing, I'm giving a score of 6 but would happily increase it if my following concerns are addressed:\n\n- The connection with the latent neural ODEs is missing. One could even present the proposed methodology as a special case of latent neural ODEs; therefore, showing and justifying the differences are important. Similarly, I would like to see empirical comparisons with latent neural ODEs to be written more clearly. As such, all tables are presented as if the competing method is the same across tables.\n- I'm not sure if the informal definition of \"stiff ODEs\" in Sec 4.6. is correct. No my knowledge, a stiff ODE system does not necessarily rapidly vary along a dimension but rather causes numerical solvers to be stuck at certain points in the space.\n- Are the number of model parameters comparable? Since the presented approach utilizes two networks, I believe the comparison should be done fairly. This is questionable looking at Sec.D. Similarly, it is questionable that the adaptive step solvers are used with correct tolerance values.\n- What are the initial values for integration? This is mentioned nowhere in the main paper. Also, the effect of the noise on the initial values should be discussed. \n- I wonder how much of the \"load\" is on the shoulders of the base ODE system. This might not be of vital importance but \"disentangling\" the contributions of the invertible layer and the ODE system might be beneficial.\n\nMinor comments: \n- The paper is written clearly. I particularly enjoyed the preliminaries section and how the concepts are explained in sufficient detail. That being said, the writing can be improved:\n  - Section 4.3 is a little repetitive. In particular, the third and fourth paragraph partially repeat the information above.\n  - To my knowledge, he phrase \"integral curve\" is not commonly used in neural ODE community. Giving its formal definition or synonyms would help the reader.\n  - Section 4.4 is highly repetitive (can be even completely removed). Also, some title along the line \"summary of our methodology\" could be better since the word \"result\" typically refers to experimental findings.\n  - The last sentence above Def. 4.2 reads strange. I would suggest authors revisit this paragraph for coherence.\n- Figure 3 seems extremely crowded (especially the right one). \n- The long sentences (especially in the table captions and the appendix) should be revised.\n- The last paragraph in page 8 is extremely long and requires a better structure.\n- It is a little difficult to see how significant the differences in Table1 are. It would be nice to see a visualization of forward trajectories over time.", "summary_of_the_paper": "This paper presents an interesting perspective that connects ordinary differential equations with diffeomorphisms. More specifically, possibly complicated and thus difficult-to-learn ODE dynamics are expressed as a morphed version of a simple \"base\" ODE system. The authors propose to learn the base dynamics alongside with the corresponding diffeomorphism. As experimentally shown, this approach leads to faster training, somewhat increases accuracy and allows for the long-term behaviour analysis of certain types of base ODE systems.", "summary_of_the_review": "Since the paper shows interesting links between seemingly unrelated topics, I find the contribution significant and suggest an accept although the writing can be improved and the model can be investigated in more detail.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "7vmlnoh_pEz", "reviewer": "Reviewer_CkYu", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The strengths of the paper are:\n- it follows a series of work in this direction of ODE estimation, which has been validated as an important direction for ML\n- it is mathematically sound\n- well written\n\nThe weaknesses are:\n- from the limited experiments presented it is not clear how applicable the method is for complex practical problems\n- existence of an diffeomorphism is a critical aspect of the paper, it is not clear if such a tractable mapping that can be estimated using using a NN can really exist\n- related to the above, the choice of the base ODE is another aspect that might be non-trivial", "summary_of_the_paper": "The paper has been well summarized by the authors in their abstract. The following is taken directly from there:\n\nLearning ordinary differential equations (ODEs), where a flexible function approximator (often a neural network) is used to estimate the system dynamics, given as a time derivative has become popular after the publication of the paper by Chen et. al. at NeurIPS 2018.. However, these integrators can be unsatisfactorily slow and unstable when learning systems of ODEs from long sequences. In this work, the authors propose to learn an ODE of interest from data by viewing its dynamics as a vector field related to another base vector field via a diffeomorphism. By learning both the diffeomorphism and the dynamics of the base ODE, the authors provide an approach to offload some of the complexity in modelling the dynamics directly on to learning the diffeomorphism. Consequently, by restricting the base ODE to be amenable to integration, we can speed up and improve the robustness of integrating trajectories from the learned system.", "summary_of_the_review": "Clearly, the paper adds to the existing literature and is interesting. It is well written and mathematically sound.  It is however, unclear how general the approach is and that brings into question the significance of the contribution.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "2fWBezbXWUa", "reviewer": "Reviewer_HKf9", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Learning an extra coordinate transformation is a very interesting idea and shared by important recent works. It is also closely related to encoder and decoder (although this is not discussed). I feel the authors have a great idea that is definitely worth pursuing, although the paper in its current form could use some significant improvements.\n\nFirst of all, putting linear and NODE-base-vector-field together appears a little unnatural to me. Is a linear base recommended or not? I would appreciate it more if they could be argued to be complementary, and a criterion is given as such when to use which. Here are some of my understandings of their difference in scopes and applicability:\n\nLet me first discuss the linear base. In this case, despite of the extra differomorphism, the topology of the learned dynamics is still going to be linear. This is very restricted. Topological conjugacy can change the quantitative appearance of dynamics but not its structures, such as the topology of limit sets. All examples in 5.1 reflect this fact. Can the author use linear base + diffeomorphism to capture both the libration and circulation regions of a pendulum, as well as the separatrix inbetween? If not, I think it is important not to exaggerate the representation power of linear bases.\n\nAn addition comment on the linear base case is, the stability criterion for autonomous linear systems is well-known, and stability is nontrivial only for truly nonlinear problems. I would rather see more details of the core treatments and discussions, instead of loose parts that try to make the paper appear more mathematical (e.g., Definition 4.2, 4.3, Theorem 4.1, and all the parts involving manifolds, e.g., 3.2, Figure 2, Definition 4.1, Proposition 4.1, as the work doesn’t really have a manifold component).\n\nThe second approach, `non-linear neural network base’, should work for general problems though. This base is not new, and the contribution is to add a coordinate transform layer. For this part, it would be great if two matters are better discussed: (1) uniqueness. Now both the differmorphism and the base dynamics are learned, and given one differmorphism there is a base dynamics. Which one will be chosen? There will be implicit bias imposed by both the architecture and the training methodology. I think theoretically understanding this would be challenging, but at least some empirical investigation would be helpful. (2) Why is this better than just learning the vector field? The authors did mention on page 5 that this is computationally more advantageous, but I’m sorry that I’m not very convinced. \nIn addition, I’d like to better understand how the new method scales with the dimension. As this paper proposes an improvement of NODE, comparisons with NODE-based approaches on machine learning baselines would be very helpful.\n\nAdditional points:\n* It seems the Lorenz example in Fig.5 is not for a chaotic trajectory, so it diffuses the point of testing on a Lorenz system.\n* Some important NODE references are given, but the expansive data-driven learning differential equation literature is almost completely missing.\n* Eq.(2): $ODESolve(f_\\omega, y_{t_0}, t_e)$ should be $ODESolve(f_\\omega, y_{t_0}, t_0, t_e)$ or $ODESolve(f_\\omega, y_{t_0}, t_e-t_0)$.\n* Right before eq.(2): Euler’s method is a Runge-Kutta method.\n", "summary_of_the_paper": "In order to better learn differential equations that produce discrete data, this paper proposes to learn a coordinate transform (i.e. the `diffeomorphism’ in the title) in addition to learning the vector field. It is argued that, after this coordinate transformation, one can either learn a linear ODE or learn a neural network parameterized ODE, and obtain improved efficiency and robustness.", "summary_of_the_review": "The \"linear base\" part and the \"non-linear neural network base\" part are disjoint. Unlike the current wordings imply, I think the former is rather restrictive. On the other hand, the latter is interesting but needs more work. Although I cannot recommend the acceptance of this paper in its current form, I encourage the authors to strengthen the latter part and consult more with the literature.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This submission proposes a new manner to learn ordinary differential equations, aiming to improve their efficiency. While judging it interesting, the reviewers are quite split on this work. Overall there was no strong consensus to accept, nor anyone willing to champion this work.

The main stated weaknesses are

- The reliance on the existence of a diffeomorphism (and its choice in the method)
- The choice of the base and its expressiveness
- A somewhat limited experimental section, not indicating strongly how amenable this would be to more complex problems.
Decision: Reject