Recognition Models to Learn Dynamics from Partial Observations with Neural ODEs
Authors: Mona Buisson-Fenet
Valery Morgenthaler
Sebastian Trimpe
Florent Di Meglio
Venue: Accepted by TMLR ()

Abstract: Identifying dynamical systems from experimental data is a notably difficult task. Prior knowledge generally helps, but the extent of this knowledge varies with the application, and customized models are often needed. Neural ordinary differential equations can be written as a flexible framework for system identification and can incorporate a broad spectrum of physical insight, giving physical interpretability to the resulting latent space. In the case of partial observations, however, the data points cannot directly be mapped to the latent state of the ODE. Hence, we propose to design recognition models, in particular inspired by nonlinear observer theory, to link the partial observations to the latent state. We demonstrate the performance of the proposed approach on numerical simulations and on an experimental dataset from a robotic exoskeleton.

Published in Transactions on Machine Learning Research (01/2023)
Recognition Models to Learn Dynamics from Partial Observations with Neural ODEs
Mona Buisson-Fence Fenett
Ansys Research Team, Ansys Prance
Centre Automatique Systèmes, Mines Paris PSL University
Institute for Data Science in Mechanical Engineering RWTH Aachen Universily
Valery Morgenthaler
mudery.norggnaallerroonnssccm
Ansys Research Team, Ansys France
Sebastian Trimpe
Institute for Data Science in Mechanical Engineering - RWTH Aachen University
Florent Di Meglio
florent. di Sas___eglliiin Treelio@minesporisso.ooo
Centre Automatique et Systémes, Mines Paris PSL University
Reviewed OpenReview: https: openreview. net/ net/forum? id= Ld=
Abstract
Identifying dynamical systems from experimental data is notably difficult task. Prior knowledge generally helps, but the extent of this knowledge varies with the application, and customized models are often needed. Neural ordinary differential equations can be written as flexible framework for system identification and can incorporate broad spectrum of physical insight, giving physical interpretability to the resulting latent space. In the case of partial observations, however, the data points cannot directly be mapped to the latent state of the ODE. Hence, we propose to design recognition models, in particular inspired by nonlinear observer theory, to link the partial observations to the latent state. We demonstrate the performance of the proposed approach numerical simulations and on experimental dataset from robotic exoskeleton.
1 Introduction
Predicting the behavior of complex systems is of great importance in many fields. In engineering, for instance, designing controllers for robotic systems requires accurate predictions of their evolution. The dynamic behavior of such systems often follows a certain structure. Mathematically, this structure is captured by differential equations, e.g., the laws of physics. However, even an accurate model cannot account for all aspects a physical phenomenon, and physical parameters can only be measured with uncertainty. Data- driven methods aim to enhance our predictive capabilities for complex systems based on experimental data.
We focus on dynamical systems and design an end-to-end method for learning them from experimental data. We investigate State-Space Models (SSMs), which common in system theory, many modern control synthesis methods build on them and the states are often amenable to physical interpretation. For many systems of interest, some degree of prior knowledge is available. It is desirable to include this knowledge into the SSM. To this end, we consider neural ordinary differential equations (NODEs), which were introduced by Chen et al. (2018) and have since sparked significant interest, e.g., Zhong et al. (2020); Rubanova et al. (2019). Their aim to approximate vector field that generates the observed data following a continuous- time ODE with a neural network. Their formulation general enough avoid needing new design for each system, but also enforce wide range of physical insight, allowing for meaningful and interpretable

---

Published in Transactions on Machine Learning Research (01/2023)
Black-box ODEs
Structured ODEs
Parametric models
Increasing prior knowledge
Regularizing priors Extra terms in the cost function
Structural priors
Constraints or form of the optimization problem
Figure 1: SSMs can include broad spectrum of physical knowledge. On the left, purely data-based formulations such as latent NODEs are general but tend to violate physical principles and have trouble generalizing. On the right, parametric models can be identified from data; they extrapolate well but are system-specific and require expert knowledge. One can bridge this gap by including the available physical knowledge in an NODE formulation (2), in particular "regularizing" priors (extra terms in the cost function) or "structural" priors (constraints form of the optimization problem).
model. Specific approaches have been proposed to include different priors, which we briefly recall; we present a unified view and include them in the proposed ead-to-end framework.
Learning an SSM satisfying these priors amounts to learning the dynamics in specific set of coordinates. However, experimental data is lypically only partial, as not all of these coordinates, slales, are measured. This is a common problem in machine learning, where the existence of an underlying state that can explain the data is often assumed. In the systems and control community, estimating this underlying state from partial observations known as state estimation or observer design. An observer an algorithm that estimates the full latent state given history of partial measurements and control inputs Bernard (2019); Bernard al. (2022). While observer design provides the theoretical framework for latent state estimation, such convergence guarantees of the estimate to the true state, it has not received much attention in the machine learning communily. Hence, we propose to leverage concepts from nonlinear observer design to learn NODEs with physical priors from partial observations.
We design so-called recognition models to map the partial observations to the latent state. We discuss several Kravaaais/Ttnnbeeereeee (KKL)
approaches, in particular based on lype nonlinear observers called observers (Kazantzis & Kravaris, 1998; Andrieu & Praly, 2006). We show that the KKL-based recognition models perform well and have desirable properties, e.g., a given size for the internal state. Such recognition models can then be embedded in the NODE formulation or any other optimization- system identification a algorithm. Our main contributions can be summarized as follows:
We formulate structured NODEs as a flexible framework for learning dynamics from partial observations, which enables enforcing a broad spectrum of physical knowledge;
We introduce recognition models to link observations and latent state, then propose several forms based on nonlinear observer design;
We compare the proposed recognition models in simulation benchmark;
We apply the proposed approach to experimental dataset obtained robotic exoskeleton, illustrating the possibility of lcarning a physically sound model of complex dynamics from real-world data.
Combining these yields an end-to-end framework for learning physical systems from partial observations.
2
Related work
The proposed method is based two research areas: nonlinear observer design and machine learning for dynamical systems. We give an overview of the main trends and the most related methods on these topics.
2.1
System theory
In system theory, many subfields concerned with the study of dynamical systems from experimental data.
2

---

Published in Transactions on Machine Learning Research (01/2023)
System identification The area of system identification aims at finding possible dynamics model given finite amount of partial measurements (L.jun, 1987; Nelles, 2001; Schoukens & Tjung, 2019). For linear systems, a suitable set of system matrices can be identified using subspace methods (Viberg, 1995). For nonlincar systems, most state-of-tho-art techniques aim at estimating the variables of a given parametric model using Bayesian parameter estimation (Galioto & Gorodetsky, 2020) or optimization- (Schittkowski, 2002; Raue et al., 2013; Villaverde et al., 2021), or a decomposition of its dynamics on a suitable basis of functions (Sjöberg et al., 1995). These classical methods tend to be system-specific: they require expert knowledge construct parametric model precisely pick the hypothesis class in which it will be approximated. NODEs are general tool for system identification in case no parametric model is available, in which a broad range of physical knowledge can be included by adapting the formulation.
met
Observer design When identifying a state-space model from partial observations, the unknown latent state must be estimated. This is the objective of state observers or estimators, which infer the state from observations by designing an auxiliary system driven by the measurement (see Bernard (2019); Bernard et al. (2022) for an overview). Observers often assume an accurate dynamics model, but designs that can deal with imperfect models are also available. In that case, the unknown parts of the dynamics can be overridden through high-gain sliding-mode designs to enable convergence (Buisson-Fenet Fenet et al., 2021; Shtessel et al., 2016). Otherwise, the unknown parameters can be seen extra states with constant dynamics, and extended state observers can be designed, such that the estimated state and parameters converge asymptotically (Praly et al., 2006). Some concepts from observer theory can be leveraged to improve upon existing approaches for learning dynamics from partial observations, which require estimating the unknown latent state.
2.2 Learning dynamical systems
Learning dynamical systems from data is also investigated in machine learning (Legaard et al., 2021; Nguyen- Tuong & Peters, 2011). We focus on settings considered realistic in system identification, ie., methods that allow for control and partial observations, and can ensure certain physical properties.
Physics-aware models The dynamics models obtained from machine learning often struggle with gener- alization and do not verify important physical principles such as conservation laws. Therefore, there have been efforts to bring together machine learning and first principles to learn physics-aware models; see Wang & Yu (2021) for overview of these efforts in deep learning. In general, there are two takes on including physical knowledge in data-driven models, as illustrated in Fig. 1. On the one hand, "regularizing" priors can be included by adding terms to the cost function to penalize certain aspects. The most common case is when a prior model of the system is available from first principles. We can then learn the residuals of this prior model, iee, the difference between the prior predictions and the observations, while penalizing the norm of the residual model to correct the prior only as much as necessary. This is investigated in Yin et al. (2021); Mchta et al. (2020) for full state obscrvations. Other quantities can be known priori and enforced similarly, such as the total energy of the system (Eichelsdörfer et al., 2021) or stability through a Lyapunov-inspired cost function (Schlaginhaufen et al., 2021). On the other hand, structural properties can be enforced by constraints or a specific form of the optimization problem. This yields a harder problem, but improves the performance and interpretability of the model. This line of work originates from Lutter et al. (2019); Greydanus et al. (2019); Cranmer et al. (2020) (see Zhong et al. (2021) for an overview) and has been extended to NODEs for Hamiltonian and Hami-tamnl systems (Zhong et al., 2020; Massaroli et al., 2020a; Zakwan el al., 2022), but also to enforce more general energy-based structures (Manek & Zico Kolter, 2019; Course et al., 2020) or the rules of electromagnetism (Zhu et al., 2019). However, little previous work on NODEs assumes partial and noisy measurements of system trajectorics.
There exist. various other methods to learn physics-aware dynamics models, such as Bayesian approaches, in which prior knowledge can be enforced in the form the kernel (Wu et al., 2019), by structural constraints (Geist & Trimpe, 2021; Rath et al., 2021; Ensinger al., 2022), or by estimating the variables of a parametric model (Galioto& & Gorodetsky, 2020). In this paper, we focus on NODEs, which leverage the predictive power of deep learning while enforcing broad range of physical knowledge in the problem formulation.
3

---

Published in Transactions on Machine Learning Research (01/2023)
Partial observations Most NODE frameworks for dynamical systems assume full state measurements. Partial observations greatly increase the complexity of the problem: the latent state is unknown, leading to a large number of possible state-space representations. In this case, the question of linking the observations with the latent state needs to be tackled. In Bayesian approaches, the distribution over the initial state can be directly conditioned on the first observations then approximated by a so-called recognition model (Eleftheriadis al., 2017; Doerr al., 2017; 2018). Such an approach has also been used for Bayesian extensions of NODEs, where the NODE describes the dynamics of the latent state while the distribution of the initial latent variable given the observations and vice versa are approximated by encoder and decoder networks (Yildiz et al., 2019; Norcliffe et al., 2021a). The encoder network, which links observations to latent state by & deterministic mapping or by approximating the conditional distribution, can also be a Recurrent Neural Network (RNN) (Rubanova et. al., 2019; Doyeon et al., 2021; de Brouwer et al., 2019; Rubanova et al., 2019) or an autoencoder (Bakarji et al., 2022). The particular case in which the latent ODE linear and evolves according the Koopman operator (that can be jointly approximated) is investigated in Lusch et al. (2018); Bevanda al. (2021). In general, little insight into the desired latent representation is provided. This leads to difficulties for the obtained models generalize, but also with their interpretability: often in a control environment, the states should have a physical meaning. Therefore, we propose to learn recognition model that maps the observations to the latent state, while enforcing physical knowledge in the latent space.
3 Problem statement
Consider a general time nonlinear system
(tt) = f(x(t), u(t)) xx0)= = xD
y(t) = h(x((t), u(t) + e(t)
(1)
where xt() E Rdd is the state, u(l) E Rd. is the control input, v(() E IR is the measured output, and f. h are the true dynamics and measurement functions, assumed continuously differentiable. We denote i(t) the derivative of x w.r.t. time 6, and generally omit the time dependency. We only have access to partial measurements corrupted by noise e, the control input u, and the measurement function h: the dynamics f and the state x are unknown. We assume that the solutions to (1) are well-defined and aim to estimate f.
The aim of NODEs (Chen et al., 2018) is learn a vector field that generates the data through an ODE, possibly up to input and output transformation; see Massaroli et al. (2020b) for an overview. While the interpretation of this formulation for general machine learning tasks remains open, it is very natural for SSMs: it amounts to approximating the dynamics with a neural network. However, there is no unifying framework for applying NODEs to dynamical systems in realistic settings, i.e., with partial and noisy trajectory data, with a control input, and using all available physical knowledge; we present one this paper.
Assume we have access to N mcasured trajectories indexed by j, denoted y' and sampled at times t, E (0..... - 1}. We approximate the true dynamics f with a neural network So of weights 0. If the initial conditions za are known, learning fe can be formulated the following optimization problem:
n
min 0
(1.) - (1:))1
(2)
2dynN
j=1 a ʃo 1o(,2u²)
s.l.
v¹ hhoo,, u²)
x (0) = 2/1-
where the constraint is valid for all j E (1. Several methods have been proposed to compute the gradient of (2); see Schittkowski (2002); Alexe & Sandu (2009); Massaroli et al. (2020b) for details. We opt for automatic differentiation through the numerical solver (torchdiffeq by Chen et al. (2018)).
This problem is not well-posod: for a given state trajectory, there cxist several statc-space representations f that can generate the data. This is known as the unidentifiability problem (Alice et al., 2021). The key problems to obtain meaningful solutions to (2) are (i) enforcing physical knowledge to learn state-space
4

---

Published in Transactions on Machine Learning Research (01/2023)
representation that not only explains the data, but is also physically meaningful, and (ii) dealing with partial observations, i.e., unknown latent state and in particular unknown To. Problem (i) has been addressed in the literature for some particular cases, as presented in Sec. 2.2, by including the available physical knowledge in the form of "regularizing" or "hard" priors (sec Fig. 1). We denote the general approach of adapting (2) to build physics-aware NODEs as structured NODEs, and apply it to examples with varying prior knowledge.
Problem (ii) remains largely open. For fixed fo and u, each prediction made during training for given measured trajectory is determined by the corresponding zn; hence, estimating this initial state is critical. We tackle (ii) by designing recognition models in Sec. 4, which is main technical contribution. We combine them with structured NODEs to simultancously address (i) and (ii). This yields an end-to-end framework for system identification based on physical knowledge and partial, noisy observations, a common setting in practice; we apply to several examples in Sec. While some of the components needed for this framework exist. the literature, e.g.; on building physics-aware NODEs (Sec. 2.2), the vision of recognition models, the presentation in a unified framework and the application to relevant practical cases are novel.
Remark 1 The considered setting is generally regarded as realistic in system identification, since y is mea- sured by sensors, and u is chosen by the user, who often knows which part of the state is being measured. However, if that is not the case, it always possible to truin an output nelwork he jointly with fa.
4 Recognition models
In the case of partial observations, the initial condition To in (2) is unknown and needs to be estimated jointly with fo. Estimating To from partial observations is directly related to state ostimation: while observers run forward in time to estimate the state asymptotically, we formulate this recoguition problem as running backward in time to estimate the initial condition. Therefore, the lens of observer design is well suited for investigating recognition models, though it has not been often considered. For example, whether the state can be estimated from the output is precise notion in system theory called obscrvability (Bernard, 2019):
Definition 1 Initial conditions To, In are uniformly distinguishable in time te if for any input u : + Rd
ya = (f,==), V Vt [0, =.
(3)
where ya (resp. )) is the output of (1) given input u and initial condition Ia (resp. zt). System (1) is observable (in Le) if all initial conditions are uniformly distinguishable.
Hence, if (2) is observable, then x0) is uniquely determined by y and u over [0, te] for to large enough. This assumption is necessary, otherwise there is no hope of learning fo from the observations only.
In system identification, the unknown initial condition is usually optimized directly as free variable: it needs to be optimized again for each new trajectory and cannot be used as such for prediction. Instead, we propose to estimate it from the observations by learning a recognition model the that links the output to the initial state. We design th as a neural network and denote its input E(tc), to be described the following. Concatenating the weights of fo and vh into leads to the modified problem:
N n- (v(( - nو (1:)11
min 0
ΣΣ
(4)
ndy
j=l
s.t.
ل fo ,w)'
y²= hhaw, u²)
x (0) = voo (1.)).
4.1 General approaches
Some recognition methods have been proposed in the literature, not necessarily for system identification with NODEs, rather for probabilistic (Doerr et al., 2017) or generative Yildiz et al. (2019) models. We draw inspiration from them and rewrite them to fit. into our general framework, leading to the following.
5

---

Published in Transactions on Machine Learning Research (01/2023)
KKL. (y(te)..... ((to)
f(t
x 1(bn-1)
Measured output
((nn-1)
1/
x
Estimated output
Latent state
x(tn-1)
x (fo)
ODE Solve (Jo: zo, lo...... ................................ (a-1)
10
Le
1--1
Figure 2: Illustration of the method: the KKL observer runs backward over the observations on [te, 0], the initial latent state is estimated, then the NODE runs forward and predicts the following trajectory.
Direct method The most straightforward approach is to stack the observations and learn a mapping from
a(te)= = Vor. ==
(5)
to the initial latent state. For nonautonomous systems, the first inputs should also be taken into account, which yields a(te) " uor.). We denote this the direct method. Variants of this approach has been used for approximating the distribution over the initial state conditioned on Mo.., : c.g., for joint inference and learning of Gaussian process state-space models (Eleftheriadis et al., 2017: Docrr et al., 2017; 2018). Augmentation strategies for NODEs (Dupont et al., 2019; Massaroli et al., 2020b; Chalvidal et al., 2021; Norclife et al., 2021b) are often particular cases with te = O. However, for many nonlinear systemss, this is too little information to estimate x(0). There are few works on NODE-basod system identification from partial observations, some of which train a recognition model from -- (Ayed al., 2020; Yildiz et al., 2019; Norcliffe et al., 2021a), or learn the dynamics of Vo.. (Schlaginhaufen et al., 2021).
As justified by the observability assumption, for te large enough this is all the information needed to estimate =00). However, for large te, the input dimension becomes arbitrarily high, so that optimizing the is more difficult. Also, the observations are not preprocessed in any way, though they may be noisy.
Recurrent recognition models Latent NODEs (Chen et al., 2018; Rubanova et al., 2019) also use a recognition model to estimate the initial latent state from observations, though this may not be the state of an SSM. This recognition model is a Recurrent Neural Network (RNN) in Chen al. (2018) and a et al. (2019). These methods filter the information contained in (Yo.,> (oot,) in backward time then feed it to the recognition network vhe which is trained jointly with the (ODE-)RNN We consider this baseline in the numerical results: we combine a Gated Recurrent Unit (GRU) of internal dimension d2, run in backward time so that a(tc) is the output of the GRU, and an output network tha We denote this method from Chen et al. (2018) as RNN+
RNN combined with sccond NODE model in Rubanova
We now propose a novel type of recognition model based nonlinear observer design, leading to different choices of a(tc). See Table for a summary of the proposed recognition methods.
4.2 Kase recognition models
Nonlinear observer design is concerned with the estimation of the state of noulinear SSMs from partial observations; see e.g., Bernard et al. (2022); Bernard (2019) for an overview. A particular method that has recently gained interest is the Kazantzis-Kravariss/ Luenberger (KKL) observer (Kazantzis & Kravaris, 1998; Andrieu & Praly, 2006). Intuitively, KKL observers rely on building a linear filter of the measurement: an auxiliary dynamical system of internal state a with known dimension d2 simulated, taking the measurement as input and filtering it to extract the information it contains. The observer state verifies
= Dz Fy
z(0) 2u
(6)
where d. with d: = dd and 2n an arbitrary initial condition. In this system, y the continuous- time measurement from (1), alternatively interpolation between the discrete observations y(44). The
6

---

Published in Transactions on Machine Learning Research (01/2023)
parameters D and F are chosen such that D is Hurwitz, ie.. all eigenvalues are in the left half-plane, and (D, F) is a controllable pair, i.e. the matrix (F DF Thanks to the stability of (6), the internal state 2 "forgets" its arbitrary initial condition 20 and converges asymptotically to a value that is uniquely determined by the history of y. Under certain conditions, this value uniquely determines, in lurn, the value of the unmeasured state x.
F)) has full rank (Kailath, 1980).
More precisely, if there exists an injective transformation from x to 2, denoted T, and its left inverse T then for any arbitrary 20. the estimate f(t) T (2(t)) converges to z(t). This yields that z(t) (x(t)) for large enough. The existence of T is studied separately for autonomous and nonautonomous systems, under mild assumptions, mainly ætt) E X compact V t and (1) is backward distinguishable, i.c., Definition 1 in backward time": the current state z(1) is uniquely determined by and u over [t le, 4] for some le-
Autonomous systems For autonomous systems, i.e., u = 0, it is shown by Andrieu & Praly (2006) that if the eigenvalues of D have sufficiently large negative real parts, then there exists an injoctive transformation T and its left inverse T such that?
- (1(())((((()))
0,
11000
meaning that (z(t)) with 2t) from (6) is an observer for x(t). However, T cannot be computed analyt- ically in general. Therefore, it has been proposed to learn it from full-state simulations (da Costa Ramos et al., 2020; Buisson- et al., 2022) or to directly learn an output predictor ho from partial observa- tions (Janny et al., 2021). Running the observer (6) backward in time? on 22.0 yields x=0) R T(=(0))) for te large enough. Hence, we propose to train a recognition model z²0) vo(2(0)), where z00) is the result of (6) run backward in time for le from an arbitrary initial condition z(to)- This is further denoted as the KKL method, illustrated in Fig. 2 (sce Table 1 for a summary of the proposed methods).
Nonautonomous systems When extending the previous results to nonautonomous systems, T not only depends on z(t) but also on time, in particular the past values of u, and becomes injective for 2 te with te from the backward distinguishability assumption (Bernard & Andrieu, 2019). In the context of recognition models, this dependency on over [0, 4] can be made explicit by rumning the observer (6) in backward time with a(te) = (z(0), o). This is still denoted as the
then training a recognition model x00) = KKL method, for nonautonomous systems.
If the signal u can be represented as the output of an auxiliary system of inner state w with dimension da it is shown in Spirito et al. (2022) that the static observer
1 = Dz + F
y V
x0) = 20
(8)
leads to the same results as for autonomous systems. The time dependency in T disappears at the cost of a higher dimension: d_ (dy + du (d, + du + 1). This functional approach leads to an alternative recognition model denoted KKLu: =00)- = (=0)) where z is solution of (8) simulated backward in time, and d is chosen large enough to generate u (e.g.; du for a sinusoidal u).
Optimizing D jointly With any KKL-based recognition model, the choice of D in (6) - resp. (8) - is critical since it controls the convergence rale of z. Hence, we propose to optimize D jointly with 0, as in Janny et al. (2021). More details are provided in the supplementary material.
For te large enough, the transformation T approximated by dhe is guaranteed to exist for a known dimension de. The KKL observer also filters the information and provides low-dimensional input to ęhe, which is expected to be casier to train. The RNN-based recognition models are close to learning discrete-time observer with unknown dynamics: this is similar, but provides no theoretical argument for choosing the internal dimension of the RNN, no guarantee for the existence of a recognition model in form of an RNN, no physical interpretation for the behavior of the obtained observer, and leads to many more free parameters.
If the solutions of (1) are unique, e.g., if f is C*, then distinguishability and backward distinguishability are equivalent.
2See the Appendix for technical details.
We simulate z backward in time, so that all samples can be used for data fitting after being inputted to vig.
7

---

Published in Transactions on Machine Learning Research (01/2023)
Method
(((e) for autonomous
2((e) for nonautonomous
te 0 direct
2(0)
(y(0), i(O))
D Dcte
(Muze. Mo:te)
RNN+
GRU over y 24.:
GRU over (Le 44,00)
KKL
KKL over 1 n/a
KKL over "0 concatenated with wo.t. functional KKL over (Kr..o Us
KKLu
Table 1: Summary of the proposed recognition methods for autonomous and nonaulonomout systems, all run backward in time. The recognition model v is trained with z00) = +o(((t))); see Sec. 4 for details.
0.25
04
0.30
0.3
10M58:
2015E
0.10
0.06
0.00
6.0
direct
RNN+
KKL
direct
KNN+
KKL
direct
RNN-
KKL
KKLu
(a) Farthquake
(b) Finzuugg-Nagumo Nagumo
(c) Van der Pol
Figure 3: Learning NODEs without structure, with recognition models direct, RNN+, KKL, and KKLu for
the last, nonantonomous system. We compare the RMSE over the predicted output for 100 test trajectories.
5 Experiments
We demonstrate the ability the proposed approach to learn dynamics from partial observations with varying degrees of prior knowledge, illustrated in Fig. 1. We first compare the different recognition models in combination with NODEs without priors. We then provide extensive case study on the harmonic oscillator, often used in the literature, learning its dynamics from partial measurements with increasing priors. Finally, we apply our approach to real-world, complex use case obtained on a robotic exoskeleton'. All models are evaluated w.r.t. their prediction capabilities: given (yo... M0aa,) for a number of test trajectories, we estimate the initial state, predict the further output, then compute the RMSE over this predicted output.
5.1 Benchmark of recognition models
We demonstrate that the proposed recognition models can estimate the initial condition of system from partial and noisy observations. We simulate three systems, the underlying physical state serves as ground truth. The first a simplified model the dynamics of a two-story building during an carthquake (Winkel, 2017; Karlsson & Svanström, 2019) with dz 4, dy 1. The second is the FitzHugh-Nagumo model, a simplified representation of a spiking neuron subject to constant stimulus (Clairon & Samson, 2020) with d, 2, d, 1. The third is the Van der Pol oscillator with d, 2, dy the supplementary material. The dynamics models are free: this corresponds to the left end of Fig. 1.
More details are provided in
We train ten direct, RNN+ KKL and KKLu recognition models as presented in Sec. 4.2 and in Table 1. The recognition models estimate z0) from the information contained in Mo., for the first, (Uxe, v0) for the Mox..) for the third system. We use N - 50 trajectories of 3 seconds; the output corrupted by Gaussian noise, and the hyperparameters are chosen be coherent between the methods and enable fair comparison. For evaluation, we randomly select 100 test trajectories (also 3s) and compute the RMSE over the predicted output. The results presented in Fig. 3 indicate that the more compressed structure from observer design helps build more effective recognition
socond system, where uo is the value of the stimulus, and (
'Implementttin details are provided the supplementary material, code reproduce the experiments available https: 4open. acience/r///trruuuurae wtteeareeeddddeeeeeeeee RODEs-7C23.
8

---

Published in Transactions on Machine Learning Research (01/2023)
0.7
0.7
0.7
0.6
0.5
0.5
0.5
KMS8
0.4
RMSE
RMSE
0.3
0.3
0.3
0.1
0.1
0.1
10
20
40
60
100
10
20
90
60
100
10
20
40
60
100
(a) direct
(b) RNN+
(c) KKL.
Figure 4: NODE and recognition model for the earthquake model, for different lengths of to (in time steps).
0.5
04
0.4
04
10008
0.3
0.3
BOSSE
(a)
direct
(b) RNN+
(c) KKL
(d) KKLa
Figure 5: NODE and recognition model for the Van der Pol oscillator, for different values o?
model. The direct method with t 0 was run but not shown here since it leads much higher error, it does not verify the observability assumption.
5.1.1 Ablation studies
In the previous benchmark, we choose all hyperparameters such that the comparison fair as possible. For example, the is the same neural network, the dimension of the internal recognition state is the same for the RNN+ and KKL baselines (dz of the standard KKT, for autonomous systems, d, of the functional KKL for nonautonomous systems). We now investigate the impact of two of the main parameters the performance of each approach for the full NODE models: to and the variance of of the Gaussian measurement noise.
For the study on te, we focus on the earthquake system. We run the same experiments as before with te in (5, 10, 20, 40, 60, 100} x At, where At = 0.03 As depicted in Fig. 4, when te is too low becomes difficult to estimate z0) from the information contained in yo.r. the system not necessarily observable. seems the threshold of observability is around te 30At = 0.9s, since the RMSE over the test trajectories stabilizes for higher values. For this system, the KKL method reaches the lowest error and keeps improving for higher values be: the higher fe, the more the observer has converged, the closer the relationship a00) (*()))) is and the casier scems learn vy. For the other methods, f, scems have less influence once the threshold of observability reached since there
notion of convergence over time.
For the study oZ, focus the Van der Pol oscillator. We test for values in {10-s 50-5, 10-4 10-3, 10-3,10-? 10-2, 10-1) and obtain the results in Fig. 5. As expected, the higher the measurement noise vari- ance, the higher the prediction error on the lest trajectories. We again observe a threshold effect, under which further reduction of the noise variance leads to little improvement in the prediction accuracy. Note that for the KKT--baood methods, we optimized once for each noise level from the same initial value, then experiments. D only optimized for specific noise level, then the
used this optimized value for performance is degraded at the others, for which this value might filter too much too little.

---

Published in Transactions on Machine Learning Research (01/2023)
04-
8.
04
-05
-12
-12
-15
4.53
(a) No structure
(b) Hamiltonian
(c) 11 32
(d) Parametric
(e) Extended state-space
Figure 6: Structured NODEs for the harmonic oscillator, with KKL recognition and increasing priors. The true trajectory in green, the prediction of a long trajectory (30s) in blue to illustrate the long-term accuracy. Increasing structure yields a more interpretable, but also more accurate model, except for (e) which solves more open problem (a new frequency is estimated for each trajectory).
Method
(a)
(b)
(c)
(d)
direct
0.040 (0.011)
0.050 (0.033)
0.035 (0.008)
0.029 (0.005)
0.080 (0.041)
RNN+
0.057 (0.014)
0.055 (0.012)
0.048 (0.003)
0.037 (0.006)
0.052 (0.011)
KKL
0.036 (0.010)
0.042 (0.011)
0.036 (0.004)
0.032 (0.003)
0.049 (0.003)
Table 2: Recognition models for the harmonic oscillator. We train ten models for each setting, then compute the median and interquartile range (in parentheses) of the RMSE on the predicted output for hundred test trajectories of 9s. In most cases, KKL recognition leads to more accurate predictions.
5.2 Harmonic oscillator with increasing priors
We now illustrate how NODEs with KKL-based recognition can be combined with physics-aware approaches to cover different degrees of structure, as illustrated in Fig. 1. We simulate an aulonomous harmonic oscillator with unknown frequency:
11 = T2
82 = -2x1,
(9)
where w² > 0 is the unknown froquency of the oscillator and xı is measured, corrupted by Gaussian noise of standard deviation = 0.01. Various designs have been proposed to identify both the state and the model, i.e., the frequency, for example subspace methods, or an extended state-space model with a nonlinear observer; see Praly et al. (2006) and references therein. We demonstrate that our unifying framework can solve this problem while enforcing increasing physical knowledge. The results are illustrated in Table. 2 with a KKL recognition model, w 1, = 20 trajectories of 3 for training and 100 trajectories of 19s for lesting.
First, the NODE is trained without any structure (a) as in (4), which leads to one of many possible state- space models: it fits the observations in 21 but finds another coordinate system for the unmeasured state, enforce Hamiltonian structure (b) by directly learning the Hamiltonian function Ho(r). This leads to a dynamics model again in another coordinate system, but that conserves energy: we learn the dynamics up to a symplectomorphism (Bertalan et al., 2019). We then impose dι = I2 and only learn के = Ho)( (c). This enforces a particular choice of Hamiltonian dynamics, such that the obtained model conserves energy and stays in the physical coordinate system (x) position, #2 velocity). Imposing even more structure, we only optimize the unknown frequency w² jointly with the recognition model, while the rest of the dynamics are considered known (d). Another possibility is to consider the extended state-space model where I3 = w² has constant dynamics (c). In that case, only a recognition model needs to be trained; however, xx0) E R3 including the frequency is estirated for each trajectory, such that this formulation is much more open. Both (d) and (e), which correspond to the right end of the spectrum in Fig. 1, also lead to energy-conservin trajectories in the physical coordinates. The results in Fig. illustrate that NODEs with recognition models can incorporate gradual priors for learning SSMs from partial and noisy observations. Note that standard methods tailored to the harmonic oscillator may perform better, however, they not as general nor flexible.
as expected for general latent NODEs. It also does not conserve energy. Then,
10

---

Published in Transactions on Machine Learning Research (01/2023)
145
Tretcied
0.161
IL144
1
0.140
0L3
0.140
01100
0.2
0.10
04188
0.188
0.3
02
0.4
106
2.5
1.0
0.9
0.2
0.4
0.0
0.5
5.0
0.2
0.5
06
1.0
002
0.8
0.1
0.0
0.0
0.0
-00
0.3
112
066
2.6
2.5
2.0
0.0
0.2
13.4 10.5
D35
1.0
0.0
0.2
0.4
15.
035
1.0
000
02
2.4 0.5
0.5
1.0
(a) Prior model
(b) No structure
(c) NODE with 11/3 22/4
(d) Residual model and (c)
Figure 7: Structured NODEs and KKL recognition on the robotics dataset. After training the NODE on trajectories of 0.2s from a subset of the input frequencies, we also test on 52 trajectories of 28 from other input frequencies, to evaluate generalization capabilities (cut at 18 on plots for visibility). Computing the prediction RMSE for the different structure settings yiclds: 110 (a), 0.72 (b), 0.66 (c), 1.2 (d). We show one such test trajectory (I1 top row, 24 bottom row) from an unknown initial condition.
5.3 Experimental dataset from robotic exoskeleton
We demonstrate the performance of the proposed framework on a real-world dataset. We use a set of measurements collected from a robotic exoskeleton at Wandercraft, presented in (Vigne, 2021) and Fig. 8. This robot features mechanical deformations at weak points of the structure that are neither captured by Computer-Assiste Design modeling nor measured by encoders. These deformations, when measured by motion capture device, can be shown to account for significant errors in foot placement. Further, they exhibit nonlinear spring-like dynamics that complicate control design. The dataset is obtained by fixing the robot basin to a wall and sending sinusoidal excitation to the front hip motor at different frequencies. The sagittal hip angle is measured by an encoder, while the angular velocity of the thigh measured by a gyroscope. In (Vigne, 2021), first results are obtained using linear system identification: the observed deformation is modeled as a linear spring in the hip, and this model is linearized around an equilibrium point, then its parameters are identified. These estimates are sufficient for tuning robust controller to compensate for the deformation", We aim to learn more accurale model of this dynamical system of dimension de 4, where (1), by identifying the nonlinear deformation terms.
Figure 8: Robotic exoskeleton by Wandercraft.
We investigate three settings: no structure, imposing á = 22 and 2t3 = 24, and learning the residual of the prior linear model on top of this structure as Yin el al. (2021), by using f fm + fo as the dynamics, where fisn is the linear prior. In each setting, we learn from N 265 trajectories of length 0.2s in subset of input frequencies. We use a recognition model with f, = 0.1s One short test trajectory in the trained frequency regime is shown in the supplementary material and illustrates the data fit. One longer test trajectory with input frequency outside of the training regime is shown in Fig. 7 and illustrates the generalization capabilities in all three settings and for the linear prior model.
The obtained results demonstrate that structured NODEs with recognition models can identify real-world nonlinear systems from partial and noisy observations. The learned models can fit data from a complex nonlinear system excited with different input frequencies, and somewhat generalize to unseen frequencies.
5 More details provided (Vigne, 2021) and in the supplementary material.
11

---

Published in Transactions on Machine Learning Research (01/2023)
Short rollouts in trained regime
Long rollouts with unseen frequencies
Long rollouts with EKF;
Long rollouts with EKF:
Recognition model
y = (01, v.4)
y - 21
direct RNN+ KKL
0.11
0.58
0.12
0.44
0.15
0,56
0.15
0.37
0.17
0.60
0.12
0.37
KKLu
0.18
0.65
0.11
0.34
Table 3: RMSE on test trajectories for the roboties dataset while imposing 41/3 22/11- We trained only one model per method; hence, the results only illustrate that all methods are comparable.
The predictions are not perfect, but much more accurate than those of the prior model, as seen in Fig. 17: this is enough to be used closed-loop tasks such as control or monitoring. Imposing $1/3 = 22/4 leads similar performance as without structure, but a physically meaningful state-space representation that can be interpreted in terms of position and velocity. Due to the inaccurate predictions of the prior model, learning its residuals leads to lower performance.
With all levels of prior knowledge, the different recognition models lead to comparable results, as illustrated in Table 3. The direct and RNN+ methods lead to slightly lower error on the test rollouts, but also take longer to train due to the higher number of free parameters. The KKL and KKLu methods lead to similar performance. To obtain these results, the choice of the gain matrix was critical. Rigorous analysis of the role of this parameter is beyond the scope of this article, and remains a relevant task for future work Munisson-Fenet et al., 2022). We also evaluate the learned model inside an Extended Kalman Filter (EKF). The EKF is classical state estimation tool for nonlinear systems, that takes the measurement and control as imput and outputs probabilistic estimate of the current underlying state. At each time step, it linearizes the dynamics model and output map then proceeds as a linear Kalman Filter to estimate the mean and covariance of the current state (Krener, 2003). We implement an EKF which receives either y = h(x) = 24) or only y == h(x) = 24 as the measurement, and uses the linear prior or an NODE as dynamics function. In both cases, the NODE estimates are more accurate than those obtained with the linear prior model, as shown in Fig. 9 for long test trajectory with input frequency outside of the training regime. As expected, the accuracy for y (x1, is high since are directly estimating the output. However, the performance difference indicates that the NODE is much more accurate and should enable meaningful state estimation for downstream tasks. When y I1, the EKF using the prior model is off, while it provides reasonable estimates in most frequency regimes when using the learned model.
6 Conclusion
The general formulation of NODEs well suited for nonlinear system identification. However, learning physically sound dynamics in realistic scttings, i.c., with control inputs and partial, noisy observations, remains challenging. To achieve this, recognition models are needed to efficiently link the observations to the latent state. We show that notions from observer theory can be leveraged to construct such models; for example, KKL observers can filter the information contained in the observations to produce an input of fixed dimension for which a suitable recognition model is guaranteed to exist. We propose to combine recognition models and existing methods for physics-aware NODEs to build a unifying framework, which can learn physically interpretable models realistic sottings. We illustrate the performance of KKL-basod recognition in numerical simulations, then demonstrate that the proposed end-to-end framework can learn SSMs from partial observations with experimental robotics dataset. While these observer- based recognition models are demonstrated in the context of NODEs, they are separate contribution, which can also be used in various system identification methods; they could be combined with e.g., Neural Controlled Differential Equations (Kidger et al., 2020), Bayesian extensions of NODEs (Norcliffe et al., 2021a; Yildiz et al., 2019) or in general optimization-based system identification methods (Schittkowski, 2002; Villaverde et al., 2021).
The results herein illustrate that observer theory and, in particular, KKL observers are suitable for building recognition models. To the best of our knowledge, this work is the first to propose this connection, hence, it
12

---

Published in Transactions on Machine Learning Research (01/2023)
0.150
1..55
6390
D3160
0LE40
0.33
0.3
03
0.2
0.
0.0
100
0.0
-0.1
(a) Prior model, (111, 374)
(b) NODE, (2r1, (54)
(c) Prior model, y 211
(d) NODE, 7 211
Figure 9: State estimation with an EKF on the robotics dataset. After training the NODE with KKL recognition while imposing $1/3 T2/4: we run the EKF on long test trajectories from unseen input frequencies. When y = (01,)44) both prior and learned model able to reconstitute the output (æ) top row, I4 bottom row), but the EKF with NODE performs better. When measuring only 21, the EKF using the prior model is off, while its estimates are reasonable in most frequency regimes with the learned model.
also points to remaining open questions. In particular, the choice of (D, F)) plays a role in the performance of KKL observers (Buisson-Fenct et al., 2022), and methods for tuning them are still needed; setting D to HiPPO matrix Gu et al. (2021; 2022) could be an interesting first step. On another note, NODEs can be combined with Convolutional Neural Networks capture spatial dependencies and learn Partial Differential Equations (PDEs), as investigated in Dulny et al. (2021); Xu et al. (2021). In this paper, we only consider dynamical systems that can be modeled by ODEs, but expect the proposed approach to extend to PDEs.
References
Mihai Alexe and Adrian Sandu. Forward and adjoint sensitivity analysis with continuous explicit Runge- Kutta schemes. Applied Mathematics and Computation, 208(2):328 346, 2009.
Hananeh Alice, Fabian J. Theis, and Niki Kilbertus. Beyond Predictions in Neural ODEs: Identification and Interventions. Xixiv preprint arxiv:2106. 12430, 2021.
Vincent Andrieu and Laurent Praly. On the existence of a KozantaisKKravais//teeeee Journal on Control and Oplimization, 45(2):422-455 252,, 2006.
observer. SIAM
Ibrahim Ayed, Emmannel De Bezenac, Arthur Pajot, and Patrick Gallinari. Learning the Dynamics of Physical Processes from Partial Observations. In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 0232-3236, 2020.
Joseph Bakarji, Kathleen Champion, J Nathan Kutz, and Steven L Brunton. Discovering Governing Equa- tions from Partial Measurements with Deep Delay Autoencoders. arXi preprint arXiv:2201.05.33 2022.
Pauline Bernard. Observer Design for Nonlinear Systems. In Leclure Noles in Control and Information Sciences, volume 479. Springer International Publishing, 2019.
Pauline Bernard and Vincent Andrieu. Luenberger Observers for Nonautonomous Nonlinear Systems. IEEE Transactions on Automatic Control, 64(1):270- 281, 2019.
Pauline Bernard, Vincent Andrieu, and Daniele Astolfi. Observer Design for Continnous- Time Dynamical Systems. Areaual Reviews in Control, 2022.
13

---

Published in Transactions on Machine Learning Research (01/2023)
Tom Bertalan, Felix Dictrich, Igor Mezić, and Ioannis G. Kevrekidis. On learning Hamiltonian systems from data. Chaos, 29(12), 2019.
Petar Bevanda, Max Beier, Sebastian Kerz, Armin Lederer, Stefan Sosnowski, and Sandra Hirche. Koop- manizingFlows: Diffeomorphicllly Learning Stable Koopman Operators. arXiv preprint arXiv:21 S0::2112.0000 12.04085. 2021.
Mona Buisson- Fenet, Valery Morgenthaler, Sebastian Trimpe, and Florent Di Meglio. Joint state and dy- namics estimation with high-gain observers and Gaussian process models. IEEE Control Systems Letters, 5(5):1627-1632 2021.
Mona Buissou-Fenet, Lukas Bahr, and Florent Di Meglio. Towards gain tuning for numerical KKL observers. arXiv preprint arXio:2204.008 003118 2022.
Mathieu Chalvidal, Matthew Ricci, Rufin VanRullen, and Thomas Serre. Go with the Flow: Adaptive Control for Neural ODEs. In Proceedings of the International Conference on Learning Representations, 2021.
Ricky T.Q. Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural Ordinary Differential Equations. In Advances in Neural Information Processing Systems 32, pp. 6572-6583, 2018.
Quentin Clairon and Adeline Samson. Optimal control for estimation in partially observed elliptic and hypoelliptic linear stochastic differential equations. Statistical Inference for Stochastic Processes, 23:105- 127, 2020.
Kevin L Course, Trefor W Evans, and Prasanth B. Nair. Weak Form Generalized Hamiltonian Learning. In Advances in Neural Information Processing Systems, 2020.
Miles Cranmer, Sam Greydanus, Stephan Hoyer, Peter Battaglia, David Spergel, and Shirley Ho. Lagrangian Neural Networks. In ICLR 2020 Workshop on Integration of Deep Neural Models and Differential Equa- tions, 2020.
L da Costa Ramos, F Di Meglio, L F Figuicra da Silva, P Bernard, and V Morgenthaler. Numerical design of Luenberger observers for nonlinear systems. In Proceedings of the 59th Conference on Decision and Control, pp. 5435-5442. 2020.
Edward Brouwer, Jaak Simm, Adam Arany, and Yves Moreau. GRU-ODE-Bayess Continuous modeling of sporadically-observed time series. In Advances in Neural Information Processing Systems 33, 2019.
Andreas Doerr, Christian Daniel, Duy Nguyen-Tuong, Alonso Marco, Stefan Schaal, Marc Toussaint, and Sebastian Trimpe. Optimizing long-term predictions for model- based policy search. Proceedings of the 1st Conference on Robot Learning, 78:227 238, 2017.
Andreas Doerr, Christian Daniel, Martin Schiegg, Duy Nguyen-Tuong Stefan Schaal, Mare Toussaint, and Sebastian Trimpe. Probabilistic recurrent state-space models. Proceedings of the 35th International Con- ference on Machine Learning, pp. 1280-1289, 2018
Timothy Doyeon, Kim Thomas, Zhihao Luo, Jonathan W Pillow, and Carlos D Brody. Inferring Latent Dynamics Underlying Neural Population Activity via Neural Differential Equations. In Proceedings of the 38th International Conference on Machine Learning, pp. 5551-5561, 2021.
Andrzej Dulny, Andreas Hotho, and Anna Krause. NeuralPDE: Modelling Dynamical Systems from Data.
arXiv preprint xrXio:2111.07671, 2021.
Emilien Dupont, Arnaud Doucet, and Yee Whye Teh. Augmented Neural ODEs. In Advances in Neural Information Processing Systems, 2019.
Jonas Eichelsdörfer, Sebastian Kaltenbach, and Phacdon-Stelios Koutsourclakis. Physics-cnhanced Neural Networks in the Small Data Regime. Workshop on Machine Learning and the Physical Sciences (NeurIPS 2021), 2021.
14

---

Published in Transactions on Machine Learning Research (01/2023)
Stefanos Eleftheriadis, Thomas F.W. Nicholson, Mare P. Deisenroth, and James Hensman. Identification of Gaussian process state space models. In Advances in Neural Information Processing Systems 31, pp. 5310 5320, Long Beach, California, 2017.
Katharina Ensinger, Friedrich Solowjow, Michael Tiemann, and Sebastian Trimpe. Structure-preseriing Gaussian Process Dynamics. arXiv preprint orXiv:2102.00 212:0002.00006, 2022.
Nicholas Galioto and Alex Arkady Gorodetsky. Bayesian system ID: optimal management of parameter, model, and mcasurement uncertainty. Nonlinear Dynamics, 102:241-267, 2020.
René Geist and Sebastian Trimpe. Structured learning of rigid-body dynamics A survey and unified view. CAMM-Mitteüüngen, (44:e202100009), 2021.
Sam Greydamus, Misko Dzamba, and Jason Yosinski, Hamiltonian Neural Networks. In Advances in Neural Information Processing Systems 33, 2019.
Albert Gu, Isys Johnson, Karan Goel, Khaled Saab, Tri Dao, Atri Rudra, and Christopher Ré. Combining Recurrent, Convolutional, and Continuous-ting Models with Linear State-Space Layers. In Advances in Newral Information Processing Systems, pp. 572-50 1225555 2021.
Albert Gu, Karan Goel, and Christopher Ré. Efficiently Modeling Long Sequences with Structured State Spaces, 2022.
Steeven Janny, Vincent. Andrieu, Madiha Nadri, and Christian Wolf. Deep KKL: Data-driven Oulput Prediction for Non-Linear Systems. In Proceedings of the IEEE Conference on Decision and Control, 2021.
Thomas Kailath. Linear Systems. Prentice Hall PTR, Englewood Cliffs, New Jersey, 1980.
Daniel Karisson and Olle Svanström. Modelling Dynamical Systems Using Neural Ordinary Differential Equations, Master's thesis in Complex Adaptive Systems, Department of Physics, Chalmers University of Technology, 2019.
Nikolaos Kazantzis and Costas Kravaris. Nonlinear observer design using Lyapunov's auxiliary theorem. Systems and Control Letters, 34:241-244 247, 1998. ISSN 01912216. doi: 10.1109 1et.0//dd...9.4999999
Patrick Kidger, James Morrill, James Foster, and Terry Lyons. Neural controlled differential equations for irregular time series. In Advances in Neural Information Processing Systems, 2020.
Diederik P. Kingma and Jimmy Lei Ba. Adam: A method for stochastic optimization. In Proceedings of the 3rd International Conference on Learning Representations, 2015.
Arthur J. Krener. The Convergence of the Extended Kalman Filter. In Directions in Mathematical Systems Theory and Optimization Lecture Noles in Control and Information Sciences, volume 286, pp. 182.12. Springer-Verte Berlin Heidelberg, 2003.
Christian Møldrup Legaard, Thomas Schranz, Gerald Schweiger, Ján Drgoña, Basak Falay, Cláudio Gomes, Alexandros Iosifidis, Mahdi Abkar, and Peter Gorm Larsen. Constructing Neural Network-Based Models for Simulating Dynamical Systems. ACM Computing Surveys, 1(1), 2021.
Lennart Ljung. System Identification: Theory for the User. Prentice Hall PTR, Englewood Cliffs, New Jersey, 1987.
Bethany Lusch, J. Nathan Kutz, and Steven L. Brunton. Deep learning for universal linear embeddings of nonlinear dynamics. Nature Communications, 9, 2018.
Michael Lutter, Christian Ritter, and Jan Peters. Deep Lagrangian Networks: Using Physics Model Prior for Deep Learning. In Proceedings of the International Conference Learning Representations, 2019.
Gaurav Manck and Zico Kolter. Learning stable deep dynamics models. In Advances in Neural Information Processing Systems, 2019.
15

---

Published in Transactions on Machine Learning Research (01/2023)
Stefano Massaroli, Michael Poli, Michelangelo Bin, Jinkyoo Park, Atsushi Yamashita, and Hajime Asama. Stable Neural Flows. orXiv preprinl. ar arXiv:2003.8806; 2020a.
Stefano Massaroli, Michael Poli, Jinkyoo Park, Atsushi Yamashita, and Hajime Asama. Dissecting neural ODEs. In Advances in Neural Information Processing Systems, pp. 3952 3963, 2020b.
Viraj Mehta, lan Char, Willie Neiswanger, Youngseog Chung, Jeff Schneider, Andrew Oakleigh Nelson, Mark Boyer, and Egemen Kolemen. Neural Dynamical Systoms. In International Conference on Learning Representations- Inlegration of Deep Neural Models and Differential Equations Workshop, 2020.
Oliver Nelles. Nonlinear System Identification. Springer, Berlin, Heidelberg, 2001.
Duy Bouyen-Tuong and Jan Peters. Model learning for robot control: A survey. Cognitive Processing, 12: 319 340, 2011.
Alexander Norcliffe, Cristian Bodnar, Ben Day, Jacob Moss, and Pietro Liò. Neural ODE Processes. In Proceedings of the International Conference on Learning Representations, 2021a.
Alexander Norcliffe, Cristian Bodnar, Ben Day, Nikola Simidjievski, and Pietro Liò. On second order behaviour in augmented neural ODEs. In The Symbiosis of Deep Learning and Differential Equations Workshop (NeurIPS 2021), 2021b.
L Praly, Marconi, and A Isidori. A new observer for an unknown harmonic oscillator. Proceedings of the 17th International Symposium on Mathematical Theory of Nelworks and Systems, pp. 996 1001, 2006.
Lucas Rath, A. René Geist, and Sebastian Trimpe. Using Physics Knowledge for Learning Rigid-body Forward Dynamics with Gaussian Process Force Priors. In Proceedings of the 5th Conference on Robot Learning, pp. 101-111, 2021.
Andreas Raue, Marcel Schilling, Julie Bachmann, Andrew Matteson, Max Schelke, Daniel Kaschek, Sabine Hug, Clemens Kreutz, Brian D. Harms, Fabian J. Theis, Ursula Klingmüller, and Jens Timmer. Lessons Learned from Quantitative Dynamical Modeling in Systems Biology. PLoS ONE, 8(9), 2013.
Yulia Rubanova, Ricky T.Q. Chen, and David Duvenaud. Latent ODEs for irregularly-samplee time series. In Advances in Neural Information Processing Systems, 2019.
Klaus Schittkowski. Numerical Data Fitting in Dynamical Systems. Springer, Boston, MA, 2002.
Andreas Schlaginhanfen, Philippe Wenk, Andreas Krause, and Florian Dörfler, Learning Stable Deep Dy- namics Models for Partially Observed or Delayed Dynamical Systems. In Advances in Neural Information Processing Systems 35, 2021.
Johan Schoukens and Lennart Ljung. Nonlinear System Identification: User-Oriented Roadmap. 1EEE Control Systems Magazine, (9))228,,,, 2019.
Yuri Shtessel, Christopher Edwards, Leonid Fridman, and Arie Levant. Sliding Mode Control and Observa- tion. In Control Engineering. Birkhäuser, New York, 2016.
Jonas Sjöberg, Qinghua Zhang, Lennart Ljung, Albert Benveniste, Bernard Delyon, Pierre Yves Glorennec, Hákan Hjalmarsson, and Anatoli Juditsky. Nonlinear black-box modeling in system identification: unified overview. Automatica, 31(12):1691 1724, 1995.
Mario Spirito, Pauline Bernard, and Lorenzo Marconi. On the existence of robust functional KKL observers.
In Proceedings of the American Control Conference, 2022.
Mats Viberg. Subspace-based methods for the identification of lincar time-invariant systems. Automatico, 31(12):1835-1851, 1995.
Matthieu Vigne. Estimation and Control of the Deformations of an Ezoskeleton using Ineriial Sensors. PhD thesis, Mines ParisTech- Université PSL, 2021.
16

---

Published in Transactions on Machine Learning Research (01/2023)
Alejandro F. Villaverde, Dilan Pathirana, Fabian Fröhlich, Jan Hasenauer, and Julio R. Banga. A protocol for dynamic model calibration. arXiv preprint Xix:19902. 11 11130, 2021.
Rui Wang and Rose Yu. Physics-Guided Deep Learning for Dynamical Systems: A survey, arXiv preprint 010002207..1172,, 2021.
Brian Winkel. 2017-Gustafson, G. B. Differential Equations Course Materials, 2017. URL https://www. simiode. simiode.org/reeourreeeeoo... org/reeoourcs//3892
Jin Long Wu, Carlos Michelén-Strröfer, and Heng Xiao. Physics-informed covariance kernel for model-form uncertainty quantification with application to turbulent flows. Computers and Fluids, 193, 2019.
Xingzi Xu, Ali Hasan, Khalil Elkhalil, Jie Ding, and Vahid Tarokh. Charactoristic Noural Ordinary Differ-
ential Equations. arXiv preprint 0rXic:2111.1222 2021.
Çagatay Yildiz, Markus Heinonen, and Harri Lähdesmäki. ODE2VAE: Deep generative second order ODEs with Bayesian neural networks. In Advances in Neural Information Processing Systems, 2019.
Yuan Yin, Vincent Le Guen, Jérémne Dona, Emmanuel de Bézenac, Ibrahim Ayed, Nicolas Thome, and Patrick Gallinari. Augmenting physical models with deep networks for complex dynamics forecasting. Tn Proceedings of the 9th International Conference on Learning Representations, 2021.
Muhammad Zakwan, Loris Di Natale, Bratislav Svetozarevic, Philipp Heer, Colin N. Jones, and Giancarlo Ferrari Trecate. Physically Consistent Neural ODEs for Learning Multi-Physics Systems. orXiv preprinl 1:Xiv:2211.06190 2022.
Yaofeng Desmond Zhong, Biswadip Dey, and Amit Chakraborty. Symplectic ODE-Net: Learning Hamilto- nian Dynamics with Control. In International Conference on Learning Representations, 2020.
Yaofeng Desmond Zhong, Biswadip Dey, and Amit Chakraborty. Benchmarking Energy- Conergy-nnnseiing Neu- ral Networks for Learning Dynamics from Data. In Proceedings of the 3rd Conference Learning for Dynamics and Control, volume 144, pp. 2218-1229) 2021.
Max Zhu, Jacob Moss, and Pietro Lio. Modular Neural Ordinary Differential Equations. arXiv preprint N.XY::210900739900 2019.
A More detailed background on KKL observers
We rocall the main existence results on KKL observers. Correctly stating these results requires more for- malism than what used in the main body of the paper. However, the assumptions and the reasoning are identical. We start with the main existence result autonomous systems, then recall the extension to nonautonomous systems.
A.1 Autonomous systems
Consider the following autonomons nonlinear dynamical system
f(),
y y= h=x)
(10)
where dd is the state, T&dy the measured output, f is C function and is a continuous function. The goal of observer design is to compute estimate of the state z(t) using the past values of the output v(s), 0 < S t. We make the following assumptions:
Assumption
There exists compact set X such that for any solution of (10), x(1) E X V120 0.
17

---

Published in Transactions on Machine Learning Research (01/2023)
Assumption 2 There erists open bounded set O containing X such that (10) is backward O- dislinguishable on X, i.e., for any Irajectories Ia and va of (10), there erists > 0 such that for ony 2 E such that (ro(t), E X x &' and (t) # xı(t), there evists s E 1t - i,t] such that
h(ra(s)) # h(xx(s))
and (I)(T) O for all T [s, In other words, their respective outputs become different in backward finite time before leaving O.
This the assumption of backward distinguishability, ie., Definition but in backward time. It means that the current state is uniquely determined by the past values of the output. On the contrary, (forward) distinguishabilityy means that the initial state is uniquely determined by the future values of the output. If the solutions of (10) are unique, e.g., if f is C², then these two notions are equivalent.
The following Theorem derived in Andricu & Praly (2006) proves the existence of a KKL observer.
Theorem (AAndricu & Praly, 2006)) Suppose Assumptions 1 and 2 hold. Define d2 = dy + +1) Then, there ecisls and a sel of zero measure in Cds such that for any malrix D Rd,xd= with eigenvalues (d,.) in Cd. A with Re A -t, and any F E Bddxdd such that (D, F) is controllable, there exists an injective mapping T Rd= + Rdd that satisfies the following equation on X
ØT
Fh(z),
(11)
Or
of and its left inverse T Rdd + Rdd such that the trajectories of (10) remaining in X and any trajectory
= Dz Fy
(12)
verify
(=(t) - T(x(1)) S M (=(0) -(x((0)) e Amiit
(13)
for some M > and with
Amin min (1 Rc Azl ..... R)
(14)
This yields
lim ++0000
(x(() (x(t) = O.
(15)
A.2 Nonautonomous systems
These results are extended to nonantonomous systems Bernard Andricu (2019). The system equations are then
④ = f(,
V == h(z, u)
(16)
where u E W is the input. Assumption 2 naturally extends to nonautonomons systems if it true for any fixed input u of interest. The following Theorem proves the existence of KKL observer in the nonautonomous case under the weak assumption of backward distinguishability.
Theorem (BBrnard & Andrieu, 2019)) Take some fixed input u E U. Suppose Assumptions 1 and 2 hold for this u with a cerlain T,, 2 0. Define d: = (d2+1) Then, there exists set S of zero measure in Qd, such that for any matrix D E xd, with eigenvalues (Ai....,da]] in Cdo S with Re X < O, and any F E Rd, such that (D, F) is controllable, there exists mapping To : RR x Rd. + Rdd that satisfies the following equation on X
8T (1,x)[(z, (()))) + 0T, (x) = DT,(1,2) + Ph(x, at
(17)
dr
18

---

Published in Transactions on Machine Learning Research (01/2023)
and mapping To RR + BR such that -u(t,-) -) and Tu (t,-) only depend on the past values of u [0,
and (t, is injective A 1 2 tu with lest-inwerse Ty (1,)) on X. Then, the trajeclories of (16) remaining in X and any trajectory of
Dz + Fy,
(18)
verify
(z(t)
S M (z - T=,(0,z(0)) e Amint
(19)
for some M > and with
Amin = min (1 R1 Re R)
(20)
This yields
lim (x(t)) T (t)) 8
(21)
Remark 2 The literature on nonlinear observer design is wide and many types of observers have been proposed; see Bernard (2019); Bernard et al. (2022) for an overview. The proposed recognition models are based on KKL observers, well-suited for the recognition problem due to the eristence of the transformation T which can be approzimated jointly with the dynamies. Other designs for general nonlinear systems include high-gain observers, which could lead to the same type of structure for the recognition problem. However, they are subject to peaking and lend to amplify measurement. noise. An exlended Kalman filler could also be used to directly estimate the trajectories using the current dynomics model. However, there is no convergence guarantee for such filters and they directly rely on linearization of the current dynamics, hence they are sensilive to model error. We expect this to be harder to train than the KKL-based methods, which decouple recognition and dynamics models.
B Implementation details
We demonstrate the proposed method in several numerical experiments and one experimental dataset ob- tained on a robotic exoskeleton. We investigate the direct method æ(t.) (V Mo:t.), the RNN+ method where E(tx) is the output of an RNN run over (Yon,, Mozt.) the KKL method E(tc) = (z(tc), No...) and the functional KKL method (denoted KKTu) E((x) x(4,) with d_ = (dy + dd, + du + 1). The KKL observer is run backward in time: we solve the ODE on backward in time on [te, 0] and learn the mapping from a0) to x0), then use all samples to train the NODE model. Similarly for the RNN.
In all cases, the choice of D is important for the KKT-based recognition models. For each considered system and for each considered method, we optimize D once jointly with all other parameters once, then reuse the obtained value of D for all corresponding experiments. We set F = 1d, and initialize with the following method. We compute the poles p, of a Butterworth filter of order de and cut-off frequency 2mw and set each block of D as
P:
if pa is real otherwise
(( Ro(ps) (m(x:) D. D;= (Boofr)) Re(r)) )
(22)
such that D is block-diagonal matrix, and its eigenvalues are the poles of the filter. This choice ensures that the pair (D, F) is controllable and that D is Hurwitz and has physically meaningful eigenvalues. Other possibilities exist, such choosing D in companion form, negative diagonal... However, we found that this strategy leads to the best performance for the considered use cases. We pick we = 1 for the systems of the recognition model benchmark and the harmonic oscillator with unknown frequency. For the experimental dataset, we initialize as D = diag(-1...., -dz). However, this choice is somewhat arbitrary, and the previous method with w.- = 10 had similar performance. Principled methods for sctting (D, F) are still needed for casing the practical use of KKL observers; setting D to HiPPO matrix Gu al. (2022; 2021) could be an interesting first step.
19

---

Published in Transactions on Machine Learning Research (01/2023)
Note also that there is a large part of randomness in the different experiments we present. Hence, results may vary, and obtaining consistent results to rigorously compare the different. methods without any statistical variations would require large computational overhead.
B.1 Benchmark of recognition models
We demonstrate on numerical systems that an NN-based recognition model can estimate the initial state of a dynamical system from partial measurements. For reproducibility, we generate the training data from simulation and choose systems that can be tested again with reasonable computational overhead.
Earthquake model A simplified model of the effects of an earthquake on a two-story building is presented in Winkel (2017), and an NODE is trained for it in Karlsson & Svanström (2019). This linear model can be written
11 32
k
42 m
(23 2x1)) Fow cos(wt)
33 I4
k
11 (x1 23)
Fow cos(wut)
m
y = xj +
(23)
where 21 and æ3 are the positions of the first and second floor respectively, 22 and I4 their velocities, Fow? (ws( is the perturbation causod by the carthquake and only 21 is observed with Gaussian noise of variance of = 10-1. We consider the oscillation caused by the earthquake as disturbance, which is known when simulating training trajectories and unknown to the recognition model: we estimate z00) from Vo:t. only.
We aim to learn a recognition ion model that estimates sx0) using only you. with the methods described above. We set fe = 40 x At = 40 x 0.03 = 1.2s which seems to be enough to reconstitute the initial condition (after trial and error), N- = 50 (each sample corresponds to a random initial condition, random Fo and random w), n 100, and design the (and eventually fe) as fully connected ferd-forward network, i.., a multi-layer perceptron, with two hidden layers containing 50 neurons each, and two fully connected input and output layers with bias terms. The RNN+ model is set have the same internal dimension d2 the KKL model. We notice that le large enough and enough parameters in va, i.e., enough flexibility of the model, are needed for good generalization performance. Also, we pick the sampling time At= 0.03 low enough such that the obtained trajectories are reasonably smooth, otherwise analyzing the results quantitatively becomes hard due to interpolation errors getting too large.
We train the recognition model with each proposed method and evaluate the results on 100 test trajectories of random initial conditions and random input oscillation. The results on one such test trajectory are illustrated in Figure 10.
We train two settings: either learning a full NODE model (main body of the paper), or having a known dynamics model in which only k/m, the main parameter of the dynamics model, is optimized jointly with the recognition model. In our example, we have k/m = 10, but we initialize its estimate to a random value in [8, 12). As usual, this problem is not well-posod and there are many local optima. Therefore, we can only hope to converge a good estimate by starting from a reasonable guess of the main parameter. We keep everything else fixed, including the optimization routine, which might not be the best choice as it has been shown that for parametric optimization, trust region optimization routines with multiple starts often lead to better results (Raue et al., 2013). For the full NODE model, we evaluate the different recognition models by computing the RMSE on the prediction of the output only, since the coordinate system for =<() is not fixed and a different coordinate system is found in each experiment. For the parametric model, we evaluate the different recognition models by computing the RMSE on the estimation of the whole trajectory over all test scenarios, since the coordinate system for æ(t) is fixed by the parametric model. The results are shown in Figure 11 and in the main body of the paper. We observe that the KKI-based models achieve higher
20

---

Published in Transactions on Machine Learning Research (01/2023)
04
1.0
0.50
Observed
03
0.25
-02
0.0
00
-45
03.50
-14
+0.25
-10
0.0
0.22
0.04
0.6
235
530
(a)
(b)
(c) di
(d) d2
Figure 10: Test Irajectory of the parametric earthquake model with KKL recognition: the initial condition is estimated from Mo., jointly with the model parameters.
04
0.6
0.3
RM55
IMSS
0.2
6.1
02
0.0
direct
RNN+
KKT.
direct
RNN+
KKI
(a) Full NODE: output error
(b) Parametric model: full state error
Figure 11: Results of the obtained earthquake recognition models. We show the RMSE on the prediction of the output when a full NODE model is learned (left column) and the whole test trajectories when a paramctric model learned (right column). Ten recognition models were trained with the methods direct (left), RNN+ (middle) and KKL (right). The direct method with 4, is not shown here for scaling, but the mean RMSE is over 0.6.
performance, which seems to indicate that the optimization problem based on z(0) is better conditioned than that based on you..
FitzHugh-Nagumo model This model represents a relaxation oscillation in an excitable system. It is a simplified represontation of the behavior of spiking ncuron: an external stimulus is recoived, loading to a short, nonlinear increase membrane voltage then slower, linear recovery channel mimicking the opening and closing of channels (Clairon Samson, 2020). The dynamics are written
ù (v 33 -u) Iext
ù TV y + C,
(24)
where v is the membrane potential, the value of the recovery channel, Tert is the value of the external stimulus (here a constant), 0.1 is time scale parameter, and 7 1.5, 0.8 are kinetic paramelers. Only is measured, corrupted by Gaussian measurement noise of variance o² 5 -
Our aim is to learn recognition model that estimates (u(0), u()) using yout and Text with the methods described above. We set I, 40 x A/ = 40 x 0.03 = 1.28, N = 50 for 50 random initial conditions and external stimulus, n = 100, and design the (and eventually fe) as a fully connected feed-forwar forward network, i.c., a multi-layer perceptron, with hidden layers containing 50 neurons cach, and two fully connected input and output layers with bias terms. The RNN+ model is set have the same internal dimension d, the KKT, model.
21

---

Published in Transactions on Machine Learning Research (01/2023)
1.2
0.5
1.0
1,0
0.
0.8
Predicted
Initill state
-05
Prodicted
0.4
Obwerved
0.02
0.05
-05
3.0
0.5
1.0
1.0
0.8
0.8
Trur
Predicted
0.6
0.6
Initial
Thar
0.4
0.4
0.2
Observed
0.00
0.02
0.04
0.06
0.08
0.10
0.00
0.02
0.04
0.00
0.08
0.10
-1.0
-05
0.0
3.0
1.0
1.0
Predicted
0.5
Obsueved
0.8
True
0.0
0.6
Predicted
0.4
0.4
Initial state
-0.5
-1.0
0.0
0.0
(a)
(b)
(c) Phose portrait
Figure 12: Test trajectory of the parametric FitzHugh-Naguna model: the initial condition is estimated from Von, jointly with the model parameters. We use direct (top), RNN+ (middle) and KKL (bottom) recognition, on three random but similar test trajectories.
We train the recognition model with each proposed method and evaluate the results 100 test trajectories with random initial conditions and random stimulus. The results in Figure 12.
such test trajectory illustrated
We either learn full NODE model (main body of the paper) or a parametric model for which we estimate the main dynamic parameters e, 8 and y jointly with the recognition model, initialized randomly in [0.05, 0.15]. (0.75, 2.25] and [0.4, 1.2] respectively. We evaluate the different recognition models as above. The results are illustrated in Figure 13 and in the main body of the paper. We observe once again that the KKI-bbaod methods lead to lower error.
Van der Pol oscillator Consider the nonlinear Van der Pol oscillator dynamics
i J2 2₂ μ(1 =2)2z +
+
(25)
where ø1, Z2 are the states, 1.2sin(wt) is a sinusoidal control input, and μ = 1 is a damping parameter. Only Di is measured, corrupted by Gaussian measurement. noise of variance o² 10-3
22

---

Published in Transactions on Machine Learning Research (01/2023)
0.5
0.20
0.2
0.15
ROSSE
0.10
RNNN
(a) Full NODE: output error
(b) Parametric model: full state error
Figure 13: Results of the obtained FitzHugh-nggumo recognition models. We show the RMSE on the prediction of the output when full NODE model is learned (left column) and of the whole test trajectories when parametric model is learned (right columm). Ten recognition models were trained with the methods direct (left), RNN+ (middle) and KKL (right). The direct method with t = 0 is not shown here for scaling, but the mcan RMSE is over 0.4.
Our aim is to learn recognition model that estimates z0) using you, and uoa, with the methods described above. We set = 40 x A/ 40 0.03 1.2s, N = 50 for 50 random initial conditions and values of w, 100, and design Vh (and eventually fo) as fully connected feed-forward network, i.c., multi-layer perceptron, with two hidden layers containing 50 neurons each, and two fully connected input and output layers with bias terms. Since u a simusoidal control input of varying frequency, it can be generated by
1 W2,
(26)
W/ W3W11,
Wi O
and == w1, where W1, w2 the internal states of the sinusoidal system and Wa >> 0 is its frequency. Hence, we can use the KKLu recognition model with du = 3. The RNN+ model is set to have the same internal dimension de as the KKLu model.
We train the recognition model with each proposed method and evaluate the results on 100 test trajectories with random initial conditions and random input frequency. The results on one such test trajectory are illustrated in Figure 14. We also consider either a full NODE model, or parametric model for which μ is jointly estimated, after from random initial guess in [0.5, 1.5). The corresponding box plots shown in Figure 15. We observe that in both settings, the performance with the different recognition models is very similar. In the main body of the paper, we show the same plot but with higher noise of variance of = 0.1 instead of of = 10-3 here. Due to the very similar performance, the hierarchy the different recognition modols slightly varies between the noise levels.
B.2 Synthetic dataset: harmonic oscillator with unknown frequency
We demonstrate the performance of structured NODEs to learn the dynamics of harmonic oscillator
with unknown frequency from partial observations with varying degrees of structure. We train on = 50
trajectories from 50 random initial states [3 and frequency w² Hz (i.e., period 6.3s), of n
time steps each for an overall length of 3s, corrupted by Gaussian measurement noise of variance o² 10-4
We use le = 20x Al 1.2s for the recognition model. We optimize the parameters using Adam (Kingma & Ba, 2015) and a learning rate of 0.005.
The obtained results are depicted in Figure 16 with the KKL recognition model. We show the prediction of a random lest. trajectory with random initial state: You, is measured for this Irajectory, used by the recognition model to estimate z0). Then, the learned NODE simulated to predict the whole state trajectory for 500 time steps, i.c., ten times longer than the training time illustrate the long-torm behavior. For the quantitative results presented in the main body of the paper, we predict on test trajectories of 150 time steps, i.e., three times the training time. These make the long-term performance difference due to the degree
23

---

Published in Transactions on Machine Learning Research (01/2023)
Tue
1.3
Observed
8
0.0
True
-0.5
Initial state
1,0
0.00
0.02
0.04
0.66
0.08
0.30
0.00
0.02
0.24
0.05
0.08
0.30
-1.0
-05
0.0
0.5
1.0
2.0
2.0
2.5
Tue
1.5
Prodicted
2.0
Observed
1.5
1.0
0.5
0.0
Predicted
-0.5
2,0
Tuee
1.3
Predicted
2.0
Observed
1.5
1.5
1.0
0.0
Trur
005
-05
Initial state
-1.0
-1.
0.00
0.02
0.04
0.06
0.08
0.10
0.00
0.02
0.84
0.06
0.08
0.10
-0.5
0.0
0.5
1.0
1.5
2.0
Predicted
Observed
47
0.0
True Predicted Initial state
0.5
(a) 21
(b)
(c) Phase portrait
Figure 14: Test trajectory of the parametric Van der Pol model: the initial condition is estimated from Vote jointly with the model parameters. We use direct (top), RNN+ KKL and KKLu (bottom) recognition, four random but similar test. trajectories.
of prior knowledge less visible, but lead to more consistent and quantitatively comparable results (with the long test trajectories, the interquartile range of the experiments was very large due to error accumulation which can possibly blow up over a long prediction horizon).
We train the dynamics and recognition model in each setting ten times, for recognition models direct, RNN+ + and KKL. The mean RMSE over hundred test trajectories is depicted in the main body of the paper.
No structure We start without imposing any structure, i.c., learning a general latent NODE model of the system in (4.2) The NODE fits the observations y 21, but not 22 as it has learned the dynamics in
24

---

Published in Transactions on Machine Learning Research (01/2023)
0.10
0.10
0.05
0.08
0.06
RORR
0000
0.94
0.02
(a) Full NODE: output error
(b) Parametric model: full state error
Figure 15: Results of the obtained Van der Pol recognition models. We show the RMSE the prediction of the output when full NODE model is learned (left column) and of the whole test trajectories when parametric model is learned (right column). Ten recognition models were trained with the methods direct (Ieft), RNN+ KKL and KKLu (right). The direct method with t. 0 is not shown here for scaling, but the mean RMSE is over 0.6.
O4
-055
-02
-09
02
-05
04
10
(a) No structure
(b) Hamiltonian
(c) dr
(d) Parametric
(c) Extended
Figure 16: Random test trajectory of the trained NODE for the harmonic oscillator: without imposing any
structure (a), imposing Hamiltonian dynamics (b), imposing Φ1 22 (c), directly identifying parametric
model (d) and learning recognition model of an extended state-space representation where 33 = w2 (e).
We show the true and predicted trajectories of æ (top) and 12 (bottom).
another coordinate system, which is expected for general latent NODEs. It also does not conserve energy, which is not surprising when no structure is imposed, as discussed e.g., in Greydanus et al. (2019).
Hamiltonian state-space model We now assume the user has some physical insight about the system hand: it derives from Hamiltonian function, i.e. there exists H such that
0H
OH
dr (z),
d2
dx (z).
(27)
We approximate directly with neural network Ho of weights such that the NODE has form (27), and inject this into the optimization problem (4). This formulation enforces the constraint that the dynamics derive from Hamiltonian function, whose choice is free. In that case, do not necessarily find the 'physical" state-space realization, as several Hamiltonian functions can fit the data. However, the obtained state-space model conserves energy due to the Hamiltonian structure.
Imposing + = x2 We now impose a somewhat stronger structure (4):
$ x2
2 = - -HH())
(28)
25

---

Published in Transactions on Machine Learning Research (01/2023)
where only the dynamics of 22 need to be learned. This enables the NODE to recover both the initial state and the unknown part of the dynamics in the imposed coordinates while also conserving energy, as this is a particular case of Hamiltonian dynamics with Hamiltonian function _ + (
Parametric system identification
We now directly learn a parametric model of the harmonic oscillator
$1 $2
22
where w > O is the unknown frequency. We approximate with parameter 0, which is initialized randomly in [0.5, 2]. We obtain excellent results with this method, as is estimated correctly up to 10-2 and the trained recognition model gives satisfying results. This demonstrates that our framework can recover both the dynamics and the recognition model in the physical coordinates imposed by the parametric model from partial and noisy measurements in this simple use case.
Extended state-space model
We now consider the exlended state-space model
11
22
42 -3
3 = 0
(29)
where 23 = w² is a constant state representing the unknwon frequency. In this case, the dynamics are completely known and only the recognition model is left to train, in order to estimate the initial condition x0) E R2, where za(0) is the unknown frequency. This is the same degree of structure as the parametric model: the dynamics known up to the frequency. However, it also is more open problem: since we learn a recognition model for x=0) E RR3, at each new trajectory, we estimate a new value of the frequency zs(0), which was considered the same across all trajectories for the previous methods. Therefore, with this setting, we also obtain models that can predict euergy-preserving trajeclories in the physical parameters, but with lower accuracy due to this extra degree of freedom.
B.3 Experimental case study: robotic exoskeleton
We use a set of measurements collected at Wandercraft on one of their exoskeletons and presented in Vigne (2021). More details on the robot, the dataset and the methods applied at Wandercraft are provided in Section 4.1.2.1 of Vigne (2021).
For this experiment, the robot basin is fixed to the wall and low amplitude sinusoidal inputs are sent to the front hip motor with different frequencies between Hz and 16 Hz. A lincar model has been identified in Vigne (2021) by modeling the deformation as linear spring in the hip motor, yielding system with I E R. The angle of the hip (z1) is measured by the encoder on this motor, while a gyrometer measures the angular velocity of the thigh (.4). The measurements are sampled with At = 1ms. The aim is to idontify the nonlinear deformations in the hip motor, which can be seen in motion capture and cause siguificant errors in gait planning, but are not captured by the known models of the exoskeleton.
We start by preprocessing the signal: for each input frequency and corresponding trajectory, we compute the FFT of y, apply Gaussian window at fc 50 Hz on the spectrum, then apply an inverse FFT and slice out the beginning and the end (100 time steps) of each signal to get rid of the border effects. For u, which is not very noisy, we rather apply a Butterworth filter of order 2 and cut-off frequency 200 Hz. We cut the long trajectories for each input frequency in slices of 200 samples, and stack these training trajectorics of length 0.2 together to form our set of training trajectories. Hence, all trajectories have the same sampling times and can be simulated in parallel casily. We choose the length of 0.2s bocause it scems long enough to capture some of the dynamics even in the low-frequency regime, but also short enough to remain acceptable in the high-frequency regime.
We then run the proposed framework on this data. We dircetly train the NODE on a random subset of training trajectories, use a subset of validation trajectories for early slopping, and a subset of test trajectories to evaluate the performance of the learned model. When no further indications are provided, we use a recognition model of type KKL with d, = 10, F F= = which yields 110 for the dimension of (z(te), uo.t.). The paraneter D was optimized after being initialized at diag(-1.... -10). We also trained recognition model of type KKTu with d₂ = 50 for which D was also optimized starting at diag(-2,... 1100), and
26

---

Published in Transactions on Machine Learning Research (01/2023)
recognition models of type direct and RNN+ using the same information contained in (Yo., (oxte) and the same size of the latent state as for KKLu, i.e, dimension 50. Both recognition and dynamics models are feed-forward networks with five hidden layers of 50 respectively 100 neurons and SiLU activation.
We notice that for this complex and nonautonomous use case, the direct and RNN+ recognition met soem casier to train. However, they also take longer due to having more parameters, and have higher generalizaiton error on the long test rollouts. We also notice that D needs to be chosen well for the KKL- based recognition models to obtain good performance, which needs to be investigated further.
Normalization is also an important aspect in the implementation: all losses and evaluation metrics are scaled to the same range, so that all loss terms play a similar role and remain within a similar range. This ensures that the values on which the optimization is based are always mumerically tractable for the chosen solver. Different scaling possibilities are discussed in Sec. 5.2.3 of Schittkowski (2002). In our case, since we do not know in advance the values that x=t) will take, we compute the mean and standard deviation of the samples in v() and u(t) and scale all outputs v(() respectively inputs u(t) according to these. We also scale all states att) or derivatives i(t) using the scaler on v(t) for the dimensions that are measured (I1 and z4), and a mean of the scaler on yt) for the other dimensions. This is not quite correct, but it is the best we can do without knowing the range of values that z(t) will take, and it enough to ensure that all scaled values of x(t) stay within a reasonable range.
We investigate three settings: no structure, imposing # 22 and 2 x4 ("structural" prior), and learning the residuals from the prior linear model on 22 and ă1 ("regularizing" prior) with À = 5 x 10-7; 7; we already have d1 = I2 and d = 24 in the prior model). For evaluating the prior linear model, we use the estimated initial states obtained by the recognition model of the last setting, in order to be in the coordinate system that corresponds to the prior. In each setting, we learn from N 265 trajectories of a subset of input frequencies: {2.5, 3.5, 4.5, 5.5, 6.5, 7.5, 8.5, 9.5, 11, 13, 15} Hz. We then evaluate on 163 test trajectories of 0.2s from these imput frequencies, to evaluate data fitting in the trained regime. We also evaluate on 52 longer (2s) lest. Irajectories from other input frequencies, lo evaluate the interpolation capabilities of the learned model: {2, 3, 4, 5, 6, 7, 8, 9, 10, 12, 15, 17] Hz. We use the Adam Kingma & Ba (2015) optimizer with decaying learning rate starting at 8 x 10-3 for the first two settings, 5 x 10-3 for the third setting,
The obtained results are described in the main body of the paper. One longer test trajectory with an input frequency outside of the training regime is presented we find that structured NODEs are able to fit this complex nonlinear dynamical system using real-world data and realistic settings. The predictions of the obtained models are not perfect, but they are much better than those of the prior model, such that they could probably be used in a closed-loop control task like the linear model currently is at Wandercraft. This is confirmed by implementing an EKF that uses the learned models for state est imation (Luming fixed for all recognition models and all degrees of prior knowledge). Adding structure leads to similar performance, but to a model that can be physically interpreted in terms of position and velocity. In the third sctting (hard constraints and residuals model), the accuracy lower. This is due to the fact that the linear prior model leads to rather inaccurate predictions.
Fig. 7 when imposing i = 22 and i33 = 2z. Overall,
27

---

Published in Transactions on Machine Learning Research (01/2023)
Ther
0.150
0.1.0
Tedcted
01145
01146
0314
01.60
0044
0.142
0340
01.00
0.00
0.35
330
6.35
0.00
040
345
030
0.55
0.70
000
006
533
0.55
0.70
0.00
045
000
3.55
0.30
0.
00
0.1
0.2
0.5
-03
-00
-01
-12
-02
-02
0.00
(a) Prior model
(b) No structure
(c) 11/3 22/4
(d) Residual model and (c)
Figure 17: Structured NODEs and KKL recognition on the robotics dataset. We test on 163 trajectories of 0.2s from the same input frequencies as the training data to evaluate data fitting in the trained regime, and compute the RMSE: we obtaín respectively 5.6 (a), 0.16 (b), 0.18 (c), 0.31 (d). We show one such test trajectory (#1 top row, J4 bottom row) from an unknown initial condition.
28


Reviews:
[{"review_id": "3yAJIY1eAX", "reviewer": "Reviewer_yCgK", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The strength of the paper is that it does not make outrageous claims and all statements are correct (deducting the normal overstatement). However, the paper has also many shortcomings.  The biggest problem is the clarity and hence the reproducibility of the paper. While the paper spends a lot of space on general approaches and related work, the actual contribution is very shallow. For the TLMR community, the presented knowledge of sections 1- 3 mostly exists. However, the TLMR community mainly consisting of ML researchers is not so fluent with observers and Kazantzis-Kravaris/Luenberger (KKL) observers, Therefore, it would be better to spend much more space on KKL observers such that the reader can understand the proposed method. The second problem is the experiments. Many of the experiments are very low dimensional and have been solved by hundreds of methods, which all beat some baselines (especially the van der pol oscillator). The exoskeleton is better from the complexity, but the reporting of single trajectories and the reported RMSE is not informative. It would be great to look at where this model is important and enables something new. The reported RMSE numbers are also not vastly better than the baselines. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a model learning technique that can learn/estimate the initial hidden state. To estimate the initial state, the paper uses a KKL-based recognition model. In the experiments, the proposed method is applied to learn dynamics models on three different datasets, the earthquake model, the FitzHugh-Nagumo neuron model, the van der pol oscillator and an exoskeleton.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "* Improve the clarity of the proposed method and underlying theory.\n\n* Make the structure of the dynamics model much clearer. Right now one can easily miss that the authors nearly assume to know the complete dynamics model. Framing some of the experiments as NODE is questionable as this overlaps with classical system identification methods.\n\n* Improve the experiments to highlight the impact of the work. Right now the main results of the trivial toy problems are not very convincing. On the exoskeleton the importance of the results is questionable. It would be great to look deeper into how this model imrpoves the control of the exoskeleton. Reporting offline RMSE on fixed datasets has barely any implications for real-world control. \n\n* All tables are missing confidence intervals. ", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "AtWcNROeLX", "reviewer": "Reviewer_5jQP", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n- Applying a KKL-based recognition model to latent ODEs or partially observed dynamical systems is a very natural idea. To my knowledge, this is the only neural ODE (NODE) encoder based on rigorous theory, which in turn comes with certain guarantees. On its own, this is a valuable contribution.\n- The experiments are very elaborate. KKL-based encoder almost always achieves better results or is on par. This is one of the strongest aspects of the work!\n- Overall, the paper is written well. In particular, I enjoyed reading the related work.\n\n### Weaknesses\n- Certain parts of the text can be improved, see the notes at the end of this section. Since the main audience will be the machine learning community, several terms/concepts in Section 4.2 should be explained in more detail. For instance,\n  - What does $z$ denote?\n  - What do \"Hurwitz\" and \"controllable pair\" mean?\n  - What does \"observer\" mean intuitively?\n  - What are the dimensionalities of $D$ and $F$, and why are they so?\n- In general, the measurement function $h$ is assumed to be known. I believe this could be slightly too restrictive.\n- Figure-1 provides a taxonomy of ODE methods but it completely disregards Gaussian process (GP) based ODE systems (this applies to the related work). I believe a better way of presenting this taxonomy would be \"black-box ODEs\", \"structured ODEs\" and \"parametric models\".\n\n\n### Comments\nI would be happy if we can chat a bit on the following:\n- I'm not sure if the word \"soft\" fits in Figure-1. Approaching from the \"additional penalty terms\" perspective, it does feel soft. However, including partially known dynamics in the model formulation is a very strong inductive bias and sounds much stronger than \"soft\".\n- _“hard” constraints in the optimization problem_: Don't the methods listed after this phrase undergo significant architecture changes (compared to vanilla NODEs)? I would call this \"hard constraints on the (dynamics) formulation / differential function\" instead of \"optimization problem\" as the latter is a much broader concept that includes completely unrelated topics, e.g., combinatorial/constrained/discrete optimization.\n- _We can then learn a residual NODE on top_: This is vague and requires a more detailed explanation.\n- $y^j(t_i)$ reads as if the data trajectory is continuous. I suggest using $y^j_i$.\n- About remark-1: Extremely expressive h would surely make identification difficult. I believe this remark requires elaboration.\n- _for $t_c$ large enough_: How would one quantify that?\n- _y is the output of (1)_: Looking at (1), the output $y$ depends on the initial value $x_0$. Then, KKL-based recognition model, which generates $x_0$, depends on $y$. Isn't this a circular dependency?\n- _The direct method with $t_c = 0$ was run but_: I'm not sure if this is reasonable/fair. Do other methods take only the initial observation as input? If so, does RNN+ make sense at all? \n- _the more the observer has converged, the closer the relationship_: That is true; however, do we expect $x(0)$ and $T^*(z(0))$ to be the same up to the added Gaussian noise?\n- Harmonic oscillator experiment is super nice!\n- It would be nice to understand more about why Figure-7d is worse compared to 7b. Why the residual model cannot correct the linear dynamics?\n- Can the model be applied to more complex datasets, i.e., images? If so, how? If not, why not?\n\n### Minor comments\nThese are very minor comments, feel free to disregard them if you don't think they fit:\n- _The behavior of such systems often follows a certain structure, which can be inferred, for example from the laws of physics_: Does the behavior or the system follow a certain structure? Here, does \"inferred\" refer to a human being observing a system and making conclusions concerning the structure or inference from data using ML tools?\n- _even an accurate model cannot account for all aspects_: all aspects of what?\n- I'm not sure if \"this experimental data\" or \"this data\" are grammatically correct.\n- _In general, learning the dynamics is an ill-posed problem_: Including references would be nice.\n- _Learning latent representations of the available data_: Maybe replace with \"Learning latent representations from data\" as there is no \"non-available data\"\n- _a continuous-time ODE_: Is there a \"discrete-time ODE\"?\n- _However, as for most machine learning methods, little insight into the desired latent representation_: Up until here, the reader is not informed about latent NODEs. In fact, the sentence \"_The aim is to approximate a vector field_\" feels like NODEs are defined solely in the data space.\n- _However, in the case of partial observations, the unknown latent state_: Again, why/when NODEs are latent should be discussed before this.\n- _We show that the KKL-based recognition models perform well_: This can be elaborated just a bit; why do they perform better, what is the performance metric, etc.\n- _We formulate NODEs as_: Maybe re-formulate as it is already formulated and is flexible? The novel bit in this \"formulation\" should be highlighted.\n- _We introduce recognition models to link observations and latent state_: This was already done in Chen'18. Maybe say something like \"new recognition models based on KKL to link...\"?\n- _In the area of system theory_: In system theory\n- _The topic of system identification aims at_: System identification aims at\n- _Observers often assume a good dynamics model_: What is meant by \"good models\"?\n-  _In that case, the unknown parts_: In which case; former or latter?\n- _can ensure certain physical properties_: Maybe a few examples?\n- _do not verify important physical principles such as conservation law_: Maybe satisfy or infer instead of verify?\n- _for an overview of this approach_: Maybe these efforts?\n- _the take on recognition models_: This sounds a bit too informal.\n- _Initial conditions $x_a$_: Maybe $x_{a,0}$ to stress initial?\n- _as a neural network that takes as input $z(t_c)$_: At least I felt like \"where does this term come from?\"\n- Only after reading Section 4.1, I understood that \"direct methods\" rely on stacking whereas recurrent methods rely on dynamical encoders. I suggest mentioning this earlier.\n- The whole Section 4.1 could be a separate \"background\" section. It seems to disrupt the natural flow of \"problem definition, methodology, experiments\".\n- _to the latent state_: I would say to the latent *initial* state.\n- In Figure-2 caption, I would say _the unknown parameters of the *known osciallating* dynamical systems_.\n- trajectories of \"3 seconds\" as it is not clear at first.\n- I would fix the y-axis scale in Figure-3.\n- I would match (a)-(e) in Figure-5 with Figure-1.\n- _learning the residual of the prior linear model on top of this structure_: This requires more explanation: Do you use an additive differential function?\n- The performance metric in Table 3 should be noted.\n- Maybe a very brief summary of results in the conclusion, e.g., in 2 sentences?\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces a new recognition model for latent ordinary differential equations (ODEs). The framework relies on nonlinear observer designs, in particular Kazantzis-Kravaris/Luenberger (KKL) observer. Under certain assumptions, the proposed encoder design is able to recover the unknown latent state (in the limit of an infinite sequence). It is exhaustively evaluated on a series of dynamical systems with varying degrees of prior knowledge. It is also compared against the encoder architectures utilized in state-of-the-art methods. The proposed method consistently outperforms alternative constructions when tested on simple simulated dynamics as well as a robotic exoskeleton dataset.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- The text could be slightly modified so that GP-ODEs are also included in the taxonomy. Testing the proposed framework on GP-ODEs would be even better but this is not necessary for the paper to be accepted. \n- The terms/concepts in Section 4.2 could be explained in more detail. This is crucial for the understandability of the work and contributions.\n- I would be happy if the author can reply to the items listed under \"Comments\" above.\n\n", "broader_impact_concerns": "I feel there is no direct adverse ethical implication of this work. ", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "QXV3sdnqOG", "reviewer": "Reviewer_Gp37", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\n- The main contribution of the paper, i.e., the construction of the KKL observer-based recognition networks, is interesting.\n- The experiments indeed show the possible utility of the proposed method.\n\n### Weaknesses\n\n- The interrelation between the two topics of the paper (i.e., recognition models and structured NODEs) is unclear.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The main topic of the paper is the construction of recognition networks for neural ODE's initial state. The authors propose to utilize the theory of the Kazantzis-Kravaris/Luenberger (KKL) observer and to build a recognition network on top of it. The effectiveness of the proposed recognition network is examined empirically.\n\nThe authors discuss another topic, the spectrum of the extent of structured-ness of neural ODEs. In some of the experiments, they examine the differences between neural ODEs with different levels of prior knowledge incorpolated. ", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "### 1.\n\nThe interrelation between the two topics of the paper (i.e., recognition models and structured NODEs) should be discussed more clearly. These are interesting yet two independent topics, at least given what I could read from the current manuscript. The proposed method is indeed applied to models with different levels of prior knowledge in Sections 5.2 and 5.3, but it is still unclear why the proposed KKL recognition model is (supposed to be) particularly suitable for such models with gradual priors.\n\n### 2. (related to Point 1)\n\nPlease elaborate on the following claim in Section 5.2:\n\n> These results illustrate that NODEs with recognition models can flexibly incorporate\n> gradual priors for learning dynamics from partial and noisy observations.\n\nAlthough I do not disagree with it, the intention of such a claim is ambiguous. Do you mean \"NODEs with recognition models\" *in general* can flexibly incorporate gradual priors? Or, do you mean the proposed KKL recognition models specifically? If the authors' intention is the former, then, again, it is unclear why the paper discusses this topic of gradual priors together with the topic of KKL recognition models. If the authors' intention is the latter, the current empirical results do not support such a claim.\n\n### 3.\n\nIn the proposed method, a sequence of $y$ is first put into the system of $z$ in Eq.(6), which is integrated backward in time. This structure reminded me of \"state space layers\" [Gu+ 2021; Gu+ 2022]. It would be great if the relationship between them and the proposed recognition models could be discussed.\n\n- [Gu+ 2021] https://openreview.net/forum?id=yWd42CWN3c\n- [Gu+ 2022] https://openreview.net/forum?id=uYLFoz1vlAC\n\n---\n\nBelow are minor points.\n\n### 4.\n\nIn the first half of Section 2.2, the categorization of papers into \"soft\" and \"hard\" constraints sounds sometimes strange.\n\n- The stable deep dynamics model by Manek & Kolter (2019) is classified as a \"soft\" approach, with which I do not agree. Their method ensures stability by construction without any need for regularization. \n- The authors define the \"hard\" approach as something in which constraints are enforced in the optimization problems, but it is not always the case in the papers introduced as examples of the \"hard\" approach. For example, the Hamiltonian neural nets are usually learned by an unconstrained optimization; there is no need for constraints because the loss function of HNNs encodes the physical knowledge by construction.\n\n### 5.\n\nIn Section 4, notations $y([0,t_c])$ and $y_{0:t_c}$ seem mixed. Please unify if they denote the same notion. If not, please specify what each notation means specifically.\n\n### 6.\n\nAt the beginning of Section 4.1, the authors say\n\n> Some recognition methods have been proposed in the literature, not necessarily for system identification\n> with NODEs, rather for probabilistic (Doerr et al., 2017) or generative Yildiz et al. (2019) models.\n\nAnd later in the bottom of Page 5, they say\n\n> There are few works on NODE-based system identification from\n> partial observations, some of which train a recognition model from $y_{-t_c:0}$ (Ayed et al., 2020; Yildiz et al.,\n> 2019; Norcliffe et al., 2021a), or ...\n\nThe work of Yildiz et al. (2019) appears in the both places, which is contradictive. In my understanding, the latter is true.\n\n### 7. \n\nIn Section 5.3, why did you defer the result on the \"longer test trajectory with an input frequency outside of the training regime\" to the appendix? I think it would be much more interesting than the rather trivial result that is currently presented in Section 5.3\n\n### 8.\n\nIn Section 5.3, the idea and the procedures of the experiment with EKF should be described in more detail.\n\n### 9.\n\nPlease verify if the subcaptions of Figure 16 are correct.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a new recognition model for latent Neural ODEs in the theme of proposing end-to-end learning physical models from experimental data. The proposed method utilizes the theory of  non-linear observer design (KKL observer) and builds a recognition model on top of it. With various empirical analysis the authors demonstrate the effectiveness of the proposed recognition network. 

While some reviewers had concerns on capabilities and presentation, overall all of them stated claims in the paper are correct and supported by the evidence and the result would be relevant to some of the TMLR audience. 

There were particular requests to improve visual presentation to guide the reader better.  The AE believes that the 12 page limit is not strict for TMLR (indeed longer paper is welcome and 12 page is just for guidance on the review timeline) and should be a soft suggestion if there's a tradeoff for clarity. Incorporating reviewer `yCgK`'s suggestion of including intuitive graphics for qualitative understanding of the method in the main body of the paper would be useful. While the paper is in a good state to be accepted, AE hopes to see some effort in improving clarity thus recommending "Accept with minor revision" for the camera-ready.

Decision: Accept (poster)