Continuous-time identification of dynamic state-space models by deep subspace encoding
Authors: Gerben I. Beintema
Maarten Schoukens
Roland Tóth
Venue: ICLR 2023 poster (2023)
Keywords: Continuous-time
State-space
Artificial Neural Networks

Abstract: Continuous-time (CT) modeling has proven to provide improved sample efficiency and interpretability in learning the dynamical behavior of physical systems compared to discrete-time (DT) models. However, even with numerous recent developments, the CT nonlinear state-space (NL-SS) model identification problem remains to be solved in full, considering common experimental aspects such as the presence of external inputs, measurement noise, latent states, and general robustness. This paper presents a novel estimation method that addresses all these aspects and that can obtain state-of-the-art results on multiple benchmarks with compact fully connected neural networks capturing the CT dynamics. The proposed estimation method called the subspace encoder approach (SUBNET) ascertains these results by efficiently approximating the complete simulation loss by evaluating short simulations on subsections of the data, by using an encoder function to estimate the initial state for each subsection and a novel state-derivative normalization to ensure stability and good numerical conditioning of the training process. We prove that the use of subsections increases cost function smoothness together with the necessary requirements for the existence of the encoder function and we show that the proposed state-derivative normalization is essential for reliable estimation of CT NL-SS models.

Published as a conference paper at ICLR 2023
CONTINUOUS-TIME IDENTIFICATION OF DYNAMIC MODELS BY DEEP SUBSPACE ENCODING
Gerben I. Beintema, Maarten Schoukens & Roland Töth'
Department of Electrical Engineering, Findhoven University of Technology, The Netherlands {g. beiintema schoukens, r. tothh@tue.....
ABSTRACT
Continuous-time (CT) modeling has proven to provide improved sample effi ciency and interpretability learning the dynamical behavior of physical systems compared to discrete-time (DT) models. However, even with numerous recent de- velopments, the CT nonlinear state-space (NL-SS) model identification problem remains to be solved in full, considering common experimental aspects such as the presence external inputs, measurement noise, latent states, and general ro- bustness. This paper presents novel estimation method that addresses all these aspects and that can obtain state-of-the-art results on multiple benchmarks with compact fully connected neural networks capturing the CT dynamics. The pro- posed estimation method called the subspace encoder approach (SUBNET) ascer- tains these results by efficiently approximating the complete simulation loss by evaluating short simulations on subsections of the data, by using an encoder func- tion estimate the initial state for each subsection and novel state-derivative normalization to ensure stability and good numerical conditioning of the training process. We prove that the use of subsections increases cost function smoothness together with the necessary requirements for the existence of the encoder func- tion and we show that the proposed verevderiv normalization is essential for reliable estimation of CT NL-SS models.
INTRODUCTION
Dynamical systems described by nonlinear state-space models with a state vector (t) E R are powerful tools of many modern sciences and engineering disciplines to understand potentially com- plex dynamical systems. One can distinguish between Discrete- Time (DT) #x+1 = f(x,, ux) and
Continuous (CT) du(t)) f (x(t), u(t)) state-space models. In general, obtaining DT dynam- ical models from data is easier than CT models since data in computers is represented as discrete elements (c.g. arrays). However, the additional implementation complexity and computational costs associated with identifying CT models can be justified in many cases. First and foremost, from the natural sciences, we know that many systems compactly described by CT dynamics which makes the continuity prior of CT models a well-motivated regularization/prion (De Brouwer et al.] 2019). has been observed that this regularization can be beneficial for sample efficiency (De Brouwer et al., 2019) which is common observation when "including physics" in learning approaches (Kar- niadakis et al.) 202T). Furthermore, the analysis of ODE equations a well-regarded field of study with many powerful results and methods which could further improve model interpretability (Fan) jet al., 202T), such as applied in [aai et al. (2019). Another inherent advantage is that these models can be used with irregularly sampled or missing data (Rudy et al. (019).. Lastly, in the control community, CT models are generally regarded desirable for control synthesis tasks as shaping the behavior of the controller is much more intuitive in CT (Garcia et al. 1989). Hence, developing robust and general CT models and estimation methods would be greatly beneficial.
In the identification of physical CT systems, it is common to encounter challenges such external inputs (u(t)). noisy measurements, latent states, unknown measurement function/distriibttion (e.g. v(() = h((((()), the need for accurate long-term predictions and need for sufficiently low com- putational cost. For instance, all these aspects need to be considered for the cascade tank benchmark
Also associated with, Systems and Control Laboratory, Institute for Computer Science and Control, Bu- dapest, Hungary.

---

Published as a conference paper at ICLR 2023
Measurements with noise
yt
ODE solver it fo (2,, u
gt(t)
(t)
Long term
2(0)
Prediction
ho
(tt)
Hidden
States
u(t)
(1))
External
Signals
Modelling Setting
Model Structure
In this work, consider the problem of estimating continuous-ting time (CT) state-space models from noisy observation (additive noise) with long-term prediction capabilities, hidden states and external signals in computationally efficient and robust manner.
Figure 1:
problem (Schoukens & Noël 2017]. These aspects and the considered state-space model summarized in Figure Many of these aspects have been studied independently, for instance, Bra- jard et al. (2020); Rudy et al. (2019) explicitly addressed the presence noise on the mcasurement data, Maulik et al. ( Chen et al. (2018) provided methods for modeling dynamics with latent states, Zhong et al. (2020) considers the presence of known external inputs, [Zhou et al. (202aa)
provides computationally tractable method for accurate long-term sequence modeling. However, formulating models and estimation methods for the combination of multiple or all aspects is in com- parison underdeveloped with only few attempts such Forgione & Pigal (2021a) that have been made.
In contrast to previous work, we present CT encoder-based method which general, robust and well well--eerfoming estimation method for CT state-space model identification. That is, the formula- tion addresses noise assumptions, external inputs, latent states, an unknown output function, and provides state-of-the-art results on multiple benchmarks of real systems. The presented subspace encoder method is summarized Figure 2 The proposed method considers a cost function evalua- tions on only short subsections of the available dataset which reduces the computational complexity. Furthermore, we show theoretically that considering subsections enhances cost function smooth- ness and thus optimization stability. The initial states of these subsections encoder function for which we present necessary requirements for its existence. Lastly, we intro- duce normalization of the state and state-derivative and we show that it is required for proper CT estimation. Moreover, we attain additional novelty as these results are obtained without needing to impose specific structure on the state-space (such as Greydanus et al. (2019); Cranmer et al. 22020)) obtaining practically widely applicable method.
estimated using the
Our main contributions are the following;
We formally derive the problem of CT state-space model estimation with latent states, external inputs, and measurement noise.
We reduce the computational loads by proposing a subspace encoder-based identification algorithm that employs short subsections, encoder function that estimates the initial latent states of these subsections, and state-derivative normalization term for robustness.
We make multiple theoretical contributions; (i) we prove that the use of short subsections increases function smoothness by Lipschitz continuity analysis, (üi) derive neces- sary conditions for the encoder function to exist and (iil) we show that a state-derivative normalization required for proper CT model estimation.
We demonstrate that the proposed estimation method obtains state-of-the-ar results on multiple benchmarks.
2 RELATED WORK
One of the most influential papers in CT model estimation is the introduction of neural ODEs (Chen et al. 2018), which showed that residual networks is presented as an Euler discretization of contin- in-depth neural network. Moreover, they also show that one employ numerical integrators
2

---

Published as a conference paper at ICLR 2023
to integrate through the depth in a computationally efficient manner. This depth can be interpreted as the time direction to be able to model dynamical systems. The ideas in the neural ODE contri- bution have been extended to/used in, for instance, normalizing flows to efficiently model arbitrary probability distributions (Papamakarios et al. 2021 Grathwohl et al. 2019). and enhance the under- standing and interpretability of neural networks (Fan et al.. (021). However, the neural ODE does not scale well for long sequences, nor does it consider external inputs or noise, and the optimization process is often unstable.
An adjacent research direction is the method/models which consider CT dynamics and directly use the state derivatives and even often the noiseless states to formulate structured and inter- pretable models such as Hamiltonian Neural Networks (HNN) (Greydanus et al. (019), Lagrangian Neural Networks (L.NN) (Cranmer et al., 2020) and Sparse Identification of Nonlinear Dynamics (SINDy) (Brunton et al. 2016). In contrast, the proposed method is formulated for an unstructured state-space and does not require the system state or the state derivatives be known.
Our method is in part related to (Ayed et al., 2019) which concerns the estimation of CT models with latent variables. They also employ an encoder function to estimate initial states, however, this encoder is only dependent on the past outputs, contains a partially known state and there is no theoretical support for the method. Furthermore, in that work, only a fixed output function is considered and the involved optimization problem is solved our formulation alters the simulation loss function to obtain a computationally desirable form. Fur- thermore, (Forgione & Piga 2021a), which compare in this work, considers CT model with latent variables, subsections, and an additional loss term for the integration error. However, they in- clude the initial states of these subsections as free optimization parameters. This increases the model complexity with the number of subsections. In contrast, our proposed method uses an encoder to estimate the initial states. This results in fixed model complexity. Furthermore, we only employ single loss function and a novel state-derivative normalization term. Additionally, we provide the- oretical insights into these existing elements and extend them to the considered setting in a robust and computationally efficient manner.
optimal control problem whereas
3 PROBLEM STATEMENT
Consider a system represented by a continuous time nonlinear state-space (CT NL-SS) description sampled at fixed interval At for simplicity:
a. (t) = f(xx(t), y/3 h(T,,,, ux) + Wk.
where the subscript notation denotes sampling as Fa,. = I. (kAt), (1) R"*, is the system state variable, u(t) RT is the input, Ry" is the output, f represents the system dynamics and gives the output function while wk R"> a iidd. zero-mean white noise process with finite variance E...
For this system, the CT model estimation problem can be expressed for a given dataset of measure- ments:
D = {uu, yo), (u1, 91),.., ..., with unknown Wy. (t), (t), in and initial state xz (0), as the following optimization problem (a.k.a. simulation loss minimization):
min (.x(0)
Σ lu(katt ((KAt))
N
(()) ha(x)t))
((t) = fo(x(t), u(t)),
where ztt) E R the model state, he and fo are the output and state-derivative functions pa- rameterized by 0 and being Lipschitz continuous in their inputs and parameterization. These two functions are formulated as multi-layer feedforward neural networks during our experiments.
To obtain the simulation output (kAtt) can integrate (+t) fo(x(t), u(t)) starting from the initial state z0). This integration can be performed with any ODE solver that allows for back- propagation such Fuler (ar(t + At) sx(t) At Alfo(a()) (3r(1), u())). RK4, or numerous adaptive step
3

---

Published as a conference paper at ICLR 2023
Ribeiro et al. 2020 To make this well well-posed optimization problem, additional information or an assumption on the inter-sample behavior of u(t) is required, since, for example, u( At/2) is not present in Dx. This behavior is often chosen to be Zero-Order Hold
methods (Chen et al. 2018
(ZOH) [Ljung] 1999) as can be viewed in Figure
Multiple major issues encountered when solving the optimization Problem 22 with gradient- descent-based method. The first issue is that computing the value of the loss function requires a forward pass on the whole length of the dataset (Ayed et al. 2019). Hence, the computational complexity grows linearly with the length of the dataset. Furthermore, a common occurrence is that the values of æxt) or its gradient grows exponentially which results in non-smooth loss functions or gradients. This causes gradient- based optimization algorithms to become unreliable since the optimization process might be unstable or converges to a local minima (Ribeiro et al. 2020). All these issues are addressed in the proposed method.
PROPOSED METHOD
We propose to consider multiple overlapping short subsections of length T&t form truncated simulation loss instead of simulating over the entire length of the dataset. We express this in the following optimization problem (note that we express the optimization problem with discrete-time notation (uy == u(k for brevity):
N-T Σ
1
minimize
yl In+k(n
N - T max (Ta, mb) +
mmmax(nn, mb)
s.t. Dn4k/n = ho ho(In+Mmm)
(3)
1o+++1/1 = ODEsolve for Fatkk Untke At),
Tn/n = =:ee Voeuun-- Un-ns Un- yn vn-n.).
Here, the pipe (1) notation indicates the current index and the starting index as (current index start index) to differentiate between different subsections. This pipe notation is similar the notation used in Kalman filtering and conditional probability distributions (Chui et al. 2017). Furthermore, ODEsolve indicates numerical scheme which integrates 1/TT fo (z. u) from the initial state zn+* for a length of At given the input Un+4k- Lastly, we introduced an encoder function dhs with encoder lengths nh, and nı, for the past output and input samples respectively, which estimates the initial states of the considered subsection. This encoder function will also be parameterized ward neural network during our experiments. A graphical summary the proposed method called the CT subspace encoder approach (abbreviated as SUBNET) can be viewed in Figure 2
feedfor-
The first observation is that optimization Problem 3 is a generalisation of (2) since if = N and na n4, = O, the original optimization Problem is recovered. However, this optimization problem is less computationally challenging to solve if < N since the first sum can be computed in parallel. In other words, computational costs scale as (TT)) for 3 and O(N) for 2 Moreover, the smoothness of the encoder cost function is also enhanced since the associated
might observe,
Lipschitz constant Lypea can scale exponentially with the subsection length (TAI) shown in
Theorem The enhanced smoothness is reflected in the ease of optimization (Ribeiro et al. 2020
Theorem 1. The Lipschitz constant Lyyows of the cost function 3
[| ylene)(O2))l S L Vio 01 - -O2112
(4)
scales as
Lpions = 0(exp(27
(5)
where Lg is the Lipschitz constant of fe.
Proof.
See Appendix 8.1
'The adjoint methods for gradient computation is not within the scope of this research.
4

---

Published as a conference paper at ICLR 2023
(t((t)
( + At
O(t + 2
hg
hg
hg
ODE solve 2
ODE solve I = FoLæ, u
x (1
(t + At(t)>
(t + 2
[ 2
4
(u(t Af).... ult mx4r) 1(( At,, g(t n.Ai))
(t)
uut + At)
Figure 2: The CT subspace encoder (SUBNET) method applied subsection of the data of length A' starting from time t. The encoder We estimates the initial state, he provides the output predictions while fo governs the state dynamics. All three functions are parameterized by fully connected neural networks. Lastly, the + factor is the novel state-derivative normalization factor which significantly increases optimization stability. All three functions are optimized together by minimizing the mean squared difference of the model outputs of multiple subsections the available training data seen in Eq. (3) which both reduces the computational cost and enhances cost function smoothness as seen in Theorem
An error in the initial state of Faln significantly bias the estimate due the short nature of the subsections. To counter this, we formulated an encoder function bh which estimates the initial state of each subsection. We do not add any additional loss term since improved initial state error estimate also minimizes the transient error (Forgione & Piga 2021a) which is present in the encoder loss.
natural question to ask is under which conditions there exists an encoder function that can map from the past inputs and outputs to this initial state. In Appendix 8.2 we formally derive necessary conditions for the existence. These necessary conditions are that, state derivative fe requires to be Lipschitz continuous x, and if the number of considered past outputs na and inputs n, are equal then 2 nz/nn, fy needs to be satisfied, among other conditions.
is widely known that input and output normalization is essential for obtaining competitive models throughout deep learning in terms of respecting the prior assumptions made in for instance Xavier Weight Initialization (Glorot & Bengio, 2010). Input and output normalization can seen to be insufficient when considering CT state-space model due to the presence of the hidden state and the state-derivative fa( u). However, as shown in Theorem 2 any CT system can transformed to become normalized by the introduction of a state transform and positive 1/T normalization factor.
Given (tt) = f(((t), u(t)) and y(t) h(x(t), u(t)) that defines the dynamics of system. For any bounded non-zero state-rajjccooy x(t) R== and input signal u(t) E R that satisfies i(l) f((((t,, u(()) for all t E R. there exists T and scalar state transformation (ā(() = x(t) such that both the equivalent state trajectory & and state- natt--erivvatve function f(ž, u) of the transformed system ã (ā(()), u(t)) normalized on the time interval [0, L] as
Theorem 2.
/''
1 nh
RMS(
dt dt
RMS(F(8 u)) =1
(6)
(MRSS((FF u)) O.
Proof. With
RMS(#) (+)
7 (M)
&
RMS(x)
the normalization conditions are satisfied, as shown below
RMS(+) = RMS(*)/7==
(8a)
RMS(f(2, w)) = -RMS ()) 7RMS(E)/> (RMM(E)//MS/) =
(8b)
5

---

Published as a conference paper at ICLR 2023
Spring
Pallyy
Pitse
BC
ostinal
altangg flter 22
Rexille bok
Power Amplifier
Figure 3: photo of the Cascade Tank with overflow (CCT) system (Schoukens & Noël 2017) and a graphical depiction of the Coupled Electric Drive (CED) system (Wigren & Schoukens, 2017]. These two systems and the EMPS benchmark are the basis of the benchmarks used in the analysis and comparison of the SUBNET method.
Hence, to assure the existence of properly normalized model where both the state-derivative function and state I are normalized, it is sufficient to include a state and state-derivative nor- malization factor. Furthermore, this proof also guides the choice of T since the amplitude of RMS(+)/RMS((i might be known from physical insight or by an approximate model.
5
EXPERIMENTS
5.1
BENCHMARK DESCRIPTIONS
The Cascade Tank with overflow (CCT) benchmark (Schoukens & Noël 2017 Schoukens et al., 2017) consists of measurements taken from two-tank fluid system with a pump. The input signal controls water pump that delivers water from the reservoir to the upper tank. Through small opening in the upper tank, the water enters the lower tank where the water level is recorded. Lastly, through a small opening in the lower tank, the water re-enters the reservoir. This benchmark is non- linear as the flow rates are governed by square root relations and the water can overflow either tank which is hard saturation nonlinearity. The benchmark consists of two datasets with measurements of 1024 samples each at a sample rate of A/ = 4s. The first dataset is used for training, the first 512 samples of the second set are used for validation (used only for early stopping) and the entire second set for testing. Most of the other methods to which we compare our approach use the entire second set as validation and test set, as no explicit test set provided in this benchmark description.
The Coupled Electric Drive (CED) benchmark |Wigren & Schoukens 2017) consists of measure- ments from a belt and pulley with two motors where both clockwise and counter-clockwise move- ment is permitted. The motors are actuated by the given inputs and the measured output is a pulse transducer that only measures the absolute velocity (i.e. insensitive to the sign of the velocity) of the belt. The system approximately has three states; the velocity of the belt, the position of the pulley, and the velocity of the pulley. The benchmark consists two datasets of measured 500 samples each at sample rate of At 20ms. The first 300 samples used for training and the other 200 samples are for testing, of those samples the first 100 samples are also used for validation with both datasets. Similar the last benchmark, even with this overlap, it is still fair comparison most of the other methods to which we compare use the entire second set as validation and test.
The Sreetro-Mechanicee Positioning System (EMPS) benchmark (Janot et al. 2019) consists of measured signals from a one-dimensional drive system used to drive the prismatic joint of robots or machine tools. The provided measurements of the position are obtained closed-loop actuated and no direct velocity measurements are available. The main source of nonlinearity are the nonlinear friction effects (e.g. static and dynamic friction). The benchmark consists of two sequences of samples 24841 with a sampling time of A/ Ims. As prescribed by the benchmark, the first sequence is used for training and validation and the second sequence for testing (i.e. the validation set and test set completely disjoint). Specifically for the CT subnet implementation, 17885 samples of the first set for training and the last 6956 samples are used for validation while the entire second set is used for testing.
utilize

---

Published as a conference paper at ICLR 2023
The test RMSE simulation on two benchmarks for the CT SUBNET method using an ensemble of models. The value given is the best RMSE simulation of all estimated models and the value between parentheses is the mean performance of the models. Note that we are unable to report the results for the neural ODE without state-derivative normalization 1/T for CCT since the optimization was unstable.
Table I:
(b) CED benchmark
(a) CCT benchmark
Method
RMSE
[ticks]]
Method
RMSE
Set
Set
RBENN- FSDE Ayala
2014
0.130
0.185
BLA (Rolan et al 2017
0.75
GP with rational
Volterra model (Bipoutsoukis
2018
0.54
quadratic kemnel
Zhou et
0.150
0.167
State space with
GP-inspired prior (Svensson Schön 2017
SCI (Forgione& & Piga 20216
0.40
0.45
GP with squared
exponential kemel
Zhou et
202
0.153
0.132
0.149
0.120
NL-SS + NLSS2 Ralan et al 2017
stable
Welgand
2021
0.39
0.34
Sparse Bayesian ML.P
al. 202th
(0.187)
(0.134)
Cascaded Splines Scarpinit
0.216
0.110
TSEM (Forgione & Piga ( b
0.33
Tensor B B-spunes (Kragoz Baweller 2020
0.30
Spurse Buyesian LSTM Zbou et al.
2021b
0.121
0.097
Vanilla neural ODE (Chen et 2018
NaN
Extended Fuzzy Logic
(0.155)
(0.126)
noural ODE with
0.18
Sabahi Alkbarzadeh-T 2015
0.150
0.092
normalization (At/T = 0.03)
(0.33)
Crey-Rox with physical
Vanilla neural (Chen et
2018
0.141
0.098
0.18
(0.362)
(0.370)
overflow model (Rogers al. 2017
neural ODE with
0.131
0.086
DT subspace encoder
0.37
normaliattion (At/T 0.12)
(0.198)
(0.15%)
(0.97)
CT subspace encoder
0.22
DT sabspace encoder
0.109
0.117
(At/r = 0.032)
(0.30)
(0.187)
(0.1557)
CT subspace encoder
0.115
0.074
(AI/T U.3)
(0.143)
5.2 RESULTS
Using the SUBNET method, we estimate models where the three functions he fo and Ve are im- plemented as 2 hidden layer neural networks with 64 hidden nodes per layer, tanh activation and a linear bypass from the input to the output for CCT and CED and hidden layer with 30 hidden nodes for EMPS. As an ODE solver, we use single RK4 step between samples and assume that the input signal has been applied in a zero-order hold sense. As for the implementation of the CT subspace encoder-based method, the following hyperparameters are considered; n2₃ 2, 1l Thb 5 and =3 for CCT, nh = 3, Tha = Th = 4 and T = 60 for CED and n2 = 3, nu = Ti = 20 and T = 200 for EMPS. These hyperparameters are chosen based on hyperparameterss analysis shown in |eeintema et al. (202I) for discrete- time. We observed similar effects of the hyperparameters for continuous-time. The training is done by using the Adam optimizer with default settings (Kingma & Ba 2015) with a batch size of 32 for CRD, 64 for CCT and 1024 for EMPS and using a simulation on the validation dataset for early stopping reduce overfitting.
We also directly compare our method with a reproduction of neural ODE on both benchmarks. We adapt the code and the example ("latent.ODE, pp""" available online (Chen et al. 2018) to include ZOH inputs, leaving the neural network unaltered and an RK4 integrator. We observed that the initial model was unstable for CCT and underperforming for CED and, hence, the neural ODE method alone was unable to provide state-of-the thrtthe-rrt results. To stabilize and improve the neural ODE method we also introduce a state-derivative normalization term 1/T motivated by Theorem The value of 1/T for CCT and CED for neural ODE was initially chosen to be the optimal value found in the SUBNET approach, however, in the CED case, it was lowered due to optimization instabilities.
We compared our obtained model to the literature in Table [ for CCT and CED, We report both the mean and the minimum of ensemble of models estimated only differing in parameter ini- tialization. This ensemble consists of 17 SUBNET models for both CCT and CED and 24 and 8 neuralODE models for CCT and CED respectively. The table also includes the discrete-time (DT) subspace encoder which has the same network structure and loss function as the CT subspace en- coder but where the ODE solver is replaced by fe. The table shows that the obtained models with the CT subspace encoder method provide state-of-the-at results. The obtained models are the best- known with a black-box modeling approach on both benchmarks. Furthermore, we use unrestricted
The code used for both SUBNET and neural ODE experiments is available at h httptt/////iiii httpa://giituuutoo
Gerben Intema/CT- Cama/CC-suunee subnet
7

---

Published as a conference paper at ICLR 2023
(a) CCT
Models (At/r 0.018)
System
200
400
600
800
1000
Index time
(b.1) CED set
(b.2) CED set
M
Models
(At/T 0.32)
System
0
50
100
150
200
0
50
100
150
200
Index time
Index time
Time-domain simulation for both the (a) CCT and (b) CED benchmarks of the obtained models by the CT SUBNET method along the test set. Since the CED benchmark contains two separate test sequences, hence, they are shown in two separate figures.
Figure 4:
Table 2: The resulting test RMSE simulation on the test set for the EMPS benchmark compared with results in the literature.
System SUBNET model (At/r= 0.01)
250
Fully
Method
[mm]
Black-box
200
BLA
Janot
2019
61.67
(mmm)
150
JynoNEr
2.64
Forgione Piga
100
10 stable CT ANN
3.68
WWigand al 2021
50
TSEM
Forgione & Piga 202
SC.
6.65
5000
10000
15000
20000
Forgione & Pigu 2021
Index time
Chen ct al. 2018
neural
NaN
CT subspace encoder (At/7 0.01)
4.61
Figure 5: Time domain simulation results the EMPS benchmark test set using the model obtained from the CT SUBNET method.
"RMS values reported in Weigand et al. (2021
state-space and fully connected neural networks model elements. Remarkably, the resulting per- formance close to the performance of a grey-box box model. Furthermore, Figure Lillustrates that the resulting models have been able to model the nonlinear behavior present in both benchmarks.
Table D also contains the results the modified neural ODE with normalization. One observation that the best and mean performance difference is significantly larger than for SUBNET. We think that this due to the availability of only a single sequence the training set for the CCT benchmark (and two sequences for CED) which results in extensive overfitting. In comparison, the subspace en- coder method is less prone to overfitting since it uses many subsections of the available sequence(s). Moreover, the subspace encoder method only requires about 20 minutes train model to the lowest validation loss, whereas, neural ODE requires about 2 hours for CED and 5 hours for CCT.
We compare the CT SUBNET method applied on the EMPS benchmark to the existing methods in Table 2 and show the simulated response on the test set in Figure 5 The obtained results show remarkable accuracy over the entire 24841 samples showing that the CT SUBNET method is able to make accurate long-term predictions while using relatively short sub-sequences of only T = 200 in length during training. In the table, dynoNET is significantly better than the proposed method, however, this method utilizes grey-box knowledge (i.e. physics-based) in the network structure but this is system specific and not easily generalizable. Lastly, 10 stable CT ANN performs slighly better than the proposed method since enforces stability which can be a problem during estimation since there is a position integrator present in system. Furthermore, since the validation and test set disjoint also show that overfitting does play role in the reported results.
8

---

Published as a conference paper at ICLR 2023
CCT
10
RMS(0)
10"
RMS(x)
10
OHO
10
RMSE
0.0001
0.0003
0.001
0.003
0.01
0.03
0.1
0.3
A//T
(b) CED
C
RMS(A
40
100
RMS(x)
RMSE
10
0.0001 0.0003
0.001
0.003
0.01
0.03
0.1
0.3
10 DT
At/r
Figure 6: The influence of the state-derivative normalization hyperparameter A1/7 as à [ar u) on different model properties for both the (a) CCT and (b) CED henchmarks. The shown model properties are the mean state amplitude RMS(z), mean state-derivative amplitude RMS and the RMSE simulation the test set(s). These two figures show that there exists range of At/r where RMS()) (MS(z) se which numerically validates Theorem 2 Furthermore, Ât/T with this property also has significantly lowered RMSE simulation as argued in its introduction.
When we introduced the state-derivative normalization factor 1/T, we argued that it would nor- malize fo and that it would increase optimization stability and, hence, the quality of the ob- tained models. Here, we provide some empirical insight for these two statements by provid- ing parameter sweep over Al/T. To eliminate variations due different initial parameters trained an ensemble of models which creates box plots with mean state amplitude defined by
RMS(x)= A V Nn. Σx [I=(kt)))) mean state-derivative amplitude RMS(f), and the RMSE sim- ulation. These box-plots as shown in Figure 6 indeed illustrate that there exists an 1/T such that both RMS()) s RMS(z) 1 and that the best performing models are close to that value of 1/T. Moreover, to illustrate that improper normalization (i.e. т = 1) can diminish the performance for both CCT and CED, observe that the RMSE simulation the test set(s) is 2.0 and 0.3 [ticks/s] for At/T = At 4s and At/T = 0.02 respectively.
CONCLUSION
In this paper, have introduced the CT subspace encoder approach identify nonlinear dynamical systems in the presence of latent states, external inputs, and measurement noise. We have shown that the proposed method can obtain highly accurate CT models only consisting of fully connected neural networks. The approach has improved computational cost and stability by considering multi- ple subsections where the initial state is estimated with an encoder function and by state-derivative normalization term to improve optimization stability. We provided multiple theoretical proofs which provide additional insight and motivation for the method. These proofs are, increased cost function smoothness. necessary conditions for the existence of the encoder function and that for proper malization in CT modeling one requires the state-derivative normalization term. Furthermore, we obtain state-of-the-art results on all three considered benchmarks.

---

Published as a conference paper at ICLR 2023
7 REPRODUCIBILITY STATEMENT
Datasets: (i) The CCT dataset described in Schoukens & Noël (2017):
Schoukens et al. (017) and is available for download hatts::/datt 4tu.
al/articles Caatase//Cascace
Tanks Benchmark Combining,
Soft and Hard Nonlinearities/ 129600004
(ii) the CED dataset is
described in Wigren & Schoukens (017) and is available for download at
http:/ /www. it. uu www.seeerseeaaee publ 1 cations /
(ii) the EMPS datset is described (Janot et al., 2019) and is available for download at
www.beenncchhheeeeee emps
Code: Both the implementation and experiments of CT SUBNET and neural ODE suubett
available at atttppes/ggtth....coo//otteteaap
Hardware: It takes about 15 minutes to estimate a single CT SUBNET model and hours for single neural ODE model consumer laptop. A notable exception is CT SUBNET for EMPS which took about 10 hours due to the increased size and difficulty of the dataset.
REFERENCES
Helon Vicente Hultmann Ayala, Luciano F da Cruz, Roberto Z Freire, and Leandro dos San- Coelho. Cascaded free search differential evolution applied nonlinear system identification based on correlation functions and neural networks. In 2014 IEEE Symposium on Computational Intelligence in Control and Automation (CICA), pp. 1-7. IEFE, 2014.
Ibrahim Ayed, Emmanuel de Bézenac, Arthur Pajot, Julien Brajard, and Patrick Gallinari. Learning dynamical systems from partial observations. Second Workshop Machine Learning and the Physical Sciences, NeurIPS, 2019.
Shaojie Bai, J Zico Kolter, and Vladlen Koltun. Deep equilibrium models. Advances in Neural Information Processing Systems (NeurIPS), 32, 2019.
Gerben I. Beintema, Roland Toth, and Maarten Schoukens. Nonlinear state-space identification using deep encoder networks. In Learning for Dynamics and Control, pp. 241-250. PMLR, 2021.
Georgios Birpoutsoukis, Péter Zoltán Csuresia, and Johan Schoukens. Efficient multidimensional regularization for volterra series estimation. Mechanical Systems and Signal Processing, 104: 896-98 9966914,222 2018.
Julien Brajard, Alberto Carrassi, Mare Bocquet, and Laurent Bertino. Combining data assimilation and machine learning to emulate dynamical model from sparse and noisy observations: A case study with the Lorenz 96 model. Journal of Computational Science, 44:101171, 2020.
Steven L Brunton, Joshua L Proctor, and Nathan Kutz. Discovering governing equations from data by sparse identification of nonlinear dynamical systems. Proceedings of the national academy of 2016.
sciences, 1 113(15):3932-33)
Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. Advances in neural information processing systems, 31, 2018.
Charles K Chui, Guanrong Chen, et al. Kalman Kalman,iltering. Springer, 2017.
Miles Cranmer, Sam Greydanus, Stephan Hoyer, Peter Battaglia, David Spergel, and Shirley Ho. Lagrangian neural networks. ICLR Workshop, 2020.
Edward De Brouwer, Jaak Simm, Adam Arany, and Yves Moreau. Gru-ode-bayes: Continuous modeling of sporadically-obsrve time series. Advances in neural information processing sys- tems, 32, 2019.
Feng-Lei Fan, Jinjun Xiong, Mengzhou Li, and Ge Wang. On interpretability of artificial neural networks: A survey. IEEE Transactions on Radiation and Plasma Medical Sciences, 5(6):411- 760, 2021.
10

---

Published as a conference paper at ICLR 2023
Marco Forgione and Dario Piga. Continuous-timg time system identification with neural networks: model structures and fitting criteria. European Journal of Control, 59:69- 11,99-81220 2021a.
Marco Forgione and Dario Piga. dynonet: A neural network architecture for learning dynamical sys- tems. International Journal of Adaplive Control and Signal Processing, (((4)6612-266 2021b.
Carlos E Garcia, David M Prett, and Manfred Morari. Model predictive control: Theory and prac- tice survey. Automatica, 25(3):335-323 1989.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference artificial intelligence and statistics, pp. 249-256. JMLR Workshop and Conference Proceedings, 2010.
Will Grathwohl, Ricky TQ Chen, Jesse Bettencourt, Ilya Sutskever, and David Duvenaud. Ffjord: Free-form continuous dynamics for scalable reversible generative models. ICLR, 2019.
Samuel Greydanus, Misko Dzamba, and Jason Yosinski. Hamiltonian neural networks. Advances in Neural Information Processing Systems, 32, 2019.
Alexandre Janot, Maxime Gautier, and Mathieu Brunot. Data set and reference models of EMPS. In Nonlinear System Identification Benchmarks, 2019.
Ridvan Karagoz and Kim Batselier. Nonlinear system identification with regularized tensor network b-splines. Automatica, 122: 109300, 2020.
George Em Karniadakis, Ioannis Kevrekidis, Lu Lu, Paris Perdikaris, Sifan Wang, and Liu Yang. Physics-informed machine learning. Nature Reviews Physics, 3(6))222 4400 2021.
Tohru Katayama. Subspace methods for system identification, volume Springer, 2005.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. ICLR, 2015.
Lennart Ljjug. System Identification: Theory for the User. Prentice Hall information and system sciences series. Prentice Hall PTR, 1999. ISBN 9780136566953.
Romit Maulik, Arvind Mohan, Bethany Lusch, Sandeep Madireddy, Prasanma Balaprakash, and Daniel Livescu. Time-series learning of latent-space dynamics for reduced-order model closure. Physica D: Nonlinear Phenomena, 405:132368, 2020.
Francis Murray and Kenneth S Miller. Existence theorems for ordinary differential equations. Courier Corporation, 2013.
George Papamakarios, Eric Nalisnick, Danilo Jimenez Rezende, Shakir Mohamed, and Balaji Lak- shminarayanan. Normalizing flows for probabilistic modeling and inference. Journal of Machine Learning Research, 22((7): 2021.
Rishi Relan, Koen Tiels, Anna Marconato, and Johan Schoukens. An unstructured flexible nonlinear model for the cascaded water-tanks benchmark. ITAC-PapersOnLine, 50(1):452-457, 2017.
Antônio H Ribeiro, Koen Tiels, Jack Umenberger, Thomas Schön, and Luis A Aguirre. On the smoothness of nonlinear system identification. Automatica, 121:109158, 2020.
Tim Rogers, Geoffrey Holmes, Elizabeth Cross, and Keith Worden. On grey box modelling frame- work for nonlinear system identification. In Special Topics in Structural Dynamies, Volume 6, pp. 167-178. Springer, 2017.
Samuel H Rudy, J Nathan Kutz, and Steven L Brunton. Deep learning of dynamics and signal-noise decomposition with time-stepping constraints. Journal of Computational Physics, 396:483-506, 2019.
Farnaz Sabahi and Mohammad Reza Akbarzadeh-1 T. Extended fuzzy logic: Sets and systems. JEEE Transactions Fuzzy Systems, 24(3):500-543 2015.

---

Published as a conference paper at ICLR 2023
Michele Scarpiniti, Danilo Comminiello, Raffacle Parisi, and Aurelio Uncini. Novel cascade spline architectures for the identification nonlinear systems. IEEE Transactions on Circuits and Sys- tems I: Regular Papers, 62(7):1825-118 2015.
Maarten Schoukens and Jean Philippe Noël. Three benchmarks addressing open challenges in non- linear system identification. 20th World Congress of the International Federation of Automatic Control, 50:446 451, 2017. doi: of..016f.f.cooo.217......
Maarten Schoukens, Mattson, Torbjöm Wigren, and Jean-Philippe Noël. Cascaded tanks bench- mark combining soft and hard nonlinearities. In STU.eesear, Dataset, 2017. doi:
10.4121/12960004.
Andreas Svensson and Thomas Schön. A flexible state-spact model for learning nonlinear dy- namical systems. Automatica,, 80:189 199, 2017.
Jonas Weigand, Michael Deflorian, and Martin Ruskowski. Input-to-state stability for system iden- tification with continuous-time runge-kutta neural networks. International Journal of Control, pp. 1-17, 2021.
Torbjörn Wigren and Maarten Schoukens. Coupled electric drives data set and reference models. Department of Information Technology, Uppsala Universitet, 2017.
Yaofeng Desmond Zhong, Biswadip Dey, and Amit Chakraborty. Symplectic ODE-Net: Learning hamiltonian dynamics with control. ICLR, 2020.
Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting, In Proceedings of AAAI, 2021a.
Hongpeng Zhou, Chahine Ibrahim, Wei Xing Zheng, and Wei Pan. Sparse Bayesian deep learning for dynamic system identification. arXiv preprint arXiv:2107.129111 2021b.
APPENDIX
8.1
PROOF OF THEOREM 1
Recall that the subspace encoder loss function as in Eq,
3 can be expressed in the following form
N-T
Vane (0) = N N T max May nb) +1
Σ
I|un-k - In
T
k=0
with On+R/n = (x( = (o(x((n k) At)mntt)
a(t/nAt) - f u(t))
T
anli(|nAl) v'v(un 1 ..... Un ney Un - 1,,,... "n yn-nn.
Our aim is to derive the scaling in 7' of the Lipschitz constant Lene as defined in
Vene - Venc (02)) < 2 10: - 02112. v01, 0g E θ c R
(10)
We aim to express Lene in terms of the following Lipschitz constants
| (x1) h4g (a²) < ( x2 11 ||0 0g 12)).
(11a)
llfe, (I1, 2) foz (I2, u) < L (1|21 - x2
12 + ||0 - 0g 122,
(11b)
De,, (Ulpast Bpast) - vho (Upast Mpast) ) 11 < 110] - -02113
(11c)
for all æ1, 22 E X C R² and 401, O2 E θ C R A sufficient condition for these Lipschitz
constants to be finite is that the derivatives are finite on compact set of inputs which is often the case in feed-forward neural networks.
Furthermore, derive the scaling of Treme known properties of the Lipschitz constants;
12

---

Published as a conference paper at ICLR 2023
The sum property: c(x) a(x) + b(x) has a Lipschitz constant of Le La + Lo-
The multiplication property: c(ar (x))((x)) has a Lipschitz Le = MaLs + MoLa where
Ma is the maximal value of a on a closed set of inputs E X and M, being similarly defined.
Since the encoder loss function as in Eq. can be written as sum:
N-T Σ
V yone (0) = N - T max(naa m6)
(n, 01)
(12a)
+
== mo)
T-1
Vsa Va (n,01) = T
Σ
1lutk Dn+A/n
(12b)
k=0
where
IV (n, 01) - (n, 02)1 < 100 - 02112.
(12c)
Then by the sum property this implies that Trene Tuse since
N-T
Lene N T max(na: nb)
L
(13)
m=
Hence, it is sufficient consider only single subsection. Take 0 and drop the bar notation for simplicity. By the sum and multiplication properties, we derive get
T-1
(0,) - V 1 (0,) S 2/T
Σ (My + Ma) - 02, k=
(14)
where My is the bound on [ly(t)]l2 assuming a stable system and Mg. the bound on l1úxll2. The Mr bound scales the same as || $02,6: shown in Ribeiro et al. (022). The || 02 ll2 expression can be expanded by using Eq. Taaaa
Ú 1 < T. (kAV)))) 22 (222) - 110: -0211))
(15)
Next, we aim derive an expression for the Lipschitz constant (t) given in terms of
[t) 22(+) L, (t) 02112.
(16)
whereby Eq. Llc
(0) Inp.
(17)
By considering a small increment in time of length h, we can express L (t + h) in terms of (t) Using the fact that h is small we can Eurler step and discard higher order terms of as;
| (t + 11) x:2 h)ll = (1(2)(t) =x)( h/T(fo (x) (21(t), u(t)) - fo, (x2(t), u(t
-1))->2(2))) (1) 24/T1 <1 (200))l2 (1), u()) - Sez (12 u()))ll by the triangle inequality. Next, we replace all the f terms by Eq. (1bb and x terms by Eq. 16) to derive an expression for T... (t. + h) as
<
s (z) 1022==111111+1111 2 < (1)² + 20 Lz +1) [t) - (z(()
- x2 (2) (() x2 (t) ||0 0
L, + h) = / Lt)2 + 2hL (t) / (t) + 1L, /T.
This expression allows us to derive that the derivative L (t) is given by L (t h) L (t)
L (t)
lim h0
(18)
/1 Lt/
(19)
13

---

Published as a conference paper at ICLR 2023
which suggests that (t) is continuous in and has the closed form solution
(t) (0) + ! (t'(
(20)
sinh(tt arcsinh(Lp))
(21)
Now by substituting Eq. sion for Lene as
21 into, 16
and using 13 we arrive at the following expres-
Lene
2/T Σ (My + Ms)Lnssinn(AAAt//+++ arcshh + 1)
(22a)
which scales in the limit of large T as
Tee = O(xxp(2TAI,,//7))
(23a)
since L > and A 0. Note that the in the exponent from the multiplication with Mr which also scales as the y term previously mentioned.
Furthermore, this bound cannot be lowered since the linear system a=() = x(()),//T already results in the scaling of Lone 22p(2TAtt,//1)
8.2 RECONSTRUCTABILITY OF THE INITIAL STATE FROM PAST INPUT AND OUTPUTS
To derive the conditions on the existence of the encoder function suppose that we have a system given by
a(t) f(x(t), ut))) 3n = hTx,)) + wn.
(24a)
(24b)
where 2, =nnAt). For this system, if state is given xtty) along the input trajectory u(t) one would in principle be able to compute xr( for all / > lg- However, since we aim to construct the state given past outputs we need xr() for < to which requires backward in time integration. This backward integration on ff is guaranteed to be unique if Lipschitz continuous in for all u as by the Bicard-Lindele theorem (Murray Miller 2013). Hence, since ff is assumed to be Lipschitz continuous construct operator fa which can integrate backwards or forwards
Intl fa(In, un) + Fn-1 fā (Z Un-1)
(25)
where u is subject to ZOH.
This operator allows us construct past outputs
Mn 1 (h o fa (Ini Un wn-1
Un-2 = (h o f/ 1-22 (In: un un-2) + wn-2
(26)
In = (ho f5z (In, Un Un- Un Un un=z) + Wn-z
y = (H o F (Ins Um=) + W-=
where
(2ns Un 11 un-p) p
(fr (Ins Un- 1), Un 2: Un-p)
is the application of fā 1 p times to obtain Tn-p- Furthermore, Y--= = [V 3/ 2n-2 2 VT__1
and, (H F=*) with W are similarly defined. To construct the initial state Fn, we need to invert Eq. (26). This inverse is also known as a reconstructability map (Katayama) (005). For the inverse to exist, several necessary requirements can be given. One such necessary requirement is solution I, should change (H FT*) otherwise these solutions are
that a small perturbation
indistinguishable from the output. This is formalized stating by that the matrix (.HePr")Kr..e
dr..
has null space rank zero which is also known as the local observability condition. This is the same the condition that the column rank of this matrix ny. A necessary requirement for this
14

---

Published as a conference paper at ICLR 2023
column rank condition is that the number of columns is equal greater than the number of rows i.c. 2ny 2 7g.-
Hence, under the right conditions, it might be possible solve Eq. 26 for singular 2n since this equation a nonlinear fixed point problem if = is known. For the case that W W is unknown one can estimate the state nn n. by solving the nonlinear regression problem;
in arg min Y (H ( U12
(27a)
arg min [(ān; = 122
(27b)
Hence, both f uniformly Lipschitz continuous and (V (V,L)*V L) v being full rank in 2n are necessary conditions for the existence of unique reconstructability map.
Computing the reconstructability map for our model thus requires solving an optimization prob- lem that becomes computationally infeasible during training. Hence, the encoder function aims to approximate the solution to Problem (27).
15


Reviews:
[{"review_id": "tChACgyUXK", "reviewer": "Reviewer_mcyj", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors consider the problem of learning models of continuous time dynamical systems in the presence of noise, latent variables and driving input. They present a method for doing this, which reduces the computational complexity of model fitting by breaking up longer observation sequences into shorter chunks.  They introduce the interesting idea of learning a neural network to estimate latent system state at the start of each of these chunks.   In addition, they provide theoretical results underlying their method and empirical results on benchmark applications.   ", "summary_of_the_review": "Summarizing all of the above, I feel the core idea of the paper is novel and important.  However, the theoretical claims need further discussion and clarification to make their full importance more apparent.  In addition, I find it likely that the authors have indeed achieved new state of the art results on the two benchmarks they provide, but additional analyses showing how their methods work when the test and validation sets are totally disjoint are needed to confirm this.   I believe addressing these points would take an already very interesting core idea and make the paper substantially stronger.  ", "strength_and_weaknesses": "The paper is well written and, as the authors do a very good job of motivating, the problem of fitting continuous models to the behavior of dynamical systems is important.  The authors also do a good job of contextualizing their work with regards to the existing literature.   While the idea of using Euler integration followed by back propagation to learn neural network descriptions of differential equations is not novel, \nI believe the idea of approaching this through (1) breaking up longer sequences into shorter ones and (2) using a neural network to predict initial state at the beginning of each shorter sequence is novel and very interesting.  This introduces the benefit that many of the computations associated with each sequence can be performed in parallel.   \n\nThe authors also provide theoretical arguments about the potential benefits of their approach.  However, there are multiple areas the presentation of these results should be clarified and improved.  First, it is not clear how surprising theorem 1 is: given that the derivative has a bounded Lipschitz constant is it surprising that when we integrate over shorter sequences, the Lipschitz constant of the integral, also decreases?  To this point, it would be helpful to know how tight the bound is.  If I understand the notation correctly to be big-O, this is an upper bound.  If the tightest upper bound possible is indeed exponential with T, then this result is indeed interesting, but we would need assurance (or at least a reason to conjecture) that this is indeed the case.   Second, while formal theoretical results are not needed, it would be helpful for the authors to discuss what, if anything, might be lost by breaking up longer observation sequences into shorter chunks.  For example, does this hinder the ability to learn dynamical systems with longer time scales?  Perhaps not if the network inferring initial state is accurate enough, but having an explicit discussion about this would help.   Third, the paragraph following theorem 2 appeals to it in a way that the theorem itself is not currently worded to support.  In particular, theorem 2 is currently worded as an existence proof: it is possible to find two constants such that both the state and derivative are normalized.  However, the paragraph immediately after this theorem (beginning \"Hence, considering only...\") appeals to this theorem as if it were a proof of necessity.  Following, the authors math shown in the proof of the theorem, I believe such a claim could be made, but the theorem would need to be re-worded (or the logic of the following paragraph made more clear).   Finally, and perhaps more importantly, I am a bit confused about how theorem 2 applies to model fitting.  In particular, I agree given a ground truth x(t) and f(t), two constants can be found such that the transformed descriptions are regularized as the authors claim.  However, when fitting models, there is flexibility in the f we chose, so that it would seem for any f and any value of 1/\\tau in equation (3), can't we always find a f' = \\tau f.  In other words, don't we have to restrict the flexibility of the function class we consider for f in some way for \\tau to have any affect?  I acknowledge the empirical results the authors show in Fig. 5 supporting their theoretical arguments, but in those results there seems to be a large range of where RMSE of the simulation error plateaus and for which the RMS of x and f are not 1. \n\nFinally, the authors provide empirical results comparing their method to others in two benchmark tests.  They claim to achieve state of the art results in both.  However, as the authors commendably make clear, part of their test sets are also used for validation.  They state this is fair because many of the existing methods also do the same.  However, to fully evaluate the claim that a new state of the art has been achieved, it would be helpful for the authors to show their existing results as well as results obtained when validation and test sets are fully distinct.   When comparing to previous methods which did in fact have overlapping test and validation sets, they could use the current set of results, but when comparing to methods that had disjoint test and validation sets, this second set of results could then be used to ensure a fair comparison. If their method still outperforms all existing results when comparing each on this more equal footing,  I believe they could then rigorously claim a new state of the art has been achieved. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "_YNQFZjeMlm", "reviewer": "Reviewer_jYDD", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces a novel approach for the estimation of continuous time state space dynamical models. \nThis field has gained significant attraction since 2018 with the Neural ODE.\nThe proposed approach is based on the evaluations on short subsections, and a novel state-derivative normalization.", "summary_of_the_review": "The paper introduces a novel approach for the identification of continuous dynamical state-space model parameters. The method is interesting, but I fail to understand the training process. The results seem to indicate performance on par with “state-of-the-art” approach but a lower computational cost.", "strength_and_weaknesses": "Strengths:\n1.\tThe authors tested their approach on two different datasets.\n2.\tThe authors have developed proof of convergence of their technique.\n3.\tThe authors have provided source code for better reproducibility purpose.\nWeaknesses:\n\n1.\tThe results in table 1 seems to suggest that for both problems the neural ODE approach with normalization seems to be performing as well or even better than the proposed technique. Do the authors suggest that the main advantage of their approach lies in the lower estimation time (lower computation complexity).\n2.\tI fail to understand how the proposed approach works, especially the joint training of 3 fully-connected neural networks.\n3.\tFigure 5 caption could be expanded for better understanding.\n\nAdditional questions:\nHave the authors analyzed the effect of the subsections size on the performance of the technique?\nWhy did the authors compare their technique with different approaches on the two datasets (for example BLA (Relan et al. 2017) and Volterra model only assessed on the CCT benchmark)?\nCould the authors comment on how the proposed technique compare to the Neural Laplace (“Neural Laplace: Learning diverse classes of differential equations in the Laplace domain”, Samuel Holt et al. ICML 2022) ?\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Wo6yG_dK94", "reviewer": "Reviewer_x1Z3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "1. The authors try to infer the underlying dynamics of some unknown dynamical system based on sparsely sampled observations and external inputs.\n2. The authors build a framework to infer such a dynamical system using 1) neural networks as function approximators for a generic continuous-time dynamical system, and 2) gradient-based optimization.\n3. The framework can account for a dynamical system that demands 1) handling noisy measurements, 2) doing long term prediction, 3) compatible with hidden states, and 4) compatible with external inputs.\n4. The quality of the inferred model is evaluated by how good the observed trajectories are reconstructed in two benchmark problems.", "summary_of_the_review": "1. I will recommend this paper if the author address and clarify the above issues.\n2. Without such clarification, it is hard to judge whether this work add to the existing effective approach: e.g., training RNN with backprop through time, or doing FORCE learning with chaotic or balanced neural network.", "strength_and_weaknesses": "**Strength:**\n\nHaving the original joint optimization problem broken down to 1) optimizing initial hidden state and 2) optimizing forward dynamics have the potential to increase optimization stability.\n\n**Weaknesses:**\n\nSince the framework uses three neural networks as function approximators for encoder, iterative dynamics, and output projection, the full model and optimization scheme look very similar to training a standard recurrent neural network. From this comparison, it is less clear about the benefits of having such elaborate framework in terms of increasing performance and reducing computational complexity.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "C00zHpmfN0m", "reviewer": "Reviewer_mnoy", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposed a new scheme for the continuous-time nonlinear state-space system identification based on overlapping short subsections rather than the whole length of time, with the help of a subspace encoder and state-derivative normalization. The paper proved that, both algorithmically and empirically, the proposed scheme will reduce computational complexity and improve cost function smoothness. ", "summary_of_the_review": "A solid and novel scheme for the continuous-time nonlinear state-space system identification. ", "strength_and_weaknesses": "Strength: The proposed work provides a solid foundation to improve the system identification modeling by the derivation of the necessary condition for encoding the initial state when performing system identification on overlapping short subsections and proving that the smoothness of the encoder cost function will be improved by doing so. Model performance shown in the experiment results, especially the comparison with neural ODE, further validates the effectiveness of the proposed model. \n\nWeakness: The structure of the SUBNET is unclear from Figure 2 and the corresponding text. As SUBNET is the key contribution of this work, more details are welcomed. The reviewer is also curious about how the discrete-time subspace encoder, as listed in Table 1, is implemented. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "9WSgBs8wb3", "reviewer": "Reviewer_jmUH", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This conference paper aims at providing a computationally efficient algorithm to identify the underlying dynamics of nonlinear physical systems using Continuous-time (CT) neural network models that consider the presence of challenging features such as external inputs, noise in the measurement process, latent states, and robustness during training. Recent models have explored the ability of general models parametrized by CT neural networks to learn unknown nonlinear dynamics from partial observations. However, these models typically seek to include only one challenging feature – thereby ignoring the others – and are computationally intensive, especially for long time series. Building on these previous works, the authors propose to break the time series into shorter overlapping chunks, each one approximated by the same neural networks and linked by an encoder function, also approximated by a neural network. The authors convincingly show analytical evidence that this approach increases the smoothness of the cost function with respect to the parameters, and hence the robustness of the learning process, and that it can reduce the computational costs by partially parallelizing operations. The performance of the proposed model is tested on two standard benchmark problems and compared to other methods from machine learning. The novel method appears to perform on par or better than state-of-the-art models such as neural Ordinary Differential Equations (ODEs).", "summary_of_the_review": "Summary of the review\nThis work addresses the important problem of reconstructing the underlying dynamics of a nonlinear system. Overall, the manuscript contains elements of interest, and the performance of the model are impressive. However, it somewhat lacks clarity and depth when discussing some key features of the model, such as the concrete advantages in computing costs, the novel aspects compared to previous works, and the advantage of utilizing the encoder to infer the initial state. Novelty is also a concern per se, since all the features of the model where already present in previous works, albeit not at simultaneously. \nThe authors may want to clarify the unclear aspects and expand the discussion before considering the paper for publication.", "strength_and_weaknesses": "Strengths\nThe problem of inferring the dynamics underlying physical, biological, and social systems is an important one, and this manuscript introduces promising results with respect to i) the robustness of the training algorithm when the problem is broken down into overlapping shorter subsections, and ii) the intriguing possibility of introducing an encoder to infer the initial state of the system at the beginning of each subsection. Particularly interesting is the analytical proof shown in the first part of the appendix, in which they derive the temporal scaling of the Lipschitz constant in the parameters space. Also significant are the insights given on the topic of the normalization terms required to avoid pathological behavior during training. The performance of the model on the two chosen benchmarks seems impressive compared to other methods. \n\nWeaknesses\nThe main claims of the paper are, in my opinion, not sufficiently explored in the results section. Specifically, a proper evaluation of the advantage in terms of computational costs compared to the other methods in Table 1 is missing. This is particularly important for other methods, like Forgione & Piga 2021 that also employ subsections. Furthermore, the authors cite Ayed et al 2019 as a similar work that also employ an encoder function, but do not compare it to their method. Along the same lines, the authors do not investigate the choice of hyperparameters such as na, nb or the length of a subsection Tdt. Exploring these hyperparameters and their impact of performance vs computing costs would greatly enhance the impact of the paper. Finally, the second benchmark only contains 500 samples in total, which seems modest if contrasted with the claims made by the authors about “accurate long-term predictions”. Perhaps a different benchmark – e.g. the EMPS dataset employed in Forgione & Piga 2021 – could be used to highlight the potential of the method, specifically the use of subsections. There’s one last aspect that I think is not particularly explored in the paper despite the initial claims: the detection of latent states. The benchmark used in the paper are not particularly well suited to evaluate the ability of the model to infer the presence of latent states. Perhaps a better benchmark could be used since this is one of the main claims of the paper.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster