Learning Physics Constrained Dynamics Using Autoencoders
Authors: Tsung-Yen Yang
Justinian P. Rosca
Karthik R Narasimhan
Peter Ramadge
Venue: NeurIPS 2022 Accept (2022)
Keywords: Autoencoder with Latent Physics
Deep Learning and its Application

Abstract: We consider the problem of estimating states (e.g., position and velocity) and physical parameters (e.g., friction, elasticity) from a sequence of observations when provided a dynamic equation that describes the behavior of the system. The dynamic equation can arise from first principles (e.g., Newton’s laws) and provide useful cues for learning, but its physical parameters are unknown. To address this problem, we propose a model that estimates states and physical parameters of the system using two main components. First, an autoencoder compresses a sequence of observations (e.g., sensor measurements, pixel images) into a sequence for the state representation that is consistent with physics by including a simulation of the dynamic equation. Second, an estimator is coupled with the autoencoder to predict the values of the physical parameters. We also theoretically and empirically show that using Fourier feature mappings improves generalization of the estimator in predicting physical parameters compared to raw state sequences. In our experiments on three visual and one sensor measurement tasks, our model imposes interpretability on latent states and achieves improved generalization performance for long-term prediction of system dynamics over state-of-the-art baselines.

Learning Physics Constrained Dynamies Using Autoencoders
-sung-Yen Yang'
Justinian Rosca²
Karthik Narasimhan'
Peter Ramadge'
!rincetonn University Siemens Corporation, Corporate Technology justinian www.aaanooossssiiiioemiomiimmmee emens. com
yangtsungyen@gmail. com (karthikn, ramiictto edu
Abstract
We consider the problem of estimating states (e.g., position and velocity) and physical parameters (e.g., friction, elasticity) from sequence of observations when provided dynamic equation that describes the behavior of the system. The dynamic equation can arise from first principles (e-g,, Newton's laws) and provide useful cues for learning, but its physical parameters are unknown, To address this problem, propose model that estimates states and physical parameters of the system using two main components. First, an autoencoder compresses sequence of observations (e.g., sensor measurements, pixel images) into a sequence for the state representation that is consistent with physics by including a simulation of the dynamic equation. Second, an estimator is coupled with the autoencoder to predict the values of the physical parameters. We also theoretically and empirically show that using Fourier feature mappings improves the generalization of the estimator in predicting physical parameters compared to raw state sequences when learning from high-frequency data. our experiments on three visual and one sensor measurement tasks, our model imposes interpretability on latent states and achieves improved generalization performance for long-term prediction of system dynamics over state-of-the-an baselines.
1 Introduction
Neural networks have become a core computational component in domains such as computer vision [1], natural language processing [2]. and deep reinforcement learning [3]. Recent work has shown that neural networks can exhibit an inductive bias that is often introduced via designing specific structures [4]. This bias be used to encode prior task knowledge that helps the network generalize to unseen data. For example, convolution neural networks capture the translation invariance of key image features. In this spirit, develop structured neural network model that leverages a dynamic equation to estimate both the state of dynamical system (e.g., position and velocity) and its physical parameters (e.g., friction constants) from a sequence of partial observations. Knowing the state and parameters of the system aids learning a control policy and tracking parameter value changes over time. For instance, for self-driving car, we want learn neural network that estimates the vehicle's 's position and velocity from a sequence of egocentric camera images. The estimated state can then be used in a control policy. In addition, we want to track physical parameters over time, e.g., friction coefficients. This is useful for vehicle maintenance and safety. We expect that exploiting neural network, regularized follow a dynamic equation, will streamline the required data-intensive operations and yield improved performance.
Observations can be direct measurements of some system variables (e.g., accelerations), take more complex form (e.g., images). We group such observations over specified time windows and also refer these groupings as observations. We obtain a compact representation for these observations that permits observation reconstruction using autoencoder (see Fig. 1(a)). {h,} in Fig. ((a) is
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

representation of an observation sequence }or} over specified time window, and } is the reconstructed observation sequence. Learning the autoencoder is both data and computationally intensive. In addition, h, may not be physically interpretable and be a "system state."
Motivated by these observations, ulator is provided that specifies the physical laws of the system. This model can arise from first princi- ples (e.g., Newton's 's laws), but its free parameters 0 (e.g.. masses, lengths) remain be specified. Given an initial state and 0, the physics simulator gener- ates a state trajectory } consistent with the laws of physics. To leverage this model, we require an estimator f()) that maps sequence of states (ær)) to an estimate of 0. We then couple the estimator and the physies simulator with the autoencoder as shown in Fig. 1(b). We train the autoencoder and f(-) to minimize the observation reconstruction loss Σ. lo. - -6,1l2- Within this process, we train the encoder 1(-)) to minimize the sum of squared state errors: Σ. || 10,--24, = The complete model (Fig. 1(b)) is called Autoencoder with Latent Physics (ALPS).
assume sim-
{og}
{0g)
Encoder h(-)
Encoder k(-)
(R.,)
Estimator ((-)
(h.)
Σ || &,ll2
Σ, llo:
0,112
Physics Simulator
Decoder g(:)
Decoder g(-)
[
(b)
(0.)
Known
Subject to
Given
Prediction
Computation
Learning
Figure 1: (a) An autoencoder learns latent representation from a block of observations. (b) Combining dynamic equation and pa- rameter estimator in (a).
The paper's contributions are three-fold. (1) ALPS the method that learns identify the system pa- rameters and the mappings between states and obser- vations from data. contrast, conventional system identification (e.g., tools in Matlab) requires knowing the function mapping and identifying the parameters online, which is costly to run.
(2) We show that one can learn periodic or vibrational behavior in this setting using Fourier feature of states. (3) We evaluate ALPS in three simulated and one real-world train dynamics. ALPS can achieve up to 4.8x and 6.3x better physical parameter and state prediction accuracy, respectively, over prior approaches.
2 Related Work
Physics-informed neural networks. There is growing interest in including physics prior or
algebraic and logical constraints into neural networks [5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18. 19, 20, 21, 22. 23, 24, 25, 26, 27, 28, 29]. For example, [30, 31. 32, 33, 34, 35, 36, 37, 38. 39, 40] exploit Lagrangian or Hamiltonian mechanics to learn an energy-conservigg system based on position, momentum, and the derivatives thereof along trajectories. These works assume the physical parameters of the system are constant and need not be estimated. In contrast, we estimate the physical parameters. This is important for fault detection and localization, and for safety. Papers [41, 42, 43, 44] learn a general physical simulation from data, but their model is required to have state information or known forward rendering engine to map states to observations. In contrast, we learn physics-based autoencoder to estimate states in an unsupervised manner. Paper [45] also uses an autoencoder with physies to predict parameters. However, we show that learning state sequences as in [45] fails to generalize to unseen parameters when the system exhibits high-frequency behavior.
coopman-inspiree neural networks. Among many physics-informee informed neural networks developed in the past years, the recent effect has also considered applying deep learning in learning Koopman operators. We highlight several papers that are similar our approach here. Koopman operators use an embedding to describe non-linear systems in linear form. They are useful in analyzing the system dynamics but require domain knowledge find the embedding. To overcome this challenge, the work [46, 47] use an autoencoder to identify Koopman eigenfunctions. In addition, the work [48, 49] also use an autoencoder for Koopman spectral analysis by learning Koopman invariant subspaces from data. The use of autoencoder structure is similar to our approach, but we explicitly use physics simulator for constraining the representation of the autoencoder. Furthermore, the work [50] propose method that optimizes neural networks by Koopman theory.

---

Encoder
(A)
Oμ-r+]
02
Estimator
Self-Attention Network h()
Fourier Trans, (Magnitude)
x r+1
X
Estimator f()
Encoded
Start State
Inputs
Decoder
Simulated State
X--r+1
{u
81-1+1
Simulator
MLP g()
MLP 8(-)
MLP a()
Physics Simulator
X A(0)x B(8)u
0z-r+1
Given
Prediction
Known Computation
Subject Learning
Pendulum
Mass- Sprinp-Dammannn
Two-Body
Figure 2: ALPS and the tasks. ALPS consists of four parts: (1) encoder network that estimates states from observations, (2) a parameter estimator network that predicts physical parameters from a state sequence, (3) a physics simulator generates state trajectory provided with initial state and values for the physical parameters, and (4) decoder network reconstructs observations from states.
Fourier features for high-frequency data. [51, 52] show that using Fourier features helps neural networks learn high-freee feet-ffrrunncy content in image regression tasks. This approach requires specifying the Fourier series coefficients and the basis frequencies. In contrast, the Fourier features in model are computed from the states, which allows us to predict physical parameters. In addition, the concurrent work [53] replaces the self-attention sublayers [54] with Fourier transformation of the input word token in natural language processing. They show that this method is sufficient to capture semantic relationships in several text classification tasks. In this paper, we provide theoretical justification for using Fourier features to learn system dynamics with high-frequency data.
System identification. The proposed approach uses the neural network to identify the system parameters from data, which is similar to conventional system identification methods such as linear grey-box model estimation (greyest) tool provided in Matlab [55]. However, ALPS does not require the mapping between the states and the observations, whereas the greyest method needs to provide full system dynamics. This offers advantage when the observation space is in high dimension. In addition, is possible to use Matlab model compilation to speed up the estimation of system parameters, which is similar to the analogy of training ALPS first and then deploying it during test time. However, even with Matlab model compilation, ALPS is still different since ALPS uses neural network to identify the system parameters.
3 Problem Formulation
We consider the continuous-time system
á Az. Bu: g(x),,
where A [aax B Ruxp and æ E E² EP, o E Ra, denote the state, input, and observation. respectively. The function provides partial observation of x. In most situations we have partial knowledge of A and from physics. This is true and reasonable many large-scale industrial applications ranging from wind turbines to aircraft, where system designers use physics knowledge to design machines. Hence we assume mappings A(.)=
Ruxn and B(.) - RExp, from physical parameters E θ to the system matrices A(0), B(0), are given (i.e., A and B obtain specific sparse and parameteric form). In addition, Eq. (1) assumes the system dynamics to linear, but approach can be extended to other non-linear cases.

---

In practice we only have observations at discrete points in time. For simplicity, we assume these are equally spaced at times t = 0. ....... At sample time t, we have the window of observations {ox)==--r-1> where r is the window length. We assume the sampling rate satisfies the Nyquist rate.
Our problem can now be stated as follows. Given functions A(.): θ - R :(:): - R"*p, seek to learn network that estimates a state sequence {I. (-p])==-+++1 physical parameters 0, and mapping g()) from finite sequence of past observations {0=)===-74 and past known inputs (= Once trained, the network can predict states and has learned to adapt physical parameters to the context without re-training. Note that even though the network operates with time sampled variables, the physics simulator be used to predict states any time. This problem setup is distinct from that of HNN [33], HGN [36], or Symplectic ODE-Net [35]. In their setups, the model is only trained on data from single physical parameter 0, and hence they need to re- re-train networks for every new set of physical parameters. In addition, compared to conventional system identification approaches, we do not need to rerun the solver each time-we only need forward pass of the network to get the estimation during deployment (testing time).
To make the learning problem well-defined, i.e., to ensure that we can uniquely identify the unknown system parameters, we make the following assumption.
Assumption 3.1. (Identifiability [56]): For sufficiently large т, the sequence of states and inputs {(z,, ) uniquely specifies the unknown parameter O.
Parameter identifiability is a well-studied problem in system identification [56, 57, 59, 60, 61]. For example, the (continuous time) system a = b
- + 6)
b
x:
+
0 u with o = [1 0] x,
-c
where 0= = (a, b, cf and g()) is a linear function, satisfies Assumption 3.1 when b o 0.
4 Nework Architecture
The network in Fig. 1(b) is expanded in Fig. 2 into its four parts: an encoder network, a parameter estimator network, a physics simulator, and decoder network.
The encoder network h(.)) (from {o,} to {æ,)). We consider two types of observations: (1) pixel images, or (2) direct measurements of some system variables. For the first case, we use a convolution neural network to compress a pixel observation o into a compact vector embedding 2' E R2, which preserves image features. For the second case, we use a feedforward network to project observation o into some higher dimensional space with a vector embedding z'. In addition, to estimating states from these vector embeddings, dynamics. One approach is to use recurrent neural networks (RNN) (e.g.. Dreamer [62]). However, RNNs suffer from vanishing gradient problems and slow computation when processing long-term sequences. Hence we use a self-attention network [54] to attend to z' of greatest importance to predict states and improve efficiency.
need to aggregate {z'} } to extract the local and global context of the
Furthermore, to inject a position signal of observations in the sequence, we add a positional encoding Rd to z' (z :- z' + p). where p are sine and cosine functions of different frequencies (see [54]). Finally, we stack embeddings over T steps to form a matrix Z E IR
The self-attention module can be formulated as querying a dictionary with key-value pairs associated with learnable weight matrices Wº E Rdxda, WK e dx and WV dddxdy (dQ dx here):
QK" Attention Q K, V) softmax( va VV, where Q = zwe, K ZW V ZWV and the softmax is taken over the sequence length r. To provide multiview of the embedding, we use the multihead variant of the attention by concatenating each attention head along the sequence axis
Multihead == Concat(head,, head,, ... head,) W head, == Attention Q, K, V),
where W E Idy xd are Iearnable matrices, I is the number of heads, and each attention head i has its own learnable weight matrices W.O E Rddxd W.K E Rdxdx and W.Y. E Rdxdy
Finally, a feedforward network takes in the Multihead embedding (of size RTxd) and produces the parameters of the distribution for each state in the sequence. The parameters are used to define posterior distribution over the encoded state ã. ~ Q-1lo.) with the prior P(ãz). For a translational coordinate, the posterior distribution is Gaussian distribution with a unit Gaussian prior. Hence the

---

network predicts a mean i E R" and standard deviation o E R" of a Gaussian distribution. In addition, for rotational coordinate, the posterior distribution is a von Mises (vM) distribution with a unit vM prior. Similar to a Gaussian distribution, vM distribution is defined by two parameters: mean fa E TR.2, ||ااا = 1 (the angular position (cos yp, sin sin))) v)), and a concentration k. E IR+ around J.. Such parameterization is found useful in practice, egg., [63]. Appendix E provides details about self-attention networks for estimating states.
The parameter estimator network f()) (from { to 0). The parameter estimator network predicts physical parameters from state sequences { { However, for systems that involve periodic or vibrational behavior, prior work [51] has shown that neural networks fail to capture the high- frequency content in the data. To improve generalization and reduce the effect of noise, we do a Fourier transform on each component j of state trajectories {æ, (i)
get (X=())))==-++)
()) = Ek= #x(() cos 2 wk i sin 2m wk)k) Note that for the systems that do not have periodic or vibrational behavior, using (1) (=1--+++11 would be enough.
One may use (i) as features for the parameter estimator network to predict physical parameters. However in Section 5 we will show that by using the neural tangent kernel (NTK) theory [64], which treats neural networks as a kernel regression, the resulting kernel matrix of (i) does not preserve high-frequency components in the data. To solve this, we will show that using the magnitude of the Fourier features { (iXx alleviates the issue. Hence the parameter estimator network takes in a concatenation of (i1I from each component of the state and predicts physical parameters O.
The physics simulator (from žt-r+1 and 0 to {.x}). Given start state žt-r+1 (from the encoder's first state prediction) and values for the physical parameters 0, we use the neural ordinary differential equation (ODE) [65], a differential ODE solver, to generate simulated state trajectory 1 &; = ODESol (er à = A(0) + B(0)u, T, A), where ODESolver takes in start state, an ODE, a window length, and a sampling time interval A. Note that 0;-7+1 = Ø1-7+1. In addition, using an ODE solver allows us to generate an accurate state trajectory compared to that of RNN in [62].
{ 1. ( +
The decoder network g()) (from S. to o.). Finally, the decoder network is either a deconvolutional network (for image observations) or feedforward network (for sensor measurements) that takes in each individual DiE-siiulated state 2, and generates a reconstructed observation ô..
Discussion. Note that we can replace the decoder g with a differentiable rendering engine. Prior work [44] use a differentiable rendering engine reconstruct the scene given the estimation of the system parameters and the states. However, using a differentiable rendering engine may introduce a simulation overhead, and using a network here is more generalizable. We think the future extension of ALPS to differentiable rendering engines is valuable future research direction. In addition, one may think that the estimator network can directly predict the system parameters given the input observation without taking the states. The reason to predict the states is that for some applications, it may be useful to know the state of the system. For example, for a self-driving car, we would like to know the speed of other vehicles by using observations from cameras. Knowing the speed of other vehicles (i.e., the state) allows the self-driving car to plan for trajectory. which is important for the safe deployment of the system. In addition, we would like to increase the interpretability of the model. The inclusion of the state allows the system designer to ensure the representation learned by neural networks is informative.
The loss function. Given a sequence of T observations, we minimize the following loss function:
C
DKL (x.(((i,,oo))(((EE.))) 8=1-T+1
Σ lo, - 8=1-++1
ı 2.112
Obs. recons. loss for h. S. and State recons. loss for f and h The variational autoencoder (VAE) [66] loss is a variational bound on the marginal lig-liieldoood of the data. It is used to train the encoder h. the estimator f. and the decoder g. Using VAEs avoids learning degenerated solutions and provides stable training of the network over deterministic network. In addition, the observation reconstruction loss encourages reconstructed observations (o.) to match true observations (o.), and the state reconstruction loss constrains encoded states ( to follow simulated states { (#.) generated by physics. The former is used to train h, f, and g, and the latter is used to train ff and h. Both observation and state reconstruction losses are
VAB loss for it, and
5

---

important for training. We find that removing the state reconstruction term reduces the state prediction performance of the encoder since the network can predict arbitrary sequences. In addition, removing the observation reconstruction term impedes the image reconstruction quality, which is vital for training the whole model. In practice, we tune the weight for each loss term to accommodate different scales. Note that for the larger loss term, we use a weight smaller than 1 to ensure learning stability.
5 Fourier Feature Mappings for Learning Periodic System Dynamics
An important feature of ALPS is that it uses Fourier feature mapping of states to learn periodic system dynamics. Prior works [67, 68] have empirically shown that using Fourier feature can help with high frequency signals. To lay the foundation for the analysis and justify the use of Fourier features, in Section 5.1 we first review the recent work that uses NTK theory [64, 51]. This allows us control the training of our parameter estimator network as fully-connected networks. Then in Section 5.2 we use these tools to analyze the effects of using Fourier features, their magnitudes, and their phases when predicting system parameters that involve the periodic behavior.
5.1 Deep networks kernel regression
Consider a set of labelled training data {(v,, ,)}," with vs E R", Yi E R, and E [1 : m]. Set
= [y...... Vnm IT E IR.. Now bring in feature map ø R + R with kernel k(v;, w,) = ((v,)" " o(v,). Let K [k(v,, U;)l E | denote the kernel matrix for the training examples and k(v) = [k(v,, v)] E iR" denote the vector of kernel evaluations k(v,, v), E [L, m), for a test sample E R The resulting kernel regression predictor is j(v) y K (k()).
Now bring the concept of NTK proposed by [64]. The theory in [64] says that when the width of the layers of cully-connected deep networks with weights initialized from Gaussian distribution / tends to infinity, and the learning rate for stochastic gradient descent tends to zero, the neural network estimator ( wv) converges to the kernel regression solution using NTK. The NTK is defined as
kNIK (U. Uÿ) Ewww [ (B0(nn:00) dw
@0(u):)))) Owe
Under asymptotic conditions, a neural network's output after t updates can be approximated as ( (v; w) R y² (e-----eie e-nK()K --k((), Spectral bias in neural networks. Now we want to compute the training error of a neural network after times update. Let K UEUT denote the eigendecomposition of the kernel matrix K which must be positive semidefinite (PSD). Here U is an orthogonal matrix and Σ is a diagonal matrix whose entries are the nonnegative eigenvalues ordered by magnitude: X 2 A2
where M is the n by n matrix M given by the power series M =E 3nd M' with M
Am 0. So the " ETA =
training error in terms of L2 norm 11-112 is ( - y112 = Udiag((e- -bi -
This shows the training convergence will decay exponentially at the rate na; Hence the components in y will be learned faster if their corresponding eigenvalue is larger. In Section 5.2 we will show that for sequence of states without doing Fourier transform, the resulting NTK will have smaller eigenvalues at the high-frequency components. This leads to slower convergence in figh-frequency components of data which are essential to identify parameters for the periodic behavior.
5.2 The effect of Fourier feature mapping
To understand the effect of the Fourier feature mapping, we first derive the kernel function of Fourier features, their magnitudes, and their phases. Then compare the spatial bias of the kernel matrices from these three kernels.
Kernels of Fourier feature mapping. Consider a state trajectory [zo, Ti, -y 1 where here we consider ID case for sealar state x E R although can be extended to a vector state R The feature maps of the Fourier feature mapping, their magnitudes, and their phases are
(1) ΦDFT(") (v) = [No...., X ......, x _ ERT; (2) ΦMAGG()) = ||ool...,|X,-... E ERT;
(3) ΦPHA (B) = [arg(Xo),.... .rg. )) E RT,

---

Eggerallsss (scaled)
Value
Port
szppiing
40
120
160
10
1.5
20
25 30
Time pointt
Hi
(a) Time series data
(b) Kernel spatial
(c) kppt
(d) KMAG (e) kPHA
(f) No mapping
Figure 3: Using Fourier feature mapping of KMAG and KPHA results in a wider spectrum, which lets neural networks learn a wide frequency content. (a) The time series data with the sine basis frequency of 25, 17.5, 11, 7.7,2, 1Hz associated with the amplitude of 1.2, 1, 1, 0.4, 1. (h) The kernel spatial of the kernel matrices. We see that KMAG can preserve high-frequency parts the signals. (c-f) The kernel matrices of the composed NTK. (best viewed in color)
i sin RTxT, then the kernel functions of these mappings
where X
Ej m cos kj
ki) Set C2 [cos
(25 ki) sin 22 kk))
(1) KDFT(VI, 02) = Σ of Cluz; (2) AMAG(UI, h=
= of CRUIuY
(3) kPHA (V1, v2) = ÓPHA (v1) (V2))(V)).
After mapping the input points into the Fourier features, we feed them into a neural network to obtain ((((v):)) w) Hence for DFT kernel function the resulting composed kernel of the neural network is (yrK((DFF(C (DFT(D>) (v2)). Similarly, we have the kernels for kMAG and kPHA-
Visualizing the composed NTK. We generate time series data composed of different frequencies and magnitude of sine waves. The length the data is 200 samples, and we set r = 100 (i.e., v E 1000 with the sampling rate being 100Hz. We slide window and hence have 101 instances in total. Fig. shows the time series data, the effects of each kernel, and its spatial plot. By construction, kMAG and kPHA have slower decay in the high-frreuency domain as shown in Fig. 3(b). In addition, KDFT and no mapping have the same kernel matrix and a narrower kernel spatial. This observation supports the idea of not using X when learning the system with periodic vibrational behavior [Xw] (magnitude) preserves frgh-freeuency information. which is useful for parameter estimations. Note that for the system that does not have high-frequency signals, both X. and | are equally well.
Comparison to [51]. Our work is inspired by [51], which also uses Fourier feature mappings. However, there few key differences. (1) Setup. The Fourier feature mapping in [51] transforms the low-dimensional x-y coordinates into high-dimensional Fourier features, which projects data Fourier feature mapping of time series data,
into a high-dimensional space. In contrast, which compresses data into more compact representations. (2) Analysís. We discuss the difference of Fourier feature mappings, their magnitudes, and their phases understand the effect of each mapping for predicting system parameters. In contrast, [51] does not have such analysis.
Simulations
We study the following questions: (1) How does ALPS perform compared to other baseline methods without the Fourier feature mapping and physies-in-the-lopp? (2) What is the effect of self-attention networks in ALPS? (3) How does ALPS perform in a full-scale train wheel system dynamics?
Visual dataset. We generate three visual datasets: pendulum, mass-spring-damaar two-body system to compare the performance of ALPS to that of the baseline in the literature. These baselines are very commonly used in the prior work. We first randomly sample an initial state and physical parameters, and then generate 125 step rollout following the true system dynamics, and render corresponding 64 by 64 by 3 pixel observation snapshots. The sampling rate is 20Hz in the
(MSD), and
7

---

pendulum, 100Hz in MSD, and 6Hz in two-body systems. The observation length is 100. In total, we generate 500 training and 500 test trajectories, resulting in 13,000 training and test sequences.
- sin (+p) where yp is an angle, & is an
(1) Pendulum. The dynamics is de [ -
angular velocity, /3 is a friction coefficient, G a gravitational constant, and L is the length of the pendulum. We fix G = 10, L = 1, then sample 3 from uniform distribution 3 ~ U(0.1, 1), an initial angle from a uniform distribution SP ~ U(-T, +T), and an initial angular velocity from a uniform distribution @o ~ ( 4). We predict 0- [3] in this task.
(2) peans--prig--daappe The dynamics of double mass-spring- damper system is
[x(1)] -()))a(()
1
0
0
Txr (1)
d dx
(2)
-a(3)=af a
m())
m())
a(1)
(1)
cx3)
x x(2)
+
m[1]
a
a(1)
T((1 u ((2)
r(3)
0
0
0
x(3)
0
0
[T(4)
a(
a(1)
[x(4]
(2))
m22
m2)
(2)
where x33) are the displacement and velocity of the primary spring; xx)), (x4) are the displace-
ment and velocity of the secondary spring; w(1), ((2 are the inputs; a(2), (4) are the stiffness and
damping ratio of the primary spring and damper; (11), ax33 are the stiffness and damping ratio of the secondary spring and damper, and m(1), m2)) are the mass of the primary and secondary spring. We fix all the parameters and the initial state except for (2), which is sampled from a uniform distribution (2) ~ U(4 x 10 4 x 109). The input excitations are sampled from a unit Gaussian distribution (O0 0.05) x A (O00, 0.05) (ie., white noise). We predict 0 = [a=(2) in this task.
(3) Two-body. In this system two particles interact with each other via an attractive force. The [ [|x(2)11 Gmn(1)m(2)
dynamics is a Hamiltonian H of the particles; (1),,(()) are the momentum of the particles; m(11, m(2) are the masses of the particles, and G is gravitational constant. We fix G = 10, m(1) = m(2) = 1, then sample H from a uniform distribution H. ~ (00.4, 1). We predict 0 = [M] in this task.
2m(1) 2m(2) [la(1) where p(1), p(2) are the positions
Time series dataset. In addition, we obtain an MSD system dataset with state measurements.
(4) MSD time series data from full-scale train wheel suspension system. To show the applicability of ALPS, we conduct experiments with the data that represent the situation when sensors are installed in the real train wheel suspension system. We use full-scale dynamics of the train wheel suspension system with 18 state elements, 12 observations, 16 input excitations, and 24 system parameters. We want to identify the stiffness and damping ratio of the 12 due to the interactions of multiple springs and dampers. In addition, this problem is challenging as the data contain noise. To align with the real world scenario, we assume that the system parameters of the system slowly change over time. As result, the data has the same set of system parameters but with different initial conditions (e.g., initial vibration speed). The dataset contains 20 trajectories with 500 steps sampled by 100Hz. We use 50-50 split to get the training and test datasets with T 100. Here we do not use the self-attention and decoder network in ALPS-the the estimator takes in state measurements directly.
The dynamics are complex
Baselines. We select the following baselines that are commonly used the literature.
(1) Context-aware dynamics model (CDM) [69]. CDM is the state-of-the-aat method to learn the system dynamics into two stages: it first learns context vector that captures the local dynamics, and then predicts the next state based on the context vector and the current state and input. CDM does not consider physics prior nor Fourier feature mapping. This is show that using physies priors improves performance and benchmark the results.
(2) Autoencoder. We remove the estimator and the physics simulator to benchmark the mismatch between the true state and the latent representation (equal dimension) of the autoencoder in Fig. 1(a) to show the interpretability of ALPS.
(3) ALPS w/o the Fourier feature mapping. We consider a variant by replacing the Fourier feature mapping with raw encoded state trajectories (..). This method similar to [45], which also uses time series data to learn system dynamics.
(4) ALPS w/o self-attention networks. We consider a variant by replacing the self-attention network with simple MLP to predict the position of the system, followed by a first-order finite-difference estimator to estimate the velocity. This uses the same approach as in [63] to estimate states. Finally,
8

---

Pendulum
Mass Masss-Spring--ee Damper
Two-body
SF:
OF
PR
SF
OF
PF
SF
OF
PF
CDM [69]
345.22
1766.70 70
0.99
5.69 x10*
50.23
2.0
Autoencoder
3041.12
600.84
7.63 7.42x 10
95.80 2.71 10
ALPS (ours)
86.48
1696.91 0.06
0.29
7,43 10
0.60x106
0.45
2.59 10
0.02
w/o Fourier feat. [45]
90.82
1773.39
0.29
0.93 7.44 10
1.28x100
1,86
2.56x 10
0.02
wo self-attention [63]
181.22
1950.77
0.06
1.85
704
187
511.49
2.72x10
0.27
Tha
ALPS
cos(p)
True
Prediction honzzn
20
30
40
50
sin(w) (4)
CDM
Prediction horbon
True
Preciction horzznn
11 222 20 44
Pendulum State Prediction for ALPS
Two-Body Observation Reconstruction
Figure 4: (Top) Evaluation for the tested networks in the visual tasks. (Bottom) Selected visualization results (we randomly sampled the ground truths and tested ALPS and CDM to show more cases). Note that for the pendulum and two-body tasks, the range of 0 is (0, 10j: for the MSD tasks, the O in the range of 10 SE: state prediction error; OE: observation prediction error; PE: parameter prediction error. And in visualization the true state here is æ, ENC state is कt and ODE state is t. ALPS achieves competitive performance in predicting physical parameters and states.
ensure that all baselines are comparable in terms of representation power and the number of parameters. The loss functions used for these baselines are included in Appendix D.
Evaluation metrics. We use mean squared error (MSE) between simulated {æ.} } and true states {æ,} to evaluate the performance: SE 1 Σ Ei || ==ll2- where N is the number of test data. We also compute MSE between reconstructed { and true observations {or} OE :=
N EN Et l|ó, 12112- Moreover, we compute the absolute value difference between estimated and true parameters PE = Σ
Results in the visual tasks. Fig. shows the results. Note that let the size of the latent representation of the autoencoder be the same as the size of the true states. The reason is that we would like know its state error when there is no constraint imposed so that we can quantify the difference from the one with imposing physics constraints. We see that (1) ALPS achieves the best performance in predicting physical parameters in all cases, with the most 4.8x lower error in the pendulum task. On the other hand, the Fourier feature mapping does not have much effect in predicting the physical parameter in the two-body system. This verifies the analysis of the Fourier feature mapping: for the task with wider frequency spectrum (e.g.. 100Hz in the MSD), the Fourier feature improves the prediction due higher eigenvalues the figgffreyencc content, whereas for the task with only low-frequenc, spectrum (e.g., 6Hz in the two-body), raw state sequences can already capture low-frequency content. This supports the idea using Fourier feature mappings for high-frequency data. (2) ALPS achieves competitive results in predicting the states, with at the most 6.3x lower eтror in the MSD task. We further visualize the state prediction results in the pudendum task for ALPS, which shows that ALPS can accurately track the true states. Without self-attention networks, the network has a substantial SE due to a large error in computing the velocity, as in the two-body task. The smaller the sampling rate is, the greater the velocity estimation error is. In addition, the high SE in autoencoder suggests that its latent representation is uninterpretable. This verifies the idea of using physics to constrain the representation of the autoencoder to estimate states.
In addition, (3) ALPS achieves comparable results in reconstructing observations. The low error for CDM in the MSD and two-body tasks is due to degenerated solutions shown in the Fig. 4. This implies that using physics stabilizes the training and improves the reconstruction of observations. (4) Finally, the autocncoder baseline has higher reconstruction loss in some tasks. This is because

---

that it learns degenerated solution, producing blur images due to high-frequeent movements of objects on the scene. Fig. 8 in Appendix E provides more qualitative visualizations of reconstructed images. Specifically, we find that CDM fails to track the state and resulting blur images in the paas--pring-damper system or duplicated objects in the two-body system. contrast, ALPS can precisely track the state and reconstruct the image well. Our remaining simulations explore ALPS's ability in predicting physical parameters from raw state measurements.
Results in MSD time series data from full-scale train wheel suspension system. To scale the pro- posed approach to more complex system, we ap- ply the approach in a system with a full-scale train wheel suspension system. The system contains 18 state elements, 12 observations, 16 input excita- tions, and 24 system parameters. Fig. 5 shows the results. Here ALPS does not use the encoder nor decoder network-th estimator takes in state measurements directly, as the system is fully-observable in the dataset. Overall we see that (1) ALPS achieves the best performance. (2) CDM has worse SE since an MLP cannot learn well from data with high-frequency content. In addition, we find that the average prediction error rate (i.e. 0-0) for these 24 parameters is 0.42% on the complex system. This also implies that by providing physics to the network, we can improve SE. These observations show ALPS can robustly identify parameters simultancously from state measurements, and support the theory of using Fourier features to learn periodic dynamics. Moreover, this result shows ALPS can be deployed in real prognostic applications for tracking physical parameters to enhance railroad safety. Our result is significant for system designers to diagnose the system and hence schedule maintenance.
SE
PE
CDM [69]
56.04
ALPS (ours)
3.02
0.44x10
Figure 5: Results in the MSD system for predict- ing physical parameters from time series data. Our method achieves the best performance.
g
7 Conclusion
We addressed the problem of predicting states and physical parameters of a system from observations with dynamic equations. We showed that the latent representation of the autoencoder is uninter- pretable. We then use dynamic equations constrain the latent representation of the autoencoder to be consistent with the laws of physics. We analyzed the effect of Fourier features for estimating physical parameters. The results showed that ALPS achieves competitive performance in the visual tasks and the time series dataset with up to 24 parameter predictions at the same time. This shows that our method scales to high-dimensional systems and it is useful for real-world applications.
Limitation and future work. Future work could improve ALPS in several ways. For instance, the understanding of underlying mechanisms in self-attention networks for estimating states from observations can be advanced. One path is to probe the network by using approaches in natural language processing. In addition, we require to have dynamic equations and assume that the system is linear and exhibits periodic or vibrational behaviors. These make it challenging to generalize to more complicated settings such as contact dynamics. One solution is to combine known physies with neural models (e.g., interaction networks [13]) to compensate for modeling error, and use time-domain and frequency-domal features together to predict physical parameters. Moreover, it would be interesting to explore other more complex tasks (e.g., non-linear systems, fluid dynamics, molecular simulation). Another limitation of ALPS is the overhead caused by simulating trajectories. For the training of ALPS, the computation cost mostly comes from running the differential solver. For example, under the same batch size of 10 and one gradient update, on average ALPS requires 498 seconds whereas the autoencoder requires 317 seconds, which is 57% slower. The training time for CDM is also similar to the autoencoder. A method to overcome this will be valuable. In addition to the challenge of computation cost, the further investigation of the full system with higher dimensional system parameters (e.g., large-scale machines with a hundred system parameters) and noise observation an important future research direction. Finally, incorporating the ideas from the literature the system identification [59, 60, 61] to ALPS will further enhance the performance of ALPS.
Acknowledgments and Disclosure of Funding
The authors would like to thank the anonymous reviewers and the area chair for their insightful and helpful comments and feedback. Tsung- Yen Yang thanks Siemens Corporation, Corporate Technology for their support.
10

---

References
[1] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages 770-778, IEEE Computer Society, 2016.
[2] Jacob Devlin, Ming-Wes Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Confer- ence of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota, 2019. Association for Computational Linguistics.
[3] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518(7540):529-533 2015.
[4] Alberto Bietti and Julien Mairal. On the inductive bias of neural tangent kernels. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Bue, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 12873-12884 2019.
[5] Russell Stewart and Stefano Ermon. Label-free supervision of neural networks with physics and domain knowledge. In Satinder Singh and Shaul Markovitch, editors, Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, February 4-9, 2017, San Francisco, California. USA, pages 2576-2582. AAAI Press, 2017.
[6] Nicholas Watters, Daniel Zoran, Theophane Weber, Peter W. Battaglia, Razvan Pascanu, and Andrea Tacchetti. Visual interaction networks: Learning physics simulator from video. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages
2017.
[7] Yinhao Zhu, Nicholas Zabaras, Phaedon-Stelios Koutsourelakis, and Paris Perdikaris. Physics- constrained deep learning for high-dimensional surrogate modeling and uncertainty quantifica- tion without labeled data. Journal of Computational Physics, 394:56 81, 2019.
[8] K. Kashinath, M. Mustafa, A. Albert, JL Wu, C. Jiang, S. Esmaeilzadeh, K. Azizzadenesheli, R. Wang, A. Chattopadhyay, A. Singh, al. Physics-informed machine learning: case stud- ies for weather and climate modelling. Philosophical Trunsactions of the Royal Society A,
379(2194):2200093, 2021.
[9] Rui Wang, Karthik Kashinath, Mustafa Mustafa, Adrian Albert, and Rose Yu. Towards physics- informed deep learning for turbulent flow prediction. In Rajesh Gupta, Yan Liu, Jiliang Tang, and B. Aditya Prakash, editors, KDD '20: The 26th ACM SIGKDD Conference Knowledge Discovery and Data Mining, Virtual Event, CA, USA, August 23-27, 2020, pages 1457-1466. ACM, 2020.
[10] Tianju Xue, Alex Beatson, Sigrid Adriaenssens, and Ryan P. Adams. Amortized finite ele- ment analysis for fast pde-constrained optimization. Proceedings of the 37th International Conference Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 10638 10687. PMLR, 2020.
[11] Rui Wang, Danielle Maddix, Christos Faloutsos, Yuyang Wang, and Rose Yu. Bridging physics-based and data-driven modeling for learning dynamical systems. arXiv preprini arXiv:2011.10616. 2020.
[12] Gustau Camps Valls, Markus Reichstein, Xiaoxiang Zhu, and Devis Tuia. Advancing deep learning for earth sciences: From hybrid modeling to interpretability. In IGARSS 2020-2020 IEEE International Geoscience and Remote Sensing Symposium, pages 3979-3982. IEEE, 2020.
11

---

[13] Peter W. Battaglia, Razvan Pascanu, Matthew Lai, Danilo Jimenez Rezende, and Koray Kavukcuoglu. Interaction networks for learning about objects, relations and physics. In Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg, Isahelle Guyon, and Roman Garnett, editors, Advances Neural Information Processing Systems 29: Annual Conference on Neu- ral Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pages 4502-4510, 2016.
[14] Alvaro Sanchez-Gonzaler, Nicolas Heess, Jost Tobias Springenberg, Josh Merel, Martin A. Riedmiller, Raia Hadsell, and Peter W. Battaglia. Graph networks as learnable physics engines for inference and control. In Jennifer G. Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmüsson, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pages 4467-4476. PMLR, 2018.
[15] Jiajun Wu, Erika Lu, Pushmeet Kohli, Bill Freeman, and Josh Tenenbaum. Learning to see physics via visual de-animation. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Advances Neural Information Processing Systems 30: Annual Conference Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 153-164, 2017.
[16] Maziar Raissi, Paris Perdikaris, and George E. Karniadakis. Physics-informe neural networks: deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics, 378:686-707, 2019.
[17] Jenny Brynjarsdottir and Anthony OHagan. Learning about physical parameters: The impor- tance of model discrepancy. Inverse problems, 30(11):11 1):1140077 2014.
[18] Yaofeng Desmond Zhong, Biswadip Dey, and Amit Chakraborty. Extending lagrangian and hamiltonian neural networks with differentiable contact models. Advances in Neural Information Processing Systems, 34, 2021.
[19] Ori Linial, Neta Ravid, Danny Eytan, and Uri Shalit. Generative ode modeling with known unknowns. In Proceedings of the Conference on Health, Inference, and Learning, pages 79-94, 2021.
[20] Pengzhan Jin, Zhen Zhang, Aiqing Zhu, Yifa Tang, and George Em Karniadakis. Sympnets: Intrinsic structure- stuuttue-preserving symplectic networks for identifying hamiltonian systems. Neural Networks, 132:166-177 179, 2020.
[21] Steven L Brunton, Marko Budišić, Eurika Kaiser, and J Nathan Kutz. Modern koopman theory for dynamical systems. arXiv preprint arXiv:2102. 12086, 2021.
[22] Tom Bertalan, Felix Dietrich, Igor Mezić, and Ioannis G Kevrekidis. On learning hamiltonian systems from data. Chaos: An Interdisciplinary Journal of Nonlinear Science, 29(12):121107, 2019.
[23] Steeven Janny, Fabien Baradel, Natalia Neverova, Madiha Nadri, Greg Mori, and Christian Wolf. Filtered-cophy: Unsupervised learning of counterfactual physics pixel space. arXiv preprint 2rXiv:2202.003363 2022.
[24] Pingchuan Ma, Tao Du, Joshua Tenenbaum, Wojciech Matusik, and Chuang Gan. Risp: Rendering-invariant state predictor with differentiable simulation and rendering for cross- domain parameter estimation. arXiv preprint arXiv:2205.05678, 2022.
[25] Niv Giladi, Zvika Ben-Haim, Sella Nevo, Yossi Matias, and Daniel Soudry. Physics-aware downsampling with deep learning for scalable flood modeling. Advances in Neural Information Processing Systems. 34, 2021.
[26] Rongye Shi, Zhaobin Mo, and Xuan Di. Physics-informed deep learning for traffic state estimation: hybrid paradigm informed by second-order traffic models. Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 540-547, 2021.
12

---

[27] Andreas Hochlehnert, Alexander Terenin, Steindór Sæemundsson, and Mare Deisenroth. Learn- ing contact dynamics using physically structured neural networks. In International Conference on Artificial Intelligence and Statistics, pages 2152-2160. PMLR, 2021.
[28] Rui Wang, Danielle Maddix, Christos Faloutsos, Yuyang Wang, and Rose Yu. Bridging physics- based and data-driven modeling for learning dynamical systems. In Learning for Dynamics and Control, pages 385-398. PMLR, 2021.
[29] Aditi Krishnapriyan, Amir Gholami, Shandian Zhe, Robert Kirby, and Michael W Mahoney. Characterizing possible failure modes in physics-infome neural networks. Advances in Neural Information Processing Systems, 34, 2021.
[30] Michael Lutter, Christian Ritter, and Jan Peters. Deep lagrangian networks: Using physics as model prior for deep learning. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
[31] Jayesh K. Gupta, Kunal Menda, Zachary Manchester, and Mykel J. Kochenderfer. A general framework for structured learning of mechanical systems. arXiv preprint 2019.
[32] Miles Cranmer, Sam Greydanus, Stephan Hoyer, Peter Battaglia, David Spergel, and Shirley Ho. Lagrangian neural networks. arXiv preprint arXiv:2003.04663 2020.
[33] Samuel Greydanus, Misko Dzamba, and Jason Yosinski. Hamiltonian neural networks. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Buuc, Emily B. Fox, and Roman Gamett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8- 8-14, 2019, Vancouver, BC, Canada, pages 15353-15363, 2019.
[34] Rui Wang, Robin Walters, and Rose Yu. Meta-learning dynamics forecasting using task inference. arXiv preprint arXiv:2102. 10271, 2021.
[35] Yaofeng Desmond Zhong, Biswadip Dey, and Amit Chakraborty. Symplectic ode-net: Learning hamiltonian dynamics with control. In 8th International Conference on Learning Representa- tions, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
[36] Peter Toth, Danilo J. Rezende, Andrew Jaegle, Sébastien Racanière, Aleksandar Botev, and Irina Higgins. Hamiltonian generative networks. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
[37] Irina Higgins, Peter Wimsberger, Andrew Jaegle, and Aleksandar Botev. Symetric: Measuring the quality learnt hamiltonian dynamics inferred from vision. Advances in Neural Information Processing Systems, 34, 2021.
[38] Alejandro Queiruga, N Benjamin Erichson, Liam Hodgkinson, and Michael W Mahoney. Stateful ode-nets using basis function expansions. Advances in Neural Information Processing Systems, 34, 2021.
[39] Yaofeng Desmond Zhong, Biswadip Dey, and Amit Chakraborty. Benchmarking energy- conserving neural networks for learning dynamics from data. In Learning for Dynamics and Control, pages 1219-1229. PMLR, 2021.
[40] Kookjin Lee, Nathaniel Trask, and Panos Stinis. Machine learning structure preserving brackets for forecasting irreversible processes. Advances in Neural Information Processing Systems, 34, 2021.
[41] Alvaro Sanchez-Gonzalee Jonathan Godwin, Tobias Pfaff, Rex Ying, Jure Leskovec, and Peter Battaglia. Learning to simulate complex physics with graph networks. Proceedings of the 37th International Conference Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 8459-8468. PMLR, 2020.
13

---

[42] Kiwon Um, Robert Brand. Philipp Holl, Nils Thuerey, et al. Solver-in Solver-in-the-logy the-loop: Learning from differentiable physies interact with iterative sde-solvers. arXiv preprint arXiv:2007.00016 2020.
[43] Eric Heiden, David Millard, Erwin Coumans, Yizhou Sheng, and Gaurav S. Sukhatme. Neuralsim: Augmenting differentiable simulators with neural networks. arXiv preprini arXiv:2011.04217 2020.
[44] Krishna Murthy Jatavallabhula, Miles Macklin, Florian Golemo, Vikram Voleti, Linda Petrini, Martin Weiss, Breandan Considine, Jerome Parent-Levesquu Kevin Xie, Kenny Erleben, et al. gradsim: Differentiable simulation for system identification and visuomotor control. arXiv preprint TrXiv:2104002646 2021.
[45] Miguel Jaques, Michael Burke, and Timothy M. Hospedales. pervised physical parameter estimation from video. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
-sve-aasr-----aaaaaaaaa Unsu-
[46] Bethany Lusch, J Nathan Kutz, and Steven L Brunton. Deep learning for universal linear embeddings of nonlinear dynamics. Nature communications, 9(1): 1-10, 2018.
[47] Samuel Rudy, Alessandro Alla, Steven Brunton, and J. Nathan Kutz. Data-driven identification of parametric partial differential equations. SIAM Journal on Applied Dynamical Systems, 18(2):643-666 2019.
[48] Naoya Takeishi, Yoshinobu Kawahara, and Takehisa Yairi. Learning koopman invariant sub- spaces for dynamic mode decomposition. Advances Neural Information Processing Systems, 30, 2017.
[49] Samuel E. Otto and Clarence W. Rowley. Linearly recurrent autoencoder networks for learning dynamics. SIAM Journal on Applied Dynamical Systems, 18(1 1811):558- -593, 2019.
[50] Akshunna S. Dogra and William Redman. Optimizing neural networks via koopman operator theory. Advances in Neural Information Processing Systems, 33:2087-2097, 2020.
[51] Matthew Tancik, Pratul P. Srinivasan, Ben Mildenhall, Sara Fridovich-Ki Keil,, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan T. Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. arXiv preprint arXiv:2006. 10739, 2020.
[52] Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. Advances in neural information processing systems, 20, 2007.
[53] James Lee-Thorp, Joshua Ainslie, Ilya Eckstein, and Santiago Ontanon. Fnet: Mixing tokens with fourier transforms. arXiv preprint arXiv:2105.03382. 2021.
[54] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin, Attention is all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30; Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 5998-6008, 2017.
[55] Linear grey-box model estimation. https://wwwt http://wwwmtttoorrss..ooe/e/ll/p html. Accessed: 2022-09-18.
iddata. wwwggeeeett...... iddata.greyest.
[56] M. Grewal and Keith Glover. Identifiability of linear and nonlinear dynamical systems. IEEE Transactions on automatic control, 21(6):833-88) 837, 1976.
[57] Hannu Pohjanpalo. System identifiability based on the power series expansion of the solution. Mathematical biosciences, 41(1-2):21-33, 1978.
[58] Hongyu Miao, Xiaohua Xia, Alan S. Perelson, and Hulin Wu. On identifiability of nonlinear ode models and applications in viral dynamics. SIAM review, 53(1):3-39, 2011.
14

---

[59] Tobias Nagel and Marco F. Huber. ieweenoooer-inspiiee identification of Iti systems. In 2027 European Control Conference (ECC), pages 2352-2357 2021.
[60] Ivan Markovsky and Florian D'orfler. Behavioral systems theory in data-driven analysis, signal processing, and control. Annual Reviews in Control, 52:42 64, 2021.
[61] Alberto Padoan, Jeremy Coulson, Henk J van Waarde, John Lygeros, and Florian D'orfler. Behavioral uncertainty quantification for data-driven control. arXiv preprint arXiv:2204.02671, 2022.
[62] Danijar Hafner, Timothy P. Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages
PMLR, 2019.
[63] Yaofeng Desmond Zhong and Naomi Leonard. Unsupervised learning of lagrangian dynamics from images for prediction and control. Advances in Neural Information Processing Systems, 33, 2020.
[64] Arthur Jacot, Clément Hongler, and Franck Gabriel. Neural tangent kernel: Convergence and generalization in neural networks. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett, editors, Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurlPS 2018, December 3-8, 2018, Montréal, Canada, pages 8580-8589, 2018.
[65] Tian Qi Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural ordinary differential equations. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett, editors, Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NewrIPS 2018, December 3-8, 2018, Montréal, Canada, pages 6572-6583, 2018
[66] Diederik Kingma and Max Welling. Auto-encoding variational bayes. In Yoshua Bengio and Yann LeCun, editors, 2nd International Conference Learning Representutions, ICLR 2014, Bansf, AB, Canada, April 14-16, 2014, Conference Truck Proceedings, 2014.
[67] Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Fourier neural operator for parametric partial differen- tial equations. arXiv preprint arXiv:2010.0882) 2020.
[68] Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Im- plicit neural representations with periodic activation functions. Advances in Neural Information Processing Systems, 33:7462-744 -773,, 2020.
[69] Kimin Lee, Younggyo Sco, Seunghyun Lee, Honglak Lee, and Jinwoo Shin. Context-aware dynamics model for generalization model-based reinforcement learning. In Proceedings of the 37th International Conference Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 3757-5766 PMLR, 2020.
[70] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] We use an autoencoder to estimate states from obser- vations with physics-in-the loop, and show that using the Fourier feature can improve generalization. Our experiments support this idea.
15

---

(b) Did you describe the limitations of your work? [Yes] See Section 7 and Section A in the supplementary material.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] We do not any potential negative societal impacts of the work. We show one example of applying the proposed model in a real engineering application Section 6, and discuss the potential impact and limitations in Section A.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] See Section and Section 5.
(b) Did you include complete proofs of all theoretical results? [Yes] See Section B and Section C in the supplementary material.
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes] See Section D in the supplementary material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section 6 and Section D in the supplementary material.
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [N/A] We follow the same style of machine learning papers to report the results.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Section D in the supplementary material.
4. If you are using existing assets (e-g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] We develop experiments for the visual tasks by modifying the code from [63]. addition, we develop own code for the MSD time series data.
(b) Did you mention the license of the assets? [Yes] The code in [63] is open-sourced.
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes] The URL can be bound in [63].
(d) Did you discuss whether and how consent obtained from people whose data you't 're
using/curating? [N/A] We do not use the data from other people. We generated the data by our own.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A] 'he data we use only contain measurements of the physical systems.
5. If you used crowdsoureing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A] We do not use crowdsourcing.
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvais, applicable? [N/A] We do not use crowdsoureing.
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A] We do not use crowdsourcing.
16


Reviews:
[{"review_id": "vEVLmNdiLcF", "reviewer": "Reviewer_dT3B", "summary": "This is an interesting paper. It addresses the task of predicting states and physical parameters of a system from observations using dynamic equations and autoencoder methods.", "questions": "1. Why not compare your approach with standard system identification methods from engineering: MATLAB will do this automatically for the types of examples you use. \n2. equation 1: is the system time-invariant (LTI)?\n3. line 171ff: what assumptions do you make about the knowledge of physics?\n", "limitations": "A better empirical analysis is needed. You must fix the fundamental mismatch between the input data (images) and the empirical validation (wheel suspension system, Mass-spring-damper, pendulum). \n\nIs this approach to be deployed on-line? Will the training be done off-line only?\n\n\nline 71, section 2 \"Related Work\" is highly limited. There is a huge literature on learning dynamical systems that is much more relevant than PINNs. \n --System identification is one area (you reference [50-52] later. \n --Nagel and Huber[2021] \"Autoencoder Identification of LTI systems\", ECC-2021.\n\nAlso see:\n\n@article{markovsky2021behavioral,\n  title={Behavioral systems theory in data-driven analysis, signal processing, and control},\n  author={Markovsky, Ivan and D{\\\"o}rfler, Florian},\n  journal={Annual Reviews in Control},\n  volume={52},\n  pages={42--64},\n  year={2021},\n  publisher={Elsevier}\n}\n@article{padoan2022behavioral,\n  title={Behavioral uncertainty quantification for data-driven control},\n  author={Padoan, Alberto and Coulson, Jeremy and van Waarde, Henk J and Lygeros, John and D{\\\"o}rfler, Florian},\n  journal={arXiv preprint arXiv:2204.02671},\n  year={2022}\n}\n", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Overall, it is difficult to understand what has been achieved. First, the empirical analysis does not provide clear evidence that the proposed approach improves quite basic engineering system identification methods, as provided in tools like MATLAB. The examples are trivial, and the firepower of the proposed approach needs more complex examples to show itself. Second, it is unclear what assumptions you make about the knowledge of physics in the basic design; the empirical analysis has very strong physics models, so reading just the empirical analysis seems to say that this is little different to LTI system identification methods.\n\nOther Issues\n============\nOne main issue that I have is the fundamental mismatch between the input data (images) and the empirical validation (wheel suspension system, Mass-spring-damper, pendulum). In real-world systems of this kind, typically the data will be from cheap sensors, e.g., accelerometer, position, etc.). This type of time-series input is easier to deal with than image data, and so the approach should be more successful.\n\nIs this meant to work with systems like autonomous vehicles, where image data is integrated with lidar, accelerometer, etc. data? What is the target application domain? It is extremely unlikely that a car manufacturer would install a camera to monitor a wheel suspension system, due to cost above all.\n\nIt would be helpful to have a diagram of your full pipeline--it's hard to piece this together from the fragmented descriptions of pieces of inference.\n\nA second fundamental issue is that the Simulations, section 6, only show system identification examples, with the help of autoencoders. The pendulum example has a well-defined ODE, and just needs to identify angle and friction coefficient. In fact, you only predict friction coefficient.\nAnalogous issues are with the other 2 examples. For mass-spring damper, the physics is pretty comprehensive, and in fact all you estimate is \\theta = [a(2)].\n\n\n\n\n\n Other issues:\n\nline 41: claim \"not guaranteed to have the Markov property, i.e., be a system state.\"----why is this true? Do you require ONLY Markov models?\n\n\nYour approach is significantly more complex than a basic autoencoder. Please address the relative computation complexity wrt a standard baseline, and describe the additional data and time (training, online) as compared to the baseline.\nObviously, there will be a tradeoff between relative computation complexity and accuracy, and this should be reported.\n\nFigure 5:a)  image is blurred since it is proprietary. Please delete it as it adds zero content.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "u7gx33o8GNm", "reviewer": "Reviewer_2274", "summary": "This paper proposes a method for estimating states of a physical system along with the system parameters. It can be seen as a VAE encoder-decoder set up where the decoder comprises a physics simulator (neural ODE solver) so that the latents (physical states and parameters) are forced to be interpretable. Important features of the encoder that let this method work well are 1. self-attention for estimating states from observations and 2. Fourier features for estimating parameters from states. Experiments confirm that these feature are crucial. The method is successfully applied to a real-world problem.", "questions": "What are the limitations of the physics simulator. Does it need to be representable as a differential equation (possibly non-linear)? What about a simulator written as a program? It seems like the only requirement is differentiability with respect to the initial state and the physical parameters.", "limitations": "N/A", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This paper tackles an important problem of learning how to estimate interpretable physical states and parameters from observations. It's clearly written. The experiments are well-selected and empirically verify the important components of the system (Fourier features and self-attention). Moreover, it is successfully applied to a real-world problem.\n\nI didn't fully grasp the significance of applying the NTK theory.\n\nI'm giving this paper a 7 (Accept) based on my understanding but I'm not an expert.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "gORU0byR5kJ", "reviewer": "Reviewer_Eodc", "summary": "This paper proposed an autoencoder-based model for system identification (estimating unknown parameters of a system defined by a dynamic equation). The model is shown to improve predicting long-term system dynamics with unknown physical parameters by predicting both the physical parameters and the system state.", "questions": "How is state error (SE) measured in the autoencoder baseline when it only reconstructs the observations without predicting states?\n\nFig 4 Two-Body Observation Reconstruction: why are the GTs for ALPS and CDM different?\n\nWeighting on loss function: it’s mentioned that `In practice, we find using the same weight for each loss term works well. However, in Fig. 4, it’s shown that SE, OE, and PE are of very different magnitude. If the same loss weight is used for each term, OE would dominate the loss for ALPS. \n\nScale differences in Fig 4 and Fig 6: \nE.g. For Two-body, the loss converged to ~4,000 in fig 6 while SE+OE ~= 2.59*10^8 in Figure 4, and loss = VAE loss + SE +OE according to Section 4\n", "limitations": "The authors are upfront about the following two limitations of the work:\n\nIt assumes prior knowledge of the system,e.g. dynamics equations.\n\nIt only works for linear systems with periodic or vibrational behaviors.\n\nOne additional limitation the reviewer would like to point out:\nAs mentioned in Weaknesses, it lacks experimental results of the full system with higher dimensional system parameters. Such results would make the paper much stronger for real system applications.\n", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\nWhile incorporating physics into Deep RL is not novel, this paper proposes to explicit predict the physical parameter which has shown promising results for predicting system dynamics\n\nProvided theoretical and empirical evidence for the benefit of Fourier Mapping of system states with a high-frequency component. As shown in the experiments, Fourier features boost the performance by a large margin, especially for State Error and Parameter Error.\n\n### Weaknesses\nExperiments for the full system (including the decoder) only include systems with one-dimensional system parameters. Experiments with 24 system parameters take the state measurement directly.\n\nCode was included in the supplemental material. It is modified from other codebases however all READMEs are from the original code base and outdated. Thus it’s impossible to reproduce the results with the code in its current state.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "KxoqWj45zvY", "reviewer": "Reviewer_qaG1", "summary": "This manuscript proposes ALPS, Autoencoder with Latent Physics, which uses neural networks as the feature extractor and parameter regressor. A differentaible physics simulator is used to assign physical meanings to the latent space so that it follows physical laws. The manuscript compare their method on several diverse tasks including a complex real-world example.", "questions": "A major question I would raise is the difference between $\\theta$ and $x$. Since $\\theta$ can be used as the most accurate descriptor of the physical dynamics (because it supports physical simulation), it makes more sense to use the network to directly regress $\\theta$ and skip $x$. Is there a reason behind it?", "limitations": "See weaknesses.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The overall presentation is very clear to me. The method makes sense to me and the pipeline is clear. There are motivating examples and also advanced examples. After training on each environment, this parameter estimation network can be deployed without re-training or re-running any costly solver.\n\nI would not call them weaknesses but I would like to know more about some choices the authors made in the manuscript. In my opinion, decoder $g(\\cdot)$ can be replaced with a differentiable rendering and save the effort of learning a visualization system. I am aware that using a network here might be more generalizable to other scenarios which are hard to model, but I would like to know if it is a possibility.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept