Symplectic Spectrum Gaussian Processes: Learning Hamiltonians from Noisy and Sparse Data
Authors: Yusuke Tanaka
Tomoharu Iwata
Naonori Ueda
Venue: NeurIPS 2022 Accept (2022)
Keywords: Gaussian processes
random Fourier features
Hamiltonian mechanics
variational Bayes

Abstract: Hamiltonian mechanics is a well-established theory for modeling the time evolution of systems with conserved quantities (called Hamiltonian), such as the total energy of the system. Recent works have parameterized the Hamiltonian by machine learning models (e.g., neural networks), allowing Hamiltonian dynamics to be obtained from state trajectories without explicit mathematical modeling. However, the performance of existing models is limited as we can observe only noisy and sparse trajectories in practice. This paper proposes a probabilistic model that can learn the dynamics of conservative or dissipative systems from noisy and sparse data. We introduce a Gaussian process that incorporates the symplectic geometric structure of Hamiltonian systems, which is used as a prior distribution for estimating Hamiltonian systems with additive dissipation. We then present its spectral representation, Symplectic Spectrum Gaussian Processes (SSGPs), for which we newly derive random Fourier features with symplectic structures. This allows us to construct an efficient variational inference algorithm for training the models while simulating the dynamics via ordinary differential equation solvers. Experiments on several physical systems show that SSGP offers excellent performance in predicting dynamics that follow the energy conservation or dissipation law from noisy and sparse data.

Symplectic Spectrum Gaussian Processes: Learning Hamiltonians from Noisy and Sparse Data
Yusuke Tanaka
Tomoharu Iwata
Naonori Ueda
NTT Communication Science Laboratories
(yusuke. tanaka, rh, tomoharu. iwwttaaaaa.... .y,, sy,noonori. ueda. frco ntt. oot.ttttt...o jo.
Abstract
Hamiltonian mechanics a well-established theory for modeling the time evolution of systems with conserved quantities (called Hamiltoniar), such as the total energy the system. Recent works have parameterized the Hamiltonian by machine learning models (e.g.. neural networks), allowing Hamiltonian dynamics to be obtained from state trajectories without explicit mathematical modeling. However, the performance of existing models is limited as we can observe only noisy and sparse trajectories in practice. This paper proposes a probabilistic model that can learn the dynamics of conservative or dissipative systems from noisy and sparse data. We introduce a Gaussian process that incorporates the symplectic geometric structure of Hamiltonian systems, which is used prior distribution for estimat- ing Hamiltonian systems with additive dissipation. We then present its spectral representation, Symplectic Spectrum Gaussian Processes (SSGPs), for which we newly derive random Fourier features with symplectic structures. This allows us to construct an efficient variational inference algorithm for training the models while simulating the dynamics via ordinary differential equation solvers. Experiments on several physical systems show that SSGP offers excellent performance in predicting dynamics that follow the energy conservation or dissipation law from noisy and sparse data.
Introduction
There is great interest the data-driven approach for learning the dynamics of physical systems. Modeling, simulating, and forecasting dynamics from data are fundamental in engineering and physical sciences; the data-driven approach also has the potential discover new laws physies [T6] 37. Many real- weal-world systems can be described by ordinary differential equations (ODEs). Classical data-driven approaches make strong assumptions about the form of the equations; thus, they are not applicable if the system's equation is unknown 31
36 4 The pioneering work of 2
neural ordinary differential equation (NODE) [4] has been presented as a general black-box model for learning vector fields, represented by functions, to output time derivatives of the system's 's state; the state of the system is continuously transformed along the vector fields (see the right part of Figure M]. The Hamiltonian neural network (HNN) [14] and variants (e.g.. 6 12 48) allow one to learn vector fields such that the total energy of the system (called Hamiltonian) is conserved. This formulation is advantageous for learning dynamics that follow the basic laws of physics (i.e., the energy conservation law). Several extended models cover Hamiltonian systems with additive dissipation 2 47 However, neural network based models implicitly assume that a large amount of training data with high temporal resolution is available.
This paper addresses the problem of learning Hamiltonian dynamics from noisy and sparse trajectories and predicting the dynamics from an arbitrary initial condition. We illustrate our problem Figure
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

The state trajectories are assumedd to be sparse in the following senses: 1) Limited observa- tion trials and 2) low temporal resolution (see Figure J. One promising approach to alleviate overfitting in sparse settings is Gaussian pro- cess (GP) B. which allows for model learn- ing while considering that the data contains uncertainties. GP models have been proposed for inferring unknown dynamics from trajec- tories [13 27 28), in which GPs are used for modeling the vector fields. They utilize GP approximation with random Fourier features (RFFs) 33 learn the dynamics via ODE solvers. The approximation needed for the following reasons. Since can obtain only trajectory data, not derivative observations (i.e., direct observations of vector fields), we cannot use the standard GP posterior conditioned on data points; thus, we require posterior approximation. The RFF-based approximation avoids the prohibitive computational complexity of generating sample paths (i.e., realizations of the vector field) from
Naisy and sparse trajectories
Learned dynamics
Noity state
Initil condition.
temporal resolution
1) Limited observation
Vector field
Figure 1: Problem setting. Color indicates time- evolution, starting at blue and ending at red.
the GP posterior (details are discussed in 44 45 ). This advantage in computation essential for
ODE-based learning. Since they do not, however, consider Hamiltonian mechanics, it is difficult to accurately capture dynamics that follow physical laws such as energy conservation and dissipation.
GP regression models for derivative observations have been proposed 38 in which the covariance function is given by matrix-valued kernels for estimating vector fields. Recently, symplectic Gaus- sian process regression (SympGPR) [65 has been proposed; the covariance function for learning conservative vector fields is derived by incorporating the theory of Hamiltonian mechanies into GP modeling. However, SympGPR assumes that one can obtain derivative observations, not trajectories, for training. Although can be naively applicable our problem by using finite differences, is difficult to learn the dynamics accurately from noisy state observations with low temporal resolution. Also, it is not applicable to Hamiltonian systems with dissipation.
This paper proposes a GP framework that can infer Hamiltonian systems with additive dissipative terms from noisy and sparse trajectories. We first extend the GP prior for Hamiltonian dynamics proposed in [35] to handle additive dissipation. The conserved quantity of the systems (i.e., Hamil- tonian) is assumed to be single-output GP. By employing Hamilton's iostoosssson equations, the vector fields are derived as a multi-output GP whose covariance function incorporates the symplectic structure for the energy conservation law. Moreover, one can handle Hamiltonian dynamics with friction by introducing the dissipation matrix. The most significant contribution of this work is the derivation of the RFFs that encode the symplectic structure to obtain the GP approximation for dissipative Hamiltonian systems, which we call Symplectic Spectrum Gaussian Processes (SSGPs). This can efficiently generate samples of the Hamiltonian vector fields from the GP posterior and construct the scalable learning algorithm based on ODE solvers. Inference in SSGP is based on variational Bayes, which allows for learning the Hamiltonian vector fields while considering the uncertainties posed by noisy and sparse data. Another benefit of SSGP is that it can be used for decomposing the dynamics into conservative and dissipative terms allowing the dynamics for unseen friction coefficients to be predicted. Such a task cannot be handled with models (e.g.. NODE) that do not use prior knowledge available in physics.
The main contributions of this work are as follows:
We introduce a GP prior for modeling Hamiltonian systems with additive dissipation.
We propose its spectral representation (called SSGP) by deriving RFFs that incorporate the symplectic structure of Hamiltonian systems.
We develop variational inference procedure for SSGP that offers numerical integration by ODE solvers as subroutine.
Experiments several physical systems show that SSGP can accurately predict the dynam- ics that follow the conservation or dissipation laws from noisy and sparse trajectories |
'Code is available at /tttps://gtthub..oo e/sssuuuussseeuuPpeeeee

---

Table 1: Comparison of the proposed model (SSGP) with the existing models. We compare the models for four items. The first and second items represent that the models can use the physies priors, i.e., (a) energy conservation law and (b) energy dissipation law, respectively, as an inductive bias for training. The third item (c) states that the models can be learned using ODE solvers from trajectory data; the models without a checkmark require derivative observations for training. The last item (d) shows that the models can handle uncertainties present in data.
HNN 14
D D-HNN
NODE
SymODEN
D D-SymODEN
SympGPR
SSGP
9 A9
4
48
35
Energy conservation law
(b) Energy dissipation law
Learning with ODE solver
(d) Uncertainty
2 Related Work
This work is positioned in the series of studies for learning black-box models that can estimate Hamiltonian dynamics from data. Here, black-box means that explicit mathematical modeling of differential equations is not required. We describe below prior works in this research line. Table T shows the comparison of the proposed model with the existing representative models.
Neural network models. Time-evolution of system states is generally described by differential equations. The neural ordinary differential equation (NODE) [ and its variants R. 19 have been proposed for learning the continuous-ting evolution of the states from data. In these studies, the vector fields that determine the state evolution are parameterized by deep neural networks (DNNs) and estimated by back-propagating errors of observed trajectories via the ODE solver. The Hamiltonian neural network (HNN) [14] introduced prior knowledge of Hamiltonian mechanics as an inductive bias for training DNNs; the core concept
parameterize the Hamiltonian (i.e., energy function) using DNNs. This formulation allows one to obtain dynamics that follow the energy conservation laws. Although the HNN assumes derivative observations for training, a learning procedure with ODE solvers from state trajectories was proposed in 6 48 Several models have been extended to handle Hamiltonian systems with additive dissipation 39
Most recent studies have further expanded the scope of application, including Hamiltonian systems with controllable inputs 19), stiff Hamiltonian dynamics [23), mnergy-conserving partial differential equation systems (25] odd-
17
dimensional chaotic systems 7. and Poisson systems (5) 17 Another approach, SympNets [18 [6].
models the symplectic map using neural networks, which are shown to universal approximators. However, existing DNN-based models implicitly assume situation wherein large amount of training data with high temporal resolution available; they might fail to capture dynamics in the noisy and sparse settings this work focuses
Gaussian process models. Gaussian process (GP) modeling has the advantage that it makes possible to learn models while considering uncertainties present in data [34]. Solak et al. [3 proposed GP regression models for derivative observations, in which the covariance function given by matrix-valued kernels for inferring vector fields. Curl-free and the divergence- preerennce-free kernels have been introduced for learning conservative vector fields [1 24]; J; however, they lack the flexibility to be extended to cover various physical dynamics as described in the previous paragraph. Rath et al. extended the model in 38 to learn Hamiltonian vector fields, which is called symplectic Gaussian process regression (SympGPR) 35). In SympGPR, the conserved quantity (i.e., Hamiltonian) is defined by a single-output GP: the covariance function for Hamiltonian vector fields can then be naturally derived on the basis of Hamilton's 's equations. However, these GP regression models assume derivative observations, not state trajectories. Although the use of finite differences allows for naively applying them to the problem of learning from trajectories, they might fail to capture dynamics, especially when the temporal resolution is lower, because each finite difference is computed from state pairs neighboring discrete time steps. Most recently, few models combined GPs with symplectic integrators [[]] 2 however, all the GP models [|]] systems with additive dissipation.
35 are not applicable to Hamiltonian
Learning with ODE solvers. Numerical integration methods play an essential role in learning dynamics from trajectory data. Chen al. [ provide convenient tool for solving ODEs and
3

---

Hamiltonian (x))
Vector field f(x)
Noiseless trajectory {
Noisy trajectory {z6g)
(a)
(b)
(c)
(d)
Figure 2: Schematic diagram of SSGP: Generative processes of noisy trajectories. (a) We model the unknown Hamiltonian H (x) by a single-output GP. Here, the color represents the magnitude of energy. (b) The vector field (x) calculated by applying the differential operator We sample the initial condition from the standard Gaussian distribution and solve the ODE defined by f(x) obtain the noiseless trajectory {æ.j} depicted by black dots. (d) The noisy trajectory {y:z) depicted by red dots is observed by adding Gaussian noise.
H (x)). (c)
learning parameters via back- pocc-propagation or memory-efficient adjoint methods, which are widely used for learning Hamiltonian dynamics (e.g..
One can also use advanced learning or discrete gradients [25], which can evaluate the loss function and its gradient while preserving energy accurately. These studies are related to ours different the research focus. They focus on how to estimate the parameters given the models proposal is model whose learning method can be chosen as needed. It might be possible to employ the above techniques for learning our model. Different from the above studies, this work can also consider uncertainties to learn the model from noisy and sparse data.
9
12
48
methods based on symplectic integrators 10.
4E
(e.g., DNNs); whereas
3 Hamiltonian Mechanics
In this section, we briefly review Hamiltonian mechanics 13 Let consider a system with N degrees of freedom. In the Hamiltonian formalism, the continuous-time evolution of the system is described in phase space, that is, the product space of generalized coordinates x = (x1 ......... x and generalized momenta æ! (xf..... .. Let where D = 2N. The system's 's evolution is determined by the Hamiltonian H(x) RDD - R. which denotes the system's total energy. Traditionally, the Hamiltonian manually designed suit the system. The dynamics of Hamiltonian system with additive dissipative terms is given by
(x* æP) E RR be state of the system,
dæ dt (S-R)VV((z) =: f(z), where S
O I )
(I)
Here, VH(2): RD + R is the gradient of the Hamiltonian with respect to state x, DxD the skew-symmetric matrix, R RDxD is the positive semi-definite dissipation matrix, is the identity matrix, and O is the zero matrix. In ①. we define the time derivatives of the state by the function f(x) RDD RD,, which is a special kind of vector field that has symplectic geometric structure (called Hamiltonian vector field or symplectic gradient). The dynamics on this vector field conserve the total energy when R O. One example of the dissipation matrix is R diag(0,.... 0, 0,,,....: .TN), representing a dissipative system with friction coefficient r, 2 0. Although assume this kind of dissipation matrix in the following, this work is extendable the general dissipation matrix, such as the state-dependent damping term predict state 8, at time &. by integrating f fxx) from 4
g Given vector field f
and initial condition æ; time 41,
as follows: x, = σ1 + Jú f(x) dt.
Model
We propose SSGP (Symplectic Spectrum Gaussian Process), a probabilistic model for learning Hamiltonian systems with additive dissipation from noisy and sparse trajectories. Figure shows schematic diagram of the proposed generative processes of noisy trajectories. the following, we first introduce a Gaussian process (GP) prior for modeling conservative and dissipative vector fields

---

by incorporating the theory of Hamiltonian mechanics. We then derive its spectral representation. for which we propose random Fourier features that encode symplectic structures. This spectral representation is used for constructing scalable inference algorithm described in Section S Finally, we describe the generative processes of noisy state observations.
GP priors for Hamiltonian systems with additive dissipation. In the proposed model, the unknown Hamiltonian (x) assumed to be single-output GP with zero mean. Let L (S R/ denote a differential operator. According to ( the vector field can be represented using L, as follows:
((x) C H (a); where H(x)) (O ((x, x')).
Here, 7(x, æ²) _R_ _D + a covariance function. Since differentiation is linear operator,
the derivative of a GP is again a GP 38 thus, f(æ) is given by multi-output GP,
f(x) ~ GP(O, (xx,
where is column vector of O's, and kxx,²') : DD x RDD + DDD the matrix-valued covariance function represented by
k(x,z')= = LC (x, - RR) xx,
(4)
Here, 2 is the Hessian operator. As described above, we can obtain the GP prior 3 with the covariance function 4 that encodes the geometric structure of Hamiltonian systems with additive dissipation. This formulation can be regarded as a generalization of the GP for non-dissipative Hamiltonian systems (35 Though one can opt for any positive definite kernel as 7(x x'), assume that (xx x') is shift-invariant in the approximation described in the next paragraph. One of the most widely used shift-invariant kernels is the ARD (Automatic Relevance Determination) Gaussian kernel,
1 7(x, x²) oJ exp (-1 (cc x') A =²)),
(x=)))
where A = diag (Af....,,23)
AB)
(5)
Here, oi E Rso and X3 E l250 are the signal variance and the length scale, respectively.
Spectral representations. We present the approximation of the GP prior L for which we newly derive random Fourier features (RFF) that encode symplectic structures of Hamiltonian systems. The RFF-based approximation advantageous for 1) estimating the GP posterior for vector fields without derivative observations and 2) efficiently sampling the vector field from the GP posterior, which has been suggested in recent studies on learning ODEs [ISI make it feasible to train our GP model for Hamiltonian systems with dissipation by utilizing ODE solvers, even when only trajectory data, not derivative observations, are available (see also the On computational complexity paragraph in Section 5 and Appendix Hamiltonian H (x) as a GP approximation,
These advantages
We begin by modeling the
H(x) = Σ Ww (m. (x), where Wm ~ N ((10
M
T T
(6)
Here, we adopt the M pairs of basis functions Φn (2) coos s x) sin TT))
parame-
terized by a D-dimensional column vector 8m of spectral points. The point 8m is sampled from the kernel's 's spectral density, which is given by
p(s) = N' (0, (4m72A)
wm E IR2 is row vector of weights for the moth pair of basis. We apply the differential operator L to the approximation 6 which yields the spectral representation (i.e., SSGP) of (3) represented by
in the case in which the ARD Gaussian kernel 5 is used 22 In 6
f( = L H
(
(8)
where (Wi.... wa)), ). and the resulting feature maps defined by (x D (z...... L as (x)); the mth feature map v pn (x) R + R dx2 is given
m (x) = 2T(S - R)sm sin JT x) cos cos(22Ts
2 An alternative version of basis functions 40 known, and proposal can easily adopt the alternative.
5

---

which we call symplectic random Fourier features (S-RFF). The derivation of g is described in Appendix A S-RFF 9 is the first incorporate symplectic structures into the random features for modeling vector fields by leveraging the knowledge of Hamiltonian mechanics. This result is not trivial in that we bridge two clearly distinct research topics, random features for kernel machines (i.e., GPs) and Hamiltonian mechanics. Let us explore the connection between the exact GP ( and its spectral approximation @ Denoting Dirac's delta function by p(flw), the distribution f given by integrating out from (8. follows
of M
p(f) - p(flu))p(w)dd = N -N0, (2, where K(x, x²)
(x)) (10)
Here, K(x, : X + RDxD is the covariance function for GP approximation. We observed that approximation quality improves with the number of spectral points (i.e., the number basis functions) by comparing the gram matrix of ((x, x') 44) with that of K(x, x') in (IO). where the spectral points 8m are sampled from 7 We show the gram matrices in Appendix B
Generative processes of noisy observations. Suppose that we have a collection of trajectories {(te). Vty)( 1..... I;j sample specified by a pair (tij. Voj], which represents the observation of noisy state yij time bjj- We treat the noiseless state Tij. the counterpart of Yij. latent variable. We assume that the observation model of Yıy is a Gaussian distribution with variance of a². Letting Y = { J. the marginal likelihood (i.e., evidence) is given by
l..... J:), where Jj is the number of samples in the ith trajectory. Each
/ D p(uulzn)p(am))
Jr II p(u b (es)pp afff df, (II)
p(Y) = P() II
where p(f) is the GP prior (TO) of the vector field, and p(=,1) is the prior distribution of the initial condition I; Given f and T1,, the state Ziz is deterministically given by solving the ODE; thus, we can write the conditional distribution p(æ:;g ff x.)) in (II) using Dirac's delta function, as follows:
p( [f, an) 5 Fij an J.m
f(x) f(x))
(12)
Note that, although we omit the observation time points {tay] in 11 is actually conditioned on {t For simplicity, we adopt this notation hereinafter.
Inference
Although would like to estimate the model parameters w. A, oi. o² and R. by maximizing the logarithm of (TT. we cannot calculate the exact marginal likelihood (TT) analytically as it includes the process of solving the ODEs. We then present a variational inference procedure for learning Hamiltonian systems with additive dissipation from noisy and sparse trajectories. The spectral representation of SSGP used for efficient sampling of the conservative or dissipative vector fields from the GP posterior, allowing the learning to proceed with ODE solvers while handling uncertainties in the vector fields.
Parameter learning. We consider the following evidence lower bound (ELBO),
log (Y)
Σ (ayy) [log (Viy (xa))] - KL (g(2=)))l(((a))) KL ([(w))))(()) (13)
where (Zxi)) and ((w) are the variational distributions of the noiseless state æij and the weights w, respectively. We assume that the variational distribution of w is given by Gaussian distribution, ((w) N (b, C), where b E RRM and C R22 X2M the mean and the covariance matrix, respectively. For j = 1, we assume that the variational distribution of initial condition æn is given
'The derivation procedure similar to that described in Chapter 4.5.2].
For simplicity, the prior is set the standard Gaussian distribution in the implementation.

---

by a Gaussian distribution with the mean being the observed state Yı1. ((x.)) = N (y:1 A), where
A E IR DxD is the covariance matrix. For j 2 2, the variational distribution of zij is given by
/ʃ p(a,,)ff of ....)
/ p(f w )q( (q(()dd
(
q ((.11)x1111
(14)
where the factor in square brackets is the variational distribution of f. The derivation of described in Appendix
13 is
We approximate the expectation
13
using Monte Carlo integration,
E (mns) [log p
(gyy Fiy]]
(
(15)
where the Monte Carlo samples are given by
(k) a il = ya + VA (k) where e(th) ~ ~N0,I)
(16)
(k) ODESolve Fil J.
(k) (x), t t.J.)
(17)
22
where the reparameterization trick 20 in 16 and we perform the numerical integration
by the ODE solvers (e.g., the Runge-Kutta method) in 17 Here, K is the number of Monte Carlo
samples to obtain (15). and v denotes matrix square root. (k)) (x)) in (17 the sample of the vector field generated from the variational posterior of f.
L
(x) (x))
1
(k.))
where (R.I) = b+ (C
(18)
T,
Here, T. is the number of Monte Carlo samples obtain (8). and ((h.)) ~N(0,, I). In the spectral representation, the randomness of function f is totally controlled by the distribution of weights w. Accordingly, we optimize both model parameters and variational parameters while considering uncertainties in the vector field by sampling w at each training iteration. The inference procedure shown in Appendix D We can easily extend our proposed approach to learn Hamiltonians from high-dimensional data (such as images) by combining an autoencoder with an SSGP, as in [4] 42 The states z and x' in the covariance function ((x, x²) are modeled as the latent vectors of the autoencoder. This formulation can be regarded as an extension of the SSGP based on deep kernel learning [33
Prediction. The variational posterior, ((f) = p(f (w) has a closed-form solution, a Gaussian distribution whose mean function and covariance function are given by r m=xx) = (x: and (x x')= = L ( T (æ'), respectively. One can predict the dynamics from an arbitrary initial condition by numerically integrating the mean function in (x). Also, one can evaluate the predictive uncertainty for the vector field by using the covariance function K* K²(xxx'),
On computational complexity. The computational bottleneck is the sampling of w; the cost of computing /C in 18 is ((3)). The cost increases with the number of basis functions; however, the most important thing is that the bottleneck is outside of the ODE solver. The sampling of w makes it possible to efficiently evaluate function values at arbitrary point æ, as in (8). and to use it in the ODE solver in (7). The advantage in computational costs discussed in Appendix L In addition, previous studies have shown experimentally that the GP models approximated by RFF have, in many cases, high predictive performance even when setting M lower than 102 (e.g.. (22) Actually, our experiments show that SSGP yields an accurate prediction using small set of RFFs. The average training time when setting M 250 was 2943.0 seconds for the dataset of the pendulum with friction; the experiments were conducted on the AMD EPYC 7313 CPU (3.0GHz).
6 Experiments
Data. We evaluated the proposed model, SSGP, using two physical systems: pendulum, and Duffing oscillator. The Hamiltonian of the pendulum system is
(xx)2
H(x) = 2mgl(1 - cos 2) +
(19)
2m
7

---

where we denote the gravitational constant by g. the mass constant by m. and the length of the pendulum by In the experiments, we set g and m = The Hamiltonian of the Duffing oscillator system is
1 (x (29)) + ((x)
B (19)4
(20)
where we set the parameters, a = == 8= 1. For each system, the dissipation matrix was set to R O (energy conservation) or R = diag(0, 0.05) (energy dissipation). We generated trajectory data by employing a numerical integrator, i.c. the Dormand-Prince method with adaptive time-stepping. We sampled initial conditions with total energies uniformly distributed across a predefined range. The energy ranges for pendulum and Duffing oscillator were [1.3, 2.3] and [0.5, 1.5]. respectively. To evaluate the robustness of our model against the degree of sparsity, we prepared f10, 15, 20, 30, 50) trajectories sampled at a frequency of {3, 5, 10) Hz for 10 seconds, and added Gaussian noise with variance o² 0.1 to each sample. We randomly split the trajectory data and used 70% for training and 30% for validation. Here, if the number of trajectories for validation was less than five, we used five trajectories for validation and the rest for training. We generated a test set of 25 trajectories independently from training and validation sets; each test trajectory was sampled five times by resampling the training and validation sets.
implemented in torchdiffed K 4
frequency of 100 Hz for 15 seconds. The experiments were conducted
Task I: Normal prediction. We evaluated performance by comparing the predicted state trajectories { with the ground truth {ar! ) (i.e., the test set described above). The evaluation metric is the mean squared error (MSE), 4 C+ l| -; = ||²), where || 11² is the Euclidean norm. In the evaluation, we used the datasets from four settings: pendulum and Duffing oscillator with or without energy dissipation.
Task 2: Predicting the dynamics for unseen friction coefficients. One benefit of SSGP is that it can decompose the dynamics into its conservative and dissipative terms, thus predicting the dynamics for arbitrary friction coefficients. To show this, we evaluated SSGP by the following procedure: 1) trained the model using the dataset from pendulum or Duffing oscillator with the friction coefficients, R diag(0, 0.05); 2) we predicted its conservative system by using the learned Hamiltonian, where we set the friction coefficients, R = O. The evaluation metric is the same as that in Task 1.
Setup of the SSGP. We trained the model using the Adam optimizer 21 with learning rate of 10-3 for 10t epochs, implemented in PyTorch [30]. We performed numerical integration by the adaptive Dormand Princce method 3 4 with the relative and absolute tolerances of 10 8 We set the numbers of Monte Carlo samples to K = 1 and L = 100. The number M of spectral points was chosen from {100, 250, 500} based on the validation error. We used block-diagonal approximation of C, namely, ((w) is factorized as two factors separating the weights for the sines and cosines, where the computational complexity is the same as that of the original. We fixed the friction coefficients R = O when training the conservative systems in Task 1.
Baselines. We compared the SSGP with the existing models (see Table I Hamiltonian neural net- work (HNN) 14, dissipative HNN (D-HNN) 39), neural ordinary differential equation (NODE) 4. symplectic ODE-Net (SymODEN) Gaussian process regression (SympGPR) 35 Since HNN, D-HNN and SympGPR require derivative observations for training, used the finite difference instead. Another baseline is the GP model using standard random Fourier features (RFF), which corresponds to the case that the vector field f(x) in SSGP is modeled as a multi-output GP approximated by the standard RFF (not consider- ing Hamiltonian mechanics). In the experiments, we call it RPF The details of each baseline are described in Appendix
48 dissipative SymODEN (D-SYMODEN) 47 and symplectic
Results. We present the experimental results that are picked up from the case where the sampling frequency was 5 Hz. Appendix shows all the results, including those for the frequencies of 3 and 10 Hz. Figure ssows MSE and standard errors for SSGP and the baselines in Task and Task 2. As expected, in many cases, MSE decreased as the number of trajectories increased. The performance of HNN, D-HNN and SympGPR was limited because their training was based on finite differences. NODE and RFF had large errors except for the case of Figure (b) because they cannot use the prior knowledge of Hamiltonian mechanics as the inductive bias. SymODEN and D-SymODEN were more competitive with SSGP; nevertheless, SSGP matched bettered their predictive performance.
hhanss/ggiiub.ccc//nnaaaetttttte (MIT License)
8

---

Task 1. Conservative systems
Task 1. Dissipative systems
Task 2. Conservative systemss
(a)
(b)
16-1
(c)
ESGP
55GP
3.0
NCDE
NOGE
NOGE
Pendulum
HPP
D-SUMODEN
2.5
D-SamODeN
MSE
2.0
1.5
1.0
Number trajectoniss
Number of trajectortes
Number of trajectories
(d)
(e)
(f)
Dutting Oscillator
RFF
1.5
RFF
SyerddeN FINN
D DyMOODDN
Symordnn
M6E
D-114
MSE
1.0
0.5
10
20
30
40
50
10
20
30
50
10
20
30
40
50
Number of trajectories
Number of trajectories
Number of trajectories
Figure 3: MSE and standard errors for the predicted trajectories when the sampling frequency was Hz. The first and second columns are the results for conservative and dissipative systems, respectively, in Task 1. The third column holds the results predicted for conservative systems in Task 2.
Table 2: MSE of the friction coefficient. All values are multiplied by 10
pendulum
Duffing
SSGP
0.519
0.506
D-SyMODEN
1.010
0.630
D-HNN
2.486
2.861
Moreover, SSGP improved the performance rather than all basclines, especially when the number of trajectories was small. These results show that SSGP is advantageous in sparse settings. In Figure (b), NODE and RFF yielded relatively low errors; however, since they cannot distinguish between the conservative and dissipative terms, their performance was worse in Task 2, as shown in Figure B (c). In contrast, SSGP can estimate the conservative and dissipative terms separately by incorporating knowledge of Hamiltonian mechanics, which yields better performance in both Task and Task than all baselines. Table 2 shows MSE of the friction coefficients estimated by SSGP, D-SymODEN and D-HNN when inferring the dissipative systems. Here. MSE was averaged for all experimental settings. This result shows that SSGP can accurately estimate the friction coefficients.
In the following, we deeply explore the results when the number of observed trajectories was 20. Figure 4 shows the visualization of the trajectories predicted by SSGP. One observes that SSGP appropriately captured the dynamics of the dissipative Hamiltonian systems (the third column of Figure 4h and better discerned the conservative and dissipative terms (the fourth and fifth columns of Figure 4.. Figure 5 shows the cumulative errors for the trajectories and energies for the pendulum data. Let ftest denote the number of time steps in the test set, and j. denote time step at time I.
We calculated the cumulative error of trajectories at time t: E)=(( yince Eie || dz. æ I!²). The cumulative error of energies was obtained similarly by calculating the squared error of the energy, Htsue (Zxy), evaluated using the predicted trajectories and the true energy, Hiuue (a
Here, H-) is the true Hamiltonian of each system. As shown in Figure 5 SSGP achieved lower enrors than the baselines for both trajectories and energies at all time points. The performance improvement of SSGP was more significant in the period [10, 15), which is out of the simulation period (10 seconds) when generating the training data. These results show that SSGP can accurately predict the dynamics with energy conservation and dissipation in both short-term and long-term simulations.

---

True
True conservative tive term
Task 1. Predicted dissipative system
Task 2. Predicted conservative term
Task 2. Predicted dissipative term
dissipative system
Pendulum
Kp 0
0
Xp
-2.5
0.0
2.5
-2.5
0.0
2.5
-2.5
0.0
2.5
Dutfing Oscillator
0 X 0
X/ 0
Xg -2.5
0.0
2.5
-2.5
0.0
2.5
-2.5
0.0
2.5
Ag
Ag
Figure 4: Prediction results of SSGP. The color indicates time-evolution, starting at blue and ending red. The first and second columns are the true trajectories for the dissipative systems and their conservative terms, respectively. The third column is the prediction for the dissipative systems in Task 1. The fourth and fifth columns are the predicted conservative and dissipative terms in Task respectively. Here, the dissipative terms are multiplied by 30 for enhanced clarity. Comparisons with other models are shown in Appendix
Task 1.
Task 1. Dissipative
Task 2.
Conservative
Conservative
1e3
163
1e3
S3GP NODE
1.00
SEGP
SSGP
Cumuissive error
NODE
Trajectory
RFF
0.75
RFF
SymELZEN
D-SYMODEN
D-SyMODEN D-HNUN
HNN
0.50
2
0.00
0
10
0
5
10
15
15
Cumuistive arror
1.00
1.0
4
3
Energy
SymODEN
D-SymODEN
D-SYMODEN D-HNN
0.50
SympGPP
HNN
DHNNN
2
0.25
1
0.00
time
time
time
Figure 5: Cumulative error for the predicted trajectories and energies when using pendulum data. The horizontal axis is simulation time in the test phase.
7 Conclusion
We have proposed the Symplectic Spectrum Gaussian Process (SSGP), which allows one to predict systems whose dynamics follow energy conservation and dissipation laws from noisy and sparse data. Our result, the symplectic random Fourier feature, is general tool and has the potential to use the design of kernel machines with prior knowledge in physics. As described in Section []] the SSGP can easily extend to learn Hamiltonians from nigh-dimensional data (e.g., images), Our future work is to evaluate the effectiveness of its extension.
Acknowledgments and Disclosure of Funding
This work was supported by JST, ACT-X Grant Number JPMIAX2IOD, Japan.
10

---

References
[1] Luca Baldassarre, Lorenzo Rosasco, Annalisa Barla, and Alessandro Verri. Multi-output learning via spectral filtering. Mach. Learn., 87(3):259-301, 2012.
[2] Christopher M. Bishop. Pattern Recognition and Machine Learning. Springer, 2006.
[3] Ricky T. Q. Chen, Brandon Amos, and Maximilian Nickel. Learning neural event functions for ordinary differential equations. In International Conference on Learning Representations, 2021.
[4] Ricky T. Q. Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. In Advances in Neural Information Processing Systems, volume 31, 2018.
[5] Yuhan Chen, Takashi Matsubara, and Takaharu Yaguchi. Neural symplectic form: Learning Hamiltonian equations on general coordinate systems. In Advances in Neural Information Processing Systems, 2021.
[6] Zhengdao Chen, Jianyu Zhang, Martin Arjovsky, and Léon Bottou. Symplectic recurrent neural networks. In International Conference Learning Representations, 2020.
[7] Kevin Course, Trefor Evans, and Prasanth Nair. Weak form generalized Hamiltonian learning. In Advances in Neural Information Processing Systems, volume 33, pages 18716-18726, 2020.
[8] P. Courtier, Jean-Noël Thépaut, and A. Hollingsworth. A strategy for operational imple- mentation of 4d-var. In Workshop on Variational Assimilation, with special emphasis on Vhrree-dimensioall Aspects, 9-12 November 1992, pages 437-464, Shinfield Park, Reading, 1992. ECMWF, ECMWE.
[9] Shaan A. Desai, Marios Mattheakis, David Sondak, Pavlos Protopapas, and Stephen J. Roberts. Port-Hamiltonian neural networks for learning explicit time-dependent dynamical systems. Phys. Rev. E, 104:034312, Sep 2021.
[10] Daniel DiPictro, Shiying Xiong, and Bo Zhu. Sparse symplectically integrated neural networks. In Advances in Neural Information Processing Systems, volume 33, pages 6074-6085, 2020.
[11] Katharina Ensinger, Friedrich Solowjow, Sebastian Ziesche, Michael Tiemann, and Sebastian Trimpe. Structure- preeeerin Gaussian process dynamics. arXiv, 2021.
[12] Marc Finzi, Ke Alexander Wang. and Andrew G Wilson. Simplifying Hamiltonian and La- grangian neural networks via explicit constraints. Advances in Neural Information Processing Systems, volume 33, pages 13880-13889, 2020.
[13] Herbert Goldstein. Classical Mechanics. Addison Addison-Wesley. 1980.
[14] Samuel Greydanus, Misko Dzamba, and Jason Yosinski. Hamiltonian neural networks. In Advances in Neural Information Processing Systems, volume 32, 2019.
[15] Pashupati Hegde, Çağatay Yıldiz, Harri Lähdesmäki, Samuel Kaski, and Markus Heinonen. Variational multiple shooting for Bayesian ODEs with Gaussian processes. In arXiv, 2021.
[16] Raban Iten, Tony Metger, Henrik Wilming, Lídia del Rio, and Renato Renner. Discovering physical concepts with neural networks. Phys. Rev. Lett., 124:010508, Jan 2020.
[17] Pengzhan Jin, Zhen Zhang, Ioannis G. Kevrekidis, and George Em Karniadakis. Learning Poisson systems and trajectories of autonomous systems via Poisson neural networks. IEEE Transactions on Neural Networks and Learning Systems, pages 1-13, 2022.
[18) Pengzhan Jin, Zhen Zhang, Aiqing Zhu, Yifa Tang, and George Em Karniadakis. SympNets: Intrinsic structure- preettrre--reervingg.... symplectic networks for identifying Hamiltonian systems. Neural Networks, 112:1661179 2020.
[19] Patrick Kidger, James Morrill, James Foster, and Terry Lyons. Neural controlled differential equations for irregular time series. In Advances Neural Information Processing Systems, volume 33, pages 6696-6707, 2020.

---

[20] D. Kingma and M. Welling. Auto-encoding variational Bayes. In International Conference on Learning Representations, 2014.
[21] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Interna- lional Conference on Learning Representations, 2015.
[22] Miguel Lázaro-Gredilla, Joaquin Quinnero-Candela, Carl Edward Rasmussen, and Aníbal R. Figueiras-Vidne Vidal. Sparse spectrum Gaussian process regression. Journal of Machine Learning Research, 1166))1885--188 1881, 2010.
[23] SENWEI Liang, Zhongzhan Huang, and Hong Zhang. Stiffness-aware aware neural network for learning Hamiltonian systems. International Conference on Learning Representations, 2022.
[24] Y. Macedo and R. Castro. Learning div-free and curl-free vector fields by matrix-valued kernels. In Technical report, Preprint A 679/2010 IMPA, 2008
[25] Takashi Matsubara, Ai Ishikawa, and Takaharu Yaguchi. Deep energy-based modeling of discrete-time physics. In Advances in Neural Information Processing Systems, volume 33, pages 13100-13111, 2020.
[26] Takashi Matsubara, Yuto Miyatake, and Takaharu Yaguchi. Symplectic adjoint method for exact gradient of neural ODE with minimal memory. In Advances Neural Information Processing Systems, volume 34, pages 2-7844,4 2021.
[27] Thomas McDonald and Mauricio Álvarez. Compositional modeling of nonlinear dynamical systems with ODE- random features. In Advances in Neural Information Processing Systems, volume 34, pages 13809-13819, 2021.
[28] Hossein Mohammadi, Peter Challenor, and Mare Goodfellow. Emulating complex dynamical simulators with random Fourier features. arXiv, 2021.
[29] C. Offen and S. Ober-Blöbaum. Symplectic integration of learned Hamiltonian systems. Chaos: An Interdisciplinary Journal of Nonlinear Science, 32(1):013122, 2022.
[30] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Vetooo, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems, pages 8024-8035. 2019.
[31] Pascal Pernot and Fabien Cailliez. critical review of statistical calivrattiooopprreiciionnn models handling data inconsistency and model inadequacy. AIChE Journal, 63(10):4642-4665 2017.
[32] Dimitris C. Psichogios and Lyle H. Ungar. hybrid neural network-first principles approach to process modeling. AIChE Journal, 38(10):1499-10 1511, 1992.
[33] Ali Rahimi and Benjamin Recht. Random features for aargesccae kernel machines. In Advances in Neural Information Processing Systems, volume 20, 2007.
[34] C. E. Rasmussen and C. K. I. Williams. Gaussian Processes for Machine Learning. MIT Press, 2006.
[35] Katharina Rath, Christopher G. Albert, Bernd Bischl, and Udo von Toussaint. Symplectic Gaussian process regression of maps in Hamiltonian systems. Chaos: An Interdisciplinary Journal Nonlinear Science, 3 31(5):053121, 2021.
[36] R. Eico-Martinez, J.S. Anderson, and L.G. Kevrekidis. Continuous-time nonlinear signal processing: A neural network based approach for gray box identification. In Proceedings of IEEE Workshop Neural Networks for Signal Processing, pages 996-605. 1994.
[37] Michael Schmidt and Hod Lipson. Distilling free-form natural laws from experimental data. Science, 324(5923)881-855 2009.
12

---

[38] E. Solak, R. in Gaussian process models of dynamic systems. In Advances in Neural Information Processing Systems, volume 15, 2003.
W. Leithead, D. Leith, and Carl Rasmussen. Derivative observations
[39] Andrew Sosanya and Sam Greydanus. Dissipative Hamiltonian neural networks: Learning dissipative and conservative dynamics separately. CoRR, abs/2201. 10085, 2022.
[40] Danica J. Sutherland and Jeff Schneider. On the enror of random Fourier features. In Proceedings of the Thirty-First Conference on Uncertainty in Artificial Intelligence, pages
2015.
[41] Michael L. Thompson and Mark A. Kramer. Modeling chemical processes using prior knowl- edge and neural networks. AIChE Journal, 40(8):1328-133 1994.
[42] Peter Toth, Danilo J. Rezende, Andrew Jaegle, Sébastien Racanière, Aleksandar Botev, and Irina Higgins. Hamiltonian generative networks. In International Conference on Learning Representations, 2020.
[43] Andrew Gordon Wilson, Zhiting Hu, Ruslan Salakhutdinov, and Eric P. Xing. Deep kernel learning, In International Conference on Artificial Intelligence and Statistics, volume 51, pages 370-378, PMLR, 2016.
[44] James Wilson, Viacheslav Borovitskiy, Alexander Terenin, Peter Mostowsky, and Mare Deisen- roth. Efficiently sampling functions from Gaussian process posteriors. In International Confer- ence on Machine Learning, volume 119, pages -0302. PMLR, 2020.
[45] James T. Wilson, Viacheslav Borovitskiy, Alexander Terenin, Peter Mostowsky, and Marc Peter Deisenroth. Pathwise conditioning of Gaussian processes. Journal of Machine Learning Research, 22(105):1-47, 2021.
[46] Shiying Xiong, Yunjin Tong, Xingzhe He, Shuqi Yang, Cheng Yang, and Bo Zhu. Nonseparable symplectic neural networks. In International Conference on Learning Representations, 2021.
[47] Yaofeng Desmond Zhong, Biswadip Dey, and Amit Chakraborty. Dissipative SymODEN: Encoding Hamiltonian dynamics with dissipation and control into deep learning. In ICLR 2020 Workshop on Integration of Deep Neural Models and Differential Equations, 2020.
[48] Yaofeng Desmond Zhong, Biswadip Dey, and Amit Chakraborty, Symplectic ODE-Net: Learn- ing Hamiltonian dynamics with control. In International Conference Learning Representa- tions, 2020.
[49] Juntang Zhuang, Nicha C Dvornek, sekhar tatikonda, and James Duncan. MALI: A memory efficient and reverse accurate integrator for neural ODEs. In International Conference on Learning Representations, 2021.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] The main contributions of this work have been
summarized in the last paragraph of Section
(b) Did you describe the limitations of your work? [Yes] The limitations have been described in Appendix H
(c) Did you discuss any potential negative societal impacts of your work? [Yes] We have discussed the societal impacts in Appendix H
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes] We have read the guidelines. No ethical impact supposed.
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? assumptions have been described in Sections and 5
[Yes] All the
13

---

(b) Did you include complete proofs of all theoretical results? [Yes] Derivation procedures of our models have been included in Appendices A and
3. If you experiments...
(a) Did you include the code, data, and instructions needed reproduce the main ex- perimental results (either the supplemental material or as a URL)? [Yes] Code is available at htthtts:// htttp:://githbbbb com/yusuk- e/SGGP
(b) Did you specify all the training details (e.g.. data splits, hyperparameters, how they were chosen)? [Yes] We have described how to generate datasets in the Data paragraph of Section 6 The setups of the proposed model and the baselines were specified in the Setup of the SSGP paragraph of Section 2 and Appendix F respectively.
(c) Did you report error bars (e.g.. with respect to the random seed after running experi- ments multiple times)? [Yes] We have provided the standard errors of the performance in Figure g
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] We used AMD EPYC 7313 CPU (3.0GHz) in the experiments, mentioned in the On computational complexity paragraph of Section
4. If you are using existing assets (c.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes] We used the python
library, torchdiffeq 3 g. which we have cited in Section
(b) Did you mention the license of the assets? [Yes] We have mentioned the license
information in the footnote on page 7.
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes] Code is available httpp:: h/pp::///1t thub. com wywyyyusuukssssss
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "GPu7yR7LP7W", "reviewer": "Reviewer_9GYm", "summary": "This paper proposed a symplectic spectrum Gaussian process method. The method can predict systems whose dynamics follow energy conservation and dissipation laws from noisy and sparse data. The proposed method is a general tool and has the potential to use the design of kernel machines with prior knowledge in physics. Although the proposed method is solid in theory, the scenarios applied are relatively simple. In addition, some classic examples such as vortex-particle and gravitational systems should be compared with the recently proposed methods.", "questions": "Overall, the exposition of the paper is pretty straightforward. However, I assume it to be difficult for readers who are not familiar with Gaussian process models. More specific comments:\n- The equations of the predicted systems should be given in the main text.\n- Is the order of the subgraphs in Figure 4 reversed.\n- The introduction of the theoretical model could be more concise.\n- The idea in Figure 2 is not clear. Can you show a clear schematic?\n- Why do the other methods in Figure 3 perform well in their papers?\n- The selection of some parameters should be discussed in detail. For example, time step, parameter size, data set size, etc.\n", "limitations": "I would love to hear some limitations of the proposed paper about failure modes that the authors might have encountered.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\nThis paper proposed a symplectic spectrum Gaussian process method for modeling Hamiltonian systems with additive dissipation. They derived a new spectral representation by incorporating the symplectic structure to handle energy dissipation and energy conservation systems. They also proposed a variational inference procedure that offers numerical integration of the ODE solver as a subroutine. Finally, they demonstrated some experiments on several physical systems to verify the accuracy of the proposed symplectic spectrum Gaussian process method. \n\nWeaknesses\n\nThe demonstrated examples seem to be cherry-picked, and recently proposed methods, such as SympNet, can also accurately predict simple Hamiltonian systems.\n- Adding some citations of recent papers related to symplectic neural networks would be helpful. \n- The symplectic neural networks are divided into separable and non-separable. The current paper seems to discuss only separable Hamiltonian systems, which needs to be clarified.\n- Limitations of the method need to be discussed.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "EZEIt_lCCG5", "reviewer": "Reviewer_AdVL", "summary": "The manuscript proposes a way to learn Hamiltonian systems with additive dissipation from (scattered) observations.\nThe learning is framed as a (Bayesian) inference of a Hamiltonian endowed with Gaussian Process prior with the rest of the generative model relating trajectories to the hamiltonian via dissipative dynamical equations, and noisy observations on top of these trajectories.\n\nThe prior covariance matrix on trajectories is approximated using RFF for an ARD base kernel and propagating the approximation via the linear operator specifying the dynamics.\n\nVariational inference under this (approximate) prior model is used, which conveniently allows to use the reparametrization trick to evaluate the ELBO.\n\nThe resulting method is evaluated on a number of experiments and compared to pre-existing methods.\n", "questions": "I don't understand the following sentence:\n\"We used a block diagonal approximation of C so that each pair of basis functions shared the same covariance\".\nIs this mean field $q(C)=\\prod_i^M q(C_i)$, or $q(C) = \\tilde{q}(C)^K$. In both case, this has strong consequence on the inference.\nThese choices should be discussed.\n", "limitations": "None", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "-- Strengths\n\nThe method is a useful extension of the SymGPR method to dissipative systems.\n\nThe combination of the prior on hamiltonian, dissipative dynamics and RFF appears original to me.\n\nThe combination of many different approximations (RFF, VI, stochastic evaluation) leads to a practical algorithm for the problem at hand.\n\nThe possibility to make predictions without or with different dissipation is expected but neat.\n\nThe results are impressive especially in the low data regime, where the method clearly outperforms alternatives in its prediction accuracy.\n\n-- Weaknesses\n\nI report here a few points that made the manuscript a bit difficult to read.\nIt is mostly about the motivation rather than the technical content.\n\n1) Previous work, especially the symGPR is not introduced which makes it difficult to understand the novelty here.\nWhat did they do exactly? did they also use RFF to approximate the covariance or do the calculation closed form?\n\n2) the many approximations introduced are not necessarily well motivated. Why do you do the RFF? Is it necessary?\n convenient for VI? for scalability? I have my guess but this needs to be more explicitely stated in the paper.\n \n3) The consequence of the approximations introduced are not discussed.\nDoes the RFF preserves the symplectic structure? What is lost by approximating the prior does it bias the inference? if so how?\nThe same applies to using VI.\n\n\nBecause of these perceived weaknesses,\nI set my score to weak accept. I m willing to change my evaluation if my concerns are adequately addressed\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Arzzd5TqHWK", "reviewer": "Reviewer_J5NT", "summary": "Paper propose a spectral representation of Hamiltonian. Demonstrated the procedure to solve the equations of motion given a set of noisy measured trajectories (line 202).\n\nThe learning process is on the parameters used in the formalism of the special Hamiltonian (line 224).", "questions": "Please read the questions given in the section \"Strengths and Weakness\".", "limitations": "Discussion on limitations is given in a short paragraph towards the end of the paper. The authors propose to extend this formalism into high dimensions.", "rating": 3, "confidence": 2, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nThe formalism and learning process is very different from a traditional Gaussian process approach. Hence there is some novelty in this paper.\n\nBack information is well given and pitch at the correct level of sophistication. e.g. Hamiltonian systems, Gaussian process and literature review.\n\n\nWeakness:\nThe main mathematical framework is given in page 5 and 6. I find that these two pages of the paper is extremely difficult to follow. This is the main weakness of this paper. I explain the reasons why these two pages are difficult to follow:\n1. Eq 5. w_m is drawn from a normal distribution. In line 224, it is said that w_m is a learned parameter. Perhaps something is not explained clearly.\n2. Line 195: why p(f|w) is a Dirac's delta function? How do we get Eq 9?\n3. Line 208: justification needs to be given for distribution of x being a delta function Eq. 11\n4. There are undefined math symbols, e.g. Eq 17, b and C\n5. Does Eq 17 contradicts Eq 5?\n6. The physical meanings of equations should be explained better.\n\n\nIt will be much better if the author rewrite the manuscript to explain more clearly and in a higher level of what their method is trying to do. After a good intuition is given to the readers, the authors may dive into details of the mathematics.\n\n\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "3hyDxvzW2bv", "reviewer": "Reviewer_zQ6w", "summary": "Many neural network-based methods for continuous-time physical systems have been proposed recently. These methods are based on ordinary differential equations and consistent with geometric structures. In practice, the available data is noisy and sparse. To deal with such situation, this study proposes to use Gaussian process (GP) consistent with geometric structures. Because tha data is composed only of observations (without derivatives), the GP uses the random Fourier features.\n", "questions": "(1) At line 67, the authors stated that \"whose covariance function incorporates the geometric structure (also called symplectic structure) for the energy conservation or dissipation laws\". The geometric structures appearing in physics are not limited to the symplectic structure, but include contact structure, Poisson structure, Dirac structure, and so on. Moreover, the symplectic structure is related to the energy conservation law, but not to the dissipation law. Please introduce and discuss the geometric structure exactly.\n\n(2) To deal with the case without the true derivatives, this study proposes the symplectic random Fourier features. The experiments demonstrated that the features worked well for pendulum and Duffing oscillator, which exhibit periodic behaviors and are easily captured in the frequency domain. The generality for non-periodic cases such as double pendulums is unclear.\n\nI am curious about the impacts of noise level on the accuracy. The comparison methods, namely HNN and SymODEN, ignore noise. In the noiseless case, is SymODEN enough?\n\n*Minor comments:*\n\nThe experimental setting is a bit unclear.\n(a) The difference between HNN and SymODEN is unclear. HNN learns the Hamiltonian using a single neural network, and SymODEN learns the weight matrix and the potential energy using two networks, obtaining the Hamiltonian. Is this OK?\n(b) At line 298, the authors stated \"Since HNN, D-HNN and SympGPR require derivative observations for training, we used the finite difference instead.\" Wasthe finite difference used as an approximation to the derivative?\n\nThe images are shown in the reverse order of the captions in the leftmost and second leftmost columns in Figure 4.\n\nDue to the inexact discussion about the geometric structures, I cannot accept the present paper as it is. In this regard, I believe that a revision of the text would be sufficient. Then, I will be happy to raise the score.", "limitations": "As discussed in Section I (and potentially shown in Figure 10), the extrapolation performance might be limited. The authors sincerely expressed this limitation. I believe that this limitation will not diminish the value of this study.\n", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "*Strengths*\n\nTo my best knowledge, this is the first study to tackle the modeling of the Hamiltonian system (with dissipation) by using GP. Like the great success of Hamiltonian neural networks (Greydanus et al., NeurIPS, 2019), this study potentially opens up a new research field.\n\nThis study tackled the observation noise, which is a crucial issue in this field, but has been ignored by most existing studies.\n\nThe impact of the sampling rate on the performance, shown in Figure 7, is interesting. The proposed method is superior especially in the case of the sparse sampling.\n\n*Weakness*\n\nThe explanation about geometric structures is inexact. See (1) below.\n\nThe generality of the spectral representation is unclear. See (2) below.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept