Machine learning structure preserving brackets for forecasting irreversible processes
Authors: Kookjin Lee
Nathaniel Trask
Panos Stinis
Venue: NeurIPS 2021 Poster (2021)
Keywords: structure preserving machine learning
neural odes
forecasting
dissipative systems

Abstract: Forecasting of time-series data requires imposition of inductive biases to obtain predictive extrapolation, and recent works have imposed Hamiltonian/Lagrangian form to preserve structure for systems with \emph{reversible} dynamics. In this work we present a novel parameterization of dissipative brackets from metriplectic dynamical systems appropriate for learning \emph{irreversible} dynamics with unknown a priori model form. The process learns generalized Casimirs for energy and entropy guaranteed to be conserved and nondecreasing, respectively. Furthermore, for the case of added thermal noise, we guarantee exact preservation of a fluctuation-dissipation theorem, ensuring thermodynamic consistency. We provide benchmarks for dissipative systems demonstrating learned dynamics are more robust and generalize better than either "black-box" or penalty-based approaches.

Machine learning structure preserving brackets for forecasting irreversible processes
Kookjin Lee School of Computing and Augmented Intelligence Arizona State University Tempe, AZ 85281
Nathaniel Trask
Center for Computing Research
Sandia National Laboratories A Albuquerque, NM 87123
natrask@sandia.
Panos Stinis
Pacific Northwest National Laboratory Richland, WA 99354
Abstract
Forecasting of time-series data requires imposition of inductive biases to obtain predictive extrapolation, and recent works have imposed Hamiltonian/Lagraan form to preserve structure for systems with reversible dynamics. In this work we present novel parameterization of dissipative brackets from metriplectic dynami- cal systems appropriate for learning irreversible dynamics with unknown a priori model form. The process learns generalized Casimirs for energy and entropy guar- anteed to be conserved and nondecreasing, respectively. Furthermore, for the case of added thermal noise, we guarantee exact preservation theorem, ensuring thermodynamic consistency. We provide benchmarks for dis- sipative systems demonstrating learned dynamics are more robust and generalize better than either "black-box" or penalty-based approaches.
coccuuationddisipaatio
1 Background and previous work
Modeling time-series data as a solution to dynamical system with learnable dynamics has been shown be effective in both data-driven modeling for physical systems and traditional machine learning (ML) tasks. Broadly, it has been observed that imposition of physics-based structure leads to more robust architectures which generalize well (T] On one end of the spectrum of inductive biases, universal differential equations (UDE) (2) assume an a priori known model form, thus imposing the strongest bias. On the other, neural ordinary differential equations (NODEs) assume a completely black-box model form with minimal bias.
Many recent approaches have turned structure preserving models of reversible dynamics obtain an inductive bias that lies in between 4 5
6 8. One may use black-box deep neural networks (DNNs) to learn energy of system with unknown model form, while the algebraic structure of Aamittonian/Lagraggiia dynamics provides a flow map which conserves energy. Typically, the learned flow map has symplectic structure so that phase space trajectories are conserved. In classifi- cation problems, this mitigates the vanishing/exploding gradient problem and improves accuracy [9]; in physics, this guarantees that extrapolated states physically realizable [TO].
Such approaches are only appropriate for reversible systems lacking friction or dissipation. In the physics literature, the theory of metriplectic dynamical systems provides generalization of the Poisson brackets of ieailltoninn/aaarangggnn mechanics which model not just conserved energy, but generalized Casimirs such entropy [2] Physical systems which can be cast in this framework
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

obtain a number of mimetic properties related to thermodynamic consistency: satisfaction of the first and second laws of thermodynamies and, for closed stochastic systems, fluctuation dissipation theorem (FDT) that guarantees thermal forcing is balanced exactly by dissipative forces in equilibrium [3]. This FDT property is particularly critical to analyzing rare events in molecular phenomena driven by thermal noise 14].
Classically, metriplectic systems are obtained by deriving a model from first principles and then observing that the system admits a requisite algebraic structure. While effective for a wide range of physical systems [3], the requisite first principles modeling may be restrictively complex for a general system, particularly for multiscale problems involving time history. In this work we reverse the process by assuming our time-series data has been generated by a metriplectic system and then inferring the requisite algebraic objects using a training strategy similar to that used in NODEs. This presents several technical challenges. First, dissipative systems typically have non-observable states (i.e. internal entropy or temperature) which may not be measured. Second, the metriplectic algebraic structure is particularly restrictive, requiring discovery of matrices with carefully designed null-spaces to separate reversible and irreversible components of the dynamics.
Finally, we note that this work is an important first step toward handling more complicated dissipative chaotic systems ubiquitous to science and engineering problems. For example, for chaotic systems the "butterfly effect" causes arbitrarily small perturbations in initial data to exponentially diverge, and it is only possible to provide long- term forecasting by learning a corresponding "strange attractor" whose latent dimension is governed by the dissipative structure 17. In reduced order-modeling, many have looked toward data-driven means of fitting dynamics to latent representations of solution space, with Hamiltonian structure particularly useful for finding long-time accurate surrogates 18, [9]. In this situation as well, structure peeeuurr-preeviingggg treatment of dissipation is critical to account for entropic/memory effects which emerge from coarse- gearrsiggiinnggggg ML is in robotic control (21). Again these models fail to account for friction due to wear, which is inevitable in realistic applications. There are also problems where we have access to time-series and no guaranteed way of modeling from first principles that can account for all the important mechanisms c.g. yystemiidentificttion in biology [22] (23. the study of cascading failures for realistic power grid models (24). social dynamics [25], and accounting for memory effects in empirical eigenfunction expansions of turbulent flows (26). All these cases can benefit by a structure preseevvnn method for identifying dynamics.
Anticipated impact:
20 Another success of reversible
2 Related work
Neural ordinary differential equations As noted, learning time-continuous dynamics in the form of a system of ODEs is an active topic with seminal works including [27 9 28 3
29 There have been many follow-up studies to enhance neural ODEs in different aspects, e.g., enhancing the expressivity
of neural ODEs by augmenting extra dimensions in state variables [30 checkpoint methods to
mitigate numerical instability and to enhance memory efficiency [31
32 33 allowing network
parameters to evolve over time together with hidden states (34] 35 and spectrally approximating
dynamics by using a set of orthogonal polynomials [36]. Applications of NODES for learning
complex physical processes (c.g.. turbulent flow) can be found in [37 38 39
Structure preserving neural networks A thorough accounting of works embedding structure- preservation into neural networks include pioneering works for Hamiltonian neural networks 4 40) followed by development of Lagragian neural networks 41 5 and neural networks that mimic the action of symplectic integrators 66 g More recently, there has been efforts to add physical invariance to learned dynamics models, e.g., time-reversal symmetry [42). Works pursuing related but distinct spatial- compaiibiiiyy related conservation structure other than geometric integration include: graph architectures with associated data-driven graph exterior calculus 43, solving optimization problems with conservation constraint in latent space [44], and adding conservation constraints as a penalty in training loss (5). The closest work to our approach is in [46), which proposed a time integrator that leverages the GENERIC (general equation for the nonequilibrium reversible-irreeersie
coupling) formalism to impose the structure, but enforces the degeneracy condition as penalty terms in the training loss objective, We will provide results demonstrating that a penalty approach is insufficient to guarantee preservation of metriplectic structure.

---

Theory and fundamentals
We consider the GENERIC formalism as particular metriplectic framework amenable to parame-
terization. Consider time series data D {(t, =tt,))) where the state a; = xxt,) ) C R has known initial condition 2o. In GENERIC, it is assumed that observable A(x) evolves under the gradient flow
AA +
(1)
where E and S denote generalized energy and entropy. {-:)) denotes Poisson bracket, and [:]]] denotes irreversible bracket. The Poisson bracket given terms of skew-symmetric Poisson matrix L and the irreversible bracket is given in terms of symmetric positive semi-definite friction matrix M,
dA AB dx
0A
3B M
{A, B)
and A. B
Ox
=
dx
dæ
A system governed by Eq.
1 is GENERIC system if the following degeneracy conditions hold
L, as = O, and aE
0.
(2)
dx
Taking æ in Eq. ① provides the evolution æ
dæ
0E
0S
+
M
dt
dx
dx
Remark 3.1 (Hamiltonian dynamics) For canonical coordinates æ =Iq. pl and canonical Pois- son matrix L
and M = O, Eq. 3 recovers Hamiltonian dynamics.
-1
Remark 3.2 (First and second laws of thermodynamics) Taking A = E and A = S, we obtain at = and ds or 2 O, respectively. This follows easily by application of the degeneracy conditions and noticing {A, A} = 0, [A 0.
Remark 3.3 (Fluctuation dissipation theorem) Introducing thermal noise to Eq. provides the stochastic differential equation (SDE)
dæ: -(1
OE Zx
OS Ox
+ M
KB
M dt / MWW
(4)
where V denotes the Cholesky factor of M, kg Boltzmann constant, and dW, Wiener process. The equilibrium statistics of this SDE reach a stationary distribution under appropriate conditions [13].
Parameterization of bracket structure
We now introduce parameterization of the dissipative and reversible brackets that exactly satisfies the degeneracy conditions described in Section 3 and review the penalty approach from [46] which imposes degeneracy conditions via soft constraints. Our approach is motivated by the work in [47 which we summarize in Sections 4. 4.3 For the remainder, we adopt the Einstein summation convention.
First, we parameterize the energy and the entropy as neural networks, i.e., E(x) x Ex(x) and S(x) (z)),, where and so are weights and biases for the neural networks E and S respectively.
4.1 Parameterizing skew-symmetric reversible dynamics
The reversible dynamics are characterized by a skew-symmetric Poisson bracket,
0A 0B as
{A, B) Eorr Dr Dz Ox,""

---

where Easy is an skew-symmetric 3d tensor. To enforce the anti-symmetry exactly, we consider generic tensor East with learnable entries and apply the following skew-symmetriiztion trick.
1 Sagy 3! (Casy Eagg + E3yy - Esa + SnaB E,,8a)
The reversible part may then be written as {ar, F. = Easy Üs. dzg dxy are given by
Bn aE as and the reversible dynamics
do
0E as
Eagh
dt
Bx dxn
4.2 Parameterizing symmetric irreversible dynamics
Next, parameterize the irreversible dynamics via the bracket,
0A dE 0B 0E [A, B] So.8.pu ax dag dr dxv
where
Sa A D Apn Here, A and D) are skew-symmetric and symmetric positive semi-definite matrices, respectively, such that
A - Bal and Dmn Dum- Again, the skew-symmetry and the symmetric positive semi-definiteness can be achieved by the parameterization tricks
A - AT), and D= DD
where Á and D are matrices with learnable entries. Finally, the irreversible part may be written as as and the irreversible part the dynamics given by
[x. S) = So Uea Bng Dap Bay
dx
DE 0S dE
S. deg Dz, O.y
dt
irr
4.3 Degeneracy conditions
With the above parameterizations the degeneracy conditions described in Eq. 2 may be easily verified by direct calculation following the definition of the brackets and the symmetry/skew-synee symmmmt conditions.
Ox as
dæ. 0S as Easy Org Ox
0S as 0,
{a, S}
EaBy dra Ozg Ox.
Ox
8x
and
dx
DE
Ox 0E 0E 0E Sag Oxg Ox dz,
OE 0E 0E
x
M
Sa Oxa drg dx dx,
0.
dx
dæ
4.4 Alternative parameterization penalty-based method
An alternative strategy to incorporate GENERIC structure is to enforce the degeneracy condition
by soft penalty advocated in [46. In this approach, E, S, L, and M, may be approximated independently of each other. Again, E and S are parameterized as neural networks (F,g and S.), and T. and M are parameterized sieexsyymeeriiaaonsiymmeerzzziii of matrices and p with learnable entries as follows
L T -T-T") and Mp - pp
With this parameterization, penalty terms,
the degeneracy conditions are simply enforced by minimizing two We stress that this penalty will be enforced only to within
L dEg
and
M
as
Bar
dm
optimization eтror.
fe. where θ consists of learnable parameters, then Table 1 summarizes the components comprising fe for black-box NODE, the penalty-bass method, and GENERIC NODE (GNODE).
If we write system of neural ODEs as dr ăt

---

Table 1: Model summary
NODE
Penalty
GNODE
fe
fe = fo
aEe fe = Li dæ
+ M
a5e
fe ar, E} + [x, S]
de
Esp and Sip (MIPs) Lr and Mp (2-tensor)
Eip and Sip (MLPs) e (3-tensor). A and D (2-tensor)
Components
black-box MI.P
=
= {Φ, (,,
= {Φ,, E. D)
Experiments
In this section, we assess the performance the three parameterizations of the ODE dynamics which apply progressively more stringent priors. We implement the algorithms in PYTHON 3.6.5, NUMPY 1.16.2, and PYTORCH 1.7.1 48.. For the time integrator, we use a PYTORCH implementation of differentiable ODE solvers, TorchDiffEq [3. All experiments are performed on MACBOOK PRo with 2.9 GHz i9 CPU and 32 GB memory.
5.1 Dataset and training
The states GENERIC systems may generally be partitioned between "observable" states (e.g.. position and momentum variables) denoted by x" and "non-observable" states (e.g., entropy, config- uration variables, ett) denoted by x"', i.e., æ = [x= _=" We assume that training data is only available for the observable states, with the non-observable states functioning as hidden variables dur- ing training. For each benchmark problem, we take as manufactured training data single trajectory of observable states obtained by integrating reference ODE with known GENERIC structure from known initial condition. We then split the sequence into three segments, [0, ttrain), (ftrain tvai], and (fval, &est] for training, validation, and test such that 0 < train < tval < &
We employ mini-batching to train all three considered architectures. Each mini-batch consists of multi- ple short sequences of length L whose initial conditions are randomly chosen from [0, feraiu]- To train "black-box" box" neural ODEs, we simply use a stochastic gradient descent (SGD) optimizer update the network weights and biases using the mini-batches on the observable states, {x² x + 2 1-11
As opposed the black-box neural ODEs, training the penalty-based approach and the GENERIC approach requires data to impose Mini-batch initial conditions (æt, #e+1. x-+ with x = [ x;]] where {x}} are unavailable. To address this is- sue, we propose a training strategy that alternately updates the model parameters and infers the non-observable states. We start with guess for the non-observable states. We then alternate between (1) updating the model parameters using SGD while fixing the current non-observable states and (2) updating the non-observable states by solving an initial value problem using the most recent model.
non-observable states, i.c.,
Algorithm 1: Neural ODE training
Initialize θ
for (i = 0; i Tmmax; = i do
Sample initial points (26 h where E(k) E [0, lerain L 1] for k 1, I,...,N, No ODESalvet fert..... f1.) for & = 1...., , No
Compute loss: L(x +m)
Update & via SGD
For ODESolve, we use the Dormand Prince method (dopri5) 449 with relative tolerance 10 and absolute tolerance 10-6 The loss function L measures the discrepancy between the ground truth states and approximate states via mean absolute errors, and the network weights and biases updated using Adamax [50] with an initial learning rate 0.01.
In the following, we test the proposed algorithms with two benchmark problems: damped nonlinear oscillator and two gas containers problems. Data for all considered benchmark problems can be found in [51).
5

---

Algorithm 2: Penalty or GENERIC training
Initialize θ and {ar ... Ftriin
Construct a dataset 2; [ x.
T for i O,... Itrain
for (i = 0; i Tmaxx; i +)
Sample initial points (a2(()) 1
1--1 where {(k) E [0, terain - 1] for k 1, No. ODESolve(zza)) (.tii..dt) for 1,...; Nob
Compute loss: ((( +m)
Update via SGD
i mod Pupdate = then "Iya..,"ttrain = ODESolve(zo. Se
Update a dataset as x; [xp 0 T for = 0,..., Israin
10
5.2 Damped nonlinear oscillator
As first benchmark problem, we consider a damped nonlinear oscillator which exhibits a natural GENERIC structure:
P p dp kinnn))) YP dt mT
dS
74
(5)
m
where (q, p) denote the position and momentum of the particle, and S is the entropy of the surrounding thermal bath. The constant parameters m, 7, and T represent the mass of the particle, the damping rate, and the constant temperature of the thermal bath. The total energy of the GENERIC system is
p² 2m
F(g. P. S) = H(, p) + TS
k cos(g) + +TS,
where H(g.p) is the Hamiltonian of the particle (the sum of the kinetic and the potential energy).
In this benchmark problem, the observable states consist of the position and the momentum variables, i.., x [q.p]" _ We consider single non-observable variable, i.e., x = Now, our goal is to learn a system of ODEs that conforms the GENERIC structure described Section 4 and infer the non-observable variable via Algorithm 2
That is, for GNODE, we model Es and S.p to take
x= P. as an input.
For black-box NODEs, we have tested MLPs with combinations of (2, 3, 4) hidden layers with (5, 10, 15} neurons and observed the best result with an MLP with 4 hidden layers with neurons in each layer and hyperbolic tangent (Tanh) activation function. For the penalty-based approach, we have tested MLPs with combinations of {2, 3, 4} hidden layers with (5. 10, 15) and Tanh for parameterizing Eg and S. and observed the best results with MLPs with hidden layers with neurons in each layer. The x 3 learnable entries for La and Mo are considered. We add the penalty terms (see Section 4.4) that are weighted by 10 to the main loss objective. We have also tested 10-2 and 10-6 for weighting, which resulted in significant misfit in data inconsistency in physics (i.e., failure to enforce the degeneracy conditions). Lastly, for the GENERIC approach, we have tested MLPs with combinations of (0, 1, 2) layers with (5, 10) neurons for parameterizing Es and S.p and we have observed the best results with an MLP with I hidden layer with 10 neurons and Tanh for parameterizing Eg,, and a linear layer for parameterizing Spp. Then, we use 3 x 3 x 3 skew-symmetric tensor to parameterize ç, 3 x 3 skew-symmetric tensor to parameterize A, and 3 x tensor, d, to parameterize D, i.e., D ddd For initializing layers in MIPs, we the PYTORCH default uniform distribution and, for initializing learnable entries, we initialize them with unit normal distribution. We initialize the non-observable variable as xb = sę = te (i.c., setting it to be monotonically increasing) in Line 1 of Algorithm 2
The dataset consists of sequence of 180,000 timesteps with fanal 180 (in second) and step size At = 0.001. We then split the dataset into training, validation, and testing sets such that Atrai 20, fyal 40, and frest 180. Each mini-batch consists of N₂ 20 subsequences of length = 120. The maximum training step is set as 7mxx 30000 and the update is performed at every Nupdate 500 training steps (in Algorithm 2.
In the experiment, we consider m = = and y 0.01. The initial condition given as x = = 0, T) where the initial condition for the non-observable variable is arbitrarily set. Results

---

of the comparison are given in Figure [ The results are obtained from 5 independent runs (i.e., 5 different random seeds). The plots of ds penalty formulation leads negative entropy production of large magnitude, while the GNODE approach enforces by construction ds both black-bor NODE and the penalty approach have large standard deviation in the predicted H. GNODE in contrast learns nearby entropy which consistently dissipates the correct amount of energy.
at reveal that the soft enforcement of constraints in the
dt 2 O. When extrapolating well beyond the training time interval,
5.3 Two gas containers
The second benchmark problem considers two (ideal) gas containers, separated by moving wall, exchanging heat and volume. Here, we are interested in the position and the momentum of the separating wall, i.e, x² [l. This problem possesses highly nonlinear expression for the entropy [51]
dq P di m
dp
2
E
E2
di
3
P
2
p
9N dS
1
ds²
9N2
1
1
dt
4E
E
Eg
dt
4E
E
E2
where (q, p) denote the position and momentum of the separating wall and S and S2 are the entropies of the two subsystems. The constants m denotes the mass the wall, 2hg is the total length of the two containers. Following [51], set Nkg 1, which fixes characteristic macroscopic unit of
Trajectory
penalty
GNODE
1.5
12.5
20
7.5
a 0.0
5.0
-20
-1.5
0.0
-2
180
180
7
(second)
(second)
(a) Trajectory
(b) 48
penalty
(c) ds
GNODE
Hlg, p) NODE
Hig. P) penalty
Hiq, GNODE
-1
-1
20 40
100
180
0
20 40
100 Isecondi
180
20 40
100 (second)
180
(seccond)
(d) NODE
(e) H penalty
(1) H GNNOD
Ground truth
Mean
Std Dev
Train
Validation
Figure 1: For the damped nonlinear oscillator, the physical entropy may be evaluated via the formula E H T.S. While all three methods fit training data reasonably well, NODE and the penalty approach rapidly deviate. An inspection of d5 for the penalty method shows that the soft penalty insufficient to ensure compatibility with the second law. GNODE is able to consistently learn an entropy 5 which closely tracks the physical entropy (E (E-H1) H)/T.

---

Trajectory
db
penalty
GNODD
10
150
100
6
4
wwwwmm
-50
0
0.5
q
(second)
(second)
Trajectory
48
penalty
48
GNODE
Hlq. p) NODE
H(q.p) penalty
H(q. p) GNODE
5
10
5
10
30
5
10
30
(ssccond)
(second)
(second)
(d) H NODE
(e) H penalty
(l) GNODE
Ground truth
Approximate
Train
Validation
Figure 2: For the two gas cylinder problem the network must learn a significantly more complex entropy-energy relationship than in the damped oscillator. The results however are similar; GNODE provides remarkable improvement in forecasting due to its faithful reproduction of the second law of thermodynamics (top row, center + right).
entropy, and a 0.5. The internal energies of the two subsystems has the relationship with the associated entropies and volumes via the Sackur-Tetrode equation for ideal gases such that
S, In (E,
1,2,
Nkg where è is constant to ensure the argument of the logarithm dimensionless (set as 102.25). The total energy is given by
p2 2m
Elg. S1. S2) = H(q p) - F1 + F2
+ F1 + E2.
Again, the observable states consist of the position and the momentum variables, i.e., x= [g. PT We consider two non-observable variables, i.e., xx = [>1, 82]] This problem is more challenging as the dynamics of the observable variables strongly depends on the dynamics of the non-obesrvable variables. Now we train GNODE to learn system of ODEs that conforms the GENERIC structure described in Section 4 and infer the non-observed variables via Algorithm Again, for GNODE, model Eg and S.P to take x [g. p, 81, s2] as an input.
For black-box NODEs, we consider an MLP with hidden layers with 5 neurons in each layer and Tanh as nonlinearity. For the penalty- based approach, we consider MLPs with 3 hidden layers with neurons in each layer and Tanh for parameterizing Eg and Sy. and 3 x learnable entries for L and Mp- We add the penalty terms that are weighted by for the GENERIC approach, we consider an MLP with 2 hidden layer with 5 neurons and Tanh for parameterizing Eeg. and hidden layer with neurons and Tanh for parameterizing Sp- Then, we use 3 x 3 3 skew-symmetric tensor to parameterize &, 4 x skew-symmetric tensor to parameterize A.
to the main loss objective. Lastly,

---

q analytic
Q- GNODE
p- -analytic
p
150
150
100
100
100
50
30
4.5 4.570042o-04
-.6827200-04
8.002106e-28
9.011495e-08
1.021312e422
1.040471-022
2:840518o-02
2,942702o-22
(a) q analytic
(b) GNODE
(c) analytic
(d) P GNODE
Figure 3: Distribution of q and p at länal 80 for the SDE The standard deviations of (g.p)
for baseline and GNODE are (9.9684 10-6 6, 1999 10-5) and (1.6436 x 10-6, respectively.
10-6,3.2977 ..3.2972 10 10-",,
and 4 tensor, d, to parameterize D, i... D dd For initializing layers in MLPs, the PYTORCH default uniform distribution and, for initializing learnable entries, we initialize them with unit normal distribution. The non-observable variables initialized 81,6 $2, 4g.
The dataset consists of a sequence of 30,000 timesteps with tinal = 30 seconds and step size At 0.001. We then split the dataset into training, validation, and testing subsets such that train 5, tval = 10, and frest = 30. Each mini-batch consists N₁ = 20 subsequences of length L = 40. The maximum training step set 7max 50000 and the update is performed every Nupdate = 500 training steps (in Algorithm
We m Lg and the initial condition is given as x 2,, Again, the initial condition for the non-observable variables are set arbitrarily. The results are depicted in Figure 2 and demonstrate similar results the damped oscillator. We depict the results from the best performing instances for each model out of five independent runs. Both NODE and the penalty method provide inaccurate forecasting beyond the training set, and the penalty method can be seen to generate negative entropy violating the second law of thermodynamics, while di 0 holds strictly for GNODE.
ds
5.4 Stochastic damped harmonic oscillator
We finally fit a GNODE model the system considered Section 5.2 and use the learned E, S, M and L in the right hand side of the SDE in 4 and compare as a baseline to using instead the analytical E, S. M, and from (5). This amounts to driving both the true system and data-driven dynamics with thermal noise which exactly balances the dissipation, and requires the FDT to hold to realize stationary statistics.
In this experiment, we consider the damped nonlinear oscillator with m = 10, and 0.16 and the same neural network architecture considered in Section [5.2] We use a sequence 80,000 timesteps (final = 80 and At 0.0001) to train the neural network. We use the same training strategy that is used in Section 5.2 (Algorithm 2); the only difference is that we use the mini-batch of size No- 40.
For Figure 3 we obtain statistics from solving 44 with linal = 80 of Maruyaaaa 52 with step size At 0.001. The mean of the resulting SDE solutions show substantial deviation, consistent with the fact that training is performed only on deterministic training data and not the SDE, which is compounded by nonlinearities in the data. However, the standard deviation of the distribution shows good agreement, suggesting that the FDT enforces thermodynamically consistent energy budget between dissipation and stochastic forcing.
6 Conclusions
We have constructed a generalization of structure preserving networks for reversible dynamics to handle dissipative processes. Unlike the reversible case, the bracket structure requires much more careful treatment of degeneracy conditions to ensure compatibility with the first and second laws of thermodynamics. Numerical examples show that our novel parameterization is able to provide non-decreasing entropy that translates improved robustness for out-of-distribution forecasting. We additionally show that exact treatment of dissipative processes allows introduction of thermal forcing

---

which satisfies discrete FDT. To achieve themmodyyamically consistent equilibrium distributions in this setting, we have shown the degeneracy condition must be imposed exactly.
While this work establishes the value of imposing bracket structure for dissipative processes in terms of generalization, robustness, and physical realizability, the training approach applied here is applicable only to relatively small systems, restricting its application e.g. to learning dynamics of reduced-ordes models. Future work will focus on developing more scalable training strategies for learning ODEs of many variables.
7 Acknowledgements
Sandia National Laboratories is multimission laboratory managed and operated by National Technol- ogy and Engineering Solutions of Sandia, LLC, a wholly owned subsidiary of Honeywell International, Inc., for the U.S. Department of Energy's 's National Nuclear Security Administration under contract DE-NA0003530. This paper describes objective technical results and analysis. Any subjective views or opinions that might be expressed in the paper do not necessarily represent the views of the U.S. Department of Energy the United States Government. SAND Number: SAND2021-7461 O.
We thank Chris Eldred for his recommendation to consider Ottingers bracket formulation of GENERIC. The work of N. Trask, and Stinis is supported by the U.S. Department of Energy, Office of Advanced Scientific Computing Research under the Collaboratory on Mathematics and Physics-Informel Learning Machines for Multiscale and Multiphysics Problems (PhILMs) project. N. Trask and K. Lee are supported by the Department of Energy early career program.
References
[1] Nathan Baker, Frank Alexander, Timo Bremer, Aric Hagherg, Yannis Kevrekidis, Habib Najm, Manish Parashar, Abani Patra. James Sethian, Stefan Wild, et Workshop report on basic research needs for scientific machine learning: Core technologies for artificial intelligence. Technical report, USDOE Office of Science (SC), Washington, DC (United States), 2019.
[2] Christopher Rackauckas, Yingbo Ma, Julius Martensen, Collin Warner, Kirill Zubov, Rohit Supekar, Dominic Skinner, Ali Ramadhan, and Alan Edelman. Universal differential equations for scientific machine learning. arXiv preprint 3rXiv:2001.04338 2020.
[3] Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural ordinary differential equations. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pages 5522-6583, 2018.
[4] Samuel Greydanus, Misko Dzamba, and Jason Yosinski. Hamiltonian neural networks. H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché- Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems. volume 32. Curran Associates. Inc., 2019.
[5] Miles Cranmer, Sam Greydanus. Stephan Hoyer, Peter Battaglia, David Spergel, and Shirley Ho. La- grangian neural networks. In ICLR 2020 Workshop on Integration of Deep Neural Models and Differential Equations, 2020.
[6] Zhengdao Chen, Jianyu Zhang, Martin Arjovsky, and Léon Bottou. Symplectic recurrent neural networks. In Intermational Conference Learning Representations, 2019.
Pengzhan Jin, Zhen Zhang, Aiqing Zhu, Yifa Tang, and George Em Kariiadakis. Sympnets: Intrinsic ceucture-preserving symplectic networks identifying hamiltonian systems. Neural Networks, 132:166- 179, 2020.
[8] Yunjin Tong, Shiying Xiong, Xingzhe He, Guanghan Pan, and Bo Zhu. Symplectic neural networks in taylor series form for hamiltonian systems. Journal of Computational Physics, page 110325, 2021.
[9] Eldad Haber and Lars Ruthotto. Stable architectures for deep neural networks. Inverse Problems, 34(1):014004, 2017.
[10] Alvaro Sanchez-Gonzalez Victor Bapst, Kyle Crammer, and Peter Battaglia. Hamiltonian graph networks with ode integrators. arXiv preprint arXiv: /909. 12790, 2019.
[11] Partha Guha. Metriplectic structure, leibniz dynamics and dissipative systems. Journal of Mathematical Analysis and Applications, 326(1):121-13) 2007.
10

---

[12] Miroslav Grmela and Hans Christian Öttinger. Dynamics and thermodynamics of complex fluids. i. development of general formalism. Physical Review E, 56(6):6620, 1997.
[13] Hans Christian Ouinger, Mark A Peletier, and Alberto Montefusco. A framework of nonequilibrium statistical mechanics. i. role and types of fluctuations, Journal of -on-qquilirriunn T'ermodynamiis, ((ahead-of-print, 2020.
[14] Alberto Montefusco, Mark A Peletier, and Hans Christian Öttinger. Coarse- graining via the fluctuation- dissipation theorem and fargedevviation theory. arXiv preprint 18X.vv8809007233 2018.
[15] Hans Christian Öutinger. Generie: Review of successful applications and a challenge for the future. arXiv preprint arXiv: aetii.810..0000 2018.
[16] Edward N Lorenz. Deterministic nonperiodic flow. Journal of atmospheric sciences, 2012):130-44 141, 1963.
[17] Peter Grassherger and Itamar Procaccia. Characterization of strange attractors. Physical review letters, 50(5):346, 1983.
[18] Zhen Gao, Qi Liu, Jan S Hesthaven, Bao-Shan Wang, Wai Sun Don, and Xiao Wen, Non-intrusive reduced order modeling of convection dominated flows using artificial neural networks with application mayleigh-taylor instability. 2021.
[19] Jan Hesthaven and Cecilia Pagliantini. Structure-peeserinng reduced basis methods for hamiltonian systems with nonlinear poisson structure. EPFL Infoxcience, 2018.
[20] Alexandre Chorin and Panagiotis Stinis. Problem reduction, renormalization, and memory. Communica- tions Applied Mathematics and Computational Science, 1(1):1-27, 2007.
[21] Michael Lutter, Christian Ritter, and Jan Peters. Deep lagrangian networks: Using physics as model prior for deep learning. arXiv preprint arXiv:1907004490, 2019.
[22] Jie Xiong and Tong Zhou. A kalman-filte based approach to identification of time-varyyng gene regulatory networks. PloS one, 8(10)=e74571, 2013.
[23] Daniel Han, Nickolay Korabel, Runze Chen, Mark Johnston, Anna Gavrilova, Victoria Allan, Sergei Fedotov, and Thomas A Waigh. Deciphering anomalous heterogeneous intracellular transport with neural networks. Elife, 9cc52224, 2020.
[24] Jacob Roth, David A Barajas-Solan Solano, Panos Stinis, Jonathan Weare, and Mihai Anitescu. A kinetic monte carlo approach for simulating cascading transmission line failure. Multiscale Modeling and Simulation, 19:208 241, 2021.
[25] Martin Short, P Jeffrey Brantingham, Andrea L Bertozzi, and George E Tita. Dissipation and displace- ment of hotspots in reaction-diffusion models of crime. Proceedings of the National Academy of Sciences, 10709)3961-3365, 2010.
[26] Lawrence Sirovich. New perspectives in turbulence. Springer Science & Business Media, 2012.
[27] E. Weinan. A proposal on machine learning via dynamical systems. Communications Mathematics and Statistics, 5(1):1-11, 2017.
[28] Yiping Lu, Aoxiao Zhong, Quanzheng Li, and Bin Dong. Beyond finite layer neural networks: Bridging deep architectures and numerical differential equations. In International Conference on Machine Learning, pages 3276-3285. PMLR, 2018
[29] Yulia Rubanova, Ricky T. Q. Chen, and David K Duvenaud. Latent ordinary differential equations for pregularly-samplee time series. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché Buc, E. Fox, and R. Gareett, editors, Advances in Neural Information Processing Systems, volume 32, Curran Associates, Inc., 2019.
[30] Emilien Dupont, Arnaud Doucet, and Yee Whye Teh. Augmented neural odes. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[31] Amir Gholami, Kurt Keutzer, and George Biros. Anode: Unconditionally accurate memory-efficient gradients for neural odes. arXiv preprint arXiv: 1902. 2019.
[32] Juntang Zhuang, Nicha Dvomek, Xiaoxiao Li, Sekhar Tatikonda, Xenophon Papademetris, and James Duncan. Adaptive checkpoint adjoint method for gradient estimation in neural ode. In International Conference on Machine Learning, pages 11639- -1649, PMLR, 2020.
11

---

[33] Juntang Zhuang. Nicha C Dvorek,, Sekhar Tatikonda, and James S Duncan. Mali: A memory efficient and reverse accurate integrator for neural odes. arXiv preprint arXiv:2102.04668, 2021.
[34] Tianjun Zhang, Zhewei Yao, Amir Gholami, Kurt Keutzer, Joseph Gonzalez, George Biros, and Michael Mahoney. Anodev2: coupled neural ode evolution framework. arXiv preprint arXiv: 190..:99990009996 2019.
[35] Stefano Massaroli, Michael Poli, Jinkyoo Park, Atsushi Yamashita, and Hajime Asma. Dissecting neural odes. In 34th Conference on Neural Information Processing Systems, NeurlPS 2020. The Neural Information Processing Systems, 2020.
[36] Alessio Quaglino. Marco Gallieri. Jonathan Masci, and Jan Koutnik. SNODE: Spectral discretization of neural odes for system identification. arXiv preprint arXiv:1906.070% 2019.
[37] Romit Maulik, Arvind Mohan, Bethany Lusch, Sandeep Madireddy, Prasanna Balaprakash, and Daniel Livescu. Time-series learning of latent-space dynamics for reduced-order model closure. Physica D: Nonlinear Phenomena, 405:132366. 2020.
[38] Gavin D Portwood, Peetak Mitra, Mateus Dias Ribeiro, Minh Nguyen, Balasubramanya T Nadiga, Juan A Saenz, Michael Chertkov, Animesh Garg, Anima Anandkumar, Andreas Dengel, et al. Turbulence forecasting via neural ode. arXiv preprint arXiv: 19t::/911.0555 1,0580, 2019.
[39] Kookjin Lee and Eric Parish. Parameterized neural ordinary differential equations: Applications to computational physics problems. arXiv preprint arXiv: 14685, 2020.
[40] Peter Toth, Danilo Rezende, Andrew Jaegle, Séhastien Racanière, Aleksandar Botev, and Irina Higgins. Hamiltonian generative networks. In International Conference on Learning Representations, 2019.
[41] Michael Lutter, Christian Ritter, and Jan Peters. Deep lagrangian networks: Using physics as model prior for deep learning. In International Conference on Learning Representations. 2018.
[42] In Huh, Funho Yang. Sung Ju Hwang, and Jinwon Shin. Time-reversal symmetric ade network. Advances in Neural Information Processing Systems, 33, 2020.
[43] Nathaniel Trask, Andy Huang, and Xiaozhe Hu. Enforcing exact physics in scientific machine learning: a data-driven exterior calculus on graphs. arXiv preprint arXi::2012. 2020.
[44] Kookjin Lee and Kevin Carlherg. Deep conservation: A latent-dynamics model for exact satisfaction of physical conservation laws. arXiv preprint (rXXv:190900975: 2019.
[45] Tom Beucler, Stephan Rasp, Michael Pritchard, and Pierre Gentine. Achieving conservation of energy in neural network emulators for climate modeling. arXiv preprint urXiv:/906.06m 2019.
[46] Quereus Heránndzz, Alberto Badías, David González, Francisco Chinesta, and Elías Cucto. Structure- preserving neural networks. Journal of Computational Physicx, 426: 00:::0991) 109950, 2021.
[47] Hans Christian Octtinger. Irreversible dynamics, onsager-casimir symmetry,, and an application to turbu- lence. Physical Review F, 1244), 2014.
[48] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang. Zachary Vito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
[49] John R Dormand and Peter Prince. A family of embedded runge-kutta formulac. Journal of computational and applied mathematics, 6(1):19-26, 1980.
[50] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412.6980, 2014.
[51] Xiaocheng Shang and Hans Christian Öttinger. Structure- Structure-reeseriing integrators for dissipative systems based reversible inveersible reversible-ineversible splitting. Proceedings of the Royal Society A, 47602234).20194446, 2020.
[52] Peter E Kloeden and Eckhard Platen. Stochastic diffcrential equations. In Numerical Solution of Stochastic Differential Equations, pages 103-160. Springer, 1992.
12


Reviews:
[{"review_id": "dip0Fb3ZQcb", "reviewer": "Reviewer_D146", "summary": "This paper tries to learn an irreversible dynamic system with dissipation that has entropy growth. The author used a rigorous treatment and parameterization for the dissipation part and separated it from the reversible part. The parameterization naturally satisfies the degeneracy conditions. The author shows that compared to other methods that try to learn degeneracy, like the penalty-based method, this approach preserves the dynamics of damped systems in the long run.", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: Preserve metriplectic structure is pretty novel compared to the Hamiltonian system, and this parametrization using the GENERIC formalism is new.\n\nQuality and Clarity: The theory and math are correct and well written. The experiments are not so appealing and convincing, although some comparisons are provided. The selection of the non-observable state is arbitrary, and no physical interpretation is provided in the paper.\n\nSignificance: The significance is not vital as some pioneering work by neural ODE and penalty-based method for dissipation treatment. This work has a more complex network while showing some noticeable improvement. In the real world, learning entropy growth is not as exciting and valuable as learning energy conservation.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I doubt the generalization of this work as the experiments are relatively simple damped systems. I suspect the treatment in this literature can forecast some more chaotic systems or predicting something like the “butterfly effect.” There have been some potential impacts, but currently, I can only see it works for some well-studied fundamental problems.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2 hours"}, "extra_text": {}}, {"review_id": "XIb49FJzL9L", "reviewer": "Reviewer_DeZu", "summary": "The paper introduces a framework for learning dissipative dynamics efficiently by imposing a physical bias in the network architecture based on the GENERIC formalism. This framework is  commonly seen in the nonequilibrium statistical mechanics literature, where one models the conservative dynamics using a Poisson bracket and the dissipative dynamics using a metric bracket. However, in contrast to what is usually done in physics, where such algebraic structures are observed after deducing the physical laws from first principles, the authors start by assuming such structure in the data and learn dynamics that conform to it. Experiments are done on simple physical systems demonstrating that this approach successfully captures correct physical behaviours (such as the second law of thermodynamics) and has better long term predictions as compared to a neural ODE baseline and another baseline that enforces a weak constraint on the GENERIC structure through a penalty term.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality:\nI believe that the approach presented in this paper, while clearly following the footsteps of recent works on imposing physical constraints in neural networks, has not been done before. In particular, the explicit parameterisations of the conservative and dissipative brackets presented in Sections 4.1 and 4.2 are appealing as it makes it easy to integrate with Neural ODE packages as opposed to other structure-preserving methods based on geometric integrators, and enables a better enforcement of the GENERIC conditions as opposed to the approach using penalty terms [41].\n\nQuality:\nOverall the paper is of good quality that reads nicely and easy to follow, except for a few places that I note below. It would have also been nice to see some experiments using real data to demonstrate the ubiquity of this modelling assumption. The experiments here use only toy problems that is exactly a GENERIC system, so it is somewhat expected that the method works well. The experiment in section 5.4 is also not very clear to me as I explain in more details below.\n\nClarity:\n- Lines 101-102. When stating that the explicit parameterisations of the brackets allows an exact enforcement of the GENERIC conditions, I understand that this holds theoretically, but is this still satisfied under the dopri5 numerical discretisation?\n- Line 160. The following line I think needs clarification: \"As opposed to the black-box neural ODEs, training the penalty-based approach and the GENERIC approach requires data to impose mini-batch initial conditions on non-observable states\". This is something that might be obvious when implementing the algorithm, but I think it is not so clear from what is discussed in the preceding paragraphs.\n- Lines 181-186: The lowercase $s$'s are perhaps meant to be uppecase $S$'s? Same for lines 224-229.\n- Equation below line 216: Please check again. I think one of the $S_1$'s should be $S_2$.\n- Figures 1 and 2: What do the dashed red lines in the top left and centre images indicate? If it is important, it should perhaps be made bolder.\n- Figure 2. The dotted yellow line makes the figure a little hard to see. Perhaps it's better to make it a solid line with some opacity?\n- I find that the experimental setup in section 5.4 is not clearly written. I am confused as to what is being learnt here: is the data now generated by the SDE? Or are you fitting the SDE on the same data as 5.2? If fitting an SDE, details are necessary as this is not straightforward.\n- Line 279. A further discussion about the computational cost (speed, memory, etc) is desired, following the statement \"the training approach applied here is applicable only to relatively small systems, restricting its application\". Is it because of the parameterisations of the symmetric and antisymmetric brackets which contain a four-tensor and a three-tensor respectively and therefore requiring large memory? Would it help to use a GPU instead?\n\nOther suggestions:\n- I suggest replacing the word \"obtain\" in line 31 with \"exhibit\"\n- Line 52. \"with Hamiltonian structure particularly useful\" -> \"with the Hamiltonian structure being particularly useful\"\n- Line 81. \"Consider time series data\" -> \"Consider the time series data\"\n\nSignificance:\nI believe that this work will provide a way to learn more realistic systems from data beyond simple conservative systems, but restricted enough to satisfy thermodynamic laws. Showing experiments on real data would have made the case stronger.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "- As the authors point out, one of the limitations of this approach is scalability to higher dimensions. A further discussion about this is certainly desired. What prevents it from scaling to high dimensions? Also how much additional speed and memory cost does this method add to training a vanilla neural ODE?\n- Discussion of how this model can be applied in real world settings is lacking.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "7dG_1Ue3kS5", "reviewer": "Reviewer_xSwS", "summary": "The paper proposes a new ML model parametrization for imposing irreversibility conditions on a learned physical system. The author then compares their approach to a black-box learner and an ML model where the irreversibility is indirectly added via a regularization term during training.", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I believe the paper has some merits. However, it also faces serval weaknesses that need to be addressed.\n\n**Writing**: From reading the paper, it is very evident that the authors do not come from ML but a physics background (e.g., use of \"penalty\" instead of \"regularization\", algorithm 1 is trivial, etc.). As a result, this makes the paper harder to read for the NeurIPS community. For instance, in section 4.1, the paper starts using $x_\\alpha$, $x_\\beta$, $x_\\gamma$, and $x_\\nu$ without defining them explicitly. Thus it is unclear how the final GNODE model is structured. Moreover, the bracket structures are defined twice, and it is not clear if one is a special case of the other.\n\nThe experiment part has three issues:\n\n**Few experiments**: I don't think one can make general claims from having only two experiment datasets (plus a stochastic version of one). Especially as the datasets are pretty toyish, i.e., the two datasets are relatively low dimensional. It is well established in ML that many models scale poorly with dimension (i.e., the curse of dimensionality). Therefore, different and higher dimensional datasets would enormously benefit the paper's claims.\n\n**Lack of quantitative comparisons**. The gas container results have no mean and standard deviation, i.e., how does the average error look over serval repeated training runs. Connecting to the issue above: More datasets are necessary to support the general claims.\n\n**Missing comparison with alternative approaches**. There is some substantial literature on incorporating physics-based priors to ML models, for instance [1]. Furthermore, the paper investigates extrapolation with ML models. This is a sub-field of ML where it is well known that existing methods yield non-optimal performance, and more task-specific models work much better [2,3]. Therefore, the paper should experimentally evaluate how these models compare to the proposed approach.\n\nIn conclusion, as long as the paper lacks more complex datasets and comparisons with more diverse existing approaches, I firmly believe this paper does not suffice the standard of this conference, and I favor rejection. I urge the authors to either resubmit the paper to a more suited computational physics venue or substantially scale up their experimental evaluation and clarify the writing.\n\n[1] Patel et al. A physics-informed operator regression framework for extracting data-driven continuum models. 2020  \n[2] Sahoo et al. Learning Equations for Extrapolation and Control. ICML 2018  \n[3] Madsen and Johansen. Neural Arithmetic Units. ICLR 2020  ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Missing comprising to alternative extrapolation and physics-informed ML models, combined with low-dimensional datasets, makes the paper fail to show its limitations.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "-efeMjCi81I", "reviewer": "Reviewer_AB4a", "summary": "This paper considers the learning and forecasting of structured dynamics, in particular Poisson mechanical systems (which is a generalization of canonical Hamiltonian systems) with additional dissipation. An existing mathematical tool called the GENERIC framework is employed to represent the latent structured dynamics, and ingredients of this representation are approximated by neural networks. Some numerical experiments are provided.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The idea of this paper is very interesting. The learning and forecasting of conservative/reversible mechanical dynamics has been an emerging but very active research area, and I can see methods for learning and forecasting dissipative mechanical dynamics corresponding to important applications. Nevertheless, the latter has not been sufficiently considered by the current research. This work nicely combines nontrivial existing tools in a novel way and provides a convincing approach. The paper is also well written and well organized.\n\nMy main questions centered around the degeneracy condition (2). Could the authors comment on how restrictive it is? Would it be harder or easier to satisfy in higher dimensions? All the numerical experiments are 1 degree-of-freedom (i.e., 2 dimensional), and more complex examples could make the paper even more convincing.\n\nIn addition, the following could reduce bias in the list of existing \"structure-reserving models of reversible dynamics\" and make it more representative: [Bertalan et al. On learning Hamiltonian systems from data. Chaos 2019], [Zhong et al. Symplectic ODE-net. ICLR 2020], [Chen & Tao. Data-driven Prediction of General Hamiltonian Dynamics via Learning Exactly-Symplectic Maps. ICML 2021]. The 2nd one is related to NODE. The 3rd one gives a strong support of the authors’ statement that symplecticity improves the accuracy of forecast; the currently cited reference [9] for supporting this statement is a great and very important paper, but it is not about forecast. In addition, [21] is not a preprint and it is the same as [36], which appeared in ICLR 201*9*.\n\nI’m willing to change my score after the authors’ responses, which I look forward to.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3.5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper underwent an involved discussion that lead one reviewer to first decrease (6 to 5) and then increase (5 to 7) their score.  
After the discussion, one reviewer is still strongly convinced about rejection, providing as main criticism insufficient empirical evidence. 
Decision: Accept (Poster)