Title: Efficiently Parameterized Neural Metriplectic Systems
Venue: NeurIPS 2024
Accepted: no
Keywords: metriplectic systems

structure preservation

energy conservation

entropy stability

neural ODEs
Abstract: Metriplectic systems are learned from data in a way that scales quadratically in both the size of the state and the rank of the metriplectic data.  Besides being provably energy conserving and entropy stable, the proposed approach comes with approximation results demonstrating its ability to accurately learn metriplectic dynamics from data as well as an error estimate indicating its potential for generalization to unseen timescales when approximation error is low.  Examples are provided which illustrate performance in the presence of both full state information as well as when entropic variables are unknown, confirming that the proposed approach exhibits superior accuracy and scalability without compromising on model expressivity.

Efficiently Parameterized Neural Metriplectic Systems

Abstract

1 Introduction 10

The theory of metriplectic, also called GENERIC, systems [1, 2] provides a principled formalism 11

for encoding dissipative dynamics in terms of complete thermodynamical systems that conserve 12

energy and produce entropy. By formally expressing the reversible and irreversible parts of state 13

evolution with separate algebraic brackets, the metriplectic formalism provides a general mechanism 14

for maintaining essential conservation laws while simultaneously respecting dissipative effects. 15

Thermodynamic completeness implies that any dissipation is caught within a metriplectic system 16

through the generation of entropy, allowing for a holistic treatment which has already found use in 17

modeling fluids [3, 4], plasmas [5, 6], and kinetic theories [7, 8]. 18

From a machine learning point of view, the formal separation of conservative and dissipative effects 19

makes metriplectic systems highly appealing for the development of phenomenological models. Given 20

21

data which is physics-constrained or exhibits some believed properties, a metriplectic system can be

learned to exhibit the same properties with clearly identifiable conservative and dissipative parts. This 22

allows for a more nuanced understanding of the governing dynamics via an evolution equation which 23

reduces to an idealized Hamiltonian system as the dissipation is taken to zero. Moreover, elements 24

in the kernel of the learned conservative part are immediately understood as Casimir invariants, 25

which are special conservation laws inherent to the phase space of solutions, and are often useful 26

for understanding and exerting control on low-dimensional structure in the system. On the other 27

hand, the same benefit of metriplectic structure as a 'direct sum' of reversible and irreversible parts 28

makes it challenging to parameterize in an efficient way, since delicate degeneracy conditions must 29

be enforced in the system for all time. In fact, there are no methods at present for learning general 30

31

metriplectic systems which scale optimally with both dimension and the rank of metriplectic data-an

issue which this work directly addresses. 32

Precisely, metriplectic dynamics on the finite or infinite dimensional phase space P are generated by a 33

free energy function(al) F : P → R , F = E + S defined in terms of a pair E,S : P → R representing 34

energy and entropy, respectively, along with two algebraic brackets {· , ·} , [ · , · ] : C ∞ ( P ) × C ∞ ( P ) → 35

36

C ∞ ( P ) which are bilinear derivations on C ∞ ( P ) with prescribed symmetries and degeneracies

{ S, ·} = [ E, · ] = 0 . Here {· , ·} is an antisymmetric Poisson bracket, which is a Lie algebra 37

realization on functions, and [ · , · ] is a degenerate metric bracket which is symmetric and positive 38

semi-definite. When P ⊂ R n for some n > 0 , these brackets can be identified with symmetric 39 matrix fields L : P → Skew n ( R ) , M : P → Sym n ( R ) satisfying { F, G } = ∇ F · L ∇ G and 40 [ F, G ] = ∇ F · M ∇ G for all functions F, G ∈ C ∞ ( P ) and all states x ∈ P . Using the degeneracy 41 conditions along with ∇ x = I and abusing notation slightly then leads the standard equations for 42 metriplectic dynamics, 43

which are provably energy conserving and entropy producing. To see why this is the case, recall that L ⊺ = -L . It follows that the infinitesimal change in energy satisfies

and therefore energy is conserved along the trajectory of x . Similarly, the fact that M ⊺ = M is positive semi-definite implies that

so that entropy is nondecreasing along x as well. Geometrically, this means that the motion of a trajectory x is everywhere tangent to the level sets of energy and transverse to those of entropy, reflecting the fact that metriplectic dynamics are a combination of noncanonical Hamiltonian ( M = 0 ) and generalized gradient ( L = 0 ) motions. Note that these considerations also imply the Lyapunov stability of metriplectic trajectories, as can be seen by taking E as a Lyapunov function. Importantly, this also implies that metriplectic trajectories which start in the (often compact) set K = { x | E ( x ) ≤ E ( x 0 ) } remain there for all time.

In phenomenological modeling, the entropy S is typically chosen from Casimirs of the Poisson bracket generated by L , i.e. those quantities C ∈ C ∞ ( P ) such that L ∇ C = 0 . On the other hand, the method which will be presented here, termed neural metriplectic systems (NMS), allows for all of the metriplectic data L , M , E, S to be approximated simultaneously, removing the need for Casimir invariants to be known or assumed ahead of time. The only restriction inherent to NMS is that the metriplectic system being approximated is nondegenerate (c.f. Definition 3.1), a mild condition meaning that the gradients of energy and entropy cannot vanish at any point x ∈ P in the phase space. It will be shown that NMS alleviates known issues with previous methods for metriplectic learning, leading to easier training, superior parametric efficiency, and better generalization performance.

Contributions. The proposed NMS method for learning metriplectic models offers the following advantages over previous state-of-the-art: (1) It approximates arbitrary nondegenerate metriplectic dynamics with optimal quadratic scaling in both the problem dimension n and the rank r of the irreversible dynamics. (2) It produces realistic, thermodynamically consistent entropic dynamics from unobserved entropy data. (3) It admits universal approximation and error accumulation results given in Proposition 3.7 and Theorem 3.9. (4) It yields exact energy conservation and entropy stability by construction, allowing for effective generalization to unseen timescales.

2 Previous and Related Work

Previous attempts to learn metriplectic systems from data separate into 'hard' and 'soft' constrained methods. Hard constrained methods enforce metriplectic structure by construction, so that the defining properties of metriplecticity cannot be violated. Conversely, methods with soft constraints relax some aspects of metriplectic structure in order to produce a wider model class which is easier to parameterize. While hard constraints are the only way to truly guarantee appropriate generalization in the learned surrogate, the hope of soft constrained methods is that the resulting model is 'close enough' to metriplectic that it will exhibit some of the favorable characteristics of metriplectic systems, such as energy and entropy stability. Some properties of the methods compared in this work are summarized in Table 1.

Soft constrained methods. Attempts to learn metriplectic systems using soft constraints rely on relaxing the degeneracy conditions L ∇ S = M ∇ E = 0 . This is the approach taken in [9], termed SPNN, which learns an almost-metriplectic model parameterized with generic neural networks through a simple L 2 penalty term in the training loss, L pen = | L ∇ E | 2 + | M ∇ S | 2 . This widens the space of allowable network parameterizations for the approximate operators L , M . While

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

82

83

84

85

86

87

88

89

90

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

the resulting model violates the first and second laws of thermodynamics, the authors show that reasonable trajectories are still obtained, at least when applied within the range of timescales used for training. A similar approach is taken in [10], which targets larger problems and develops an almost-metriplectic model reduction strategy based on the same core idea.

Hard constrained methods. Perhaps the first example of learning metriplectic systems from data was given in [11] in the context of system identification. Here, training data is assumed to come from a finite element simulation, so that the discrete gradients of energy and entropy can be approximated as ∇ E ( x ) = Ax , ∇ S ( x ) = Bx . Assuming a fixed form for L produces a constrained learning problem for the constant matrices M , A , B which is solved to yield a provably metriplectic surrogate model. Similarly, the work [12] learns M , E given L , S by considering a fixed block-wise decoupled form which trivializes the degeneracy conditions, i.e. L = [ ⋆ 0 ; 0 0 ] and M = [ 0 0 ; 0 ⋆ ] . This line of thought is continued in [13] and [14], both of which learn metriplectic systems with neural network parameterizations by assuming this decoupled block structure. A somewhat broader class of metriplectic systems are considered in [15] using tools from exterior calculus, with the goal of learning metriplectic dynamics on graph data. This leads to a structure-preserving network surrogate which scales linearly in the size of the graph domain, but also cannot express arbitrary metriplectic dynamics due to the specific choices of model form for L , M .

A particularly inspirational method for learning general metriplectic systems was given in [16] and termed GNODE, building on parameterizations of metriplectic operators developed in [17]. GNODE parameterizes learnable reversible and irreversible bracket generating matrices L , M in terms of stateindependent tensors ξ ∈ ( R n ) ⊗ 3 and ζ ∈ ( R n ) ⊗ 4 : for 1 ≤ α, β, γ, µ, ν ≤ n , the authors choose L αβ ( x ) = ∑ γ ξ αβγ ∂ γ S and M αβ ( x ) = ∑ µ,ν ζ αµ,βν ∂ µ E∂ ν E, where ∂ α F = ∂F/∂x α , ξ is totally antisymmetric, and ζ is symmetric between the pairs ( α, µ ) and ( β, ν ) but antisymmetric within each of these pairs. The key idea here is to exchange the problem of enforcing degeneracy conditions L ∇ E = M ∇ S = 0 in matrix fields L , M with the problem of enforcing symmetry conditions in tensor fields ξ , ζ , which is comparatively easier but comes at the expense of underdetermining the problem. In GNODE, enforcement of these symmetries is handled by a generic learnable 3-tensor ˜ ξ ∈ ( R n ) ⊗ 3 along with learnable matrices D ∈ Sym r ( R ) and Λ s ∈ Skew n ( R ) for 1 ≤ s ≤ r ≤ n , leading to the final parameterizations ξ αβγ = 1 3! ( ˜ ξ αβγ -˜ ξ αγβ + ˜ ξ βγα -˜ ξ βαγ + ˜ ξ γαβ -˜ ξ γβα ) and ζ αµ,βν = ∑ s,t Λ s αµ D st Λ t βν . Along with learnable energy and entropy functions E,S parameterized by multi-layer perceptrons (MLPs), the data L , M learned by GNODE guarantees metriplectic structure in the surrogate model and leads to successful learning of metriplectic systems in some simple cases of interest. However, note that this is a highly redundant parameterization requiring ( n 3 ) + r ( n 2 ) + ( r +1 2 ) + 2 learnable scalar functions, which exhibits O ( n 3 + rn 2 ) scaling in the problem size because of the necessity to compute and store ( n 3 ) entries of ξ and r ( n 2 ) entries of Λ . Additionally, the assumption of state-independence in the bracket generating tensors ξ , ζ is somewhat restrictive, limiting the class of problems to which GNODE can be applied.

A related approach to learning metriplectic dynamics with hard constraints was seen in [18], which proposed a series of GFINN architectures depending on how much of the information L , M , E, S is assumed to be known. In the case that L , M are known, the GFINN energy and entropy are parameterized with scalar-valued functions f ◦ P ker A where f : R n → R ( E or S ) is learnable and P ker A : R n → R n is orthogonal projection onto the kernel of the (known) operator A ( L or M ). It follows that the gradient ∇ ( f ◦ P ker A ) = P ker A ∇ f ( P ker A ) lies in the kernel of A , so that the degeneracy conditions are guaranteed at the expense of constraining the model class of potential energies/entropies. Alternatively, in the case that all of L , M , E, S are unknown, GFINNs use learnable scalar functions f for E,S parameterized by MLPs as well as two matrix fields Q E , Q S ∈ R r × n with rows given by q f s = ( S f s ∇ f ) ⊺ for learnable skew-symmetric matrices S f s , 1 ≤ s ≤ r , f = E,S . Along with two triangular ( r × r ) matrix fields T L , T M , this yields the parameterizations L ( x ) = Q S ( x ) ⊺ ( T L ( x ) ⊺ -T L ( x )) Q S ( x ) and M ( x ) = Q E ( x ) ⊺ ( T M ( x ) ⊺ T M ( x )) Q E ( x ) , which necessarily satisfy the symmetries and degeneracy conditions required for metriplectic structure. GFINNs are shown to both increase expressivity over the GNODE method as well as decrease redundancy, since the need for an explicit order-3 tensor field is removed and the reversible and irreversible brackets are allowed to depend explicitly on the state x . However, GFINNs still exhibit cubic scaling through the requirement of rn ( n -1) + r 2 +2 = O ( rn 2 ) learnable functions, which is well above the theoretical minimum required to express a general metriplectic system and leads to difficulties in training the resulting models.

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

Model reduction. Finally, it is worth mentioning the closely related line of work involving model reduction for metriplectic systems, which began with [19]. As remarked there, preserving metriplecticity in reduced-order models (ROMs) exhibits many challenges due to its delicate requirements on the kernels of the involved operators. There are also hard and soft constrained approaches: the already mentioned [10] aims to learn a close-to-metriplectic data-driven ROM by enforcing degeneracies by penalty, while [20] directly enforces metriplectic structure in projection-based ROMs using exterior algebraic factorizations. The parameterizations of metriplectic data presented here are related to those presented in [20], although NMS does not require access to nonzero components of ∇ E, ∇ S .

3 Formulation and Analysis

The proposed formulation of the metriplectic bracket-generating operators L , M used by NMS is based on the idea of exploiting structure in the tensor fields ξ , ζ to reduce the necessary number of degrees of freedom. In particular, it will be shown that the degeneracy conditions L ∇ S = M ∇ E = 0 imply more than just symmetry constraints on these fields, and that taking these additional constraints into account allows for a more compact representation of metriplectic data. Following this, results are presented which show that the proposed formulation is universally approximating on nondegenerate systems (c.f. Definition 3.1) and admits a generalization error bound in time.

3.1 Exterior algebra

Developing these metriplectic expressions will require some basic facts from exterior algebra, of which more details can be found in, e.g., [21, Chapter 19]. The basic objects in the exterior algebra ∧ V over the vector space V are multivectors, which are formal linear combinations of totally antisymmetric tensors on V . More precisely, if I ( V ) denotes the two-sided ideal of the free tensor algebra T ( V ) generated by elements of the form v ⊗ v ( v ∈ V ), then the exterior algebra is the quotient space ∧ V ≃ T ( V ) /I ( V ) equipped with the antisymmetric wedge product operation ∧ . This graded algebra is equipped with natural projection operators P k : ∧ V → ∧ k V which map between the full exterior algebra and the k th exterior power of V , denoted ∧ k V , whose elements are homogeneous k -vectors. More generally, given an n -manifold M with tangent bundle TM , the exterior algebra ∧ ( TM ) is the algebra of multivector fields whose fiber over x ∈ M is given by ∧ T x M .

For the present purposes, it will be useful to develop a correspondence between bivectors B ∈ ∧ 2 ( R n ) and skew-symmetric matrices B ∈ Skew n ( R ) , which follows directly from Riesz representation in terms of the usual Euclidean dot product. More precisely, supposing that e 1 , ..., e n are the standard basis vectors for R n , any bivector B ∈ ∧ 2 T R n can be represented as B = ∑ i<j B ij e i ∧ e j where B ij ∈ R denote the components of B . Define a grade-lowering action of bivectors on vectors through right contraction (see e.g. Section 3.4 of [22]), expressed for any vector v and basis bivector e i ∧ e j as ( e i ∧ e j ) · v = ( e j · v ) e i -( e i · v ) e j . It follows that the action of B is equivalent to

where B ⊺ = -B ∈ R n × n is a skew-symmetric matrix representing B , and we have re-indexed under the second sum and applied that B ij = -B ji for all i, j . Since the kernel of this action is the zero bivector, it is straightforward to check that this string of equalities defines an isomorphism M : ∧ 2 R n → Skew n ( R ) from the 2 nd exterior power of R n to the space of skew-symmetric ( n × n ) -matrices over R : in what follows, we will write B ≃ B rather than B = M ( B ) for notational convenience. Note that a correspondence in the more general case of bivector/matrix fields follows in the usual way via the fiber-wise extension of M .

3.2 Learnable metriplectic operators

It is now possible to explain the proposed NMS formulation. First, note the following key definition which prevents the consideration of unphysical examples.

̸

Definition 3.1. A metriplectic system on K ⊂ R n generated by the data L , M , E, S will be called nondegenerate provided ∇ E, ∇ S = 0 for all x ∈ K .

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

With this, the NMS parameterizations for metriplectic operators are predicated on an algebraic result proven in Appendix A.

̸

Lemma 3.2. Let K ⊂ R n . For all x ∈ K , the operator L : K → R n × n satisfies L ⊺ = -L and L ∇ S = 0 for some S : K → R , ∇ S = 0 , provided there exists a non-unique bivector field A : U → ∧ 2 R n and equivalent matrix field A ≃ A such that

̸

Similarly, for all x ∈ K a positive semi-definite operator M : K → R n × n satisfies M ⊺ = M and M ∇ E = 0 for some E : K → R , ∇ E = 0 , provided there exists a non-unique matrix-valued B : K → R n × r and symmetric matrix-valued D : K → R r × r such that r ≤ n and

where b s denotes the s th column of B . Moreover, using P ⊥ f = ( I -∇ f ∇ f ⊺ |∇ f | 2 ) to denote the orthogonal projector onto Span( ∇ f ) ⊥ , these parameterizations of L , M are equivalent to the matricized expressions L = P ⊥ S AP ⊥ S and M = P ⊥ E BDB ⊺ P ⊥ E .

Remark 3.3 . Observe that the projections appearing in these expressions are the minimum necessary for guaranteeing the symmetries and degeneracy conditions necessary for metriplectic structure. In particular, conjugation by P ⊥ f respects symmetry and ensures that both the input and output to the conjugated matrix field lie in Span( ∇ f ) ⊥ .

Lemma 3.2 demonstrates specific parameterizations for L , M that hold for any nondegenerate metriplectic data and are core to the NMS method for learning metriplectic dynamics. While generally underdetermined, these expressions are in a sense maximally specific given no additional information, since there is nothing available in the general metriplectic formalism to determine the matrix fields A , BDB ⊺ on Span( ∇ S ) , Span( ∇ E ) , respectively. The following Theorem, also proven in Appendix A, provides a rigorous correspondence between metriplectic systems and these particular parameterizations.

Theorem 3.4. The data L , M , E, S form a nondegenerate metriplectic system in the state variable x ∈ K if and only if there exist a skew-symmetric A : K → Skew n ( R ) , symmetric postive semidefinite D : K → Sym r ( R ) , and generic B : K → R n × r such that

Remark 3.5 . Note that the proposed parameterizations for L , M are not one-to-one but properly contain the set of valid nondegenerate metriplectic systems in E,S , since the Jacobi identity on L necessary for a true Poisson manifold structure is not strictly enforced. For 1 ≤ i, j, k ≤ n , the Jacobi identity is given in components as ∑ ℓ L iℓ ∂ ℓ L jk + L jℓ ∂ ℓ L ki + L kℓ ∂ ℓ L ij = 0 . However, this requirement is not often enforced in algorithms for learning general metriplectic (or even symplectic) systems, since it is considered subordinate to energy conservation and it is well known that both qualities cannot hold simultaneously in general [23].

3.3 Specific parameterizations

Now that Theorem 3.4 has provided a model class which is rich enough to express any desired metriplectic system, it remains to discuss what NMS actually learns. First, note that it is unlikely to be the case that E,S are known a priori , so it is beneficial to allow these functions to be learnable alongside the governing operators L , M . For simplicity, energy and entropy E,S are parameterized as scalar-valued MLPs with tanh activation, although any desired architecture could be chosen for this task. The skew-symmetric matrix field A = A tri -A ⊺ tri used to build L is parameterized through its strictly lower-triangular part A tri using a vector-valued MLP with output dimension ( n 2 ) ,

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

which guarantees that the mapping A tri ↦→ A above is bijective. Similarly, the symmetric matrix field D = K chol K ⊺ chol is parameterized through its lower-triangular Cholesky factor K chol , which is a vector-valued MLP with output dimension ( r +1 2 ) . While this choice does not yield a bijective mapping K chol ↦→ D unless, e.g., D is assumed to be positive definite with diagonal entries of fixed sign, this does not hinder the method in practice. In fact, D should not be positive definite in general, as this is equivalent to claiming that M is positive definite on vectors tangent to the level sets of E . Finally, the generic matrix field B is parameterized as a vector-valued MLP with output dimension nr . Remarkably, the exterior algebraic expressions in Lemma 3.2 require less redundant operations than the corresponding matricized expressions from Theorem 3.4, and therefore the expressions from Lemma 3.2 are used when implementing NMS. Figure 1 summarizes this information.

Remark 3.6 . With these choices, the NMS parameterization of metriplectic systems requires only (1 / 2) ( ( n + r ) 2 -( n -r ) ) +2 learnable scalar functions, in contrast to ( n 3 ) + r ( n 2 ) + ( r +1 2 ) +2 for the GNODE approach in [16] and rn ( n -1) + r 2 +2 for the GFINN approach in [18]. In particular, NMS is quadratic in both n, r with no decrease in model expressivity, in contrast to the cubic scaling of previous methods.

3.4 Approximation and error

Besides offering a compact parameterization of metriplectic dynamics, the expressions used in NMS also exhibit desirable approximation properties which guarantee a reasonable bound on state error over time. To state this precisely, first note the following universal approximation result proven in Appendix A.

̸

Proposition 3.7. Let K ⊂ R n be compact and E,S : K → R be continuous such that L ∇ S = M ∇ E = 0 and ∇ E, ∇ S = 0 for all x ∈ K . For any ε > 0 , there exist two-layer neural network functions ˜ E, ˜ S : K → R , ˜ L : K → Skew n ( R ) and ˜ M : K → Sym n ( R ) such that ∇ ˜ E, ∇ ˜ S = 0 on K , ˜ M is positive semi-definite, ˜ L ∇ ˜ S = ˜ M ∇ ˜ E = 0 for all x ∈ K , and each approximate function is ε -close to its given counterpart on K . Moreover, if L , M have k ≥ 0 continuous derivatives on K then so do ˜ L , ˜ M .

̸

Remark 3.8 . The assumption x ∈ K of the state remaining in a compact set V is not restrictive when at least one of E, -S : R n → R , say E , has bounded sublevel sets. In this case, letting x 0 = x (0) it follows from ˙ E ≤ 0 that E ( x ( t )) = E ( x 0 ) + ∫ t 0 ˙ E ( x ( τ )) dτ ≤ E ( x 0 ) , so that the entire trajectory x ( t ) lies in the (closed and bounded) compact set K = { x | E ( x ) ≤ E ( x 0 ) } ⊂ R n .

Leaning on Proposition 3.7 and classical universal approximation results in [24], it is further possible to establish the following error estimate also proven in Appendix A which gives an idea of the error accumulation rate that can be expected from this method.

Theorem 3.9. Suppose L , M , E, S are nondegenerate metriplectic data such that L , M have at least one continuous derivative, E,S have Lipschitz continuous gradients, and at least one of E, -S have bounded sublevel sets. For any ε > 0 , there exist nondegenerate metriplectic data ˜ L , ˜ M , ˜ E, ˜ S defined by two-layer neural networks such that, for all T > 0 ,

Input/Output

Learnable

Calculated

Integrate

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

where a, b ∈ R are constants depending on both sets of metriplectic data and ˙ ˜ x = ˜ L (˜ x ) ∇ ˜ E (˜ x ) + ˜ M (˜ x ) ∇ ˜ S (˜ x ) .

Remark 3.10 . Theorem 3.9 provides a bound on state error over time under the assumption that the approximation error in the metriplectic networks can be controlled. On the other hand, notice that Theorem 3.9 can also be understood as a generic error bound on any trained metriplectic networks

˜

L

,

˜

M

,

˜

E,

˜

S

provided universal approximation results are not invoked in the estimation leading to

εb

.

This result confirms that the error in the state x for a fixed final time T tends to zero with the approximation error in the networks ˜ L , ˜ M , ˜ E, ˜ S , as one would hope based on the approximation capabilities of neural networks. More importantly, Theorem 3.9 also bounds the generalization error of any trained metriplectic network for an appropriate (and possibly large) ε equal to the maximum approximation error on K , where the learned metriplectic trajectories are confined for all time. With this theoretical guidance, the remaining goal of this work is to demonstrate that NMS is also practically effective at learning metriplectic systems from data and exhibits reasonable generalization to unseen timescales.

4 Algorithm

Similar to previous approaches in [16] and [18], the learnable parameters in NMS are calibrated using data along solution trajectories to a given dynamical system. This brings up an important question regarding how much information about the system in question is realistically present in the training data. While many systems have a known metriplectic form, it is not always the case that one will know metriplectic governing equations for a given set of training data. Therefore, two approaches are considered in the experiments below corresponding to whether full or partial state information is assumed available during NMS training. More precisely, the state x = ( x o , x u ) will be partitioned into 'observable' and 'unobservable' variables, where x u may be empty in the case that full state information is available. In a partially observable system x o typically contains positions and momenta while x u contains entropy or other configuration variables which are more difficult to observe during physical experiments. In both cases, NMS will learn a metriplectic system in x according to the procedure described in Algorithm 1.

Algorithm 1 Training neural metriplectic systems

Note that the batch-wise training strategy in Algorithm 1 requires initial conditions for x u in the partially observed case. There are several options for this, and two specific strategies will be considered here. Suppose the data are drawn from the training interval [0 , T ] with initial state x 0 and final state x T . The first strategy sets x u 0 = 0 , x u T = 1 (where 1 is the all ones vector), and x u s = 1 /T , 0 < s < T , so that the unobserved states are initially assumed to lie on a straight line. The second strategy is more sophisticated, and involves training a diffusion model to predict the distribution of x u given x o . Specific details of this procedure are given in Appendix E.

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

336

337

338

339

340

341

5 Examples

The goal of the following experiments is to show that NMS is effective even when entropic information cannot be observed during training, yielding superior performance when compared to previous methods including GNODE, GFINN, and SPNN discussed in Section 2. The metrics considered for this purpose will be mean absolute error (MAE) and mean squared error (MSE) defined in the usual way as the average ℓ 1 (resp. squared ℓ 2 ) error between the discrete states x , ˜ x ∈ R n × n s . For brevity, many implementation details have been omitted here and can be found in Appendix B. An additional experiment showing the effectiveness of NMS in the presence of both full and partial state information can be found in Appendix C.

Remark 5.1 . To facilitate a more equal parameter count between the compared metriplectic methods, the results of the experiments below were generated using the alternative parameterization D = KK ⊺ where K : K → R r × r ′ is full and r ′ ≥ r . Of course, this change does not affect metriplecticity since D is still positive semi-definite for each x ∈ K .

5.1 Two gas containers

The first test of NMS involves two ideal gas containers separated by movable wall which is removed at time t 0 , allowing for the exchange of heat and volume. In this example, the motion of the state x = ( q p S 1 S 2 ) ⊺ is governed by the metriplectic equations:

where ( q, p ) are the position and momentum of the separating wall, S 1 , S 2 are the entropies of the two subsystems, and the internal energies E 1 , E 2 are determined from the Sackur-Tetrode equation for ideal gases, S i /Nk B = ln ( ˆ cV i E 3 / 2 i ) , 1 ≤ i ≤ 2 . Here, m denotes the mass of the wall, 2 L is th the total length of the system, and V i is the volume of the i container. As in [16, 25] Nk B = 1 and α = 0 . 5 fix the characteristic macroscopic unit of entropy while ˆ c = 102 . 25 ensures the argument of the logarithm defining E i is dimensionless. This leads to the total entropy S ( x ) = S 1 + S 2 and the total energy E ( x ) = (1 / 2 m ) p 2 + E 1 ( x ) + E 2 ( x ) , which are guaranteed to be nondecreasing and constant, respectively.

The primary goal here is to verify that NMS can accurately and stably predict gas container dynamics without the need to observe the entropic variables S 1 , S 2 . To that end, NMS has been compared to GNODE, SPNN, and GFINN on the task of predicting the trajectories of this metriplectic system over time, with results displayed in Table 2. More precisely, given an intial condition x 0 and an interval 0 < t train < t valid < t test , each method is trained on partial state information (in the case of NMS) or full state information (in the case of the others) from the interval [0 , t train ] and validated on ( t train , t valid ] before state errors in q, p only are calculated on the whole interval [0 , t test ] . As can be seen from Table 2 and Figure 2, NMS is remarkably accurate over unseen timescales even in this unfair comparison, avoiding the unphysical behavior which often hinders soft-constrained methods like SPNN. The energy and instantaneous entropy plots in Figure 2 further confirm that the strong enforcement of metriplectic structure guaranteed by NMS leads to correct energetic and entropic dynamics for all time.

5.2 Thermoelastic double pendulum

Next, consider the thermoelastic double pendulum from [26] with 10-dimensional state variable x = ( q 1 q 2 p 1 p 2 S 1 S 2 ) ⊺ , which represents a highly challenging benchmark for metriplectic methods. The equations of motion in this case are given for 1 ≤ i ≤ 2 as

where κ > 0 is a thermal conductivity constant (set to 1), m i is the mass of the i th spring (also set to 342 1) and T i = ∂ S i E i is its absolute temperature. In this case, q i , p i ∈ R 2 represent the position and 343

344

345

346

347

348

349

350

351

352

353

354

355

(a) Hamiltonian

H

=

p

2

2

m

(b) Position

q

(c) Momentum

p

(d) Energy

E

=

H

+

∑

E

i

˙

S

SPNN

GNODE

GFINN

NMS

2.0

1.2

1.5

AWW

1.0

1.0

0.5

0.8

0.0

0.6

0

20

40

60

80

100

10

15

20

25

30

35

40

5

0.015

0.5

0

MAA

0.010

0.0

FA

Q

0.005

0.5

-5

0.000

0.02.5

10

15

20

25

30

35

40

0

20

40

60

80

100

0

20

40

60

80

100

i

momentum of the i th mass, while S i represents the entropy of the i th pendulum. As before, the total entropy S ( x ) = S 1 + S 2 is the sum of the entropies of the two springs, while defining the internal energies E i ( x ) = (1 / 2)(ln λ i ) 2 +ln λ i + e S i -ln λ i -1 , λ 1 = | q i | , λ 2 = | q 2 -q 1 | , leads to the total energy E ( x ) = (1 / 2 m 1 ) | p 1 | 2 +(1 / 2 m 2 ) | p 2 | 2 + E 1 ( x ) + E 2 ( x ) .

The task in this case is prediction across initial conditions. As in [18], 100 trajectories are drawn from the ranges in Appendix B and integrated over the interval [0 , 40] with ∆ t = 0 . 1 , with an 80/10/10 split for training/validation/testing. Here all compared models are trained using full state information. As seen in Table 2, NMS is again the most performant, although all models struggle to approximate the dynamics over the entire training interval. It is also notable that the training time of NMS is greatly decreased relative to GNODE and GFINN due to its improved quadratic scaling; a representative study to this effect is given in Appendix D.

6 Conclusion

Neural metriplectic systems (NMS) have been considered for learning finite-dimensional metriplectic 356 dynamics from data. Making use of novel non-redundant parameterizations for metriplectic operators, 357 NMS provably approximates arbitrary nondegenerate metriplectic systems with generalization error 358 bounded in terms of the operator approximation quality. Benchmark examples have shown that 359 NMS is both more scalable and more accurate than previous methods, including when only partial 360 state information is observed. Future work will consider extensions of NMS to infinite-dimensional 361 metriplectic systems with the aim of addressing its main limitation: the difficulty of scaling NMS 362 (among all present methods for metriplectic learning) to realistic, 3-D problems of the size that would 363 be considered in practice. A promising direction is to consider the use of NMS in model reduction, 364 where sparse, large-scale systems are converted to small, dense systems through a clever choice of 365 encoding/decoding. 366

367

368

369

370

371

372

373

374

375

376

377

References

Philip J. Morrison. A paradigm for joined hamiltonian and dissipative systems. Physica D: Nonlinear Phenomena , 18(1):410-419, 1986.

Miroslav Grmela and Hans Christian Öttinger. Dynamics and thermodynamics of complex fluids. i. development of a general formalism. Phys. Rev. E , 56:6620-6632, Dec 1997.

P. J. Morrison. Some observations regarding brackets and dissipation. Technical Report PAM-228, Center for Pure and Applied Mathematics, University of California, Berkeley, 1984.

PJ Morrison. Thoughts on brackets and dissipation: old and new. In Journal of Physics: Conference Series , volume 169, page 012006. IOP Publishing, 2009.

Allan N. Kaufman and Philip J. Morrison. Algebraic structure of the plasma quasilinear equations. Physics Letters A , 88(8):405-406, 1982.

Emmanuele Materassi, M.; Tassi. Metriplectic framework for dissipative magneto-hydrodynamics. Physica 378 D: Nonlinear Phenomena , 2012. 379

Allan N. Kaufman. Dissipative hamiltonian systems: A unifying principle. Physics Letters A , 100(8):419380 422, 1984. 381

Darryl D Holm, Vakhtang Putkaradze, and Cesare Tronci. Kinetic models of oriented self-assembly. 382 Journal of Physics A: Mathematical and Theoretical , 41(34):344010, aug 2008. 383

Quercus Hernández, Alberto Badías, David González, Francisco Chinesta, and Elías Cueto. Structure384 preserving neural networks. Journal of Computational Physics , 426:109950, 2021. 385

Quercus Hernández, Alberto Badías, David González, Francisco Chinesta, and Elías Cueto. Deep learning 386 of thermodynamics-aware reduced-order models from data. Computer Methods in Applied Mechanics and 387 Engineering , 379:113763, 2021. 388

389

390

David González, Francisco Chinesta, and Elías Cueto. Thermodynamically consistent data-driven computational mechanics. Continuum Mechanics and Thermodynamics , 31(1):239-253, 2019.

391

392

393

394

395

396

397

398

399

400

401

402

403

404

405

406

407

408

409

410

411

412

413

414

D. Ruiz, D. Portillo, and I. Romero. A data-driven method for dissipative thermomechanics. IFACPapersOnLine , 54(19):315-320, 2021.

Baige Xu, Yuhan Chen, Takashi Matsubara, and Takaharu Yaguchi. Learning generic systems using neural symplectic forms. In International Symposium on Nonlinear Theory and Its Applications , number A2L-D03 in IEICE Proceeding Series, pages 29-32. The Institute of Electronics, Information, and Communication Engineers (IEICE), 2022.

Baige Xu, Yuhan Chen, Takashi Matsubara, and Takaharu Yaguchi. Equivalence class learning for GENERIC systems. In ICML Workshop on New Frontiers in Learning, Control, and Dynamical Systems , 2023.

Anthony Gruber, Kookjin Lee, and Nathaniel Trask. Reversible and irreversible bracket-based dynamics for deep graph neural networks. In Thirty-seventh Conference on Neural Information Processing Systems , 2023.

Kookjin Lee, Nathaniel Trask, and Panos Stinis. Machine learning structure preserving brackets for forecasting irreversible processes. Advances in Neural Information Processing Systems , 34:5696-5707, 2021.

Hans Christian Öttinger. Irreversible dynamics, onsager-casimir symmetry, and an application to turbulence. Phys. Rev. E , 90:042121, Oct 2014.

Zhen Zhang, Yeonjong Shin, and George Em Karniadakis. Gfinns: Generic formalism informed neural networks for deterministic and stochastic dynamical systems. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences , 380(2229):20210207, 2022.

Hans Christian Öttinger. Preservation of thermodynamic structure in model reduction. Phys. Rev. E , 91:032147, Mar 2015.

Anthony Gruber, Max Gunzburger, Lili Ju, and Zhu Wang. Energetically consistent model reduction for metriplectic systems. Computer Methods in Applied Mechanics and Engineering , 404:115709, 2023.

Loring W. Tu. Differential Geometry: Connections, Curvature, and Characteristic Classes . Springer 415 International Publishing, 2017. 416

417

418

Leo Dorst, Daniel Fontijne, and Stephen Mann. Geometric Algebra for Computer Science: An Objectoriented Approach to Geometry . Morgan Kaufmann, Amsterdam, 2007.

419

420

421

422

423

424

425

426

427

428

429

430

431

432

433

434

435

Ge Zhong and Jerrold E. Marsden. Lie-poisson hamilton-jacobi theory and lie-poisson integrators. Physics Letters A , 133(3):134-139, 1988.

Xin Li. Simultaneous approximations of multivariate functions and their derivatives by neural networks with one hidden layer. Neurocomputing , 12(4):327-343, 1996.

Kookjin Lee, Nathaniel Trask, and Panos Stinis. Structure-preserving sparse identification of nonlinear dynamics for data-driven modeling. In Mathematical and Scientific Machine Learning , pages 65-80. PMLR, 2022.

Ignacio Romero. Thermodynamically consistent time-stepping algorithms for non-linear thermomechanical systems. International Journal for Numerical Methods in Engineering , 79(6):706-732, 2023/05/14 2009.

John R Dormand and Peter J Prince. A family of embedded runge-kutta formulae. Journal of computational and applied mathematics , 6(1):19-26, 1980.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

Xiaocheng Shang and Hans Christian Öttinger. Structure-preserving integrators for dissipative systems based on reversible-irreversible splitting. Proceedings of the Royal Society A , 476(2234):20190446, 2020.

Haksoo Lim, Minjung Kim, Sewon Park, and Noseong Park. Regular time-series generation using sgm. arXiv preprint arXiv:2301.08518 , 2023.

Pascal Vincent. A connection between score matching and denoising autoencoders. Neural Computation , 436 23(7):1661-1674, 2011. 437

Simo Särkkä and Arno Solin, editors. Applied stochastic differential equations , volume 10. Cambridge 438 University Press, 2019. 439

Yang Song, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. 440 Score-based generative modeling through stochastic differential equations. CoRR , abs/2011.13456, 2020. 441

442

443

444

445

446

447

448

449

450

451

452

it follows that

since ∇ S · A ∇ S = -∇ S · A ∇ S = 0 . Moving to the case of M , notice that M = D st v s ⊗ v t for a particular choice of v , meaning that

since D is a symmetric matrix. Additionally, it is straightforward to check that, for any 1 ≤ s ≤ r , 453

So, it follows immediately that 454

Now, observe that 455

456

457

458

459

460

461

462

463

464

465

and therefore M is expressible as

With Lemma 3.2 established, the proof of Theorem 3.4 is straightforward.

Proof of Theorem 3.4. The 'if' direction follows immediately from Lemma 3.2. Now, suppose that L and M define a metriplectic system, meaning that the mentioned symmetries and degeneracy conditions hold. Then, it follows from L ∇ S = 0 that the projection P ⊥ S LP ⊥ S = L leaves L invariant, so that choosing A = L yields P ⊥ S AP ⊥ S = L . Similarly, from positive semi-definiteness and M ∇ E = 0 it follows that M = U Λ U ⊺ = P ⊥ E U Λ U ⊺ P ⊥ E for some column-orthonormal U ∈ R N × r and positive diagonal Λ ∈ R r × r . Therefore, choosing B = U and D = Λ yields M = P ⊥ E BDB ⊺ P ⊥ , as desired.

since A ⊺ = -A and hence v ⊺ Av = 0 for all v ∈ R n . Similarly, it follows that for every 1 ≤ s ≤ r ,

A Proof of Theoretical Results

This Appendix provides proof of the analytical results in Section 3 of the body. First, the parameterizations of L , M in terms of exterior algebra are established.

Proof of Lemma 3.2. First, it is necessary to check that the operators L , M parameterized this way satisfy the symmetries and degeneracy conditions claimed in the statement. To that end, recall that a ∧ b ≃ ab ⊺ -ba ⊺ , meaning that ( ab ⊺ -ba ⊺ ) ⊺ ≃ b ∧ a = -a ∧ b . It follows that A ⊺ ≃ ˜ A = -A where ˜ A denotes the reversion of A , i.e., ˜ A = ∑ i<j A ij e j ∧ e i . Therefore, we may write

showing that L ⊺ = -L . Moreover, using that

466

467

468

469

470

471

472

473

474

475

Looking toward the proof of Proposition 3.7, we also need to establish the following Lemmata which give control over the orthogonal projectors P ⊥ ˜ E , P ⊥ ˜ S . First, we recall how control over the L ∞ norm |·| ∞ of a matrix field gives control over its spectral norm |·| .

Lemma A.1. Let A : K → R n × n be a matrix field defined on the compact set K ⊂ R n with m continuous derivatives. Then, for any ε > 0 there exists a two-layer neural network ˜ A : K → R n × n such that sup x ∈ K ∣ ∣ ∣ A -˜ A ∣ ∣ ∣ < ε and sup x ∈ K ∣ ∣ ∣ ∇ k A -∇ k ˜ A ∣ ∣ ∣ ∞ < ε for 1 ≤ k ≤ m where ∇ k is the (total) derivative operator of order k .

Proof. This will be a direct consequence of Corollary 2.2 in [24] provided we show that | A | ≤ c | A | ∞ for some c > 0 . To that end, if σ 1 ≥ ... ≥ σ r > 0 ( r ≤ n ) denote the nonzero singular values of A -˜ A , it follows that for each x ∈ K ,

On the other hand, it also follows that 476

477

478

479

480

481

482

483

484

485

486

and therefore the desired inequality holds with c = n . Now, for any ε > 0 it follows from [24] that there exists a two layer network ˜ A with m continuous derivatives such that sup x ∈ K ∣ ∣ ∣ A -˜ A ∣ ∣ ∣ ∞ < ε/n and sup x ∈ K ∣ ∣ ∣ ∇ k A -∇ k ˜ A ∣ ∣ ∣ ∞ < ε/n < ε for all 1 ≤ k ≤ m . Therefore, it follows that

completing the argument.

Next, we bound the deviation in the orthogonal projectors P ⊥ ˜ E , P ⊥ ˜ S .

̸

Lemma A.2. Let f : R n → R be such that ∇ f = 0 on the compact set K ⊂ R n . For any ε > 0 , there exists a two-layer neural network ˜ f : K → R such that ∇ ˜ f = 0 on K , sup x ∈ K ∣ ∣ ∣ f -˜ f ∣ ∣ ∣ <

Proof. Denote ∇ f = v and consider any ˜ v : K → R . Since | v | ≤ | ˜ v | + | v -˜ v | , it follows for all x ∈ K that whenever | v -˜ v | < (1 / 2) inf x ∈ K | v | ,

so that ˜ v = 0 in K , and since the square function is monotonic, 487

On the other hand, we also have | ˜ v | ≤ | v | + | ˜ v -v | < | v | +(1 / 2) inf x ∈ K | v | , so that, adding and 488 subtracting ˜ vv ⊺ and applying Cauchy-Schwarz, it follows that for all x ∈ K , 489

Now, by Corollary 2.2 in [24], for any ε > 0 there exists a two-layer neural network ˜ f : K → R such 490 that 491

̸

̸

and also sup x ∈ K ∣ ∣ ∣ f -˜ f ∣ ∣ ∣ < ε. Letting ˜ v = ∇ ˜ f , it follows that for all x ∈ K , 492

and therefore, taking the supremum of both sides and applying the previous work yields the desired 493 estimate, 494

495

496

497

498

499

With these intermediate results established, the proof of the approximation result Proposition 3.7 proceeds as follows.

Proof of Proposition 3.7. Recall from Theorem 3.4 that we can write L = P ⊥ S ( A tri -A ⊺ tri ) P ⊥ S and similarly for ˜ L . Notice that, by adding and subtracting P ⊥ ˜ S A tri P ⊥ S and P ⊥ ˜ S ˜ A tri P ⊥ S , it follows that for all x ∈ K ,

̸

where we have used that P ⊥ S , P ⊥ ˜ S have unit spectral norm. By Lemma A.1, for any ε > 0 there exists 500 a two layer neural network ˜ A tri such that sup x ∈ K ∣ ∣ ∣ A tri -˜ A tri ∣ ∣ ∣ < ε 4 , and by Lemma A.2 there exists 501 a two-layer network ˜ S with ∇ ˜ S = 0 on K such that 502

It follows that ˜ S, ∇ ˜ S are ε -close to S, ∇ S on K and 503

Therefore, the estimate 504

implies that ˜ L is ε -close to L on K as well. 505

Moving to the case of M , we see that for all x ∈ K , by writing M = U Λ U ⊺ = K chol K ⊺ chol for 506 K chol = U Λ 1 / 2 and repeating the first calculation with K chol in place of A tri and P ⊥ E in place of 507 P ⊥ S , 508

Moreover, if ∣ ∣ ∣ K chol -˜ K chol ∣ ∣ ∣ < (1 / 2) inf x ∈ K | K chol | for all x ∈ K then similar arguments as used 509 in the proof of Lemma A.2 yield the following estimate for all x ∈ K , 510

As before, we now invoke Lemma A.1 to construct a two-layer lower-triangular network ˜ K chol such 511 that 512

̸

as well as (using Lemma A.2) a network ˜ E satisfying ∇ ˜ E = 0 on K and 513

514

515

516

517

518

519

520

521

522

523

524

525

526

Again, it follows that ˜ E, ∇ ˜ E are ε -close to E, ∇ E on K , and by the work above we conclude

as desired.

It is now possible to give a proof of the error bound in Theorem 3.9. Recall the L 2 ([0 , T ]) error metric ∥ x ∥ and Lipschitz constant L f , defined for all x , y ∈ R n and Lipschitz continuous functions f as

Proof of Theorem 3.9. First, note that the assumption that one of E, -S (without loss of generality, say E ) has bounded sublevel sets implies bounded trajectories for the state x as in Remark 3.8, so we may assume x ∈ K for some compact K ⊂ R n . Moreover, for any ε > 0 it follows from Proposition 3.7 that there are approximate networks ˜ E, ˜ S which are ε -close to E,S on K . Additionally, it follows that ˜ E, ˜ S have nonzero gradients ∇ ˜ E, ∇ ˜ S which are also ε -close to the true gradients ∇ E, ∇ S on K . This implies that for each x ∈ K , E = ˜ E + ( E -˜ E ) ≤ ˜ E + ε , so it follows that the sublevel sets { x | ˜ E ( x ) ≤ m } ⊆ { x | E ( x ) ≤ m + ε } are also bounded. Therefore, we may assume (by potentially enlarging K ) that both x , ˜ x ∈ K lie in the compact set K for all time.

Now, let y = x -˜ x . The next goal is to bound the following quantity: 527

To that end, notice that by adding and subtracting L ( x ) ∇ E (˜ x ) , ˜ L ( x ) ∇ E (˜ x ) , ˜ L (˜ x ) ∇ E (˜ x ) , it fol528 lows that 529

530

531

532

533

By Proposition 3.7 there exists a two-layer neural network ˜ L with one continuous derivative such that sup x ∈ K ∣ ∣ ∣ L -˜ L ∣ ∣ ∣ < ε , which implies that ˜ L is Lipschitz continuous with (uniformly wellapproximated) Lipschitz constant. Using this fact along with the assumed Lipschitz continuity of ∇ E and the approximation properties of the network ˜ E already constructed then yields

Similarly, by adding and subtracting M ( x ) ∇ S (˜ x ) , ˜ M ( x ) ∇ S (˜ x ) , ˜ M (˜ x ) ∇ S (˜ x ) , it follows that 534

535

536

537

538

539

540

541

542

543

544

545

546

547

548

549

550

551

552

553

554

555

556

557

558

559

560

561

562

563

By Proposition 3.7, there exists a two-layer network ˜ M with one continuous derivative such that sup x ∈ K ∣ ∣ ∣ M -˜ M ∣ ∣ ∣ < ε , with ˜ M Lipschitz continuous for the same reason as before. It follows from this and sup x ∈ K ∣ ∣ ∣ ∇ S -∇ ˜ S ∣ ∣ ∣ < ε that

Now, recall that ∂ t | y | = | y | -1 ( ˙ y · y ) ≤ | ˙ y | by Cauchy-Schwarz, and therefore the time derivative of | y | is bounded by

This implies that ∂ t | y | -a | y | ≤ b , so multiplying by the integrating factor e -at and integrating in time yields

where we used that y (0) = 0 since the initial condition of the trajectories is shared. Therefore, the L 2 error in time can be approximated by

establishing the conclusion.

B Experimental and Implementation Details

This Appendix records additional details related to the numerical experiments in Section 5. For each benchmark problem, a set of trajectories is manufactured given initial conditions by simulating ODEs with known metriplectic structure. For the experiments in Table 2, only the observable variables are used to construct datasets, since entropic information is assumed to be unknown. Algorithm 2 summarizes the training of the dynamics models used for comparison with NMS.

For each compared method, integrating the ODEs is done via the Dormand-Prince method (dopri5) [27] with relative tolerance 10 -7 and absolute tolerance 10 -9 . The loss is evaluated by measuring the discrepancy between the ground truth observable states x o and the approximate observable states ˜ x o in the mean absolute error (MAE) metric. The model parameters Θ (i.e., the weights and biases) are updated by using Adamax [28] with an initial learning rate of 0.01. The number of training steps is set as 30,000, and the model parameters resulting in the best performance for the validation set are chosen for testing. Specific information related to the experiments in Section 5 is given in the subsections below.

For generating the results reported in Table 2, we implemented the proposed algorithm in Python 3.9.12 and PyTorch 2.0.0. Other required information is provided with the accompanying code. All experiments are conducted on Apple M2 Max chips with 96 GB memory. To provide the mean and the standard deviation, experiments are repeated three times with varying random seeds for all considered methods.

564

565

566

567

568

569

570

571

572

573

574

575

576

577

578

579

580

581

582

583

584

585

586

587

588

589

590

591

592

593

594

595

596

597

598

599

600

601

602

603

604

605

606

607

608

609

B.1 Two gas containers

As mentioned in the body, the two gas container (TGC) problem tests models' predictive capability (i.e., extrapolation in time). To this end, one simulated trajectory is obtained by solving an IVP with a known TGC system and an initial condition, and the trajectory of the observable variables is split into three subsequences, [0, t train ], ( t train , t val ] , and ( t val , t test ] for training, validation, and test with 0 < t train < t val < t test .

In the experiment, a sequence of 100,000 timesteps is generated using the Runge-Kutta 4thorder (RK4) time integrator with a step size 0.001. The initial condition is given as x = (1 , 2 , 103 . 2874 , 103 . 2874) following [29]. The training/validation/test split is defined by t train = 20 , t val = 30 , and t test = 100 . For a fair comparison, all considered models are set to have a similar number of model parameters, ∼ 2,000. The specifications of the network architectures are:

NMS: The total number of model parameters is 1959. The functions A tri , B , K chol , E, S are parameterized as MLPs with the Tanh nonlinear activation function. The MLPs parameterizing A tri , B , K chol , E are specified as 1 hidden layer with 10 neurons, and the on parameterizing S is specified as 3 hidden layers with 25 neurons.

NODE: The total number of model parameters is 2179. The black-box NODE is parameterized as an MLP with the Tanh nonlinear activation function, 4 hidden layers and 25 neurons.

SPNN: The total number of model parameters is 1954. The functions E and S are parameterized as MLPs with the Tanh nonlinear activation function; each MLP is specified as 3 hidden layers and 20 neurons. The two 2-tensors defining L and M are defined as learnable 3 × 3 matrices.

GNODE: The total number of model parameters is 2343. The functions E and S are parameterized as MLPs with the Tanh nonlinear activaton function; each MLP is specified as 2 hidden layers and 30 neurons. The matrices and 3-tensors required to learn L and M are defined as learnable 3 × 3 matrices and 3 × 3 × 3 tensor.

GFINN: The total number of model parameters is 2065. The functions E and S are parameterized as MLPs with Tanh nonlinear activation function; each MLP is specified as 2 hidden layers and 20 neurons. The matrices to required to learn L and M are defined as K learnable 3 × 3 matrices, where K is set to 2.

B.2 Thermoelastic double pendulum

The equations of motion in this case are given for 1 ≤ i ≤ 2 as

where κ > 0 is a thermal conductivity constant (set to 1), m i is the mass of the i th spring (also set to 1) and T i = ∂ S i E i is its absolute temperature. In this case, q i , p i ∈ R 2 represent the position and momentum of the i th mass, while S i represents the entropy of the i th pendulum. As before, the total entropy S ( x ) = S 1 + S 2 is the sum of the entropies of the two springs, while defining the internal energies

leads to the total energy E ( x ) = (1 / 2 m 1 ) | p 1 | 2 +(1 / 2 m 2 ) | p 2 | 2 + E 1 ( x ) + E 2 ( x ) .

The thermoelastic double pendulum experiment tests model prediction across initial conditions. In this case, 100 trajectories are generated by varying initial conditions that are randomly sampled from [0.1,1.1] × [-0.1,0.1] × [2.1, 2.3] × [-0.1,0.1] × [-1.9,2.1] × [0.9,1.1] × [-0.1, 0.1] × [0.9,1.1] × [0.1,0.3] ⊂ R 10 . Each trajectory is obtained from the numerical integration of the ODEs using an RK4 time integrator with step size 0.02 and the final time T = 40 , resulting in the trajectories of length 2,000. The resulting 100 trajectories are split into 80/10/10 for training/validation/test sets. For a fair comparison, all considered models are again set to have similar number of model parameters, ∼ 2,000. The specifications of the network architectures are:

610

611

612

613

614

615

616

617

618

619

620

621

622

623

624

625

626

627

628

629

630

631

632

633

634

635

636

637

638

639

640

641

642

643

644

645

646

647

648

649

650

651

652

653

654

655

656

657

658

659

NMS: The total number of model parameters is 2201. The functions A , B , K , E, S are parameterized as MLPs with the Tanh nonlinear activation function. The MLPs parameterizing are specified as 1 hidden layer with 15 neurons.

NODE: The total number of model parameters is 2005. The black-box NODE is parameterized as an MLP with the Tanh nonlinear activation function, 2 hidden layers and 35 neurons.

SPNN: The total number of model parameters is 2362. The functions E and S are parameterized as MLPs with the Tanh nonlinear activation function; each MLP is specified as 3 hidden layers and 20 neurons. The two 2-tensors defining L and M are defined as learnable 3 × 3 matrices.

GNODE: The total number of model parameters is 2151. The functions E and S are parameterized as MLPs with the Tanh nonlinear activaton function; each MLP is specified as 2 hidden layers and 15 neurons. The matrices and 3-tensors required to learn L and M are defined as learnable 3 × 3 matrices and 3 × 3 × 3 tensor.

GFINN: The total number of model parameters is 2180. The functions E and S are parameterized as MLPs with Tanh nonlinear activation function; each MLP is specified as 2 hidden layers and 15 neurons. The matrices to required to learn L and M are defined as K learnable 3 × 3 matrices, where K is set to 2.

C Additional experiment: Damped nonlinear oscillator

Consider a damped nonlinear oscillator of variable dimension with state x = ( q p S ) ⊺ , whose motion is governed by the metriplectic system

Here q , p ∈ R n denote the position and momentum of the oscillator, S is the entropy of a surrounding thermal bath, and the constant parameters m,γ,T are the mass, damping rate, and (constant) temperature. This leads to the total energy E ( x ) = (1 / 2 m ) | p | 2 -k cos q + TS, which is readily seen to be constant along solutions x ( t ) .

It is now verified that NMS can accurately and stably predict the dynamics of a nonlinear oscillator x = ( q p S ) ⊺ in the case that n = 1 , 2 , both when the entropy S is observable as well as when it is not. As before, the task considered is prediction in time, although all compared methods NODE, GNODE, and NMSknown are now trained on full state information from the training interval, and test errors are computed over the full state x on the extrapolation interval ( t valid , t test ] , which is 150% longer than the training interval. In addition, another NMS model, NMS diff , was trained using only the partial state information x o = ( q , p ) ⊺ and tested under the same conditions, with the initial guess for x u generated as in Appendix E. As can be seen in Table 3, NMS is more accurate than GNODE or NODE in both the 1-D and 2-D nonlinear oscillator experiments, improving on previous results by up to two orders of magnitude. Remarkably, NMS produces more accurate entropic dynamics even in the case where the entropic variable S is unobserved during NMS training and observed during the training of other methods. This illustrates another advantage of the NMS approach: because of the reasonable initial data for S produced by the diffusion model, the learned metriplectic system produced by NMS remains performant even when metriplectic governing equations are unknown and only partial state information is observed.

To describe the experimental setup precisely, data is collected from a single trajectory with initial condition as x = ( 2 , 0 , 0) following [16]. The path is calculated at 180,000 steps with a time interval of 0.001, and is then split into training/validation/test sets as before using t train = 60 , t val = 90 and t test = 180 . Specifications of the networks used for the experiments in Table 3 are:

NMS: The total number of parameters is 154. The number of layers for A tri , B , K chol , E, S is selected from {1,2,3} and the number of neurons per layer from {5,10,15}. The best hyperparameters are 1 hidden layer with 5 neurons for each network function.

GNODE: The total number of model parameters is 203. The number of layers and number of neurons for each network is chosen from the same ranges as for NMS. The best hyperparameters are 1 layer with 10 neurons for each network function.

660

661

662

663

664

665

666

667

668

669

670

671

672

673

674

675

676

677

678

679

NODE: The total number of model paramters is 3003. The NODE architecture is formed by stacking MLPs with Tanh activation functions. The number of blocks is chosen from {3,4,5} and the number of neurons of each MLP from {30,40,50}. The best hyperparameters are 4 and 30 for the number of blocks and number of neurons, respectively.

D Scaling study

To compare the scalability of the proposed NMS architecture design with existing architectures, different realizations of GNODE, GFINN, and NMS are generated by varying the dimension of the state variables, n = { 1 , 5 , 10 , 15 , 20 , 30 , 50 } . The specifications of these models (i.e., hyperparameters) are set so that the number of model parameters is kept similar between each method for smaller values of n . For example, for n = 1 , 5 the number of model parameters is ∼ 20,000 for each architecture. The results in Figure 3(a) confirm that GNODE scales cubically in n while both GFINN and NMS scale quadratically. Note that only a constant scaling advantage of NMS over GFINN can be seen from this plot, since r is fixed during this study.

It is also worthwhile to investigate the computational timings of these three models. Considering the same realizations of the models listed above, i.e., the model instances for varying n = { 1 , 5 , 10 , 15 , 20 , 30 , 50 } , 1,000 random samples of states { x ( i ) } 1 , 000 i =1 are generated. These samples are then fed to the dynamics function L ( x ( i ) ) ∇ E ( x ( i ) ) + M ( x ( i ) ) ∇ S ( x ( i ) ) for i = 1 , . . . , 1000 , and the computational wall time of the function evaluation via PyTorch's profiler API is measured. The results of this procedure are displayed in Figure 3(b). Again, it is seen that the proposed NMSs require less computational resources than GNODEs and GFINNs.

×

GNODE

GFINN

NMS

5.2

6.4

number of model parameters

5.0

Computational wall time

6.3

4.8

6.2

X

4.6

6.1

6.0

4.4

5.9

1

5

10152030

50

4.3

4.4

4.5

4.6

4.7

number of states, (n)

number of model parameters

(a) State dimension n versus number of model parameters

(b) Model parameters versus wall time in microseconds

680

681

682

683

684

685

686

687

688

689

690

691

692

693

694

695

696

697

698

699

700

701

702

703

704

705

706

707

708

709

710

711

712

713

714

715

716

717

718

719

720

721

722

723

724

E Diffusion model for unobserved variables

Recent work in [30] suggests the benefits of performing time-series generation using a diffusion model. This Appendix describes how this technology is used to generate initial conditions for the unobserved NMS variables in the experiments corresponding to Table 3. More precisely, we describe how to train a conditional diffusion model which generates values for unobserved variables x u given values for the observed variables x o .

Training and sampling: Recall that diffusion models add noise with the following stochastic differential equation (SDE):

where w ∈ R dim( x ) is a multi-dimensional Brownian motion, f ( t, · ) : R dim( x ) → R dim( x ) is a vector-valued drift term, and g : [0 , 1] → R is a scalar-valued diffusion function.

For the forward SDE, there exists a corresponding reverse SDE:

which produces samples from the initial distribution at t = 0 . This formula suggests that if the score function, ∇ x ( t ) log p ( x ( t )) , is known, then real samples from the prior distribution p ( x ) ∼ N ( µ, σ 2 ) can be recovered, where µ, σ vary depending on the forward SDE type.

In order for a model M θ to learn the score function, it has to optimize the following loss:

where t is uniformly sampled over [0 , 1] with an appropriate weight function λ ( t ) : [0 , 1] → R . However, using the above formula is computationally prohibitive. Thanks to [31], this loss can be substituted with the following denoising score matching loss:

Since score-based generative models use an affine drift term, the transition kernel p ( x ( t ) | x (0)) follows a certain Gaussian distribution [32], and therefore the gradient term ∇ x ( t ) log p ( x ( t ) | x (0)) can be analytically calculated.

Experimental details On the other hand, the present goal is to generate unobserved variables x u given values for the observed variables x o = ( q , p ) , i.e., conditional generation. Therefore, our model has to learn the conditional score function, ∇ x u ( t ) log p ( x u ( t ) | x o ) . For example, in the damped nonlinear oscillator case, S ( t ) is initialized as a perturbed t ∈ [0 , 1] , from which the model takes the concatenation of q , p , S ( t ) as inputs and learns conditional the score function ∇ S ( t ) log( S ( t ) | q , p ) .

For the experiments in Table 3, diffusion models are trained to generate x u variables on three benchmark problems: the damped nonlinear oscillator, two gas containers, and thermolastic double pendulum. On each problem, representative parameters such as mass or thermal conductivity are varied, with the total number of cases denoted by N . Full trajectory data of length T is then generated using a standard numerical integrator (e.g., dopri5), before it is evenly cut into ⌊ T/L ⌋ pieces of length L . Let V, U denote the total number of variables and the number of unobserved variables, respectively. It follows that the goal is to generate U unobserved variables given V -U observed ones, i.e., the objective is to generate data of shape ( NT/L,L,U ) conditioned on data of shape ( NT/L,L,V -U ) . After the diffusion model has been trained for this task, the output data is reshaped into size ( N,T,U ) , which is used to initialize the NMS model. Note that the NODE and GNODE methods compared to NMS in Table 3 use full state information for their training, i.e., x u = ∅ in these cases, making it comparatively easier for these methods to learn system dynamics.

As in other diffusion models e.g. [33], a U-net architecture is used, modifying 2-D convolutions to 1-D ones and following the detailed hyperparameters described in [33]. Note the following probability flow ODE seen in [33]:

Although models trained to mimic the probability flow ODE do not match the perofrmance of the forward SDE's result in the image domain, the authors of [30] observe that the probability flow ODE outperforms the forward SDE in the time-series domain. Therefore, the probability flow ODE is used with the default hyperparameters of [33].

725

726

727

728

729

730

731

732

733

734

735

736

737

738

739

740

741

742

743

744

745

746

747

748

749

750

751

752

753

754

755

756

757

758

759

760

761

762

763

764

765

766

767

768

769

770

771

772

NeurIPS Paper Checklist

The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.

Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:

You should answer [Yes] , [No] , or [NA] .

[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.

Please provide a short (1-2 sentence) justification right after your answer (even for NA).

The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.

The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation. While "[Yes] " is generally preferable to "[No] ", it is perfectly acceptable to answer "[No] " provided a proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] " or "[NA] " is not grounds for rejection. While the questions are phrased in a binary way, we acknowledge that the true answer is often more nuanced, so please just use your best judgment and write a justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.

IMPORTANT, please:

Delete this instruction block, but keep the section heading 'NeurIPS paper checklist" ,

Keep the checklist subsection headings, questions/answers and guidelines below.

Do not modify the questions and only use the provided macros for your answers .

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The claims made in the abstract and contributions paragraph at the end of the introduction are justified in detail throughout the rest of the paper.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]

Justification: Limitations are discussed in the Conclusion section.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: All theoretical results are clearly stated along with the necessary assumptions. All formal arguments are complete and contained in the Appendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: All information necessary to implement the proposed architecture is included in the body of the manuscript. In addition, all relevant experimental details are included in the Appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Code for running the proposed algorithm is included in the supplemental material and will be released publicly upon publication.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: All relevant experimental details are presented in the Appendix at an appropriate level of detail.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: All experiments in the body contain means and standard deviations as the initialization is varied.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: All necessary information is included in the Appendix.

Guidelines:

932

933

934

935

936

937

938

939

940

941

942

943

944

945

946

947

948

949

950

951

952

953

954

955

956

957

958

959

960

961

962

963

964

965

966

967

968

969

970

971

972

973

974

975

976

977

978

979

980

981

982

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: This is explained in the "broader impacts" section.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: This paper investigates a novel machine learning method tailored to physicsbased simulations and fundamental science. While subsequent applications of this work may have societal impact, the research presented here is strictly foundational and only serves to improve the production of physically realistic dynamics from data.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

983

984

985

986

987

988

989

990

991

992

993

994

995

996

997

998

999

1000

1001

1002

1003

1004

1005

1006

1007

1008

1009

1010

1011

1012

1013

1014

1015

1016

1017

1018

1019

1020

1021

1022

1023

1024

1025

1026

1027

1028

1029

1030

1031

1032

1033

1034

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: N/A

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: N/A

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: N/A

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

1035

1036

1037

1038

1039

1040

1041

1042

1043

1044

1045

1046

1047

1048

1049

1050

1051

1052

1053

1054

1055

1056

1057

1058

1059

1060

1061

1062

1063

1064

1065

1066

1067

1068

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: N/A

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: N/A

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new parameterization for neural metriplectic systems which explicitly incorporates structural information about the degeneracy conditions $\\{ S, \\cdot \\} = 0, [E,\\cdot ] = 0$ into the model. The model requires $\\sim O(n^2)$ learnable parameters for a problem with $n$ state variables instead of some prior methods which need $O(n^3)$. Further, it also encodes this degeneracy condition in a hard constraint, leading to models which will by construction respect these desired physical conditions. The authors provide a deep learning implementation scheme for their method which involves learning $E(x), S(x)$ and using $\\nabla E, \\nabla S$ to construct the matrices $L, M$ needed for the bracket from observed trajectories of the physical system. The gradients $\\nabla E, \\nabla S$ needed for the brackets are computed with autodifferentiation. The authors show that this system is trained end-to-end on simple physical systems including a two-gas system and a thermo-elastic pendulum and can outperform existing methods on these benchmarks.\n\nSTRENGTHS:\nThis paper motivates the need for metriplectic systems which can be efficiently implemented and that incorporate the dynamical constraints required for these systems (energy conservation and entropy production). This captures a potentially interesting class of physical systems that could be modeled by machine learning methods like those employed in the present work. The method that they present as Algorithm 1 is straightforward and improves upon the cubic time complexity of GNODE or GFINN. The authors also provide an approximation result for their algorithm and support their claims with some experiments.\n\nWEAKNESSES:\nWhile the authors improve the scaling from cubic to quadratic in the number of state variables, the total complexity (quadratic) still scales poorly with size of the problem (number of state variables / dimensions). Further, the current experiments and comparisons were performed on small benchmarks. However, since this paper is the first to point out that the cubic scaling can be improved by reflecting constraints due to degeneracy, I think the experimental component of the contribution is not the most important.\n\nQUESTIONS:\nLemma 3.2 involves projection matrices to construct the $L,M$ matrices. How are these chosen in the experiments? \n\nI am not an expert in this area of ML for physical modeling. Could the authors provide some insight into the ultimate goal system they would eventually wish to model with metriplectic neural systems? Are there real world physical data which could be well captured by these kinds of models?\n\nLIMITATIONS:\nThe authors do mention the primary limitations of this present work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a parameter efficient parameterization for neural networks simulating metripletic systems, which are differential equations that have both an energy dissipative piece and an energy conservative piece. The method works by learning several of the required quantities (L and M, which trade off dissipation and conservation, I believe), while also using a small neural network to estimate the dissipation and conservation pieces (E(x) and S(x) ). As not all quantities in the state, x, can be observed, they use a time based diffusion model to emulate the hidden states (e.g. entropy) to develop initial conditions for these. Experiments are performed on two systems of this class, where it seems like the method performs better (probably due to having better inductive biases).\n\n\nUnfortunately, due to not having a strong physics background, I feel somewhat unqualified to judge many of the technical strengths although things seem reasonable from a skim. I don’t know if I can properly assess novelty and significance as a result.\n\nSTRENGTHS:\nSignificance:\n-\tBuilding better emulators of physical systems that are complicated is a good first step in what the authors term “phenomenonological” understanding of these systems.\n\n-\tEven quicker training time (and demonstrated both practically and theoretically) is quite helpful. I remember one of the original issues with NODE was that it took a very long time to converge.\n\nClarity: \n-\tOverall, the paper is pretty well written, even if quite dense, and okay to follow for a non expert physicist. I was able to follow at least the ML pieces and the experiments section quite well.\n\n-\tThe relevant literature is reasonably well signposted; I learned a fair bit about the state of this field by checking the references.\n\nNovelty:\n\n-\tThe approach seems to have a clear inductive bias win over the prior works GFINN and GNODE due to better parameterization of the system.\n\nWEAKNESSES:\nUnfortunately, the writing ends up being quite dense and technical with minimal outside applications. \n\nSure, emulating these physical systems in the experiments is quite nice, but what types of applications does this lead to? This is more of a writing based thing and the paper could be refactored around one of these applications if possible.\n\nQUESTIONS:\n**Questions**\n\nL138: Why do GFINNs require cubic scaling if they only need to learn O(rn^2) functions?\n\nIn general, what is the lower theoretical bound for learning metripletic systems?\n\nL363: what are relevant 3-d problems of the size that you’d like to tackle?\n\nFig 3b: is model parameters measured in thousands here? Don’t think we can have 4.3 parameters…\n\nCould the authors generate plots like Fig 2 for the other two experiments? I think it may be useful to understand why the double pendulum example shows less relative improvement in that system?\n\nAre the two main text experiments fully observed? Could the authors run these without full observations (e.g. with the diffusion for the unobserved states) if that makes sense in this setting?\n\nL226: why tanh activation for this task?\n\n**Writing comments:**\n\n-\tI would personally suggest moving a bit more of appendix E into the main text, as well as Fig 3. These are both interesting and pretty compelling. \n\n-\tI would also suggest then moving section 3.1 into the appendix as I don’t think you really need much of that material. Pieces of Section 2 are much too dense and could be moved into the appendix as well; it should probably be half a page at most.\n\nLIMITATIONS:\nn/a", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work presents a method for learning metriplectic systems from data. Metriplectic systems are a model which conserve energy and produce entropy, two desirable features. Their method, termed “neural metriplectic systems” (NMS), is based on a more efficient system parametrization. The authors also prove universal approximation results on non-degenerate systems, and generalization error bounds. They verify that their method outperforms other metriplectic-learning baselines, GNODE and GFINN, on two physical experiments.\n\nSTRENGTHS:\nOriginality: Although I am not at all an expert in this field and therefore cannot properly judge, it seems that the main theorem (Theorem 3.4) is novel and non-trivial.\n\nQuality: The proposed method outperforms baselines in two experimental settings, verifying the expected gain from having a more efficient parametrization. The corresponding theoretical results, on universality and generalization, provide a fairly thorough picture of the method. \n\nSignificance: Within the field of learning metriplectic systems, this paper seems to make a valuable contribution and improve on prior work.\n\nClarity: The paper is very well-written, and mathematically rigorous. Although the details are not accessible to someone without a background that matches the subject material rather closely, the high-level ideas about the benefits of metriplectic systems, what past work has done, and the advantages of their new method, are conveyed well.\n\nWEAKNESSES:\nClarity:\n1. Although well-written, the paper is not accessible to most machine learning audiences, and seemingly requires the reader to already have a physics background in phenomenological modeling, or exterior algebra.\n\n2. Mathematical terms such as algebraic brackets, Poisson brackets, degenerate metric brackets, etc. should be defined in the beginning of the paper, or with a reference to a textbook or other paper defining them. The “exterior algebra” background is suitable for only those with a strong mathematical background already, using terms like “wedge product” and “graded algebra” without definition. (Admittedly, it would be impossible to fully explain all of these concepts in only 9 pages — perhaps a citation to a textbook would be helpful here, but in practice if the reader needs to understand the decomposition result properly to grasp the contribution, then this work may be more suitable for a venue other than a machine learning conference.)\n\nQuality: The baselines in experiments, as well as the methods discussed in the exposition, are all metriplectic. However, it seems like other methods (e.g. which preserve energy but do not increase entropy, or even those which are not physics-informed at all), should be included too.\n\nSignificance: I am not sure how widely applicable metriplectic learning systems are, or what alternative (non-metriplectic) methods can be used for the same problems. The paper would be improved by providing more of this background/motivation.\n\nOverall, as a non-expert, my main concern is with the suitability of this work for a machine learning conference - I defer to the AC on this point. It seems that the machine learning techniques used within NMS are fairly straightforward, while it is the parametrization in Theorem 3.4 that seemingly constitutes the crux of the method. However, the statement and proof of Theorem 3.4 would be more accessible to a physics or math audience, than an ML audience.\n\nQUESTIONS:\n1. NMS is a more efficient parametrization than GNODE and GFINN. Is it also more efficient computationally (end-to-end)? (For example, one could in theory imagine a system with very few parameters, that uses them in a forward pass in a computationally intense way.)\n\n2. Are there non-metriplectic methods that are suitable for comparison in the experiments (for example, methods that preserve energy like Hamiltonian networks)? \n\n3. What is the motivation for metriplectic methods overall — do they have drawbacks relative to methods that do not preserve energy/increase entropy? It would be helpful to include these in related work.\n\nLIMITATIONS:\nLimitations are discussed.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 1, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}