Title: Learning Dynamical Systems from Noisy Data with Inverse-Explicit Integrators
Venue: NeurIPS 2023
Accepted: no
Keywords: Deep neural networks

Hamiltonian systems

ODE discretization

Runge-Kutta

Geometric numerical integration
Abstract: We introduce the mean inverse integrator (MII), a novel approach to increase the accuracy when training neural networks to approximate vector fields of dynamical systems from noisy data. This method can be used to average multiple trajectories obtained by numerical integrators such as Runge--Kutta methods. We show that the class of mono-implicit Runge--Kutta methods (MIRK) has particular advantages when used in connection with MII. When training vector field approximations, explicit expressions for the loss functions are obtained when inserting the training data in the MIRK formulae, unlocking symmetric and high order integrators that would otherwise be implicit for initial value problems. The combined approach of applying MIRK within MII yields a significantly lower error compared to the plain use of the numerical integrator without averaging the trajectories. This is demonstrated with experiments using data from several (chaotic) Hamiltonian systems. Additionally, we perform a sensitivity analysis of the loss functions under normally distributed perturbations, supporting the favourable performance of MII.

1

Abstract

We introduce the mean inverse integrator (MII), a novel approach to increase the accuracy when training neural networks to approximate vector fields of dynamical systems from noisy data. This method can be used to average multiple trajectories obtained by numerical integrators such as Runge-Kutta methods. We show that the class of mono-implicit Runge-Kutta methods (MIRK) has particular advantages when used in connection with MII. When training vector field approximations, explicit expressions for the loss functions are obtained when inserting the training data in the MIRK formulae, unlocking symmetric and high order integrators that would otherwise be implicit for initial value problems. The combined approach of applying MIRK within MII yields a significantly lower error compared to the plain use of the numerical integrator without averaging the trajectories. This is demonstrated with experiments using data from several (chaotic) Hamiltonian systems. Additionally, we perform a sensitivity analysis of the loss functions under normally distributed perturbations, supporting the favourable performance of MII.

1 Introduction

Recently, many deep learning methodologies have been introduced to increase the efficiency and quality of scientific computations [1, 2, 3, 4]. In physics-informed machine learning, deep neural networks are purposely built so to enforce physical laws. As an example, Hamiltonian neural networks (HNNs) [5] aim at learning the Hamiltonian function from temporal observations. The Hamiltonian formalism was derived within classical mechanics for modelling a wide variety of physical systems. The temporal evolution of such systems is fully determined when the Hamiltonian function is known, and it is characterized by geometric properties such as the preservation of energy, the symplectic structure and the time-reversal symmetry of the flow [6, 7].

Numerical integrators that compute solutions preserving such properties are studied in the field of geometric numerical integration [7, 8]. Thus, deep learning, classical mechanics and geometric numerical integration are all relevant to the development of HNNs. In this work, we try to identify the optimal strategy for using numerical integrators when constructing loss functions for HNNs that are trained on noisy and sparse data.

Generally, we aim at learning autonomous systems of first-order ordinary differential equations (ODE)

In the traditional setting, solving an initial value problem (IVP) means computing approximated 31 solutions y n ≈ y ( t n ) when the vector field f ( y ) and an initial value y ( t 0 ) = y 0 are known. The 32 focus of our study is the corresponding inverse problem; assuming knowledge of multiple noisy 33 samples of the solution, S N = { ˜ y n } N n =0 , the aim is to approximate the vector field f with a neural 34

network model f θ . We will assume that the observations originate from a (canonical) Hamiltonian 35 system, with a Hamiltonian H : R 2 d → R , where the vector field is given by 36

This allows for learning the Hamiltonian function directly by setting f θ ( y ) = J ∇ H θ ( y ) , as proposed 37 initially in [5]. 38

Recently, many works highlight the benefit of using symplectic integrators when learning Hamiltonian 39 neural networks [9, 10, 11, 12]. Here, we study what happens if, instead of using symplectic methods, 40 efficient and higher-order MIRK methods are applied for inverse problems. We develop different 41 approaches and apply them to learn highly oscillatory and chaotic dynamical systems from noisy data. 42 The methods are general, they are not limited to separable Hamiltonian systems, and could indeed be 43 used to learn any first-order ODE. However we focus our study on Hamiltonian systems, in order to 44 build on the latest research on HNNs. Specifically, we compare our methods to the use of symplectic 45 integrators to train Hamiltonian neural networks. Our contributions can be summarized as follows: 46

47

48

49

We introduce the mean inverse integrator (MII), which efficiently averages trajectories of MIRK methods in order to increase accuracy when learning vector fields from noisy data (Definition 5.1).

50

51

52

53

54

55

We present an analysis of the sensitivity of the loss function to perturbations giving insight into when the MII method yields improvement over a standard one-step scheme (Theorem 5.2).

We show that symplectic MIRK methods have at most order p = 2 (Theorem 4.4). Particularly, the second-order implicit midpoint method is the symplectic MIRK method with minimal number of stages.

Finally, numerical experiments on several Hamiltonian systems benchmark MII against one-step 56 training and symplectic recurrent neural networks (SRNN) [10], which rely on the Störmer-Verlet 57 integrator. The structural difference between these three approached is presented in Figure 2. Ad58 ditionally, we demonstrate that substituting Störmer-Verlet with the classic Runge-Kutta method 59 (RK 4 ) in the SRNN framework yields significant reduction in error and allows accurate learning of 60 non-separable Hamiltonian systems. 61

62

2 Related work

Hamiltonian neural networks was introduced in [5]. The numerical integration of Hamiltonian ODEs 63 and the preservation of the symplectic structure of the ODE flow under numerical discretization 64 have been widely studied over several decades [8, 7]. The symplecticity property is key and could 65 inform the neural network architecture [13] or guide the choice of numerical integrator, yielding a 66 theoretical guarantee that the learning target is actually a (modified) Hamiltonian vector field [14, 9], 67 building on the backward error analysis framework [8]. Discrete gradients is an approach to numerical 68 integration that guarantees exact preservation of the (learned) Hamiltonian, and an algorithm for 69 training Hamiltonian neural networks using discrete gradient integrators is developed in [15] and 70 extended to higher order in [16]. 71

Since we for the inverse problem want to approximate the time-derivative of the solution, f , using 72 only ˜ y n , we need to use a numerical integrator when specifying the neural network loss function. 73 For learning dynamical systems from data, explicit methods such as RK 4 are much used [5, 17, 18]. 74 However, explicit methods cannot in general preserve time-symmetry or symplecticity, and they often 75 have worse stability properties compared to implicit methods [19]. Assuming that the underlying 76 Hamiltonian is separable allows for explicit integration with the symplectic Störmer-Verlet method, 77 which is exploited in [10, 20]. Symplecticity could be achieved without the limiting assumption 78 of separability by training using the implicit midpoint method [12]. As pointed out in [12], this 79 integrator could be turned into an explicit method in training by inserting sequential training data ˜ y n 80 and ˜ y n +1 . In fact, the MIRK class [21, 22] contains all Runge-Kutta (RK) methods (including the 81 midpoint method) that could be turned into explicit schemes when inserting the training data. This 82 is exploited in [23], where high-order MIRK methods are used to train HNNs, achieving accurate 83

interpolation and extrapolation of a single trajectory with large step size, few samples and assuming 84 zero noise. 85

The assumption of noise-free data limits the potential of learning from physical measurements 86 or applications on data sets from industry. This issue is addressed in [10], presenting symplectic 87 recurrent neural networks (SRNN). Here, Störmer-Verlet is used to integrate multiple steps and is 88 combined with initial state optimization (ISO) before computing the loss. ISO is applied after training 89 f θ a given number of epochs and aims at finding the optimal initial value ˆ y 0 , such that the distance 90 to the subsequent observed points ˜ y 1 , . . . , ˜ y N is minimized when integrating over f θ . While [10] is 91 limited by only considering separable systems, [24] aims at identifying the optimal combination of 92 third order polynomial basis functions to approximate a cubic non-separable Hamiltonian from noisy 93 data, using a Bayesian framework. 94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

3 Background on numerical integration

Some necessary and fundamental concepts on numerical integration and the geometry of Hamiltonian systems are presented below to inform the discussion on which integrators to use in inverse problems. Further details could be found in Appendix C.

Fundamental concepts: An important subclass of the general first-order ODEs (1) is the class of Hamiltonian systems, as given by (2). Often, the solution is partitioned into the coordinates y ( t ) = [ q ( t ) , p ( t )] T , with q ( t ) , p ( t ) ∈ R d . A separable Hamiltonian system is one where the Hamiltonian could be written as the sum of two scalar functions, often representing the kinetic and potential energy, that depend only on q and p respectively, this means we have H ( q, p ) = H 1 ( q ) + H 2 ( p ) .

The h flow of an ODE is a map ϕ h,f : R n → R n sending an initial value y ( t 0 ) to the solution of the ODE at time t 0 + h , given by ϕ h,f ( y ( t 0 )) := y ( t 0 + h ) . A numerical integration method Φ h,f : R n → R n is a map approximating the exact flow of the ODE, so that

Here, y ( t n ) represents the exact solution and we denote with y n the approximation at time t n = t 0 + nh . It should be noted that the flow map satisfies the following group property:

In other words, a composition of two flows with step sizes h 1 , h 2 is equivalent to the flow map over f with step size h 1 + h 2 . This property is not shared by numerical integrators for general vector fields. The order of a numerical integrator Φ h,f characterizes how the error after one step depends on the step size h and is given by the integer p such that the following holds:

Mono-implicit Runge-Kutta methods: Given vectors b, v ∈ R s and a strictly lower triangular matrix D ∈ R s × s , a MIRK method is a Runge-Kutta method where A = D + vb T [25, 26] and we assume that [ A ] ij = a ij is the stage-coefficient matrix. This implies that the MIRK method can be written on the form

Specific MIRK methods and further details on Runge-Kutta schemes is discussed in Appendix C.2.

Symplectic methods: The flow map of a Hamiltonian system is symplectic, meaning that its Jacobian Υ ϕ := ∂ ∂ y ϕ h,f ( y ) satisfies Υ T ϕ J Υ ϕ = J , where J is the same matrix as in (2). As explained in [8, Ch. VI.2], this is equivalent to the preservation of a projected area in the phase space of [ q, p ] T . Similarly, a numerical integrator is symplectic if its Jacobian Υ Φ := ∂ ∂ y n Φ h,f ( y n ) satisfies Υ T Φ J Υ Φ = J . It is possible to prove [8, Ch. VI.4] that a Runge-Kutta method is symplectic if and only if the coeffients satisfy

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141 142

143

144

145

146

147

148

149

150

151

152

153

154

155

4 Numerical integration schemes for solving inverse problems

We will now consider different ways to use numerical integrators when training Hamiltonian neural networks and present important properties of MIRK methods, a key component of the MII that is presented in Chapter 5.

Inverse ODE problems in Hamiltonian form: We assume to have potentially noisy samples S N = { ˜ y } N n =0 of the solution of an ODE with vector field f . The inverse problem can be formulated as the following optimization problem:

where f θ = J ∇ H θ is a neural network approximation with parameters θ of a Hamiltonian vector field f , and Φ h,f θ is a one-step integration method with step length h . In the setting of inverse ODE problems, the availabil- ity of sequential points S N could be exploited when a numerical method is used to form interpolation conditions, for f θ ≈ f for each n in the optimization problem (6). For example, ˜ y n and ˜ y n +1 could be inserted in the implicit midpoint method, turning a method that is implicit for IVPs into an explicit method for inverse problems:

n n +1 → n n +1

We denote this as the inverse injection, which defines an inverse explicit property for numerical integrators. Definition 4.1 (Inverse injection) . Assume that ˜ y n , ˜ y n +1 ∈ S N . Let the inverse injection for the integrator Φ h,f ( y n , y n +1 ) be given by the substitution (˜ y , ˜ y ) ( y , y ) such that

ERK

RK

SympRK

SymRK

GL4, GL6

MIRK4, MIRK6

Midpoint

Definition 4.2 (Inverse explicit) . A numerical one-step method Φ is called inverse explicit if it is explicit under the inverse injection.

This procedure is utilized successfully by several authors when learning dynamical systems from data, see e.g. [12, 27]. However, this work is the first attempt at systematically exploring numerical integrators under the inverse injection, by identifying the MIRK methods as the class consisting of inverse explicit Runge-Kutta methods.

Proposition 4.3. MIRK-methods are inverse explicit.

Proof. Since the matrix D in (4) is strictly lower triangular, the stages are given by

meaning that if y n and y n +1 are known, all stages, and thus the next step ˆ y n +1 = y n + h ∑ s i =1 b i k i , could be computed explicitly.

Because of their explicit nature when applied to inverse ODE problems, MIRK methods are an attractive alternative to explicit Runge-Kutta methods; in contrast to explicit RK methods, they can be symplectic or symmetric, or both, without requiring the solution of systems of nonlinear

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

˜

y

0

˜

y

1

ˆ

y

1

˜

y

2

ˆ

y

2

(a) ERK, one-step.

˜

y

0

˜

y

1

ˆ

y

1

˜

y

2

ˆ

y

2

(b) MIRK, one-step.

ˆ

y

0

˜

y

1

ˆ

y

1

˜

y

2

ˆ

y

2

(c) SRNN with ISO.

equations, even when the Hamiltonian is non-separable. Figure 1 illustrates the relation between various subclasses and the specific methods are described in Table 1 in Appendix C. In addition, for s -stage MIRK methods, it is possible to construct methods of order p = s + 1 [22]. This is in general higher order than what is possible to obtain with s -stage explicit Runge-Kutta methods. Further computational gains could also be made by reusing evaluations of the vector field between multiple steps, which using MIRK methods allow for, as explained in Appendix I. The dependency structure on the data S N of explicit RK (ERK) methods, MIRK methods and the SRNN method [10] is illustrated in Figure 2.

Maximal order of symplectic MIRK methods: From the preceding discussion, it is clear that symplectic MIRK methods are of interest when learning Hamiltonian systems from data, since they combine computational efficiency with the ability to preserve useful, geometric properties. Indeed, symplectic integrators in the training of HNNs have been considered in [9, 10, 11, 12, 13]. The subclass of symplectic MIRK methods is represented by the middle, dark blue field in the Venn diagram of Figure 1. The next result gives an order barrier for symplectic MIRK methods that was, to the best of our knowledge, not known up to this point.

Theorem 4.4. The maximum order of a symplectic MIRK method is p = 2 .

Proof. This is a shortened version of the full proof, which can be found in Appendix F. A MIRK method is a Runge-Kutta method with coefficients a ij = d ij + v i b j . Requiring d ij , b i and v i to satisfy the symplecticity conditions of (5) in addition to D being strictly lower triangular, yields the following restrictions

/negationslash

These restrictions result in an RK method that could be reduced to choosing a coefficient vector 176 b ∈ R s and choosing stages on the form k i = f ( y n + h 2 ∑ s j b j k j ) for i = 1 , . . . , s . It is then trivial 177 to check that this method can only be of up to order p = 2 . Note that for s = 1 and b 1 = 1 we get the 178 midpoint method. 179

180

181

182

183

184

185

186

187

188

189

190

191

192

Numerical integrators outside the RK class: While this paper is mainly concerned with MIRK methods, several other types of numerical integrators could be of interest for inverse problems. Partitioned Runge-Kutta methods are an extension and not a subclass of RK methods, and can be symplectic and symmetric, while also being explicit for separable Hamiltonian systems. The Störmer-Verlet integrator of order p = 2 is one example. Higher order methods of this type are derived in [28] and used for learning Hamiltonian systems in [29, 30]. Discrete gradient methods [31, 32] are inverse explicit and well suited to train Hamiltonian neural networks using a modified automatic differentiation algorithm [15]. This method could be extended to higher order methods as shown in [16]. In contrast to symplectic methods, discrete gradient methods preserve the Hamiltonian exactly up to machine precision. A third option is elementary differential Runge-Kutta methods [33], where for instance [34] show how to use backward error analysis to construct higher order methods from modifications to the midpoint method. This topic is discussed further in Appendix H, where we also present a novel, symmetric discrete gradient method of order p = 4 .

193

194

195

5 Mean inverse integrator for handling noisy data

Noisy ODE sample: It is often the case that the samples S N are not exact measurements of the system, but perturbed by noise. In this paper, we model the noise as independent, normally distributed

perturbations 196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

where N (0 , σ 2 I ) represents the multivariate normal distribution. With this assumption, a standard result from statistics tells us that the variance of a sample-mean estimator with N samples converges to zero at the rate of 1 N . That is, assuming that we have N samples ˜ y (1) n , . . . , ˜ y ( N ) n , then

Using the inverse injection with the midpoint method, the vector field is evaluated in the average of ˜ y n and ˜ y n +1 , reducing the variance of the perturbation by a factor of two, compared to evaluating the vector field in ˜ y n , as is done in all explicit RK methods. Furthermore, considering the whole data trajectory S N , multiple independent approximations to the same point y ( t n ) can enable an even more accurate estimate. This is demonstrated in the analysis presented in Theorem 5.2 and in Figure 4.

/negationslash

Averaging multiple trajectories: In the inverse ODE problem, we assume that there exists an exact vector field f whose flow interpolates the discrete trajectories S N , and the flow of this vector field satisfies the group property (3). The numerical flow Φ h,f for a method of order p satisfies this property only up to an error O ( h p +1 ) over one step. In the presence of noisy data, compositions of one-step methods can be used to obtain multiple different approximations to the same point y ( t n ) , by following the numerical flow from different nearby initial values ˜ y j , j = n , and thus reduce the noise by averaging over these multiple approximations. Accumulation of the local truncation error is expected when relying on points further away from t n . However, for sufficiently small step sizes h compared to the size of the noise σ , one can expect increased accuracy when averaging over multiple noisy samples.

As an example, assume that we know the points { ˜ y 0 , ˜ y 1 , ˜ y 2 , ˜ y 3 } . Then y ( t 2 ) can be approximated by computing the mean of the numerical flows Φ h,f starting from different initial values:

where we by Φ ∗ mean the adjoint method of Φ , as defined in [8, Ch. V], and we let Ψ n,n +1 be the increment of an inverse-explicit numerical integrator, so that

For example, for the midpoint method, we have that Ψ n,n +1 = f ( ˜ y n +˜ y n +1 2 ) . When stepping in negative time in (10), we use the adjoint method in order to minimize the number of vector field evaluations, also when non-symmetric methods are used (which implies that we always use e.g. Ψ 1 , 2 and not Ψ 2 , 1 ). Note that in order to derive the approximation in (10), repeated use of the inverse injection allows the known points ˜ y n to form an explicit integration procedure, where composition of integration steps are approximated by summation over increments Ψ n,n +1 . This approximation procedure is presented in greater detail in Appendix D.

Mean inverse integrator: The mean approximation over the whole trajectory y n , for n = 0 , . . . , N , could be computed simultaneously, reusing multiple vector field evaluations in an efficient manner. This leads to what we call the mean inverse integrator. For example, when N = 3 we get

and the same structure is illustrated in Figure 3.

Definition 5.1 (Mean inverse integrator) . For a sample S N and an inverse-explicit integrator Ψ n,n +1 , 230 the mean inverse integrator is given by 231

By substituting the known vector field f with a neural network f θ and denoting the matrix containing 234 vector field evaluations by Ψ θ such that Y θ := 1 N ( U ˜ Y + hW Ψ θ ) , we can formulate an analogue to 235 the inverse problem (6) by 236

Analysis of sensitivity to noise: Consider the optimiza237 tion problems using integrators either as one-step methods 238 or MII by (6) resp. (12). We want to investigate how 239 uncertainty in the data ˜ y n introduces uncertainty in the op240 timization problem. Assume, for the purpose of analysis, 241 that the underlying vector field f ( y ) is known. Let 242

be the optimization target or the expression one aims to minimize using a one-step method (OS) and the MII, where Y is given by Definition 5.1. For a matrix A with eigenvalues λ i ( A ) , the spectral radius is given by

y

0

y

1

y

2

y

3

-

h

Ψ

2

,

3

-

2

h

Ψ

1

,

2

-

3

h

Ψ

0

,

1

y

0

y

1

y

2

y

3

h

Ψ

0

,

1

-

2

h

Ψ

1

,

2

-

h

Ψ

2

,

3

y

0

y

1

y

2

y

3

h

Ψ

0

,

1

2

h

Ψ

1

,

2

-

h

Ψ

2

,

3

y

0

y

1

y

2

y

3

h

Ψ

0

,

1

2

h

Ψ

1

,

2

3

h

Ψ

2

,

3

ρ ( A ) := max i | λ i ( A ) | . An analytic expression that approximates ρ ( T OS n ) and ρ ( T MII n ) by linearization of f for a general MIRK method is provided below.

Theorem 5.2. Let S N = { ˜ y n } N n =0 be a set of noisy samples, equidistant in time with step size h , with Gaussian perturbations as defined by (9) with variance σ 2 . Assume that a MIRK integrator Φ h,f is used as a one-step method. Then the spectral radius is approximated by

/negationslash

where f ′ := f ′ ( y n ) and P nj , Q OS and Q MII (defined in (24) in Appendix G) are matrices independent of the step size h .

The proof is found in Appendix G. Let α := b T ( -2 v ) denote the coefficients of the first order term in h of Equation (13). For any explicit RK method we have that v = 0 and since b T = 1 (method of at least order one) we find that α ERK = 1 . Considering the Butcher tableau of MIRK4 in Figure 9 we find that α MIRK 4 = 0 . Thus, as h → 0 we would expect quadratic convergence of MIRK4 and linear convergence of RK4 for ρ OS n to 2 σ 2 . Considering MII (14) one would expect linear convergence for ρ MII n to σ 2 if N is large, as h → 0 .

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

A numerical approximation of ρ OS n and ρ MII n could be real264 ized by a Monte-Carlo estimate. We compute the spectral 265 radius ˆ ρ n of the empirical covariance matrix of T OS n and 266 T MII n by sampling 5 · 10 3 normally distributed perturbations 267 δ n with σ 2 = 2 . 5 · 10 -3 to each point y n in a trajectory 268 of N + 1 points and step size h . We then compute the 269

0

.

10

0

.

22

0

.

34

0

.

48

0

.

60

0

.

80

Step size

h

0

.

3

0

.

4

0

.

5

0

.

6

0

.

7

0

.

8

0

.

9

1

.

0

Spectral radius

ρ

×

10

-

2

Propagation of noise, Double pendulum

RK4 OS

MIRK4 OS

/negationslash

0

.

0

2

.

5

5

.

0

7

.

5

10

.

0

12

.

5

15

.

0

17

.

5

20

.

0

t

-

0

.

4

-

0

.

2

0

.

0

0

.

2

0

.

4

y

1

Flow roll-out Double pendulum

h

= 0.8,

σ

= 0.05

0

.

0

2

.

5

5

.

0

7

.

5

10

.

0

12

.

5

15

.

0

17

.

5

20

.

0

t

-

0

.

4

-

0

.

2

0

.

0

0

.

2

0

.

4

y

1

Flow roll-out Double pendulum

h

= 0.1,

σ

= 0.05

Midpoint

Midpoint

RK4

ISO RK4

h

= 0.1,

σ

= 0.05

Midpoint

RK4

ISO RK4

MIRK4

h

= 0.8,

σ

= 0.05

Midpoint

MIRK4

Ground truth

Given data

Flow roll-out FPUT

RK4

MIRK4

Exact flow

= 0.05

10

.

t

15

20

trajectory average ρ = 1 N +1 ∑ N n =0 ˆ ρ n , fix the end time T = 2 . 4 , repeat the approximations for 270 decreasing step sizes h and increasing N and compute the average of ρ for 10 randomly sampled 271 trajectories S N from the double pendulum system. The plot in Figure 4 corresponds well with what 272 one would expect from Theorem 5.2 and confirms that first MIRK (with v = 0 ) and secondly MII 273 reduces the sensitivity to noise in the optimization target. 274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

6 Experiments

Flow roll-out H´ enon-Heiles h = 0.1, σ = 0.05 0 . 2 Flow roll-out H´ enon-Heiles h = 0.1, σ 0 . 0 0 . 2 y 1 Flow roll-out H´ enon-Heiles 0 . 2 0 . 4 Methods and test problems: We train HNNs using different integrators and methods in the inverse problem (6). We use MIRK4 together with the MII method and compare to the implicit midpoint method, RK4 and MIRK4 applied as one-step methods, as well as ISO followed by Störmer-Verlet and RK4 integrated over multiple time-steps. The latter strategy, illustrated in Figure 2, was suggested in [10], where Störmer-Verlet is used. Separable networks H θ ( q, p ) = H 1 , θ ( q ) + H 2 , θ ( p ) are trained on data from the Fermi-Pasta-Ulam-Tsingou (FPUT) problem and the Hénon-Heiles system. For the double pendulum, which is non-separable, a fully connected network is used for all methods except StörmerVerlet, which requires separability in order to be explicit. The Hamiltonians are described in Appendix A and all systems have solutions y ( t ) ∈ R 4 .

0

.

2

0 . 0 2 . 5 5 . 0 7 . 5 10 . 0 12 . 5 15 . 0 17 . 5 t -0 . 2 0 . 0 y 1 0 . 0 2 . 5 5 . 0 7 . 5 10 . 0 12 . 5 t -0 . 2 0 . 0 y 1 0 . 0 2 . 5 5 . 0 7 . 5 -0 . 2 0 . 0 2 . 5 -0 . 6 -0 . 4 -0 . 2 0 . 0 y 0 After using the specified integrators in training, approximated solutions are computed for each learned vector field f θ using the Scikit-learn implementation of DOP853 [35], which is also used to generate the training data. The error is averaged over M = 10 points and we find what we call the flow error by

.

0

y

ˆ

n

+1

=

Φ

h,f

θ

(

y

n

)

.

Training data: Training data is generated by sampling N 2 = 300 random initial values y 0 requiring that 0 . 3 ≤ ‖ y 0 ‖ 2 ≤ 0 . 6 . The data S N 1 ,N 2 = { ˜ y ( j ) n } N 1 ,N 2 n =0 ,j =0 is found by integrating the initial values with DOP853 with a tolerance of 10 -15 for the following step sizes and number of steps: ( h, N 1 ) = (0 . 4 , 4) , (0 . 2 , 8) , (0 . 1 , 16) . The points in the flow are perturbed by noise where σ ∈ { 0 , 0 . 05 } . Error is measured in M = 10 random points in the flow, within the same domain as the initial values. Furthermore, experiments are repeated with a new random seed for the generation of data and initialization of neural network parameters five times in order to compute the standard deviation of the flow error. The flow error is shown in Figure 6. Additional results are presented in Appendix B.

Neural network architecture and optimization: For all test problems, the neural networks have 3 layers with a width of 200 neurons and tanh ( · ) as the activation function. The algorithms are implemented using PyTorch [36] and the code for performing ISO is a modification of the implementation by [10] 1 . Training is done using the quasi-Newton L-BFGS algorithm [37] for 20 epochs without batching. This optimization algorithm is often used to train physics-informed neural networks [1] and in this setting it proved to yield superior results in comparison to the often used Adam optimizer. Further details are provided in Appendix E.

Results: As observed in Figure 6 and supported by the analytical result illustrated in Figure 4, the MII approach facilitates more accurate training from from noisy data than one-step methods. However, training with multiple integration steps in combination with ISO yields lower error when RK4 is used

Midpoint

RK4

ISO RK4

MIRK4

Exact flow

Given data

Midpoint

RK4

ISO RK4

MIRK4

Exact flow

Given data

10

-

2

10

-

1

10

0

e

(

f

θ

)

Flow error

FPUT,

σ

= 0.05

Time and accuracy

10

-

2

6

×

10

-

3

2

×

10

-

2

3

×

10

-

2

4

×

10

-

2

e

(

f

θ

)

H´ enon-Heiles,

σ

= 0.05

h

= 0.8,

N

1

= 3

h

= 0.4,

N

1

= 6

h

= 0.2,

N

1

= 12

h

= 0.1,

N

1

= 24

10

-

2

2

×

10

-

2

3

×

10

-

2

4

×

10

-

2

e

(

f

θ

)

Double pendulum,

σ

= 0.05

0

50

100

150

200

250

300

350

Training time

Midpoint

RK4

ISO RK4

MIRK4

for the Hénon-Heiles problem and similar performance as MII on the double pendulum. We notice 317 that the SRNN approach, i.e. ISO with Störmer-Verlet, is improved when switching to RK4, which 318 means sacrificing symplecticity to achieve higher order. The results for FPUT stand out in Figure 6, 319 since both ISO methods have large errors here. The roll-out in time of the learned vector fields is 320 presented in Figure 8 in Appendix B, where the same can be observed. As also could be seen here, 321 the FPUT Hamiltonian gives rise to highly oscillatory trajectories, and the errors observed in Figure 322 6 might indicate that ISO is ill-suited for this kind of dynamical systems. 323

324

325

326

327

328

329

Two observations could be made regarding the one-step methods without averaging or ISO. First, it is likely that the midpoint method has weaker performance for large step sizes due to its lower order, compared to both RK4 and MIRK4, despite the fact that it is a symplectic method. The same is clear from Figure 7 in Appendix B, which display the flow error when training on data without noise. Secondly, building on the sensitivity analysis, we observe that MIRK4 consistently attains higher accuracy than RK4, as expected from the Monte-Carlo simulation found in Figure 4.

330

331

332

333

334

335

336

337

338

339

340

341

7 Conclusion

In this work we present the mean inverse integrator, which allows both chaotic and oscillatory dynamical systems to be learned with high accuracy from noisy data. Within this method, integrators of the MIRK class are a key component. To analyse how noise is propagated when training with MII and MIRK, compared to much used explicit methods such as RK4, we developed a sensitivity analysis that is verified both by a Monte-Carlo approximation and reflected in the error of the learned vector fields. Finally, we build on the SRNN [10] by replacing Störmer-Verlet with RK4, and observer increased performance. When also considering the weak performance of the implicit midpoint method, this tells us that order might be of greater importance than preserving the symplectic structure when training HNNs. Both the MIRK methods, the mean inverse integrator and initial state optimization form building blocks that could be combined to form novel approaches for solving inverse problems and learning from noisy data.

Limitations: The experiments presented here assume that both the generalized coordinates q n and 342 the generalized momenta p n could be observed. In a setting where HNNs are to model real and not 343 simulated data, the observations might lack generalized momenta [38] or follow Cartesian coordinates, 344 requiring the enforcement of constraints [17, 39]. Combining approaches that are suitable for data 345 that is both noisy and follow less trivial coordinate systems is a subject for future research. 346

347

348

349

350

351

352

353

354

355

References

Maziar Raissi, Paris Perdikaris, and George E Karniadakis. Physics-informed neural networks: Adeep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational physics , 378:686-707, 2019.

Christopher Rackauckas, Yingbo Ma, Julius Martensen, Collin Warner, Kirill Zubov, Rohit Supekar, Dominic Skinner, Ali Ramadhan, and Alan Edelman. Universal differential equations for scientific machine learning. Aug 2020.

Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. Advances in neural information processing systems , 31, 2018.

Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, 356 Andrew Stuart, and Anima Anandkumar. Fourier neural operator for parametric partial differen357 tial equations. arXiv preprint arXiv:2010.08895 , 2020. 358

359

360

Sam Greydanus, Misko Dzamba, and Jason Yosinski. Hamiltonian neural networks. CoRR , abs/1906.01563, 2019.

361

362

363

364

365

366

367

368

369

Herbert Goldstein, Charles Poole, and John Safko. Classical Mechanics . Addison Wesley, 3 edition, 2001.

Benedict Leimkuhler and Sebastian Reich. Simulating Hamiltonian Dynamics . Cambridge Monographs on Applied and Computational Mathematics. Cambridge University Press, 2005.

Ernst Hairer, Christian Lubich, and Gerhard Wanner. Geometric Numerical Integration: Structure-Preserving Algorithms for Ordinary Differential Equations; 2nd ed. Springer, Dordrecht, 2006.

Christian Offen and Sina Ober-Blöbaum. Symplectic integration of learned Hamiltonian systems. Chaos: An Interdisciplinary Journal of Nonlinear Science , 32(1):013122, 2022.

Zhengdao Chen, Jianyu Zhang, Martin Arjovsky, and Léon Bottou. Symplectic recurrent neural 370 networks. In International Conference on Learning Representations , 2020. 371

Aiqing Zhu, Pengzhan Jin, and Yifa Tang. Deep Hamiltonian networks based on symplectic 372 integrators. arXiv preprint arXiv:2004.13830 , 2020. 373

Marco David and Florian Méhats. Symplectic learning for Hamiltonian neural networks. arXiv 374 preprint arXiv:2106.11753 , 2021. 375

Pengzhan Jin, Zhen Zhang, Aiqing Zhu, Yifa Tang, and George Em Karniadakis. SympNets: 376 Intrinsic structure-preserving symplectic networks for identifying Hamiltonian systems. Neural 377 Networks , 132:166-179, 2020. 378

Aiqing Zhu, Pengzhan Jin, Beibei Zhu, and Yifa Tang. Inverse modified differential equations 379 for discovery of dynamics. arXiv preprint arXiv:2009.01058 , 2020. 380

Takashi Matsubara, Ai Ishikawa, and Takaharu Yaguchi. Deep energy-based modeling of 381 discrete-time physics. Advances in Neural Information Processing Systems , 33:13100-13111, 382 2020. 383

Sølve Eidnes. Order theory for discrete gradient methods. BIT , 62(4):1207-1255, 2022. 384

Elena Celledoni, Andrea Leone, Davide Murari, and Brynjulf Owren. Learning Hamiltonians 385 of constrained mechanical systems. J. Comput. Appl. Math. , 417:Paper No. 114608, 12, 2023. 386

Alvaro Sanchez-Gonzalez, Victor Bapst, Kyle Cranmer, and Peter Battaglia. Hamiltonian Graph 387 Networks with ODE Integrators. 388

Gerhard Wanner and Ernst Hairer. Solving ordinary differential equations II , volume 375. 389 Springer Berlin Heidelberg, 1996. 390

Senwei Liang, Zhongzhan Huang, and Hong Zhang. Stiffness-aware neural network for learning 391 Hamiltonian systems. 2022. 392

Jeff R Cash. A class of implicit Runge-Kutta methods for the numerical integration of stiff 393 ordinary differential equations. Journal of the ACM (JACM) , 22(4):504-511, 1975. 394

K Burrage, FH Chipman, and Paul H Muir. Order results for mono-implicit Runge-Kutta 395 methods. SIAM journal on numerical analysis , 31(3):876-891, 1994. 396

Håkon Noren. Learning Hamiltonian systems with mono-implicit Runge-Kutta methods. arXiv 397 preprint, arXiv:2303.03769 , 2023. 398

Harsh Sharma, Nicholas Galioto, Alex A Gorodetsky, and Boris Kramer. Bayesian identification 399 of nonseparable Hamiltonian systems using stochastic dynamic models. In 2022 IEEE 61st 400 Conference on Decision and Control (CDC) , pages 6742-6749. IEEE, 2022. 401

W. M. G. van Bokhoven. Efficient higher order implicit one-step methods for integration of 402 stiff differential equations. BIT , 20(1):34-43, 1980. 403

404

405

J. R. Cash and A. Singhal. Mono-implicit Runge-Kutta formulae for the numerical integration of stiff differential systems. IMA J. Numer. Anal. , 2(2):211-227, 1982.

406

407

408

409

410

411

412

413

414

415

416

417

418

419

420

421

422

423

424

425

426

427

428

429

430

431

432

433

434

435

436

Sølve Eidnes, Alexander J Stasik, Camilla Sterud, Eivind Bøhn, and Signe Riemer-Sørensen. Pseudo-Hamiltonian neural networks with state-dependent external forces. arXiv preprint, arXiv:2206.02660 , 2022.

Haruo Yoshida. Construction of higher order symplectic integrators. Physics letters A , 150(57):262-268, 1990.

Shaan A Desai, Marios Mattheakis, and Stephen J Roberts. Variational integrator graph networks for learning energy-conserving dynamical systems. Physical Review E , 104(3):035310, 2021.

Daniel DiPietro, Shiying Xiong, and Bo Zhu. Sparse symplectically integrated neural networks. Advances in Neural Information Processing Systems , 33:6074-6085, 2020.

GRWQuispel and Grant S Turner. Discrete gradient methods for solving ODEs numerically while preserving a first integral. Journal of Physics A: Mathematical and General , 29(13):L341, 1996.

Robert I McLachlan, G Reinout W Quispel, and Nicolas Robidoux. Geometric integration using discrete gradients. Philosophical Transactions of the Royal Society of London. Series A: Mathematical, Physical and Engineering Sciences , 357(1754):1021-1045, 1999.

Ander Murua. Métodos simplécticos desarrollables en P-series . PhD thesis, PhD thesis. Valladolid: Universidad de Valladolid, 1995.

Philippe Chartier, Ernst Hairer, and Gilles Vilmart. Numerical integrators based on modified differential equations. Mathematics of Computation , 76(260):1941-1953, October 2007.

J.R. Dormand and P.J. Prince. A family of embedded Runge-Kutta formulae. Journal of Computational and Applied Mathematics , 6(1):19-26, 1980.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. PyTorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems , 32:8026-8037, 2019.

Jorge Nocedal and Stephen J Wright. Numerical optimization . Springer, 1999.

Yuhan Chen, Takashi Matsubara, and Takaharu Yaguchi. Neural symplectic form: learning hamiltonian equations on general coordinate systems. Advances in Neural Information Processing Systems , 34:16659-16670, 2021.

Marc Finzi, Ke Alexander Wang, and Andrew Gordon Wilson. Simplifying Hamiltonian and Lagrangian neural networks via explicit constraints. arXiv preprint arXiv:2010.13581 , 2020.

Enrico Fermi, P Pasta, Stanislaw Ulam, and Mary Tsingou. Studies of the nonlinear problems. 437 Technical report, Los Alamos National Lab.(LANL), Los Alamos, NM (United States), 1955. 438

439

440

441

442

443

444

445

446

447

448

449

450

451

452

453

454

455

456

457

458

459

460

461

E. Hairer, S. P. Nørsett, and G. Wanner. Solving ordinary differential equations. I , volume 8 of Springer Series in Computational Mathematics . Springer-Verlag, Berlin, second edition, 1993. Nonstiff problems.

P. H. Muir. Optimal discrete and continuous mono-implicit Runge-Kutta schemes for BVODEs. Adv. Comput. Math. , 10(2):135-167, 1999.

J. R. Cash and D. R. Moore. A high order method for the numerical solution of two-point boundary value problems. BIT , 20(1):44-52, 1980.

Philippe Chartier. Symmetric Methods. In Björn Engquist, editor, Encyclopedia of Applied and Computational Mathematics , pages 1439-1448. Springer, Berlin, Heidelberg, 2015.

J. R. Cash and A. Singhal. High order methods for the numerical solution of two-point boundary value problems. BIT , 22(2):184-199, 1982.

Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nelson, Eric Jones, Robert Kern, Eric Larson, C J Carey, ˙ Ilhan Polat, Yu Feng, Eric W. Moore, Jake VanderPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, E. A. Quintero, Charles R. Harris, Anne M. Archibald, Antônio H. Ribeiro, Fabian Pedregosa, Paul van Mulbregt, and SciPy 1.0 Contributors. SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python. Nature Methods , 17:261-272, 2020.

Philippe Chartier, Ernst Hairer, and Gilles Vilmart. Numerical integrators based on modified differential equations. Math. Comp. , 76(260):1941-1953, 2007.

Ge Zhong and Jerrold E. Marsden. Lie-Poisson Hamilton-Jacobi theory and Lie-Poisson integrators. Phys. Lett. A , 133(3):134-139, 1988.

Takashi Matsubara and Takaharu Yaguchi. FINDE: Neural differential equations for finding 462 and preserving invariant quantities. arXiv preprint, arXiv:2210.00272 , 2022. 463

Peer reviews:
comments: SUMMARY:
This paper introduces a new integration method (mean inverse integrator) for learning dynamics from noisy data. Experiments on Hamiltonian systems show the effectiveness of the proposed method.

STRENGTHS:
- The problem of learning physical dynamics from noisy data is an interesting one.
- It combines techniques from the field of numerical computation with machine learning.

WEAKNESSES:
- The usefulness and significance of the proposed method is not clear.
- I feel that comparative experiments are not sufficient.
- It is not clear what advantages the proposed method has over the naive noise handling method.

QUESTIONS:
- To handle noisy data, the most common method is to introduce an observation model (e.g., Normal distribution) and learn the noise variance by maximum likelihood estimation. Can you tell us what advantages the proposed method has over such naive methods?

- It would be good to clarify situations that can only be solved by the proposed method and emphasize the significance of this paper. For example, when using Gaussian likelihood, the noise variance is usually assumed to be constant. Can the proposed method relax this assumption? In other words, can the proposed method be applied to cases where the noise variance depends on time and state?

- A thorough comparison with the latest methods (e.g., [9-12, 15]) would be helpful.

- The font size in Fig. 5 and 6 is too small and should be enlarged.

LIMITATIONS:
It would be good to add a careful discussion of the advantages and disadvantages of the proposed method.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper investigates mono-implicit Runge--Kutta (MIRK) methods for learning dynamical systems from data. In particular, MIRK methods can be made explicit by introducing the external data into the solver step itself, leading to a more efficient integrator while keeping favorable stability, symmetry, and symplecticity properties. To handle noisy data, the paper proposes the \"mean inverse integrator\" as an efficient way to average multiple trajectories and learn meaningful vector fields from these. The methods are demonstrated in multiple numerical experiments.\n\nSTRENGTHS:\nThe paper is well-written, the proposed method is presented very clearly, and the main claims made are well-supported.\n\nWEAKNESSES:\nI find the presentation of the explicit Runge--Kutta (ERK) baseline a bit confusing, and in particular counter-intuitive of my current understanding of the usage of ERK methods for inverse problems in the context of ODEs. Concrete questions are below, in \"Questions\".\nThis criticism also extends to insufficiently detailed baselines, and potentially to insufficient baselines overall.\n\nQUESTIONS:\nSee also \"Weaknesses\". Notably: \n1. How exactly is the RK4 / ISO+RK4 baseline defined?\n2. Why is (6) _the_ appropriate loss function? I would rather assume something of the type $\\sum || \\tilde{y}_n - \\hat{y}_n ||$, where $\\hat{y}_n$ is a state estimate computed by the method; for instance by applying a numerical solver to simulate a whole trajectory. In particular, extrapolating exactly from the last data point seems to be a strong restriction, in particular if the data point can be very noisy. I would not consider this \"the inverse problem formulation\", but rather a specific choice that is made, geared towards the specific algorithm of interest.\n3. Related to the last question: Could you commend on the standard least-squares approach of integrating the whole trajectory from some initial value (estimate) with RK, as opposed to the described method of just integrating from one data point to the next? The latter seems to be related to \"multiple shooting\"-type approaches which, while they can be very beneficial, seems to be far from the standard for neural ODEs. Though, to the best of my knowledge, even in multiple shooting it is common to extrapolate some state estimates $\\hat{y}$; though part of the objective is the closeness of these estimates to the data, and they are initialized on the data.\n\nOther remarks:\n- l. 201: \"evaluationg the vector filed in $\\tilde{y}_n$, as is done in all explicit RK methods\": To the best of my understanding this is not done in all explicit RK methods, but only in multiple shooting-type approaches (and even there this might not hold). A standard least-squares approach simulates a whole trajectory and then computes an L2-loss---and thus does not need to evaluate $f$ on the data points themselves.\n- l. 129: $\\tilde{y}$ is missing an index\n- Related work: The write-up is very helpful; but I think it could be further improved by highlighting better how the proposed paper relates to these methods and highlight similarities and differences.\n- l. 72-73: I agree with the sentiment of the statement, but I think the strict _necessity for numerical integrators_ is somewhat disputed in the literature. Notably, \"gradient matching\" methods compute first an interpolant, and then adjust the vector field to match the known derivative of the interpolant---this circumvents using RK or any related numerical method, which is why such methods often claim to be \"simulation-free\".\n\nLIMITATIONS:\nThe authors address limitations in a dedicated section, which is much appreciated.\n\nOne additional limitation that is not explicitly mentioned in the section that comes to mind is the influence of the noise on the data: Since the data is explicitly included into the numerical solver, as opposed to being just part of some L2 loss to guide some estimated trajectory, I would expect that for very noisy data other methods might be preferrable (in particular a least-squares approach with RK from a learned initial value).\n\nAnother potential limitation: I assume that this method is not able to be a plug-in replacement in some latent neural ODE setting, where the ODE trajectory is not observed directly but only in some transformed space, e.g. when having video data and modeling an ODE in latent space? This is of course far from a trivial setting and I do not expect that such specific scenarios need to be mentioned explicitly; but the necessity of actual trajectory observations, as opposed to partial or non-linear observations, could indeed be another limitation.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

comments: SUMMARY:
The authors present a novel method, Mean Inverse Integrator (MII), used to aggregate data generated through numerical integration of the vector field characterizing Hamiltonian systems. In particular, the objective is to improve the training of Hamiltonian Neural Networks (HNNs) when the data used is noisy.
The authors train HNNs on two tasks, using data generated by different methods of numerical integration.

STRENGTHS:
- The paper is well written and complete. My background on the topic is limited, and so I was happy that Sections 3 and 4 were included.
- The theoretical analysis is thorough and convincing.

WEAKNESSES:
- I am not sure how much the paper fits with the themes of NeurIPS, but it should still be of interest to some of the audience.

QUESTIONS:
N/A

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe presented work considers a novel class of integrators that are used to train Hamiltonian Neural Networks (HNNs). This class is called mean inverse integrator and it averages the trajectories from mono implicit RK methods (MIRK) to obtain higher accuracy. The authors provide theoretical results on how MIRK convergence suffers from noisy data. Also, experimental results on test dynamical systems illustrate the performance of the proposed approach.\n\nSTRENGTHS:\nThe paper presents a comprehensive introduction to the different types of integrators and smoothly introduces the new one. The novel integrator is investigated w.r.t. the relations with classical RK methods (Theorem 4.4 and Proposition 4.3). Also, theoretical results on the robustness w.r.t. noise in the data are presented.\n\nWEAKNESSES:\n1) the experimental results show that the proposed approach is not always better than the alternative methods in terms of accuracy, but always slower in terms of runtime. So the use cases for the MII-related approaches should be stated more strictly.\n2) the manuscript is mostly about integrators theory rather than learning dynamical systems from data. Revision of the structure and shift of the focus from integrators to its application in learning HNN can be very helpful\n3) the connection of the proposed integrator and its usage in forward or in backward passes is ignored. Thus, it is unclear from the text how this new integrator should be incorporated into the existing pipeline of training HNN\n4) experiments are performed only for systems, where the states are small dimensional. The scalability and robustness w.r.t the large dimension of states are unclear\n5) since no batching is used for training, it indicates that a large amount of data are not tested yet and all available data are fitted in the GPU memory. It will be interesting to study how the stochasticity of the data and corresponding gradient estimates affect the convergence of the training curves.\n\nQUESTIONS:\n1) how do different integrators affect the way to perform backward passes? Please provide a detailed explanation and complexity analysis\n2) even for 4d states the proposed approach (MII MIRK4) shows the worst runtime compared to the alternatives (Fig. 6). So, how do the MII-related integrators scale w.r.t. the state dimension? How are they applicable to learning dynamical systems with high-dimensional states?\n3) all experimental results are shown for fixed noise level $\\sigma= 0.05$, but authors highlight that the proposed approach is robust to noisy data. So please show how the test accuracy depends on the noise level in training data for different integrators \n4) please provide analytical forms of the considered test dynamical systems for the reader's convenience\n\nLIMITATIONS:\nThe authors explicitly state the limitations of the proposed method.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel method aimed at learning the vector field of a dynamical system. The proposed approach is called the mean inverse integrator, which utilizes a neural network (e.g., SRNN) to accurately estimate the integrator in the presence of noisy data. The authors provide theoretical insights into the sensitivity of both the one-step target function and the mean inverse integrator to data noise. Additionally, the paper presents empirical evaluations by comparing the method to five different types of integrators.\n\nSTRENGTHS:\n- The paper effectively conveys its ideas and arguments with clarity.\n\n- The research addresses an important question and presents a new approach to handle noise when learning data dynamics.\n\n- Theoretical analysis shows how the proposed method and the baseline approach respond to noise, contributing to a better understanding of their performance.\n\n- Empirical results demonstrate the effectiveness of the proposed method, showing significant advantages over the baseline approaches.\n\nWEAKNESSES:\n- The paper lacks clarity on why the mean inverse integrator yields more accurate estimates compared to the one-step baseline.\n\n- Scalability to high-dimensional systems is not explored, as current experiments primarily focus on trivial test cases. It would be beneficial to investigate potential challenges arising from increased computational complexity, instability of estimation in the presence of chaotic dynamics, and provide examples demonstrating the method's efficacy.\n\nQUESTIONS:\nQ1: The paper would benefit from running experiments on complex, high-dimensional data to showcase the method's capabilities.\n\nQ2: It remains unclear whether the proposed method can generalize to more general dynamical systems beyond Hamiltonian systems.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}