Title: HHD-GP: Incorporating Helmholtz-Hodge Decomposition into Gaussian Processes for Learning Dynamical Systems
Venue: NeurIPS 2024
Accepted: yes
Keywords: Gaussian process

Helmholtz-Hodge decomposition

dynamical system
Abstract: Machine learning models provide alternatives for efficiently recognizing complex patterns from data, but the main concern in applying them to modeling physical systems stems from their physics-agnostic design, leading to learning methods that lack interpretability, robustness, and data efficiency. This paper mitigates this concern by incorporating the Helmholtz-Hodge decomposition into a Gaussian process model, leading to a versatile framework that simultaneously learns the curl-free and divergence-free components of a dynamical system. Learning a predictive model in this form facilitates the exploitation of symmetry priors. In addition to improving predictive power, these priors make the model indentifiable, thus the identified features can be linked to comprehensible scientific properties of the system. We show that compared to baseline models, our model achieves better predictive performance on several benchmark dynamical systems while allowing physically meaningful decomposition of the systems from noisy and sparse data.

HHD-GP: Incorporating Helmholtz-Hodge Decomposition into Gaussian Processes for Learning Dynamical Systems

Abstract

Machine learning models provide alternatives for efficiently recognizing complex patterns from data, but the main concern in applying them to modeling physical systems stems from their physics-agnostic design, leading to learning methods that lack interpretability, robustness, and data efficiency. This paper mitigates this concern by incorporating the Helmholtz-Hodge decomposition into a Gaussian process model, leading to a versatile framework that simultaneously learns the curl-free and divergence-free components of a dynamical system. Learning a predictive model in this form facilitates the exploitation of symmetry priors. In addition to improving predictive power, these priors make the model indentifiable, thus the identified features can be linked to comprehensible scientific properties of the system. We show that compared to baseline models, our model achieves better predictive performance on several benchmark dynamical systems while allowing physically meaningful decomposition of the systems from noisy and sparse data.

1 Introduction

Adynamical system describes how the state of a system evolves over time [64]. Data-driven modeling of dynamical systems has become a fundamental task in many modern science and engineering applications, such as physical emulation [21] and robotics control [13]. Mathematically, a dynamical system is often a set of first-order ordinary differential equations (ODEs) or, equivalently, a smooth vector field on a manifold [28]. Given the functional form of ODEs, classical data-driven methods typically involve optimizing their parameters [52]. However, for many complex systems it is practically difficult to determine the form of the ODEs governing their dynamics.

Recent advances in machine learning (ML) focus on the use of neural networks [11] and nonparametric Bayesian models [63, 27, 26] for the black-box approximation of vector fields. Despite the rich expressive power, the physics-agnostic design of these models hinders their performance when applied to physical systems. To address this issue, a popular approach is to develop ML models that incorporate strong physical priors as inductive biases. Such prior knowledge commonly stems from basic physical principles related to certain differential invariants of vector fields. For example, in scenarios of learning Hamiltonian systems [23, 67, 54] and incompressible fluid dynamics [73, 33], ML models are constructed to learn divergence-free (div-free) vector fields, as a consequence of conservation laws of energy or mass. By constraining the solution space, the powerful physical principles effectively improve the extrapolation performance of ML models and enhance their interpretability. However, enforcing the model's behavior to adhere to centain fundamental physical

principles also restricts the applicability of the model. For example, a div-free vector field fails to describe a dynamical system with dissipation, but real-world dynamical systems always suffer from non-negligible dissipation.

To develop a predictive model covering more dynamical systems, we explore supplementing the divfree vector field with a curl-free vector field. This is inspired by the Helmholtz-Hodge decomposition (HHD) [3, 44, 8], which states that any sufficiently smooth vector field can be expressed as the sum of a curl-free vector field and a div-free vector field. HHD is widely used in the study of Navier-Stokes equations [45, 20, 9], but in this work we explore its connections with more general dynamical systems. For example, as shown in Fig. 1, HHD can be used to characterize the dynamics of a dissipative Hamiltonian system, where the div-free component represents its conservative dynamics and the curl-free component describes the friction-induced dissipation. In addition to dissipative Hamiltonian systems, HHD is also widely used to study chaotic systems and ocean dynamics. The connection between HHD and these dynamical systems is described in more detail in Appendix A.

In this work, we construct Gaussian process (GP) priors for div-free and curl-free dynamics separately, resulting in an additive GP model in the form of HHD for learning vector fields of dynamical systems. Our resulting HHD-GP allows one to leverage more prior knowledge than modeling the system as a whole. In particular, we investigate its potential in exploiting priors of symmetries, motivated by the observation that the div-free and curl-free components of a dynamical system usually exhibit more symmetry properties than the system itself.

1

0

1

q

1.5

1.0

0.5

0.0

0.5

1.0

1.5

p

(a) dynamics

=

1

0

1

q

1.5

1.0

0.5

0.0

0.5

1.0

1.5

p

(b) div-free part

+

1

0

1

q

1.5

1.0

0.5

0.0

0.5

1.0

1.5

p

(c) curl-free part

For example, the damped mass-spring system in Fig. 1(a) exhibits odd symmetry, but its div-free (Fig. 1(b)) and curl-free (Fig. 1(c)) components additionally present rotation and translation symmetry, respectively. Therefore, we further build symmetry-preserving div-free GPs and curl-free GPs by exploiting closure of GPs under linear transformation. The symmetry prior not only improves the predictive performance of HHD-GP, but also makes it identifiable, thus identified div-free and curl-free features can be physically-meaningful. In particular, the learned div-free features are closely related to the energy of dynamical systems.

The main contributions of this work are summarized as:

We introduce a GP prior (called HHD-GP) for learning the Helmholtz-Hodge decomposition of a general dynamical system.

We construct a symmetry-preserving extension of the HHD-GP that can learn physically meaningful representations of dynamical systems.

Experiments on several benchmark systems show that our model can both accurately predict their dynamics and HHDs from sparse, noisy observations.

2 Related work

Learning with div/curl-free constraints Div-free vector fields are a focal point of mathematical physics and have been well exploited by machine learning models to learn system dynamics, with the most well-known examples being neural networks (NNs) [23, 67] and GPs [54, 59] for learning Hamiltonian dynamics. All Hamiltonian vector fields are div-free, but not vice versa. To learn more general div-free vector fields as solutions of the continuity equation, [57] introduced an NN architecture to parameterize a universal representation of div-free vector fields. Based on the same representation, we constructed div-free kernels for GPs from matrix-valued kernels, which is an extension of the method of constructing div-free kernels from scalar kernels [50, 41, 75]. According to Maxwell's equations, div-free kernels were combined with curl-free kernels in [72, 71] to model magnetic fields. They assumed direct access to noisy observations of the div-free and curl-free components separately, whereas in this work, our goal is to recover the individual components from noisy observations of their sum. A similar prediction problem was studied in [6], they combined 2D

div-free and curl-free GP priors in the form of HHD to reconstruct planar ocean current fields, and recovered their divergence. Their model has the same formulation as ours when the dimension of HHD-GP is two, but we further developed a symmetry-preserving extension of HHD-GP to solve its non-identifiability problem. Another similar work is Dissipative Hamiltonian neural network (D-HNN) [22], which compensated HNN [23] with a curl-free part to model both conservative and dissipative dynamics simultaneously, but D-HNN is only applicable to dissipative Hamiltonian systems due to its construction.

Learning with symmetry Symmetries are another important aspect of priors that can be incorporated into machine learning models. Motivated by the success of the translation-invariant NNs [38], network architectures with symmetries to more general transformations have been proposed, such as steerable CNNs [74, 10] and graph NNs [46, 61] equivariant to Euclidean symmetries. They achieved great success in improving generalization of the models. With the same motivation, kernel methods incorporating symmetries have also been developed. To make predictions invariant to transformations of inputs, [24] constructed kernels using Haar integration. And based on similar integration technique, [56] developed the Group Integration Matrix Kernel (GIM-kernel) to learn equivariant functions, which was later used by [58] to learn dynamical systems with symmetries. In this work, we constructed GP models to impose Euclidean symmetries to div/curl-free vector fields, which to the best of our knowledge has not been explored by the machine learning community.

3 Background

3.1 HHDand Problem Setup

We consider an autonomous system governed by the following ODEs:

which defines a vector field by assigning a vector f ( x ) ∈ R n to every state x ∈ R n . We assume that the vector field f ∈ L 2 ( R n , R n ) is smooth, and any such vector field can be decomposed into the sum of a curl-free vector field f curl : R n → R n ( ∇∧ f curl = 0 , ∀ x ∈ R n ) and a divergence-free vector field f div : R n → R n ( ∇· f div = 0 , ∀ x ∈ R n ), according to the Helmholtz-Hodge decomposition (HHD) [44, 8]. In this work, we are interested in learning f , f curl and f div simultaneously from a collection of noisy observations denoted by D = { ( x i , y i ) } m i =1 , with a noisy observation y i at a state x i given by

where an additive noise ϵ ∈ R n follows a zero-mean Gaussian distribution defined by a covariance matrix Ω = diag ( σ 2 1 , . . . , σ 2 n ) ∈ R n × n modeling noise variance in each output dimension.

3.2 Vector-valued GP Model

We are interested in using Gaussian processes (GPs) to infer unknown vector fields. A GP is a stochastic process commonly used as a distribution for functions, assuming that any finite number of function values has a joint Gaussian distribution [53]. To learn an unknown vector field f : R n → R n , we assume a vector-valued GP prior as f ( x ) ∼ GP ( 0 , κ ( x , x ′ )) , where the mean of the function values is set to zero, and the covariance is captured by a matrix-valued kernel κ : R n × R n → R n × n , whose ( i, j ) -th entry expresses the correlation between the i -th dimension of f ( x ) and the j -th dimension of f ( x ′ ) . In the GP framework, the kernel controls the properties of possible functions under a GP, leading to various efforts of problem-specific design of kernels [76, 16, 18]. And to be a valid covariance function, the kernel should be symmetric and positive semidefinite [78].

GPs provide a Bayesian non-parametric approach for solving regression tasks. According to the GP prior, function values at inputs X = [ x 1 , . . . , x m ] T are jointly distributed as N ( f ( X ) ; 0 , K ) , where K = [ κ ( x i , x j ) ∈ R n × n ] m i,j =1 is a block-partitioned covariance matrix. Then the marginal likelihood of the noisy observations Y = [ y 1 , . . . , y m ] T given by Eq. 2 can be calculated by

where K ∗ = K + Σ , Σ = Ω ⊗ I m is a diagonal matrix whose elements are the variance of the observation noise. Training a GP model refers to maximizing the log of Eq. 3 to optimize the kernel parameters and the noise variance. Then by conditioning on these observations using Bayes' rule, the predictive posterior for a new state x ∗ is still Gaussian with its mean µ and variance v given by

where k = [ κ ( x 1 , x ∗ ) , . . . , κ ( x m , x ∗ )] T ∈ R mn × n . The derived mean function is used for regression results, and the associated variance quantifies prediction uncertainty. Due to their nonparametric nature, GPs can self-adapt to the complexity of the target function based on the data provided, without being restricted to specific parametric forms.

4 HHD-GP Model

We consider the problem of learning a continuous-time dynamical model in the form of HHD (Eq. 1) with GPs. HHD points out the prevalence of an additive structure in dynamical systems, so the key idea here is to exploit two GPs to model f curl and f div , respectively, f curl ∼ GP ( 0 , κ curl ) , f div ∼ GP ( 0 , κ div ) . Then the sum of these two GPs results in a new GP modeling the dynamical system f , with a new kernel function defined as the sum of the curl-free and divergence-free ones. And the additive kernel κ = κ curl + κ div inherits the symmetric and positive-semidefinite properties of κ curl and κ div , so the GP predictor (Eq. 4) is valid for the additive GP model. And the additivity of the kernels implies the additivity of GP means, so the mean function µ in Eq. 4 can be split into a curl-free part µ curl and a divergence-free part µ div , and we have

where K ∗ = K curl + K div +Σ . It can be seen that the effects of f curl and f div can be treated as observation noises for each other, so their prediction variances at x ∗ are obtained by

Consequently, observations of f ( x ) given by Eq. 2 can be used to make predictions for its hidden components f curl and f div . So we then construct GPs with realizations in the space of curl-free and div-free vector fields.

In the following parts, we construct the kernels κ curl and κ div from the representations of f curl and f div respectively, exploiting closure of GPs under linear transformation. Let R = R [ ∂ x 1 , . . . , ∂ x n ] be the polynomial ring in the partial derivatives, and L ∈ R a × b be a matrix of differential operators acting on functions g : R n → R b distributed as GP ( µ ( x ) , κ ( x , x ′ )) . Then, the transformation of g under L is again distributed as a GP with

where L x and L x ′ denote the operation of L on the first and second argument of κ ( x , x ′ ) , respectively [29, 1, 36, 7]. To make L x κ ( x , x ′ ) L T x ′ a valid covariance function, its underlying kernel κ ( x , x ′ ) must be twice differentiable in R n . See Appendix B for details on linear operations on GPs.

4.1 Curl-free Kernel

The gradient operator ( ∇ := [ ∂ x 1 , . . . , ∂ x n ] T ∈ R n × 1 ) defines a surjective mapping from the space of smooth scalar fields to the space of curl-free vector fields [14], so f curl can be represented by f curl = ∇ V , where V ∈ C ∞ ( R n , R ) is called the scalar potential of f curl . Since the gradient operation defines a linear transformation, if a GP with a scalar kernel κ V is assumed on V , the distribution of f curl is again a GP. According to Eq. 7, the curl-free GP over f curl is given by

where κ curl : R n × R n → R n × n is a matrix-valued kernel constructed by the Hessian of the scalar kernel κ V , consisting of all second-order partial derivatives in x and x ′ , with the entry in the i -th row and j -th column given by

By this construction, if κ V induces a GP with realizations dense in C ∞ ( R n , R ) , the set of realizations of GP ( 0 , κ curl ) is dense in the space of curl-free vector fields, because a surjective mapping maps dense sets to dense sets.

4.2 Divergence-free Kernel

Adiv-free vector field can be constructed from a skew-symmetric matrix field [5, 32, 57]. Specifically, let A : R n → R n × n be a skew-symmetric matrix-valued function, then a div-free vector field f div can be represented by taking row-wise divergence of A , i.e. ,

where A i : R n → R n is the i -th row of A . The skew-symmetric matrix field A of size n × n can be compactly represented by m = n ( n -1) / 2 2 scalar functions u ij ∈ C ∞ ( R n , R ) :

where Φ ij ∈ R n × n is a matrix with its ( i, j ) -th entry equal to 1, ( j, i ) -th entry equal to -1, and all other entries equal to 0. Then, the div-free vector field given by Eq. 9 can be reformulated as a linear transformation:

where ψ ij = Φ ij ∇ ∈ R n × 1 is a column vector obtained by linearly transforming the gradient operator. The m column vectors ψ ij are aggregated in a matrix Ψ ∈ R n × m , and the m corresponding scalar functions u ij are collected in a vector-valued function u : R n → R m . Ψ[ · ] is a matrix of linear differential operators, so to use a GP to model f div , we can proceed by assuming a GP prior over u ∼ GP ( 0 , κ u : R n × R n → R m × m ) , then based on the closure of GPs under linear transformation (Eq. 7), the GP prior over f div (Eq. 11) is constructed by

where κ u is a scalar-valued kernel for two dimensional systems ( n = 2 , m = 1 ), and is a matrixvalued kernel for n > 2 . Notice that this div-free kernel can be treated as a generalization of the div-free kernel derived from a scalar kernel [50, 41, 75], which can be recovered by setting κ u = κ · I . Therefore, it is more expressive and flexible. Theoretically, the GP model given by Eq. 12 can be used to approximate arbitrary div-free vector fields, because the representation given by Eq. 9 is shown to be maximally expressive ( i.e. universal) in [57].

4.3 Identifiability and Constraints

With the curl-free and div-free kernels, our objective is to learn physically interpretable representations of a dynamical system based on the HHD-GP model. However, the HHD is always not unique due to the existence of harmonic components f harm (vector fields satisfying both ∇∧ f harm = 0 and ∇ · f harm = 0 , e.g. , constant vector fields). For the HHD of a dynamical system with the true functional decomposition f ∗ curl and f ∗ div ,

is a valid HHD for arbitrary f harm , which thus makes the HHD-GP model non-identifiable, meaning that from the same training data, we may learn different decompositions giving the same predictions. This is not desirable because we expect the learned dynamical model to be interpretable: the curl-free and div-free components f curl , f div are physically meaningful.

To mitigate the identifiability problem in additive regression models, an effective method is to impose constraints on their component models [16, 17, 47, 42]. The imposed constraints can affect the decomposition results of the additive models. Therefore, to ensure that HHD-GP can produce a scientific decomposition, we desire constraints that respect the inherent characteristics of dynamical systems. And, as another primary goal, incorporating prior knowledge of a system into a GP model can also improve its prediction accuracy and learning efficiency. Therefore, in the next section, we present how to impose symmetry-based constraints on the curl-free and div-free GP models.

5 Symmetry Constraints

5.1 Equivariance and Invariance

Symmetry is a fundamental geometric property prevalent in dynamical systems in natural [39], and is usually described by the concept of equivariance and invariance:

Definition 5.1 (Equivariance and Invariance) . Let G be a group acting on R n through a smooth map L : G × R n → R n . The dynamical system f : R n → R n is said to be G -equivariant if

where L g ( x ) := L ( g, x ) , and J L g denotes the Jacobian matrix of L g . Then, G is termed the symmetry group of the dynamical system. In particular, if J L g is the identity matrix ( i.e. , f ◦ L g = f , ∀ g ∈ G ), the dynamical system f is said to be G -invariant.

From the equivariance condition (Eq. 14) of the vector field, it follows the system's trajectory commutes with the action map. For vector fields on R n , the symmetry group G is commonly a subgroup of the Euclidean group E ( n ) , which comprises all intuitive geometric transformations in R n (see Appendix C for a brief introduction). The symmetry constraints refer to that we expect the learned curl-free and div-free vector fields to be G -equivariant. With the representation of their GP models, we demonstrate that the symmetries can be enforced by designing suitable kernels.

5.2 Symmetry-preserving Curl-free GP

The curl-free GP (Eq. 8) is constructed by transforming another GP over a potential function, implying that we can impose constraints of symmetry on the curl-free GP by designing a suitable potential GP. Therefore, we start by exploring how to construct potential functions to obtain curl-free vector fields with the desired equivariance. As expected, the following theorem holds:

Theorem 5.2. Let G be a Euclidean group or its subgroup, and let V : R n → R be a G -invariant scalar function. Then, the curl-free vector field f curl : R n → R n defined by f curl ( x ) = ∇ V ( x ) is G -equivariant.

See Appendix D.1 for the proof. Theorem 5.2 shows that a G -invariant scalar potential V can yield a G -equivariant gradient field, indicating that if any realization V of GP (0 , κ V ) is constrained to be G -invariant, then its pushforward GP over ∇ V ∼ GP ( 0 , ∇ x κ V ∇ T x ′ ) can induce the space of G -equivariant curl-free vector fields.

It is obvious that a G -invariant scalar potential V can be constructed by integrating some non-invariant function h : R n → R over the symmetry group: V = ∫ G ( h ◦ L g ) dg , where the measure dg is called Haar measure , which exists for locally compact topological groups and finite groups. Therefore, by assuming that h is distributed as h ∼ GP (0 , κ h ) , we can construct the GP prior over the G -invariant scalar potential as V ∼ GP (0 , κ V ) , with its kernel κ V given by

This kernel is called the Haar-integration kernel [24]. While it provides a general method for constructing kernels for G -invariant functions, the double integral can be computationally expensive. If the kernel κ h is invariant to any g ∈ G in the sense that κ ( x , x ′ ) = κ ( L g ( x ) , L g ( x ′ )) 3 , a complexity reduction of Eq. 15 by one square-root can be performed by

where |G| = ∫ G dg , and it denotes the cardinality of G when the group is finite.

5.3 Symmetry-preserving Divergence-free GP

To incorporate the equivariance condition (Eq. 14) into realizations of the div-free GP (Eq. 12), we construct the skew-symmetric matrix field A from a vector-valued function. Specifically, given a smooth vector field v ∈ C ∞ ( R n , R n ) , A is constructed by A = J v -J T v , where J v denotes the Jacobian of v with its ( i, j ) -th entry given by ∂v i /∂x j . Then the component function u ij in Eq. 10 is given by u ij = ∂v i /∂x j -∂v j /∂x i . By this construction, the symmetry of the div-free vector field f div is governed by the symmetry of its vector potential v . In particular, a G -equivariant v can produce a G -equivariant f div , and it is formalized in the following theorem:

Theorem 5.3. Let G be a Euclidean group or its subgroup, and let v : R n → R n be a G -equivariant vector field. Then the divergence-free vector field f div = [ ∇· A 1 , . . . , ∇· A n ] T is G -equivariant, where A i denotes the i -th row of the skew-symmetric matrix-valued function A = J v -J T v .

See Appendix D.2 for the proof. By this theorem, we then proceed by assuming a GP prior over the vector potential v ∼ GP ( 0 , κ v ) , and to constrain v to be G -equivariant, we build its kernel κ v ∈ R n × n in the form of the Group Integration Matrix kernel (GIM-kernel) [56, 55], which is constructed by:

where κ is some arbitrary scalar-valued kernel satisfying κ ( x , x ′ ) = κ ( L g ( x ) , L g ( x ′ )) for all g ∈ G . The GIM-kernel spans a Reproducing Kernel Hilbert Space (RKHS) of functions with the desired equivariance [55]. So we can then use κ v (Eq. 17) to construct the GP prior over u in Eq. 11, where the covariance between components u ij and u kq is given by

Finally, this matrix-valued kernel κ u is transformed by Eq. 12 to construct the div-free GP, of which the realizations are guaranteed to be G -equivariant div-free vector fields, according to Theorem 5.3.

6 Experiments

We evaluated the performance of our proposed method in several representative physical systems. Through the experiments, we found that our model can not only accurately capture the system dynamics, but also predict correct decompositions.

6.1 Learning Dissipative Hamiltonian Dynamics

We first evaluated our method on a damped mass-spring system and a damped pendulum. Their governing equations are detailed in Appendix E.1. We generated the training data { ( x , ˙ x ) } by randomly sampling states x in their phase space, and each of their derivative observations ˙ x is corrupted by an additive Gaussian noise with a standard deviation of 0 . 05 .

The models are first evaluated in terms of learning ODEs. Specifically, our evaluation focused on the accuracy of the models in predicting state derivatives, as measured by the root mean squared error (RMSE), and their ability to accurately predict state trajectories over time, as indicated by the valid prediction time (VPT). In addition to these metrics for evaluating the regression results, we further use the mean negative log likelihood (MNLL) to evaluate the prediction uncertainty provided by the GP models. These evaluation metrics and the generation of test data are detailed in Appendix F. We compared our models, HHD-GP and its symmetry-preserving extension, SPHHD-GP, with Dissipative Hamiltonian neural network (D-HNN) [22], GPs involving div-free kernels for learning Hamiltonian dynamics (Div-GP) [54, 59], and GPs with Group Integration Matrix Kernels (GIM-GP) [56] that can incorporate symmetries. Another baseline is GPs with independent kernels (Ind-GP), which model each dimension of a dynamical system with an independent scalar GP. See Appendix G for the implementation details of these models.

The results for 20 training data are shown in Table 1, and the results for an increasing number of training data are provided in Appendix J.1. The performance of Div-GP is limited because it can only model conservative dynamics. HHD-GP improves its performance by compensating with a curl-free kernel, which offsets the strong inductive bias imposed by the div-free kernel. And the performance of HHD-GP is better than that of another HHD-based model, D-HNN, because the low data efficiency of NNs makes it hard for D-HNN to capture dynamics using noisy and sparse training data, so actually the performance of D-HNN is worse than either of the GP methods. As another model without inductive bias, Ind-GP performs similarly to HHD-GP in most cases. Then, by incorporating symmetry priors into GPs, GIM-GP performs better than the above models but not as well as SPHHDGP, because learning in the form of HHD allows the model to exploit more implicit symmetries in the dynamical systems. SPHHD-GP performs best overall in learning ODEs. Appendix J.2 presents the plots of trajectory predictions for each system.

Another advantage of our model is that it can decompose the dynamics into its div-free and curl-free components. According to Eq. 19, the div-free component can be used to recover the system's Hamiltonian, as long as the correct form of HHD is learned by our model. To show this, we evaluated our model by another task: we predicted the Hamiltonian ˆ H ( x t ) (energy) along the system's trajectory { x t } . For HHD-GP and SPHHD-GP, the Hamiltonian is predicted from a joint GP prior over the Hamiltonian and its underlying dynamics (see Appendix G.7.1 for details). Fig. 2 shows the

0.0

2.5

5.0

7.5

10.0

12.5

15.0

Time

0.0

0.2

0.4

0.6

0.8

Energy

Mass-spring System

Ground Truth

D-HNN

HHD-GP

SPHHD-GP

0.0

2.5

5.0

7.5

10.0

12.5

15.0

Time

1

2

3

4

5

6

Energy

Pendulum

Ground Truth

D-HNN

HHD-GP

SPHHD-GP

predicted energy evolution of the two systems, where we can find that D-HNN and HHD-GP fail to provide physically plausible results, because the energy should continue to decrease due to friction, but they provide oscillating predictions, along with significant variances. In contrast, predictions of SPHHD-GP are highly accurate and closely aligned with the true values. One reason is that the symmetry priors used by SPHHD-GP improves the generalization performance of the model, but more importantly, the priors solve the problem of non-identifiability suffered by HHD-GP and D-HNN. See Appendix J.3 for an visualization of predicted decompositions, which shows that although the models capture similar system dynamics, they can learn completely different decompositions.

Consistency is a necessary condition for a learning model to be identifiable and refers to the property that its parameter estimates should converge to the true values as the amount of data increases [19, 40]. Therefore, to further explore the non-identifiability problem, we provide the RMSE of energy prediction with an increasing number of training data in Fig. 3. The result shows that HHD-GP and D-HNN always generate significant errors in predicting energy evolution, meaning that the model parameters cannot converge as the amount of data

0

100

200

300

400

Number of data

10

3

10

2

10

1

RMSE of Energy

Mass-spring System

D-HNN

HHD-GP

SPHHD-GP

0

100

200

300

400

Number of data

10

2

10

1

10 0

RMSE of Energy

Pendulum

D-HNN

HHD-GP

SPHHD-GP

increases. In contrast, the energy errors of SPHHD-GP exhibits a decreasing trend, which reflects

the consistency of SPHHD-GP. In addition to this empirical validation, in Appendix H we provide a theoretical verification that for the dynamical systems in our experiments, the non-identifiability of our model ( i.e. non-uniqueness of HHD) is solved through forced symmetries.

Accurate prediction of energy demonstrates the interpretability of learned div-free features. To further show the interpretability of curl-free features, in Appendix J.4 we added experiments of adapting the learned models to predict dynamics with unseen friction coefficients. And Appendix J.5 presents some additional experimental results of increasing the noise in training data, showing that SPHHD-GP maintains high gain in large noise compared to the baselines.

6.2 Learning Chaotic dynamics

Learning the correct HHD also has important implications for studying chaotic systems. [60] stated that the div-free component of a chaotic system is always orthogonal to the gradient of its energy function, i.e. , ∇ H T f div ( x ) = 0 . This energy function can be used to synchronize two chaotic systems [60], analyze their stability [77], and design energymodulated controllers [43]. This experiment aims to learn the Chua circuit [49], which is a chaotic system with applications in various fields. Its governing equa-

tion and the HHD are detailed in Appendix E.2. The models are trained on 100 randomly sampled data, corrupted with additive Gaussian noise (standard deviation: 0 . 05 ). The generation of test data is the same as in the experiment in Section 6.1. The results are given in Table 2, which shows that SPHHD-GP consistently outperforms the other approaches. Appendix J.6 plots the trajectory and energy predictions. These results again confirm the advantage of incorporating symmetry constraints into our method. Please note that D-HNN is not applicable to this system because the div-free part of D-HNN only applies to Hamiltonian systems.

6.3 Learning ocean current fields

To investigate the performance of our model in learning real-world dynamical systems, we then evaluated our model on an ocean current field, which is a complex dynamical system intricately governed by the interplay of multiple factors such as Earth's rotation, wind patterns, temperature gradients, and coastal interactions. Based on sparse observations of buoy velocities, oceanographers are interested in estimating ocean currents away from buoys and identifying divergences of the ocean current field.

In this experiment, we used a dataset from [6], containing 1183 velocity data points from 12 buoys distributed across the northern Gulf of Mexico, as shown by the red arrows in Fig. 4. We compared our model with the Helmholtz-GP model [6], which combines 2D div-free and curl-free kernels in the form of HHD to reconstruct planar ocean current fields, but without considering the severe non-identifiability problem caused by the non-uniqueness of HHD. To make the HHD of an ocean current field unique, an effective way in the field of fluid dynamics is to enforce the parallel boundary condition , which states that the HHD of a vector field in a bounded domain is unique if its div-free component is parallel to the domain boundary (cf. page 36 in [12], Section 5.1 in [8]). For our proposed model, the method of incorporating symmetry can be used to enforce the parallel boundary condition . Specifically, if a vector field has mirror symmetry with respect to a hyperplane, then the vectors on this hyperplane must be parallel to the hyperplane. Therefore, to make our model identifiable, the div-free kernel in our model was constructed to incorporate mirror symmetry with respect to the rectangular boundary of the ocean current field. To reduce the computational complexity, the kernels were fitted into a sparse GP framework [62], which reduced the computational complexity from O ( m 3 ) to O ( M 2 m ) , where m = 1183 is the number of data points and M = 200 is the number of inducing points. See Appendix I for the details of computational complexity of our model.

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

Prediction of Ocean Currents

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

Divergence-free Component (Vorticity)

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

Curl-free Component (Divergence)

0.03

0.02

0.01

0.00

0.01

0.02

0.03

0.000

0.005

0.010

0.015

0.020

0.025

0.030

0.035

(a) Rresults of the Helmholtz-GP [6]

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

Prediction of Ocean Currents

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

Divergence-free Component (Vorticity)

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

Curl-free Component (Divergence)

0.03

0.02

0.01

0.00

0.01

0.02

0.03

0.000

0.005

0.010

0.015

0.020

0.025

0.030

0.035

(b) Rresults of the SPHHD-GP (ours)

The predictions of the ocean current field and its HHD are shown in Fig. 4. Although the ground truth is not available for this field of real-world ocean currents, we can still assess performance against oceanographers' expert knowledge (cf. Section I.3 in [6]). For the predictions of ocean currents (the first column of Fig. 4), the oceanographers expect to see continuous currents with no sharp deviations. However, we can clearly observe that the prediction of Helmholtz-GP shows an abrupt drop in positions away from observed data, especially in the lower left, upper right and center regions. In contrast, our model (SPHHD-GP) presents a more continuous current prediction. The predicted divergence field ( i.e. , the scalar potential of the curl-free component) is provided in the third column of Fig. 4. The oceanographers expect to find a rich structure in divergence predictions. However, the Helmholtz-GP fails to recover the divergence field and instead predicts an almost constant curl-free vector field, as shown in the third block of Fig. 4(a). This is caused by the non-identifiability of the Helmholtz-GP, since harmonic components usually exist in the form of constant vector fields (cf. Section 4.3). Our model, instead, recovers the rich structure of the divergence field. Therefore, we can conclude that our model provides more realistic ocean current predictions and divergence identification, offering a better alternative for the simulation of ocean dynamics.

7 Conclusion and future work

Our work develops an additive GP model whose component is either free of divergence or of curl, the two most ubiquitous differential invariants of vector fields in natural, and we constrain the div/curl-free kernels to preserve desired symmetries. These symmetry-preserving kernels not only improve the accuracy of predictions but also make the model identifiable, thus a physically meaningful decomposition can be predicted. Our future work is to extend our model to exploit the connection of HHD with more dynamical systems. For example, there are recent advances in using the HHD to construct Lyapunov functions [65]. So, our model has potential to achieve good performance in learning stable dynamics.

Acknowledgments

This research is partially supported by the Innovation and Technology Commission of the HKSAR Government under the InnoHK initiative and GHP/126/21GD, and GRF 17200924.

References

Christian Agrell. Gaussian processes with linear operator inequality constraints. Journal of Machine Learning Research , 20:1-36, 2019.

Martin Andersen, Joachim Dahl, and Lieven Vandenberghe. CVXOPT: Convex optimization. Astrophysics Source Code Library , pages ascl-2008, 2020.

George B Arfken and Hans J Weber. Mathematical methods for physicists, 1999.

Vladimir Igorevich Arnol'd. Mathematical methods of classical mechanics , volume 60. Springer Science & Business Media, 2013.

Cristian Barbarosie. Representation of divergence-free vector fields. Quarterly of applied mathematics , 69(2):309-316, 2011.

Renato Berlinghieri, Brian L. Trippe, David R. Burt, Ryan James Giordano, Kaushik Srinivasan, Tamay Özgökmen, Junfei Xia, and Tamara Broderick. Gaussian processes at the helm(holtz): A more fluid model for ocean currents. In Proceedings of the 40th International Conference on Machine Learning , pages 2113-2163, 2023.

Andreas Besginow and Markus Lange-Hegermann. Constraining Gaussian processes to systems of linear ordinary differential equations. Advances in Neural Information Processing Systems , 35:29386-29399, 2022.

Harsh Bhatia, Gregory Norgard, Valerio Pascucci, and Peer-Timo Bremer. The HelmholtzHodge decomposition-a survey. IEEE Transactions on visualization and computer graphics , 19(8):1386-1404, 2012.

Jean-Paul Caltagirone. On a reformulation of Navier-Stokes equations based on HelmholtzHodge decomposition. Physics of Fluids , 33(6), 2021.

Gabriele Cesa, Leon Lang, and Maurice Weiler. A program to build E(N)-equivariant steerable CNNs. In International Conference on Learning Representations , 2021.

Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. Advances in neural information processing systems , 31, 2018.

Alexandre Joel Chorin, Jerrold E Marsden, and Jerrold E Marsden. A mathematical introduction to fluid mechanics , volume 3. Springer, 1990.

Marc Peter Deisenroth, Dieter Fox, and Carl Edward Rasmussen. Gaussian processes for data-efficient learning in robotics and control. IEEE Transactions on Pattern Analysis and Machine Intelligence , 37(2):408-423, 2015.

Manfredo P Do Carmo. Differential forms and applications . Springer Science & Business Media, 1998.

JR Dormand and PJ Prince. A reconsideration of some embedded runge-kutta formulae. Journal of computational and applied mathematics , 15(2):203-211, 1986.

Nicolas Durrande, David Ginsbourger, and Olivier Roustant. Additive covariance kernels for high-dimensional Gaussian process modeling. In Annales de la Faculté des sciences de Toulouse: Mathématiques , volume 21, pages 481-499, 2012.

Nicolas Durrande, David Ginsbourger, Olivier Roustant, and Laurent Carraro. ANOVA kernels and RKHS of zero mean functions for model-based sensitivity analysis. Journal of Multivariate Analysis , 115:57-67, 2013.

David Duvenaud. Automatic model construction with Gaussian processes . PhD thesis, University of Cambridge, 2014.

Arne Gabrielsen. Consistency and identifiability. Journal of Econometrics , 8(2):261-263, 1978.

Abdou Garba and Pierre Haldenwang. A Helmholtz-Hodge Projection Method Using an Iterative Gauge Computation to Solve the 3D Generalized Stokes Problem. SIAM Journal on Scientific Computing , 35(3):A1560-A1583, 2013.

Roman Garnett, Shirley Ho, and Jeff Schneider. Finding galaxies in the shadows of quasars with Gaussian processes. In International Conference on Machine Learning , pages 1025-1033. PMLR, 2015.

Sam Greydanus and Andrew Sosanya. Dissipative Hamiltonian neural networks: Learning dissipative and conservative dynamics separately. arXiv preprint arXiv:2201.10085 , 2022.

Samuel Greydanus, Misko Dzamba, and Jason Yosinski. Hamiltonian neural networks. Advances in neural information processing systems , 32, 2019.

Bernard Haasdonk, A Vossen, and Hans Burkhardt. Invariance in kernel methods by Haarintegration kernels. In Image Analysis: 14th Scandinavian Conference, SCIA 2005, Joensuu, Finland, June 19-22, 2005. Proceedings 14 , pages 841-851. Springer, 2005.

Marc Härkönen, Markus Lange-Hegermann, and Bogdan Rai¸ t˘ a. Gaussian process priors for systems of linear partial differential equations with constant coefficients. arXiv preprint arXiv:2212.14319 , 2022.

Pashupati Hegde, Ça˘ gatay Yıldız, Harri Lähdesmäki, Samuel Kaski, and Markus Heinonen. Variational multiple shooting for Bayesian ODEs with Gaussian processes. In Uncertainty in Artificial Intelligence , pages 790-799. PMLR, 2022.

Markus Heinonen, Cagatay Yildiz, Henrik Mannerström, Jukka Intosalmi, and Harri Lähdesmäki. Learning unknown ODE models with Gaussian processes. In International Conference on Machine Learning , pages 1959-1968. PMLR, 2018.

Morris W Hirsch, Stephen Smale, and Robert L Devaney. Differential equations, dynamical systems, and an introduction to chaos . Academic press, 2012.

Carl Jidling, Niklas Wahlström, Adrian Wills, and Thomas B Schön. Linearly constrained Gaussian processes. Advances in Neural Information Processing Systems , 30, 2017.

Pengzhan Jin, Zhen Zhang, Aiqing Zhu, Yifa Tang, and George Em Karniadakis. Sympnets: Intrinsic structure-preserving symplectic networks for identifying Hamiltonian systems. Neural Networks , 132:166-179, 2020.

Sanket Kamthe and Marc Deisenroth. Data-efficient reinforcement learning with probabilistic model predictive control. In Proceedings of the Twenty-First International Conference on Artificial Intelligence and Statistics , volume 84 of Proceedings of Machine Learning Research , pages 1701-1710. PMLR, 09-11 Apr 2018.

James Kelliher. Stream functions for divergence-free vector fields. Quarterly of Applied Mathematics , 79(1):163-174, 2021.

Jonas Kiessling, Emanuel Ström, and Raúl Tempone. Wind field reconstruction with adaptive random fourier features. Proceedings of the Royal Society A , 477(2255):20210236, 2021.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

Markus Lange-Hegermann. Algorithmic linearly constrained Gaussian processes. Advances in Neural Information Processing Systems , 31, 2018.

Markus Lange-Hegermann. Linearly constrained Gaussian processes with boundary conditions. In International Conference on Artificial Intelligence and Statistics , pages 1090-1098. PMLR, 2021.

Markus Lange-Hegermann and Daniel Robertz. On boundary conditions parametrized by analytic functions. In International Workshop on Computer Algebra in Scientific Computing , pages 225-245. Springer, 2022.

Yann LeCun, Bernhard Boser, John Denker, Donnie Henderson, Richard Howard, Wayne Hubbard, and Lawrence Jackel. Handwritten digit recognition with a back-propagation network. Advances in neural information processing systems , 2, 1989.

Mario Livio. Why symmetry matters. Nature , 490(7421):472-473, 2012.

Lennart Ljung. On consistency and identifiability. In Stochastic Systems: Modeling, Identification and Optimization, I , pages 169-190. Springer, 2009.

Svenja Lowitzsch. Approximation and interpolation employing divergence-free radial basis functions with applications . Texas A&M University, 2002.

Xiaoyu Lu, Alexis Boukouvalas, and James Hensman. Additive Gaussian processes revisited. In International Conference on Machine Learning , pages 14358-14383. PMLR, 2022.

Jun Ma, Fuqiang Wu, Wuyin Jin, Ping Zhou, and Tasawar Hayat. Calculation of hamilton energy and control of dynamical systems with different types of attractors. Chaos: An Interdisciplinary Journal of Nonlinear Science , 27(5), 2017.

Andrew J. Majda and Andrea L. Bertozzi. Vorticity and Incompressible Flow . Cambridge Texts in Applied Mathematics. Cambridge University Press, 2001.

Filippo Maria Denaro. On the application of the Helmholtz-Hodge decomposition in projection methods for incompressible flows with general boundary conditions. International Journal for Numerical Methods in Fluids , 43(1):43-69, 2003.

Haggai Maron, Heli Ben-Hamu, Nadav Shamir, and Yaron Lipman. Invariant and equivariant graph networks. In International Conference on Learning Representations , 2019.

Kaspar Martens. Enabling feature-level interpretability in non-linear latent variable models: a synthesis of statistical and machine learning techniques . PhD thesis, University of Oxford, 2019.

Takashi Matsubara and Takaharu Yaguchi. FINDE: Neural differential equations for finding and preserving invariant quantities. In The Eleventh International Conference on Learning Representations , 2023.

Takashi Matsumoto. A chaotic attractor from Chua's circuit. IEEE Transactions on Circuits and Systems , 31(12):1055-1058, 1984.

Francis J Narcowich and Joseph D Ward. Generalized hermite interpolation via matrix-valued conditionally positive definite functions. Mathematics of Computation , 63(208):661-687, 1994.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems , 32, 2019.

Jim O Ramsay, Giles Hooker, David Campbell, and Jiguo Cao. Parameter estimation for differential equations: a generalized smoothing approach. Journal of the Royal Statistical Society: Series B (Statistical Methodology) , 69(5):741-796, 2007.

Carl Edward Rasmussen and Christopher K. I. Williams. Gaussian Processes for Machine Learning . The MIT Press, 11 2005.

Katharina Rath, Christopher G Albert, Bernd Bischl, and Udo von Toussaint. Symplectic Gaussian process regression of maps in Hamiltonian systems. Chaos: An Interdisciplinary Journal of Nonlinear Science , 31(5):053121, 2021.

Marco Reisert. Group integration techniques in pattern analysis: a kernel view . PhD thesis, Freiburg (Breisgau), Univ., Diss., 2008.

Marco Reisert and Hans Burkhardt. Learning equivariant functions with matrix valued kernels. Journal of Machine Learning Research , 8(15):385-408, 2007.

Jack Richter-Powell, Yaron Lipman, and Ricky T. Q. Chen. Neural conservation laws: A divergence-free perspective. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems , 2022.

Steffen Ridderbusch, Christian Offen, Sina Ober-Blöbaum, and Paul Goulart. Learning ODE models with qualitative structure using Gaussian processes. In 2021 60th IEEE Conference on Decision and Control (CDC) , pages 2896-2896. IEEE, 2021.

Magnus Ross and Markus Heinonen. Learning energy conserving dynamics efficiently with Hamiltonian Gaussian Processes. Transactions on Machine Learning Research , 2023.

C Sarasola, FJ Torrealdea, A d'Anjou, A Moujahid, and M Graña. Energy balance in feedback synchronization of chaotic systems. Physical Review E , 69(1):011606, 2004.

Vıctor Garcia Satorras, Emiel Hoogeboom, and Max Welling. E(n) equivariant graph neural networks. In International conference on machine learning , pages 9323-9332. PMLR, 2021.

Edward Snelson and Zoubin Ghahramani. Sparse Gaussian processes using pseudo-inputs. In Y. Weiss, B. Schölkopf, and J. Platt, editors, Advances in Neural Information Processing Systems , volume 18. MIT Press, 2005.

Ercan Solak, Roderick Murray-Smith, WE Leithead, D Leith, and Carl Rasmussen. Derivative observations in Gaussian process models of dynamic systems. Advances in neural information processing systems , 15, 2002.

Steven H Strogatz. Nonlinear dynamics and chaos: with applications to physics, biology, chemistry, and engineering . CRC press, 2018.

Tomoharu Suda. Construction of lyapunov functions using helmholtz-hodge decomposition. Discrete and Continuous Dynamical Systems , 39(5):2437-2454, 2019.

Michalis Titsias. Variational learning of inducing variables in sparse gaussian processes. In Artificial intelligence and statistics , pages 567-574. PMLR, 2009.

Peter Toth, Danilo J Rezende, Andrew Jaegle, Sébastien Racanière, Aleksandar Botev, and Irina Higgins. Hamiltonian generative networks. In International Conference on Learning Representations , 2019.

Ilari Vallivaara, Janne Haverinen, Anssi Kemppainen, and Juha Röning. Simultaneous localization and mapping using ambient magnetic field. In 2010 IEEE Conference on Multisensor Fusion and Integration , pages 14-19, 2010.

Ilari Vallivaara, Janne Haverinen, Anssi Kemppainen, and Juha Röning. Magnetic field-based SLAM method for solving the localization problem in mobile robot floor-cleaning task. In 2011 15th International Conference on Advanced Robotics (ICAR) , pages 198-203, 2011.

Pantelis R Vlachas, Jaideep Pathak, Brian R Hunt, Themistoklis P Sapsis, Michelle Girvan, Edward Ott, and Petros Koumoutsakos. Backpropagation algorithms and reservoir computing in recurrent neural networks for the forecasting of complex spatiotemporal dynamics. Neural Networks , 126:191-217, 2020.

Niklas Wahlström. Modeling of magnetic fields and extended objects for localization applications . PhD thesis, Linköping University Electronic Press, 2015.

Niklas Wahlström, Manon Kok, Thomas B. Schön, and Fredrik Gustafsson. Modeling magnetic fields using Gaussian processes. In 2013 IEEE International Conference on Acoustics, Speech and Signal Processing , pages 3522-3526, 2013.

Nils Wandel, Michael Weinmann, and Reinhard Klein. Learning incompressible fluid dynamics from scratch - towards fast, differentiable fluid models that generalize. In International Conference on Learning Representations , 2021.

Maurice Weiler and Gabriele Cesa. General E(2)-equivariant steerable CNNs. Advances in neural information processing systems , 32, 2019.

Holger Wendland. Divergence-free kernel methods for approximating the stokes problem. SIAM Journal on Numerical Analysis , 47(4):3158-3179, 2009.

Andrew Wilson and Ryan Adams. Gaussian process kernels for pattern discovery and extrapolation. In International conference on machine learning , pages 1067-1075. PMLR, 2013.

Ping Zhou, Xikui Hu, Zhigang Zhu, and Jun Ma. What is the most suitable Lyapunov function? Chaos, Solitons & Fractals , 150:111154, 2021.

Mauricio A. Álvarez, Lorenzo Rosasco, and Neil D. Lawrence. Kernels for vector-valued functions: A review. Foundations and Trends® in Machine Learning , 4(3):195-266, 2012.

Appendix

A HHDand dynamical systems

This section provides a brief introduction to the connections between the Helmholtz-Hodge decomposition (HHD) and some dynamical systems needed to understand this work.

A.1 HHDand dissipative Hamiltonian systems

We begin with a brief review of Hamiltonian mechanics [4]. For a dynamical system with N degrees of freedom, the Hamiltonian formalism describes the system by defining a scalar function H ( x ) known as the Hamiltonian , where the system state x = ( q , p ) ∈ R 2 N is described by generalized coordinates q ∈ R N and p ∈ R N in the phase space, corresponding to generalized position and momentum, respectively. The time evolution of ( q , p ) is governed by the symplectic gradient of its Hamiltonian, i.e. , ˙ q = ∂H ∂ p , ˙ p = -∂H ∂ q . If the Hamiltonian is not explicitly time-dependent, [ ˙ q , ˙ p ] T defines a divergence-free vector field ( ∂ ∂ q ∂H ∂ p -∂ ∂ p ∂H ∂ q = 0 ) whose flows conserve the Hamiltonian. By interpreting the Hamiltonian as energy, the Hamiltonian vector field can model systems with energy conservation. However, real-life systems often suffer from energy dissipation. In physical systems governed by an autonomous ODE, any energy variation occurs along a volume change in phase space and vice versa. Therefore, to account for energy dissipation, a term responsible for the volume contraction of the phase space can be introduced into the Hamiltonian dynamics. This term represents the energy lost by the system due to various dissipative forces and is typically modeled by gradient fields of some scalar functions D ( · ) , so the motion equations of the dissipative Hamiltonian system can be given by

In a dissipative Hamiltonian system, the Hamiltonian still represents the system's total energy, but the additional damping term causes the system to lose energy over time. One common example of the damping term is induced by the Rayleigh function , D ( p ) = -1 2 p T Q p , where Q ∈ R N × N is a symmetric positive-definite matrix called the Rayleigh dissipation matrix . The Rayleigh function provides an elegant way to include dissipative forces-such as friction, air resistance, and viscosity-in the context of Hamiltonian mechanics. When the Hamiltonian H ( · ) represents mass instead of energy, Eq. 19 can also be used to model 2-dimensional compressible fluid dynamics, such as the dynamics of ocean current fields.

A.2 HHDand chaotic systems

The dissipative Hamiltonian system (Eq. 19) is in an explicit form of HHD, because the Hamiltonian vector field is divergence-free and the dissipative field ∇ D is curl-free. However, as its divergencefree part is governed by the Hamiltonian equations, Eq. 19 can only be used to describe a subset of even-dimensional systems. To extend its scope to study chaotics systems, [60] proposed to approach energy for dimensionless dynamical systems by using HHD as follows,

where f div and f curl are divergence-free and curl-free components of a dynamical system, respectively. According to this criterion (Eq. 20), HHDs of chaotic systems were approached to calculate the energy for synchronizing two chaotic systems [60], to analyze the stability of chaotic systems [77], and to design energy modulation-based controllers [43].

B GPs and linear operators

Gaussian processes (GPs) are closed under linear transformations (cf. Lemma 2.2 in [36], Lemma 2.1 in [25]). Let L be a linear operator acting on realizations of g ∼ GP ( µ g ( x ) , κ g ( x , x ′ )) , then under

the assumption that L commutes with expectation, the mean of L g is given by

and the covariance is such that

A common application of this technique is to construct GPs with realizations in the solution set of linear differential equations, assuming that L is a linear differential operator [29, 35, 36, 7, 25]. And similarly, we make use of the closure of GPs under linear differential operators to construct the curl-free and div-free kernels.

Although the transformed kernel (Eq. 22) has been widely used, its validity as a covariance function is rarely discussed in existing works. In a GP framework, a kernel κ ( x , x ′ ) : R n × R n → R m × m is a valid covariance function if it is:

(ii) Positive semidefinite, i.e. , ∑ ij c T i κ ( x i , x j ) c j ≥ 0 , for any finite set { x i } ⊂ R n and { c i } ⊂ R m .

These two conditions for a kernel can be verified if and only if there exists a feature map ϕ ( x ) such that κ ( x , x ′ ) = ϕ ( x ) ϕ ( x ′ ) T [53]. So, if κ g ( x , x ′ ) is a valid kernel, it holds that

Therefore, the transformed kernel L x κ g ( x , x ′ ) L T x ′ is guaranteed to be a valid covariance function provided that its underlying kernel κ g ( x , x ′ ) is. However, when L is a differential operator, κ g should be twice differentiable, which is satisfied by most of the standard kernels, such as the squared exponential kernel.

C Basics for Euclidean group

This section gives the basic definitions about the Euclidean group . In the context of this work, the most important example of a symmetry group is the Euclidean group E ( n ) or its subgroups. The set of all elements in E ( n ) can be denoted as

where O ( n ) = { A ∈ R n × n | AA T = I } is the orthogonal group . Any element g = ( A, b ) ∈ E ( n ) represents a translation followed by an orthogonal transformation, the action of g on a point x ∈ R n is given by a linear mapping:

Therefore, E ( n ) comprises all isometries of a Euclidean space, i.e. for all x , x ′ ∈ R n and g ∈ E ( n ) , we have

where ∥·∥ is the Euclidean norm. All intuitive geometric transformations in R n can be described by subgroups of E ( n ) , such as

Translation: The group of all translations in R n is denoted by ( R n , +) . For any v ∈ R n , A translation is a transformation that moves a point x ∈ R n by L v ( x ) = x + v .

Rotation: The group of all rotations in R n is represented by the set of special orthogonal matrices SO ( n ) = { R ∈ O ( n ) | det R = 1 } , where a rotation matrix R transforms a point by L R ( x ) = R x .

Reflection: Reflections in R n forms subgroups of the orthogonal group O ( n ) . Reflections correspond to mirror symmetries. They mirror points across a hyperplane. For a hyperplane with a unit normal vector n , the action of a reflection is defined as L n ( x ) = x -2( x · n ) n .

D Proofs

This section restates and proves the theorems in Section 5, which give the theoretical foundations that we use to enforce symmetry constraints to our HHD-GP model.

D.1 Proof of Theorem 5.2

Theorem 5.2. Let G be a Euclidean group or its subgroup, and let V : R n → R be a G -invariant scalar function. Then, the curl-free vector field f curl : R n → R n defined by f curl ( x ) = ∇ V ( x ) is G -equivariant.

Proof. To prove that ∇ V ( x ) is G -equivariant, by Definition 5.1 we need to show that for any x ∈ R n and g ∈ G , we have

where L g ( x ) : R n → R n is the action of g on x , and J L g is the Jacobian matrix of L g . To show this, we first recall the chain rule of the gradient operator,

where J T L g denotes the transpose of J L g . Since V ( x ) is G -invariant, for all g ∈ G we have

Now taking the gradient of both sides, and combining with the chain rule, we obtain

Then by multiplying J L g on both sides, we have

Considering G is a subgroup of the Euclidean group, the action of any group element g ∈ G on R n can be represented by

where A ∈ O ( n ) is an orthogonal matrix. Therefore, J L g J T L g = AA T = I , where I is an identity matrix, so we have J L g ∇ V = ∇ V ◦ L g , which means that the curl-free vector field f curl = ∇ V is G -equivariant, as desired.

D.2 Proof of Theorem 5.3

Theorem 5.3. Let G be a Euclidean group or its subgroup, and let v : R n → R n be a G -equivariant vector field. Then the divergence-free vector field f div = [ ∇· A 1 , . . . , ∇· A n ] T is G -equivariant, where A i denotes the i -th row of the skew-symmetric matrix-valued function A = J v -J T v .

Proof. To prove this theorem, we need to show that the divergence-free vector field f div satisfies the G -equivariance condition, i.e., for all g ∈ G and x ∈ R n ,

where L g ( x ) : R n → R n is the action of g on x , and J L g is the Jacobian matrix of L g .

Define the divergence-free vector field f div = [ ∇· A 1 , ..., ∇· A n ] T , where A i is the i th row of the skew-symmetric matrix field A = J v -J T v , and J v is the Jacobian of some vector field v : R n → R n . Given that v is G -equivariant, for all g ∈ G and x ∈ R n we have

Now computing the Jacobian of both sides, by the chain rule of the Jacobian, we obtain

Then applying J T L g on both sides, we have J T L g ( J v ◦ L g ) J L g = J T L g J L g J v . Since G is a subgroup of the Euclidean group, the Jacobian of the action of any group element g ∈ G on R n is an orthogonal

matrix, i.e., J L g ∈ O ( n ) . Therefore, J T L g J L g = I , where I is an identity matrix, so it holds that J v = J T L g ( J v ◦ L g ) J L g . Then by substituting it into A = J v -J T v , we obtain

where the equation at the ( i, j ) -th entries of both sides is given by

Then by substituting A ij into the construction of f div , for any g ∈ G and x ∈ R n , we have

Then, applying J L g on both sides, we obtain J L g f div = f div ◦ L g , which completes the proof.

E Physical systems in the experiments

This section reviews the physical systems used in our experiments, listing their governing equations, HHDs and symmetry properties.

E.1 Dissipative Hamiltonian systems

The connection between HHD and dissipative Hamiltonian systems is reviewed in Appendix A.1. In our experiments, the models are evaluated with the following two dissipative Hamiltonian systems.

Damped Mass Spring A damped mass-spring system is a mass attached to a spring that oscillates periodically around an equilibrium position. Its Hamiltonian in natural units is given by

where q ∈ R is its position, and p is the momentum conjugate to q . The Hamiltonian represents the total energy of the oscillator. Without energy dissipation, the motion of the oscillator is described by the Hamiltonian vector field (div-free vector field f div ) derived from the Hamiltonian:

Then we show that the Hamiltonian vector field has SO(2)-equivariance. Apply a rotation by an angle θ that transforms any state vector [ q, p ] T to [ q ′ , p ′ ] T , where q ′ = q cos θ -p sin θ , p ′ = q sin θ + p cos θ , then for all θ ∈ R , the equivariance condition (Eq. 14) can be easily verified by

Therefore, the Hamiltonian vector field is equivariant with respect to the 2D rotations. However, this rotation symmetry is broken when the system suffers from energy dissipation, i.e. , a dissipative field is added to the motion equations. The dissipative vector field (curl-free vector field f curl ) is induced by the Rayleigh function D ( p ) = ρp 2 / 2 :

where ρ is the friction coefficient and we set it at 0.1. This dissipative field has the symmetries of the q -axis translation ( f curl ( q + g, p ) = f curl ( q, p ) , ∀ g ∈ R ) and the rotation by an angle of π radians (the so-called odd symmetry , i.e. , f curl ( -q, -p ) = -f curl ( q, p ) ). Then by summing the curl-free vector field (Eq. 42) and the Hamiltonian vector field (Eq. 40), the dynamics of the damped harmonic oscillator is characterized by

which only demonstrates the odd symmetry.

Damped Pendulum A damped pendulum is a physical system consisting of a weight suspended from a pivot, subjected to a resistive force that gradually reduces its oscillation over time. By defining its state in the phase space through [ q, p ] T ∈ R 2 , the dynamics of a damped pendulum and its HHD can be given by

where the Hamiltonian H and the Rayleigh function D are given by

In our experiments, the gravitational constant g , the mass m , and the pendulum length l were set as g = 3 and m = l = 1 , and the friction coefficient ρ was set as ρ = 0 . 1 . The dynamics of the damped pendulum, as well as its div-free and curl-free components, exhibit the odd symmetry, but the curl-free part additionally exhibits the translation invariance along the q -axis.

E.2 Chaotic system

The Chua circuit [49] is a simple electronic circuit that exhibits chaotic behavior, and it has applications in various fields, such as secure communication systems and random number generators. The ODE of a Chua circuit with its HHD is given by

where x , y , z are the phase space variables, and α , c , β are the system parameters, which were set to 10 , -0 . 143 , and 16 respectively in the experiments. The Chua circuit is equivariant under a π

rotation about the origin: L π : ( x, y, z ) ↦→ ( -x, -y, -z ) . If the Chua circuit is decomposed by the HHD in Eq. 46, more symmetries are exhibited. Specifically, its div-free component inherits the odd symmetry, but additionally presents the translation invariance along x = z . The curl-free vector field is invariant along the z -axis and equivariant under mirror reflections across the coordinate planes,

This HHD not only uncovers more knowledge of the Chua circuit's symmetry but can also be used to analysis the energy of the system. According to the generalized Hamiltonian formalism (Eq. 20), the energy function H associated with the Chua circuit satisfies the following PDE:

which is satisfied by the quadratic form:

As presented by [77], this energy function indicates that the Chua circuit keeps oscillatory when the energy release along the x -axis is enough to balance the energy pumping along the y -axis and z -axis.

F Evaluation metrics for learning ODEs

RMSE and MNLL of state derivatives To evaluate the models' performance in terms of learning ˙ x = f ( x ) , we first compute the root mean squared error (RMSE) between the predicted time derivatives ˆ ˙ x i and the ground truth ˙ x i , i.e.

where ∥·∥ 2 is the Euclidean norm, and m is the number of test data. The lower the RMSE, the better. The test set { ( x i , ˙ x i ) } m i =1 was generated by sampling a grid of points with the resolution of 10 . In addition to the RMSE for evaluating the regression results, we further use the mean negative log likelihood (MNLL) to evaluate the prediction uncertainty provided by the GP models, and it is calculated by

where var ( x i ) is the prediction variance at a test state x i . The lower the MNLL, the more effectively the forecast uncertainty reflects the prediction error.

VPT of state trajectories To obtain a more comprehensive evaluation of the learned ODE models, we considered additional metrics for evaluating the accuracy of predicting state trajectories. Recent studies [48, 30, 70] have suggested that RMSEs of state trajectories over long time horizons can be misleading indicators. So we used the valid prediction time (VPT) to measure the model's ability to do such long-term extrapolation in a phase space. Following the definition in [30, 70], VPT calculates the first time step t at which the normalized root mean square error (NRMSE) between the predicted state ˆ x t and the ground truth x t exceeds a given threshold ϵ ,

where NRMSE(ˆ x t , x t ) = (ˆ x t -x t ) T Σ(ˆ x t -x t ) /n , Σ = diag (1 /σ 1 , . . . , 1 /σ n ) , with σ i denoting the variance of the i -th dimension of the true trajectory. The VPT measures how long the predicted trajectory remains close to the true trajectory, so the higher this indicator, the better. In the experiments, we set ϵ = 0 . 01 . To alleviate the dependency on the initial condition, we reported the VPT averaged over trajectories simulated from 50 randomly sampled initial conditions. And a trajectory from an initial condition was solved by the Dormand-Prince method (dopri5) [15] implemented in torchdiffeq 4 , integrating forward in time at a frequency of 25 Hz for 15 seconds, with the relative and absolute tolerances of 10 -6 .

G Implementation details

The experiments were performed on a single Nvidia GeForce GTX 3050 Ti GPU, and all of the models were implemented with PyTorch [51]. The GP-based models (Ind-GP, GIM-GP, Div-GP, HHD-GP and SPHHD-GP) were trained by maximizing the log of their marginal likelihood:

where |·| computes the determinant of the covariance matrix K +Σ . Training a GP model refers to optimizing the kernel parameters in K and the noise variances in Σ , and these hyperparameters were initialized randomly in our experiments. The GP-based models were trained by the ADAM optimizer [34], with a learning rate 0 . 01 for 3000 gradient steps.

G.1 Ind-GP

The Ind-GP models each dimension of a n -dimensional dynamical system independently, so its matrix-valued kernel is constructed by

where κ i , i = 1 , . . . , n are independent scalar kernels. And a standard choice for κ i is the squared exponential (SE) kernel,

which has two parameters: σ determines the variation of function values from their mean, and l controls the length scale on which the function varies. Realizations of GPs with SE kernels are dense in the set of smooth functions C ∞ ( R n , R ) (cf. Prop.1 in [37]). For a fair comparison, we also used the SE kernel (Eq. 55) to construct kernels for the other GP-based models.

G.2 GIM-GP

The GIM-GP produces predictions with the desired symmetry. Each of the systems in our experiments exhibits the odd symmetry as a whole. So their symmetry groups can be given by G odd = { I n , -I n } , where I n is the n -dimensional identity matrix, and the group elements are linear representations of the group actions, i.e. , L g ( x ) = g x , ∀ g ∈ G . Therefore, according to Eq. 17, the GIM-kernel for G odd was constructed by

where the SE kernel (Eq. 55) was used as the basis kernel κ in Eq. 17.

G.3 Div-GP

GPs with a div-free kernel can be used to approximate conservative dynamics. According to Eq. 12, a two-dimensional div-free kernel was constructed by

where κ u in Eq. 12 is denoted by κ H instead, indicating that it is the kernel for the Hamiltonian functions of the damped mass spring and the damped pendulum, and the SE kernel (Eq. 55) was used for κ H . The partial derivatives in Eq. 57 were calculated by automatic differentiation in PyTorch, so we did not need to derive its analytic expression.

Then we constructed the div-free kernel for the Chua circuit system. According to Eq. 11, a threedimensional div-free vector field is given by f div ( x ) = Ψ u ( x ) , where Ψ = [ ψ 12 , ψ 13 , ψ 23 ] ∈ R 3 × 3

with its components given by

Then by assuming u = [ u 12 , u 13 , u 23 ] T ∼ GP ( 0 , κ u = κ ind ) , we constructed the div-free kernel for the Chua circuit according to Eq. 12:

where κ 1 , κ 2 and κ 3 are all independent SE kernels (Eq. 55).

G.4 HHD-GP

The HHD-GP consists of two independent GPs added together, modeling curl-free and div-free dynamics respectively. GPs with div-free kernels have been constructed earlier, so here we build curl-free kernels according to Eq. 8. For the damped mass spring and the damped pendulum, their two-dimensional curl-free kernel was constructed by

Similarly, the three-dimensional curl-free kernel for the Chua circuit was constructed by

where κ V is a SE kernel.

G.5 SPHHD-GP

G.5.1 Symmetry-preserving curl-free kernels

According to Theorem 5.2, a G -equivariant curl-free vector field is constructed by constraining its scalar potential V to be G -invariant. Therefore, a G -equivariant curl-free kernel is obtained by constructing its potential kernel κ V according to Eq. 16.

The curl-free vector field of the damped mass spring and the damped pendulum has two types of symmetry:

(i) translation along the q -axis, i.e. , L g ( x ) = x +( g, 0) , for all g ∈ R ;

(ii) rotation by an angle of π radians, i.e. , L g ( x ) = g x , for all g ∈ { I 2 , -I 2 } .

These symmetry groups were enforced to a SE kernel sequentially according to Eq. 16, where |G| can be ignored. Specifically, the translation invariance was enforced using the Gaussian integral formula:

then based on which the π rotation invariance was enforced by

The curl-free vector field of the Chua circuit (Eq. 46) has similar two types of symmetry:

(i) translation along the z -axis, i.e. , L g ( x ) = x +(0 , 0 , g ) , for all g ∈ R ;

(ii) mirror reflections across the coordinate planes, i.e. , L i,j : ( x, y, z ) ↦→ ( ( -1) i x, ( -1) j y, z ) , for all i, j ∈ { 0 , 1 } .

Therefore, κ V for the Chua circuit was constructed by

G.5.2 Symmetry-preserving div-free kernels

Theorem 5.3 shows that a G -equivariant div-free vector field is constructed from a vector potential v with the same equivariance. For the div-free vector field of the damped mass-spring system with SO(2)-equivariance, its κ v was constructed by the GIM-kernel (Eq. 17):

Then this κ v (Eq. 70) is substituted into Eq. 18 to construct κ H for its div-free kernel κ div (Eq. 57):

which admits no closed-form solution, so we used a numerical approximation of the integral (Eq. 72) by sampling discrete rotations of { θ = π 4 n | n = 0 , . . . , 7 } , whose rotation matrices form a finite group. And by setting θ ∈ { 0 , π } , Eq. 72 was used to construct κ H for the div-free vector field of the damped pendulum, which has the equivariance under a π rotation (odd symmetry).

The div-free vector field of the Chui circuit (Eq. 46) has the odd symmetry and the translation symmetry along x = z . κ v respecting the translation symmetry was constructed by κ v = κ · I 3 , where κ is given by

where p = [ 1 √ 2 ( x + z ) , y ] T . And we further enforced the odd symmetry by

Then, κ v = κ · I 3 was substituted into Eq. 18 to construct κ u by

Finally, this κ u (Eq. 79) was substituted into Eq. 12 to construct the symmetry-preserving div-free kernel for the Chua circuit.

G.6 D-HNN

We used the released code 5 of D-HNN and ran their training routine for our systems.

G.7 Implementation details of predicting energy

G.7.1 Predicting energy of the mass-spring system and the pendulum

When HHD-GP and SPHHD-GP are used to learn the damped mass-spring system and the damped pendulum, the potential kernel κ H for constructing their div-free kernels (Eq. 57) can be interpreted as placing a GP prior on the Hamiltonian function. Therefore, a joint GP describing both the Hamiltonian function H ( x ) and the dynamical system f ( x ) is given by

where κ H, f ( x , x ′ ) = κ f ,H ( x ′ , x ) T with

After training the model f ( x ) ∼ GP ( 0 , κ hhd ( x , x ′ )) on noisy observations Y = [ y 1 , . . . , y m ] T 6 at states X = [ x 1 , . . . , x m ] T , we are interested in predicting the value of Hamiltonian function H ( x ∗ ) at a new test state x ∗ . Since these data determine H ( · ) only up to an additive constant, we assume that we have an anchor point H ( x 0 ) which can be chosen arbitrarily. Then, according to the GP prior (Eq. 80), H ( x ∗ ) and Y H = [ H ( x 0 ) , Y ] T are jointly distributed as

where k = [ κ H ( x ∗ , x 0 ) , κ H, f ( x ∗ , X )] , and

Then, we obtain the posterior distribution

where the mean function is used for energy prediction.

G.7.2 Predicting energy of the Chua circuit

[60] stated that the div-free component of a chaotic system is always orthogonal to the gradient of its energy function, i.e. , ∇ H T f div ( x ) = 0 , for all x ∈ R n . The energy function of the Chua circuit is in a quadratic form:

So we parameterize the energy function of the Chua circuit by ˆ H ( x ) = 1 2 ( a 1 x 2 + a 2 y 2 + a 3 z 2 ) , where a = [ a 1 , a 2 , a 3 ] are parameters. Then by learning a divergence-free vector field ˆ f div ( · ) through our model, we can estimate the parameters a by minimizing ∑ m i =1 ( ∇ ˆ H T ˆ f ( x i ) ) 2 at a finite number of sample points { x i } m i =1 ( m = 500 in our experiments). And to eliminate the solution at a = [0 , 0 , 0] , we add an equality constraint at a random point of the ground truth { x 0 , H ( x 0 ) } . Therefore, the parameters a of ˆ H are solved in a convex quadratic program (QP):

where Q = q T q with q = [ ˆ f div ( x 1 ) , . . . , ˆ f div ( x m ) ] T ∈ R m × 3 . And in the experiments, we solved the QP using the solver provided by CVXOPT [2].

H Uniqueness and Symmetries

The two constituent kernels of our model define the space of divergence-free vector fields ( f div ∈ F div ) and the space of curl-free vector fields ( f curl ∈ F curl ), respectively. These two spaces overlap partially due to the presence of harmonic vector fields ( f harm ∈ F div ∩ F curl ). To eliminate this overlap and make HHD unique, we propose to impose symmetry constraints on the two spaces separately, with the corresponding symmetry groups defined as G div and G curl . Therefore, the uniqueness property of HHD depends on the space of harmonic vector fields that respects the union of two symmetry groups, i.e.

If F harm = ∅ , the HHD is unique, i.e. our model is identifiable. A harmonic vector field can always be represented by the gradient field of a harmonic function h , which is a scalar function satisfying ∇ · ∇ h = 0 (Laplace's equation). For the three dynamical systems used in our experiments, it can be easily verified that the presence of harmonic components can be eliminated through forced symmetries. The symmetry group union of a damped mass-spring system consists of a rotation group G div = SO (2) and a translation group G curl = { ( g, 0) | g ∈ R } . A harmonic vector field f harm = ∇ h that satisfies this translation symmetry implies that its harmonic function h ( q, p ) is independent of the variable q , so the harmonic vector field is given by f harm = (0 , ∂ p h ) and ∂ p h should be a constant to satisfy the Laplace's equation, but the harmonic vector field in the form of constant clearly contradicts rotation symmetry G div . Therefore, there is no harmonic vector field that respects both the symmetry groups G div and G curl . Similar conclusions can be drawn for the damped pendulum and the Chua circuit. The Laplace's equation and translation symmetry imply that harmonic vector fields can only exist in the form of constant vector fields. However, constant vector fields obviously contradict odd symmetry or mirror symmetry.

I Computational complexity

Compared to the diagonal kernel that independently models each dimension of a dynamical system [13, 31, 68, 69], our kernel can capture correlations among output dimensions because its off-diagonal elements are non-zero. However, this advantage comes together with a low computational efficiency. The exact inference of HHD-GP over an n -dimensional system with m data points has a cost of O ( m 3 n 3 ) in time, due to the inversion of its full covariance matrix K ∗ ∈ R mn × mn . Although in this work we do not focus on the computational challenge in large-scale applications, one can plug our kernel in a variety of sparse GP frameworks [62, 66], reducing the computational complexity to O ( M 2 mn ) , where M ( M ≪ m ) is the number of inducing point to approximate the covariance matrix.

J Additional Experimental Results

J.1 Results with increasing number of training data

To further compare the predictive performance of the models, we evaluated them by increasing the amount of training data. The results are shown in Fig. 5. As expected, the performance of the models improves as the amount of training data increases, but we can observe that our model (SPHHD-GP) performs best at every data amount relative to the baseline models, verifying the robustness of our model against data sparsity.

0

100

200

300

400

Number of data

10

2

10

1

RMSE

Mass-spring System

D-HNN

Div-GP

GIM-GP

HHD-GP

Ind-GP

SPHHD-GP

0

100

200

300

400

Number of data

10

2

10

1

10 0

VPT

Mass-spring System

D-HNN

Div-GP

GIM-GP

HHD-GP

Ind-GP

SPHHD-GP

0

100

200

300

400

Number of data

5

4

3

2

1

MNLL

Mass-spring System

Div-GP

GIM-GP

HHD-GP

Ind-GP

SPHHD-GP

0

100

200

300

400

Number of data

10

1

10 0

RMSE

Pendulum

D-HNN

Div-GP

GIM-GP

HHD-GP

Ind-GP

SPHHD-GP

0

100

200

300

400

Number of data

10

2

10

1

VPT

Pendulum

D-HNN

Div-GP

GIM-GP

HHD-GP

Ind-GP

SPHHD-GP

0

100

200

300

400

Number of data

3

2

1

0

MNLL

Pendulum

Div-GP

GIM-GP

HHD-GP

Ind-GP

SPHHD-GP

J.2 Trajectory prediction of the mass-spring system and the pendulum

Fig. 6 and Fig. 7 present the trajectory predictions for the damped mass-spring system and the damped pendulum, respectively.

J.3 Example Decomposition

Fig. 8 presents predicted decompositions of HHD-GP and SPHHD-GP. It can be observed that although they capture highly similar system dynamics (the first column in Fig. 8(a)), they learn completely different decompositions (the second and third columns in Fig. 8(a)). Compared with the ground truth in Fig. 1, SPHHD-GP learns the physically correct decomposition, so it can accurately predict the system energy. From Fig. 8(b) we can find that the predictions of HHD-GP have large variance, meaning that the model is less certain in isolating individual effects from other terms.

J.4 Experiments of predicting dynamics with unseen frictions

To demonstrate the interpretability of the learned curl-free features, we adapted the trained model to predict dynamics with different friction coefficients. As detailed in Appendix A.1, the curl-free dynamics in the mass-spring system and the pendulum is caused by friction forces in the system. Utilizing this interpretation and the additive structure of the HHD-based models, we can generalize the models to perform inference over dynamics with different friction coefficients. We performed this experiment by first training HHD-GP, SPHHD-GP and D-HNN with data when ρ = 0 . 1 ( ρ is the friction coefficients), then generalizing the trained models to dynamics when ρ = 0 . 05 and ρ = 0 . 5 by multiplying the learned curl-free component with constants 0 . 5 and 5 . 0 , respectively. The results

0

10

20

30

40

50

time

1.0

0.5

0.0

0.5

1.0

q

Ground Truth

HHD-GP

SPHHD-GP

Div-GP

D-HNN

Ind-GP

GIM-GP

40

42

44

46

48

50

time

0.15

0.10

0.05

0.00

0.05

0.10

0.15

q

Ground Truth

HHD-GP

SPHHD-GP

Div-GP

D-HNN

Ind-GP

GIM-GP

0

10

20

30

40

50

time

1.0

0.5

0.0

0.5

1.0

p

Ground Truth

HHD-GP

SPHHD-GP

Div-GP

D-HNN

Ind-GP

GIM-GP

40

42

44

46

48

50

time

0.15

0.10

0.05

0.00

0.05

0.10

0.15

p

Ground Truth

HHD-GP

SPHHD-GP

Div-GP

D-HNN

Ind-GP

GIM-GP

0

5

10

15

20

time

1

0

1

q

Ground Truth

HHD-GP

SPHHD-GP

Div-GP

D-HNN

Ind-GP

GIM-GP

19

20

21

22

23

24

time

0.75

0.50

0.25

0.00

0.25

0.50

0.75

q

Ground Truth

HHD-GP

SPHHD-GP

Div-GP

D-HNN

Ind-GP

GIM-GP

0

5

10

15

20

time

4

2

0

2

4

p

Ground Truth

HHD-GP

SPHHD-GP

Div-GP

D-HNN

Ind-GP

GIM-GP

19

20

21

22

23

24

time

2

1

0

1

2

p

Ground Truth

HHD-GP

SPHHD-GP

Div-GP

D-HNN

Ind-GP

GIM-GP

are shown in Fig. 9, where we can observe that the three models accurately predict the trajectories of the systems with a friction coefficient of 0 . 1 (first column in Fig. 9), but HHD-GP and D-HNN are difficult to generalize to cases with friction coefficients of 0.05 and 0.5 (second and third columns in Fig. 9). In contrast, SPHHD-GP effectively captures the dynamics of different friction conditions, even though it has not been trained for these specific friction coefficients. This adaptability is valuable because it allows the model to be applied to real-world scenarios where friction coefficients may change.

J.5 Results of increasing noise level

To evaluate the robustness of the models, we added the experiments of increasing the standard deviation σ = { 0 . 01 , 0 . 05 , 0 . 10 , 0 . 20 } of Gaussian noise in the training data. Fig. 10 presents the results on the pendulum. As expected, the performance of the models degrades as noise increases. However, it can be observed that SPHHD-GP still performs best at all noise levels compared to the baselines, meaning that our model is more robust to data noise. Furthermore, from Fig. 10(d) we observe that the energy predictions of HHD-GP and D-HNN generate consistently significant errors, insensitive to noise level in training data, which again demonstrate the non-identifiability suffered by these models. In contrast, the energy errors of SPHHD-GP exhibit an increasing trend with increasing

HHD-GP

=

+

SPHHD-GP

=

+

0.5

1.0

1.5

2.0

2.5

3.0

3.5

(a)

1.7275

1.7280

1.7285

1.7290

1.7295

2

4

6

1e

6

(b)

1.5

1.0

0.5

0.0

0.5

1.0

1.5

q

1.5

1.0

0.5

0.0

0.5

1.0

1.5

p

Mass Spring

Pendulum

=0.1

Ground Truth

HHD-GP

SPHHD-GP

D-HNN

1.5

1.0

0.5

0.0

0.5

1.0

1.5

q

1.5

1.0

0.5

0.0

0.5

1.0

1.5

p

=0.05

Ground Truth

HHD-GP

SPHHD-GP

D-HNN

1.5

1.0

0.5

0.0

0.5

1.0

1.5

q

1.5

1.0

0.5

0.0

0.5

1.0

1.5

p

=0.5

Ground Truth

HHD-GP

SPHHD-GP

D-HNN

2.0

1.5

1.0

0.5

0.0

0.5

1.0

1.5

2.0

q

3

2

1

0

1

2

3

p

Ground Truth

HHD-GP

SPHHD-GP

D-HNN

2.0

1.5

1.0

0.5

0.0

0.5

1.0

1.5

2.0

q

3

2

1

0

1

2

3

p

Ground Truth

HHD-GP

SPHHD-GP

D-HNN

2.0

1.5

1.0

0.5

0.0

0.5

1.0

1.5

2.0

q

3

2

1

0

1

2

3

p

Ground Truth

HHD-GP

SPHHD-GP

D-HNN

noise, further demonstrating the effectiveness of addressing model identifiability through symmetry constraints.

J.6 Trajectory and energy prediction of the Chua circuit

Fig. 12 presents the comparison of trajectory predictions for the Chua circuit, and Fig. 11 compares the energy prediction along the trajectory.

0.05

0.10

0.15

0.20

SD of noise

10

1

10 0

RMSE

D-HNN

Div-GP

GIM-GP

HHD-GP

Ind-GP

SPHHD-GP

(a) RMSE of derivatives (

↓

)

0.05

0.10

0.15

0.20

SD of noise

3

2

1

0

1

MNLL

(b) MNLL (

↓

)

0.05

0.10

0.15

0.20

SD of noise

10

2

10

1

VPT

(c) VPT (

↑

)

0.05

0.10

0.15

0.20

SD of noise

10

2

10

1

10 0

RMSE

(d) RMSE of energy (

↓

)

0.0

2.5

5.0

7.5

10.0

12.5

15.0

Time

0.01

0.00

0.01

0.02

Energy

Chua circuit

Ground Truth

HHD-GP

SPHHD-GP

K Societal impact

Generally, our model can have a significant impact on a wide range of applications, such as physical simulation and robotic control. However, it is imperative that we meticulously evaluate the performance of prediction and uncertainty estimation when implementing it in societal contexts.

x

0.4

0.2

0.0

0.2

0.4

0.6

y

0.10

0.05

0.00

0.05

z

0.8

0.6

0.4

0.2

0.0

0.2

0.4

0.6

HHD-GP

Ground Truth

Start Point

0.4

0.2

0.0

0.2

0.4

0.6

x

0.8

0.6

0.4

0.2

0.0

0.2

0.4

0.6

z

HHD-GP

Ground Truth

Start Point

0.10

0.05

0.00

0.05

y

0.8

0.6

0.4

0.2

0.0

0.2

0.4

0.6

z

HHD-GP

Ground Truth

Start Point

0.4

0.2

0.0

0.2

0.4

0.6

x

0.10

0.05

0.00

0.05

y

HHD-GP

Ground Truth

Start Point

x

0.4

0.2

0.0

0.2

0.4

y

0.075

0.050

0.025

0.000

0.025

0.050

0.075

z

0.6

0.4

0.2

0.0

0.2

0.4

0.6

SPHHD-GP

Ground Truth

Start Point

0.4

0.2

0.0

0.2

0.4

x

0.6

0.4

0.2

0.0

0.2

0.4

0.6

z

SPHHD-GP

Ground Truth

Start Point

0.075

0.050

0.025

0.000

0.025

0.050

0.075

y

0.6

0.4

0.2

0.0

0.2

0.4

0.6

z

SPHHD-GP

Ground Truth

Start Point

0.4

0.2

0.0

0.2

0.4

x

0.075

0.050

0.025

0.000

0.025

0.050

0.075

y

SPHHD-GP

Ground Truth

Start Point

x

0.6

0.4

0.2

0.0

0.2

0.4

0.6

0.8

1.0

y

1

0

1

2

3

z

4

2

0

2

4

Div-GP

Ground Truth

Start Point

0.6

0.4

0.2

0.0

0.2

0.4

0.6

0.8

1.0

x

4

2

0

2

4

z

Div-GP

Ground Truth

Start Point

1

0

1

2

3

y

4

2

0

2

4

z

Div-GP

Ground Truth

Start Point

0.6

0.4

0.2

0.0

0.2

0.4

0.6

0.8

1.0

x

1

0

1

2

3

y

Div-GP

Ground Truth

Start Point

x

0.4

0.2

0.0

0.2

0.4

y

0.08

0.06

0.04

0.02

0.00

0.02

0.04

0.06

0.08

z

0.8

0.6

0.4

0.2

0.0

0.2

0.4

0.6

Ind-GP

Ground Truth

Start Point

0.4

0.2

0.0

0.2

0.4

x

0.8

0.6

0.4

0.2

0.0

0.2

0.4

0.6

z

Ind-GP

Ground Truth

Start Point

0.08

0.06

0.04

0.02

0.00

0.02

0.04

0.06

0.08

y

0.8

0.6

0.4

0.2

0.0

0.2

0.4

0.6

z

Ind-GP

Ground Truth

Start Point

0.4

0.2

0.0

0.2

0.4

x

0.08

0.06

0.04

0.02

0.00

0.02

0.04

0.06

0.08

y

Ind-GP

Ground Truth

Start Point

x

0.4

0.2

0.0

0.2

0.4

y

0.075

0.050

0.025

0.000

0.025

0.050

0.075

z

0.6

0.4

0.2

0.0

0.2

0.4

0.6

GIM-GP

Ground Truth

Start Point

0.4

0.2

0.0

0.2

0.4

x

0.6

0.4

0.2

0.0

0.2

0.4

0.6

z

GIM-GP

Ground Truth

Start Point

0.075

0.050

0.025

0.000

0.025

0.050

0.075

y

0.6

0.4

0.2

0.0

0.2

0.4

0.6

z

GIM-GP

Ground Truth

Start Point

0.4

0.2

0.0

0.2

0.4

x

0.075

0.050

0.025

0.000

0.025

0.050

0.075

y

GIM-GP

Ground Truth

Start Point

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The main contributions are summarized in the last paragraph of Section 1.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: High computational complexity is the main limitation of our proposed model, which has been discussed in Appendix I.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: They are provided in Section 4, Section 5 and Appendix D.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The information needed to reproduce the experimental results is provided in Section 6, Appendix E, Appendix F and Appendix G.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: The code will be released afterwards.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The training and test details are provided in Section 6, Appendix E, Appendix F and Appendix G.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We have provided the standard errors in Section 6.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We used a single Nvidia GeForce GTX 3050 Ti GPU in the experiments, as mentioned in the first paragraph of Appendix G.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conducted in the paper conform with the NeurIPS Code of Ethics. Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: [TODO]

Guidelines: We have discussed the societal impacts in Appendix K.

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The code packages and datasets are properly cited in the paper.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel dimensionality reduction method.\nThe method relies on the Helmholz-Hodge decomposition to identify the dynamical system through a decomposition into a curl-free and a divergence-free part.\nFurthermore, the method introduces a way to incorporate priors that constrain the identified vector fields based on symmetry constraints.\nThis method is applied to various toy models (mass-spring system, pendulum, Chua circuit) and to real world data (ocean current fields).\n\nSTRENGTHS:\nThis paper addresses an important problem in dynamical systems modeling.\nThis paper is the first to introduce the use of symmetries to construct unique GP models of dynamics that are decomposed into div- and curl-free vector fields.\nThis method is shown to be consistent in terms of convergence to the true parameter values as the amount of data increases.\nThe paper discusses the computational complexity of the proposed method.\nThe paper demonstrates that the presented framework is outperforms baselines on various toy models\nFurthermore, the method is shown to be able to handle noisy data.\nFinally, the method is applied to a real world dataset, which indicates practical significance.\n\nWEAKNESSES:\nThe overall applicability of this method is questionable due to not completely addressing the identifiability issue of the decomposition. The issue is only resolved for systems with known symmetries, see also Questions.\n\nThe paper mentions that the accurate prediction of energy demonstrates the interpretability of learned div-free features. \nHowever, this is not well-explained. The paper claims that the curl- and div-free components are physically meaningful, but this should be better justified.\n\nThe paper is missing comparison to other existing approaches for modeling physical systems. See for example: \nGuo, Q., Mandal, M. K., \\& Li, M. Y. (2005). Efficient Hodge–Helmholtz decomposition of motion fields. Pattern Recognition Letters, 26(4), 493-501.\n\nCompare to other dynamical systems reconstruction methods:\n- Hess, F., Monfared, Z., Brenner, M., \\& Durstewitz, D. (2023). Generalized teacher forcing for learning chaotic dynamics. arXiv preprint arXiv:2306.04406.\n\n-I. Mac\\^edo and R. Castro, “Learning Divergence-Free and Curl-Free Vector Fields with Matrix-Valued Kernels,” technical report, Instituto de Matema ́tica Pura e Aplicada, Rio de Janeiro, Brazil, 2010.\n\nFinally, there is no code provided to show the method's working and not all hyperparameters are described (see also Questions).\n\nQUESTIONS:\nWhat is the relevance of $\\mathcal{G}$-equivariance and uniqueness/identifiability?\nAppendix H: Uniqueness and Symmetries\n- How do you know what priors to choose? Isn't an agnostic method better?\n- How do you know which symmetries to enforce?\n\nCould it be more fruitful to restrict yourself to a compact manifold with boundary so that the decomposition is unique?\n\n\n\n# Implementation details\n- How were the kernel parameters and noise variances initialized? Does training depend on initialization?\n(Here you mention that these are hyperparameters, but typically one refers to non trained parameters as hyperparamters.)\n-What were the ADAM optimizer paramters used for training?\n\n\n# Choice of evaluation metrics \n-Why did you mostly focus on performance of the vector field? If the point is learning the dynamics, metrics that look at the dynamics directly would be preferable.\nWhy not prediction of the future state/trajectory? Why prediction only measured through VPT?\n\nLIMITATIONS:\nThe authors should comment on limitations and societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors tackle the problem of modelling dynamical systems in scenarios where it may not be possible to straightforward to determine the exact form of the ODEs governing the system and optimise their parameters directly. Whilst physics-informed Bayesian models which learn divergence-free vector fields are capable of extrapolating dynamics effectively in an interpretable manner, they cannot represent certain common real-world behaviours such as dissipation. Proposed in this work is a Gaussian process (GP) which combines the utility of the div-free vector field with a curl-free vector field, an approach which is theoretically motivated by the Helmholtz-Hodge decomposition which states that many vector fields may be decomposed into a sum of a div-free and curl-free field. This approach (termed the SPHHD-GP) allows for a much wider range of dynamical systems to be modelled without strict assumptions on the form of the system, and also permits a symmetry preserving representation which allows for enhanced interpretability of the learned dynamics. Experimental results are presented which show the efficacy of the approach on tasks such as learning Hamiltonian dynamics from noisy data, learning chaotic dynamics and modelling ocean currents.\n\nSTRENGTHS:\n-\tThe approach taken is novel, but not unnecessarily complex. The theoretical insight regarding the Helmholtz-Hodge decomposition is combined with a basic building block of GP-based modelling (additive combination of GPs) in order to yield a technique which consistently outperforms all the relevant baselines presented by the authors.\n-\tThe quality of the work carried out in general is very good; the experiments are thorough and well documented overall. The Hamiltonian dynamics experiments are effectively a standard benchmark in this area, the chaotic system presents a further challenge for the model, and the ocean current experiment is a real-world example of how this approach may be useful in a practical setting. The performance of the SPHHD-GP across all of these experiments is impressive.\n-\tThe clarity of presentation is also very good, and I believe that the work would be accessible both to researchers from a dynamical systems background without in-depth knowledge of GPs, and vice versa. The narrative of the paper is well crafted, from problem setting to initial formulation of the model, onto discussion of limitations regarding identifiability, and finally the implementation of symmetry-based constraints which address these limitations.\n\nWEAKNESSES:\n-\tFrom the text it appears the ocean current experiment was performed using a sparse GP implementation, whereas the rest of the experiments were performed using exact inference, is this correct? This is only mentioned very briefly in passing and few details given; I know the details of the sparse GP formulation are probably a given to readers with a background in the area, but I think adding some detail to the appendix on how this was implemented would be useful to many readers.\n-\tSeveral of the derivations, proofs and discussions in the appendix are of sufficient relevance that it would be great if they could be included in the main text, however I’m obviously aware that the authors are constrained by the page limit so I wouldn’t expect any alterations based on this. I think the amount of content more just speaks to the fact the work has been carried out in a thorough manner.\n\nQUESTIONS:\n-\tPlease address the point mentioned in the weaknesses section regarding adding some brief details about the sparse implementation into the appendix and make some reference to this in the main text (maybe where the sparse GP framework is mentioned in Section 6.3).\n-\tRelated to the above point, could you speak to how the sparse formulation performs empirically compared to the exact approach on problems for which both are computationally feasible (i.e., sections 6.1 and 6.2)?\n\nLIMITATIONS:\nThe authors address the main limitation of the model in Appendix I which is the considerable computational complexity of the model, $\\mathcal{O}(m^3 n^3)$, although this approach can also be implemented using sparse GPs instead to reduce this. The authors also briefly mention in Appendix K how the model has a wide range of applications, but from a societal impact perspective it is important that predictions and uncertainty estimates are rigorously evaluated. As I don’t think the work has significant potential for negative impact, I think this is adequate.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-08"}

comments: SUMMARY:
The paper formulates a vector-valued GP model to infer unknown vector fields. The authors discuss how to impose symmetry-based constraints on the GP models to ensure physically meaningful decompositions. The paper provides theoretical proofs to support the construction of curl-free and divergence-free vector fields that preserve desired symmetries. The model's effectiveness is validated through experiments on various dynamical systems, including dissipative Hamiltonian dynamics and chaotic systems.

STRENGTHS:
- The curl-free and divergence-free kernel considering symmetry constraints is novel.
- The manuscript is generally easy to read.
- The model is evaluated on multiple dynamical systems.

WEAKNESSES:
- There are missing citations for relevant existing methods.
- There is no discussion on computational complexity.

QUESTIONS:
- In [R1], a Gaussian Process for dissipative Hamiltonian systems has been proposed. Can HHD-GP be considered a generalization of [R1]? What are the essential differences between SSGP and HHD-GP? While SPHHD-GP is clearly novel, it was unclear whether HHD-GP is new compared to [R1].
- Please add a discussion on computational complexity. Specifically, numerical integration is used to consider symmetry constraints, but could this be a computational bottleneck?
- In Figure 3, the error of HHD-GP appears to increase with the number of samples, which seems unnatural. Shouldn't this be analyzed in more detail?

[R1] Yusuke Tanaka, Tomoharu Iwata, Naonori Ueda, Symplectic Spectrum Gaussian Processes: Learning Hamiltonians from Noisy and Sparse Data, NeurIPS, 2022.

LIMITATIONS:
The differences between HHD-GP and existing methods (such as [R1]) should be thoroughly discussed.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-09

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper derives a G-equivariant versions of the both curl (through the Haar Integration kernel) and divergence free (through GIM kernel) kernels. This is then used to define a prior over the Helmholtz decomposition and is identifiable wrt to relevant functions in the Euclidean group (translation, rotation, reflection etc). This is demonstrated on a range of examples.\n\nSTRENGTHS:\nThis is a well written, clear paper, that improves on recent work with convincing experiments.\n\nWEAKNESSES:\n1.\tThe justification for how and why the curl and div free kernels should be invariant is not given until the experiments/the appendix. It would be nice if this was discussed in section 5.\n2.\tThe fact that the actual symmetries required is problem specific is not really clear in the main paper. In the appendix it is stated, but it would be good if this was made more explicit in the main paper.\n\nQUESTIONS:\n1.\tIs the proposed symmetry-preserving Helmholtz decomposition as expressive as non-symmetry preserving one? Ie any (sufficient smooth) field can be decomposed into curl and divergence free part. Is the same for the symmetric-preserving decomposition?\n2.\tOn line 328 it is stated that mirror symmetry is incorporated into the div-free kernel. Where is this kernel given?\n3.\tAre there general symmetries that should be held or is this always going to be problem/application specific? \n4.\tWhen the kernels do not admit a closed form how sensitive are they to approximations? And does this significantly affect the computational cost?\n5.\tIs there any impact of using sparse GPs? Ie are the sparse models still guaranteed to be G-invariant? \n6.\tRelevant GP citations:\n    o\tEquivariant Learning of Stochastic Fields: Gaussian Processes and Steerable Conditional Neural Processes, Holderrieth et al\n    o\tVector-valued Gaussian Processes on Riemannian Manifolds via Gauge Equivariant Projected Kernels – Hutchinson et al\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-15"}