Neural Symplectic Form: Learning Hamiltonian Equations on General Coordinate Systems

1 Introduction . In recent years , the applications of deep learning to learn the fundamental equations of classical mechanics have been actively studied . Analytical mechanics , which is a theory of classical mechanics , is classified into Lagrangian mechanics and Hamiltonian mechanics [ 1 , 3 , 18 ] . The equations of motion of Lagrangian mechanics are called the Euler–Lagrange equation , for which several neural network models were proposed [ 7 , 17 ] . In Lagrangian mechanics , the equations are described using the state variables and the time derivatives of them . This feature of Lagrangian mechanics makes it easy to prepare the data necessary for learning . On the other hand , Hamiltonian mechanics can describe more general equations which are not covered by Lagrangian mechanics . In the previous 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . models for Hamiltonian equations , the equation d dt ( q p ) = ( O I −I O ) ( ∂H ∂q ∂H ∂p ) ( 1 ) is typically assumed , where q is the state variable , and p is a variable called the generalized momentum . It is known that Hamiltonian equations can be written in this form by using these variables ; more precisely , any Hamiltonian equation can be locally written as ( 1 ) by using a special coordinate system called the Darboux coordinate system [ 20 ] . However , this coordinate system depends on the generally unknown Hamiltonian ( the energy function ) , and it is usually not possible to prepare data in this coordinate system . On the other hand , Hamiltonian equations also have a coordinate-free representation using the symplectic 2-form [ 20 ] . In this paper , we propose a method to learn the symplectic 2-form from data by using neural networks , thereby introducing a method to learn the Hamilton equation from data represented in general coordinate systems , not restricted to the generalized momentum . As explained below , general 2-forms are corresponding to skew symmetric matrices . Hence , a naive way to learn the symplectic 2-form is learning a skew symmetric matrix ; however , actually , the equation learned in this way may not be Hamiltonian because not all skew symmetric matrices are corresponding to symplectic 2-forms . In the proposed method , instead of learning the 2-form directly , a 1-form that derives the symplectic 2-form is learned ( see Figure 1 ) . Because the proposed model can be applied to data in any coordinate system , it can be employed for extracting the unknown underlying Hamiltonian structure behind the data that are expected to have such a structure . For example , many polynomial ordinary differential equations are known to have a Hamiltonian structure , but it is not easy to find the structure analytically . The proposed method extracts the hidden structure in such cases from data . If the Hamiltonian structure can be found , it is possible to make predictions while preserving the energy conservation law . In addition to the energy , other hidden conserved quantities can be also extracted . Main contributions of this paper include : Symplectic geometric approach to learning symplectic 2-forms . The symplectic 2-form required to describe the Hamilton equation corresponds to a skew-symmetric matrix , but conversely , not all skew-symmetric matrices correspond to a symplectic 2-form . In this paper , we propose an efficient model by learning the symplectic 1-form that derives the symplectic 2-form with neural networks . Learning the Hamilton equation from data in arbitrary coordinate systems . By using the coordinate-free representation of the Hamilton equation , it is possible to learn the Hamilton equation from data represented in a general coordinate system , not restricted to the Darboux coordinate system . In this way , the proposed method can determine whether the given data can be explained by the hidden theory of classical mechanics or not . A universal approximation theorem is also provided . 2 Related Work . Neural Networks for Hamiltonian Mechanics Neural ordinary differential equation ( NODE , [ 5 ] ) is a neural network that models the time-derivative of the states , thereby defining an ordinary differential equation ( ODE ) in a general way . Due to the generality , this model does not admit the energy conservation law . Hamiltonian neural network ( HNN ) is a neural network that models the HamiltonianH and defines the dynamics following the Hamiltonian mechanics , thereby ensuring the energy conservation law [ 11 ] . Although models of the form ( 1 ) are often used in the previous studies , the Hamilton equation has this form only in the Darboux coordinates [ 20 ] . The Darboux coordinate system , which is essentially the generalized momentum p , is defined using the Hamiltonian H , which is the target to be learned . Hence , training data in the coordinate is usually unavailable . In addition , Hamiltonian equations are defined on general symplectic manifolds ; however , in the existing studies , cotangent bundles are typically assumed as the symplectic manifold . A numerical integration of the Hamiltonian system is known to destroy the symplectic structure and does not conserve the Hamiltonian H , unless the integrator is carefully designed ( see , e.g. , [ 12 , 26 ] ) . Several studies focus on the numerical integration that conserves the Hamiltonian H [ 6 , 9 , 19 , 27 , 28 ] . The HNN was extended to energy-conserving partial differential equation ( PDE ) systems , such as the Korteweg–De Vries ( KdV ) equation [ 19 ] , to dissipative systems , such as a pendulum with friction [ 19 , 29 ] . In [ 15 ] , a discrete-time model is proposed for Poisson systems , which are extension of Hamiltonian systems , where the skew symmetric matrix can be degenerate . In particular , in [ 15 ] , dynamics with state-dependent skew symmetric matrices are leaned by introducing coordinate transformations for learning the dynamics in the latent space . The proposed method is different from this study in that our method does not use the coordinate transformations and hence has an advantage in interpretability . Another approach is employed in [ 4 , 16 ] , where the symplectic map is modeled . SympNets [ 16 ] are also shown to be universal approximators , and , in [ 4 ] , a bound on the prediction error is provided . The proposed method can be combined with these discrete-time approaches . Modeling using machine learning has also been performed in the field of quantum mechanics , for example , by Tkatchenko and coworkers ( e.g . [ 22 , 25 ] ) . Some breakthroughs have been reported that have not been possible with conventional computational chemistry methods . The relationship with these studies needs to be investigated in the future . Neural Networks for Lagrangian Mechanics Another branch of studies focus on Lagrangian mechanics . Lagrangian neural network ( LNN ) is a neural network that models the Lagrangian L in a general way [ 7 ] , and deep Lagrangian network explicitly defines the kinetics energy with a trainable mass matrix [ 17 ] . Lagrangian mechanics defines Lagrangian systems on tangent bundles [ 18 ] , where the state is the pair of the position q and velocity q̇ . The systems have specific symplectic structures , which are equivalent to Hamiltonian systems in general coordinate systems on cotangent bundles . Because the LNN does not assume equations of a specific form , it can learn a wider class of systems , including a double pendulum , in addition to the systems that the HNN can learn ( see Table 1 ) . Similarly to the HNN , numerical integrators that preserve the symplectic structure have been investigated [ 8 , 23 ] . Neural network architectures that ensure translational and rotational symmetries have also been investigated [ 10 , 24 ] . As mentioned before , not all Hamiltonian systems are defined on cotangent bundles . For examples , the some polynomial equations including the Lotka–Volterra equation have a different symplectic structure [ 13 ] . In fact , the Lotka–Volterra equation is actually a Hamiltonian system , even though its states are not position , velocity , nor generalized momentum . These equations are out of the scopes of the HNN and LNN ( see Table 1 ) . 3 Methods . Coordinate-Free Representation of Hamiltonian Equations In this paper , we propose a neural network model based on a coordinate-free representation of Hamiltonian equations . First , we describe this representation . Because detailed knowledge of geometry is required for a precise description of this representation , the details are omitted here . For more details , see Appendix A . The model in this paper can be used on a general symplectic manifold , but for simplicity , we describe the case where the phase space isM = R2N . A differential 2-form ω onM is a skew-symmetric bilinear function that maps given two vectors into a real number , depending on each point u onM . The skew-symmetric bilinear function defined by ω has the following matrix representation : ωu ( v1 , v2 ) = v > 1 Wuv2 , for all v1 , v2 ∈ R2N , where Wu is a skew-symmetric matrix , and the subscript u denotes that ω and hence its matrix representation Wu depend on u . A symplectic 2-form is a differential 2-form that is nondegenerate and closed . See Appendix A for the precise meanings of these terms . The following is the coordinate-free form of Hamiltonian equations [ 18 ] du dt = XH , ω ( XH , v ) = dH ( v ) for all v ∈ R2N . ( 2 ) Here , dH is the Fréchet derivative of the Hamiltonian H , and XH is a vector field depending on H . This equation is satisfied regardless of the coordinate system in which the state variable u is expressed . Therefore , by using this equation as a model , as long as the given data is described by the Hamilton equation , it is possible to learn both the symplectic 2-form and the Hamiltonian that define the Hamilton equation , no matter what coordinate system the data is given in . Naive method : the skew matrix learning By replacing the symplectic 2-form with the matrix Wu , ( 4 ) can be rewritten as du dt = XH , X > HWuv = v · ∇H for all v ∈ R2N ⇔ du dt = W− > u ∇H . A natural model based on this representation would be a model in which Wu and H are modeled by multilayer perceptrons : du dt = W− > u , NN∇HNN ( u ) , ( 3 ) where HNN is a function of u defined by a multilayer perceptron , and Wu , NN is a skew-symmetric matrix depending on u represented by another multilayer perceptron . We refer this model as the skew matrix learning . If u is represented in the Darboux coordinate system , ( 3 ) becomes ( 1 ) , and hence the model is the same as the Hamiltonian neural network . Similarly to Hamiltonian neural networks , this model has the energy conservation law . Theorem 1 . Solutions to ( 3 ) satisfy dHNN/dt = 0 . See Appendix C for the proof . Neural Symplectic Form Although the model in the previous section is natural , it is a redundant model and inefficient for learning as explained below . As explained in Introduction , although a differential 2-form corresponds to a skew-symmetric matrix , not all skew-symmetric matrices define a symplectic 2-form . Symplectic 2-forms have the characteristic feature of being closed . In this paper , we propose a model in which the learned 2-form is guaranteed to be closed . We refer the learned 2-form as the neural symplectic form . First , the necessary terminology is briefly explained . See Appendix A for details . A differential 0-form onM = R2N is a function fromM to R. A differential 1-form θ onM = R2N is a field of linear functions each of which maps a vector v ∈ R2N to θu ( v ) ∈ R , depending on each point u ∈ M. In general , a linear function from R2N to R can be expressed as an inner product with a vector , so a differential 1-form can be expressed as a vector field depending on u . A differential operation called the exterior derivative d is defined for differential forms . The exterior derivative is a graded linear map , i.e . a linear map depending on an integer k , which transfers a differential k-form to a differential ( k + 1 ) -form and has the property that dd = 0 . The differential form in Ker d is called a closed form . Since a symplectic 2-form is a closed form , in order to learn the symplectic 2-form by neural networks , neural networks should be designed so that differential 2-forms represented by the neural networks are contained in Ker d. Meanwhile , due to the property of the exterior derivative , dd = 0 , it holds that Im d ⊂ Ker d. Actually , according to the de Rham theorem , when the phase space is R2N , these two spaces coincide : Im d = Ker d. The difference between these two spaces Im d/Ker d is called the cohomology space . The de Rham theorem states that the cohomology space is isomorphic to the homology space , which is roughly a space of `` spatial holes . '' Because R2N contains no holes , the homology space must vanish , and hence Im d = Ker d holds ( see Figure 2 . ) Even when the phase space has a hole , in many cases the space can be embedded in a large Euclid space without holes , and the model should be defined on that space . See Appendix B . Therefore , in this paper , instead of learning the symplectic 2-form directly , we propose a method to learn the symplectic 2-form by learning the differential 1-form of which exterior derivative gives the symplectic 2-form . The following is the coordinate-free form of the proposed model : ω̃ = dθNN , du dt = X̃HNN , ω ( X̃HNN , v ) = dHNN ( v ) for all v ∈ R2N ( 4 ) Unlike ( 3 ) , in this model skew-symmetric matrices that do not correspond to symplectic forms are not explored . Therefore , this model can be trained much more efficiently than ( 3 ) . In fact , suppose that the dimension of the phase space is 2N . Then , the number of components of the skew-symmetric matrix is N ( 2N − 1 ) , while the differential 1-form is represented as a vector field , the number of which components is 2N . Consequently , a model using the neural symplectic form can reduce the order of the number of functions to be learned from O ( N2 ) to O ( N ) without sacrificing the expressive power . As per the above , the differential 1-form can be expressed as a vector field . Hence the neural network for modeling the 1-form in the proposed model essentially models a vector field YNN , which represents the differential 1-form θNN . As shown in Appendix A , the vector field YNN is transformed to the matrix W̃u representing the symplectic 2-form as follows : ( W̃u ) i , j = ∂ ( YNN ) i ∂uj − ∂ ( YNN ) j ∂ui . In the actual model , YNN given by the neural network is differentiated by the automatic differentiation , and Wu , NN is obtained by substituting the derivatives into the above equation . Thus we have the model expressed in terms of vectors and a matrix , without using the differential forms : du dt = W̃− > u ∇HNN ( u ) , ( W̃u ) i , j = ∂ ( YNN ) i ∂uj − ∂ ( YNN ) j ∂ui . ( 5 ) The proposed model has the energy conservation law and the universal approximation property . For the proofs , see Appendix C. Theorem 2 . Solutions to ( 4 ) , or equivalently , to ( 5 ) satisfy dHNN/dt = 0 . Theorem 3 . Suppose that the Hamilton equation to be learned can be represented in the form ( 2 ) using a symplectic 2-form ω derived from a 1-form θ ∈ W d , p and the Hamiltonian H ∈ W 1 , p . Suppose also that the phase space is compact . The model ( 4 ) with multilayer perceptrons with a sufficiently smooth activation function has the universal approximation property in the sense that θ and H can be approximated by the multilayer perceptrons with arbitrary accuracy . W 1 , p and W d , p are Sobolev spaces ; see [ 2 , 14 ] . Learning dynamics by using the neural symplectic form The proposed model learns the symplectic form and the Hamiltonian by minimizing the squared error between the left- and right-hand sides of ( 4 ) , assuming that time series data of state vectors { u ( n ) } and its time-derivatives { dudt ( n ) } are given ; minimize ∑ n ∣∣∣∣∣dudt ( n ) − W̃− > u ( n ) ∇HNN ( u ( n ) ) ∣∣∣∣∣ 2 , ( W̃u ( n ) ) i , j = ∂ ( YNN ) i ∂uj ( u ( n ) ) − ∂ ( YNN ) j ∂ui ( u ( n ) ) . If the time-derivatives are not available , interpolated data should be used . For each state vector u ( n ) , first YNN ( u ( n ) ) is computed . Then , the derivatives of YNN ( u ( n ) ) are calculated by the automatic differentiation to obtain ( W̃− > u ) i , j . Meanwhile , the gradient of the Hamiltonian is computed in the same way ; first HNN ( ( n ) ) is computed , and then ∇HNN ( ( n ) ) is obtained by the automatic differentiation . In the standard Hamiltonian neural networks only the computation of the gradient of the Hamiltonian is required . Hence the computational cost of the neural symplectic form is roughly 2N times the computational cost of Hamiltonian neural networks , where 2N is the number of the state variables , because the number of components of YNN ( u ( n ) ) is 2N . When N is large , such as in many-body problems , the most computationally expensive part may be computation of the inverse of the matrix W̃u ( n ) , which requires O ( N3 ) operations . To avoid the computation of the inverse , one may be tempted to minimize ∑ n ∣∣∣∣∣W̃ > u ( n ) dudt ( n ) −∇HNN ( u ( n ) ) ∣∣∣∣∣ 2 . However , this model does not work because this is trivially minimized by setting W̃ > u ( n ) = O and HNN = ( constant . ) Learning the inverse while maintaining the symplectic structure is future work .

Summary: Authors present a ML-based approach to learning dynamics in the space of Hamiltonian systems. The approach is based on generic formulation of Hamiltonian mechanics in the language of differential geometry, effectively avoiding the requirement of knowing the appropriate generalized coordinates beforehand. The proposed approach is based on modeling the symplectic 2-form (known to be closed) as an exterior derivative of a parametrized (learned) 1-form. This method is termed "neural symplectic form" and provides a compact parametrization of the underlying governing equations.  The proposed model is trained by minimizing the MSE between estimated ground truth time derivatives of the dynamical system and predictions of the model on several datasets: mass-spring system; double pendulum; Lotka-Volterra equations (which are not captured by a traditional formulation of Hamiltonian mechanics with fixed symplectic form).  The model is compared to baselines (HNN, LNN and a naive implementation termed skew matrix learning) on test loss and errors of the conserved quantity in the predicted dynamics accumulated during simulation for a fixed time.  Authors find that the proposed model produces best results across most of the comparisons, especially on the Lotka-Volterra test case.