Nonseparable Symplectic Neural Networks

1 INTRODUCTION . A Hamiltonian dynamic system refers to a formalism for modeling a physical system exhibiting some specific form of energy conservation during its temporal evolution . A typical example is a pendulum whose total energy ( referred to as the system ’ s Hamiltonian ) is conserved as a temporally invariant sum of its kinematic energy and potential energy . Mathematically , such energy conservation indicates a specific geometric structure underpinning its time integration , named as a symplectic structure , which further spawns a wide range of numerical time integrators to model Hamiltonian systems . These symplectic time integrators have proven their effectiveness in simulating a variety of energy-conserving dynamics when Hamiltonian expressions are known as a prior . Examples encompass applications in plasma physics ( Morrison , 2005 ) , electromagnetics ( Li et al. , 2019 ) , fluid mechanics ( Salmon , 1988 ) , and celestial mechanics ( Saari & Xia , 1996 ) , to name a few . On another front , the emergence of the various machine learning paradigms with their particular focus on uncovering the hidden invariant quantities and their evolutionary structures enable a faithful prediction of Hamiltonian dynamics without knowing its analytical energy expression beforehand . The key mechanics underpinning these learning models lie in a proper embedding of the strong mathematical inductive priors to ensure Hamiltonian conservation in a neural network data flow . Typically , such priors are realized in a variational way or a structured way . For example , in Greydanus et al . ( 2019 ) , the Hamiltonian conservation is encoded in the loss function . This category of methods does not assume any combinatorial pattern of the energy term and therefore relies on the inherent expressiveness of neural networks to distill the Hamiltonian structure from abundant training datasets ( Choudhary et al. , 2019 ) . Another category of Hamiltonian networks , which we refer to as structured approaches , implements the conservation law indirectly by embedding a symplectic time integrator ( DiPietro et al. , 2020 ; Tong et al. , 2020 ; Chen et al. , 2020 ) or composition of linear , activation , and gradient modules ( Jin et al. , 2020 ) into the network architecture . ∗shiying.xiong @ dartmouth.edu . One of the main limitations of the current structured methods lies in the separable assumption of the Hamiltonian expression . Examples of separable Hamiltonian systems include the pendulum , the Lotka–Volterra ( Zhu et al. , 2016 ) , the Kepler ( Antohe & Gladwell , 2004 ) , and the Hénon–Heiles systems ( Zotos , 2015 ) . However , beyond this scope , there exist various nonseparable systems whose Hamiltonian has no explicit expression to decouple the position and momentum energies . Examples include incompressible flows ( Suzuki et al. , 2007 ) , quantum systems ( Bonnabel et al. , 2009 ) , rigid body dynamics ( Chadaj et al. , 2017 ) , charged particle dynamics ( Zhang et al. , 2016 ) , and nonlinear Schrödinger equation ( Brugnano et al. , 2018 ) . This nonseparability typically causes chaos and instability , which further complicates the systems ’ dynamics . Although SympNet in Jin et al . ( 2020 ) can be used to learn and predict nonseparable Hamiltonian systems , multiple matrices of the same order with system dimension are needed in the training process of SympNet , resulting in difficulties in generalizing into high-dimensional large-scale N-body problems which are common in a series of nonseparable Hamiltonian systems , such as quantum multibody problems and vortexparticle dynamics problems . Such chaotic and large-scale nature jointly adds shear difficulties for a conventional machine learning model to deliver faithful predictions . In this paper , we propose an effective machine learning paradigm to predict nonseparable Hamiltonian systems . We build a novel neural network architecture , named nonseparable symplectic neural networks ( NSSNNs ) , to enable accurate and robust predictions of long-term Hamiltonian dynamics based on short-term observation data . Our proposed method belongs to the category of structured network architectures : it intrinsically embeds the symplectomorphism into the network design to strictly preserve the symplectic evolution and further conserves the unknown , nonseparable Hamiltonian energy . The enabling techniques we adopted in our learning framework consist of an augmented symplectic time integrator to asymptotically “ decouple ” the position and momentum quantities that were nonseparable in their original form . We also introduce the Lagrangian multiplier in the augmented phase space to improve the system ’ s numerical stability . Our network design is motivated by ideas originated from physics ( Tao , 2016 ) and optimization ( Boyd et al. , 2004 ) . The combination of these mathematical observations and numerical paradigms enables a novel neural network architecture that can drastically enhance both the scale and scope of the current predictions . We show a motivational example in Figure 1 by comparing our approach with a traditional HNN method ( Greydanus et al. , 2019 ) regarding their structural designs and predicting abilities . We refer the readers to Section 6 for a detailed discussion . As shown in Figure 1 , the vortices evolved using NSSNN are separated nicely as the ground truth , while the vortices merge together using HNN due to the failure of conserving the symplectic structure of a nonseparable system . The conservative capability of NSSNN springs from our design of the auxiliary variables ( red x and y ) which converts the original nonseparable system into a higher dimensional quasi-separable system where we can adopt a symplectic integrator . 2 RELATED WORKS . Data-driven physical prediction . Data-driven approaches have been widely applied in physical systems including fluid mechanics ( Brunton et al. , 2020 ) , wave physics ( Hughes et al. , 2019 ) , quantum physics ( Sellier et al. , 2019 ) , thermodynamics ( Hernandez et al. , 2020 ) , and material science ( Teicherta et al. , 2019 ) . Among these different physical systems , data-driven fluid receives increasing attention . We refer the readers to Brunton et al . ( 2020 ) for a thorough survey of the fundamental machine learning methodologies as well as their uses for understanding , modeling , optimizing , and controlling fluid flows in experiments and simulations based on training data . One of the motivations of our work is to design a versatile learning approach that can predict complex fluid motions . On another front , many pieces of research focus on incorporating physical priors into the learning framework , e.g. , by enforcing incompressibility ( Mohan et al. , 2020 ) , the Galilean invariance ( Ling et al. , 2016 ) , quasistatic equilibrium ( Geng et al. , 2020 ) , the Lagrangian invariance ( Cranmer et al. , 2020 ) , and Hamiltonian conservation ( Hernandez et al. , 2020 ; Greydanus et al. , 2019 ; Jin et al. , 2020 ; Zhong et al. , 2020 ) . Here , inspired by the idea of embedding physics priors into neural networks , we aim to accelerate the learning process and improve the accuracy of our model . Neural networks for Hamiltonian systems . Greydanus et al . ( 2019 ) introduced Hamiltonian neural networks ( HNNs ) to conserve the Hamiltonian energy of the system by reformulating the loss function . Inspired by HNN , a series of methods intrinsically embedding a symplectic integrator into the recurrent neural network was proposed , such as SRNN ( Chen et al. , 2020 ) , TaylorNet ( Tong et al. , 2020 ) and SSINN ( DiPietro et al. , 2020 ) , to solve separable Hamiltonian systems . Combined with graph networks ( Sanchez-Gonzalez et al. , 2019 ; Battaglia et al. , 2016 ) , these methods were further generalized to large-scale N-body problems induced by interaction force between the particle pairs . Jin et al . ( 2020 ) proposed SympNet by directly constructing the symplectic mapping of system variables within neighboring time steps to handle both separable and nonseparable Hamiltonian systems . However , the scale of parameters in SympNet for training N dimensional Hamiltonian system is O ( N2 ) , which makes it hard to be generalized to the high dimensional N-body problems . Our NSSNN overcomes these limitations by devising a new Hamiltonian network architecture that is specifically suited for nonseparable systems ( see details in Section 5 ) . In addition , the Hamiltonianbased neural networks can be extended to further applications . Toth et al . ( 2020 ) developed the Hamiltonian Generative Network ( HGN ) to learn Hamiltonian dynamics from high-dimensional observations ( such as images ) . Moreover , Zhong et al . ( 2020 ) introduced Symplectic ODE-Net ( SymODEN ) , which adds an external control term to the standard Hamiltonian dynamics . 3 FRAMEWORK . 3.1 AUGMENTED HAMILTONIAN EQUATION . We start by considering a Hamiltonian system with N pairs of canonical coordinates ( i.e . N generalized positions and N generalized momentum ) . The time evolution of canonical coordinates is governed by the symplectic gradient of the Hamiltonian ( Hand & Finch , 2008 ) . Specifically , the time evolution of the system is governed by Hamilton ’ s equations as dq dt = ∂H ∂p , dp dt = −∂H ∂q , ( 1 ) with the initial condition ( q , p ) |t=t0 = ( q0 , p0 ) . In a general setting , q = ( q1 , q2 , · · · , qN ) represents the positions and p = ( p1 , p2 , ... pN ) denotes their momentum . Function H = H ( q , p ) is the Hamiltonian , which corresponds to the total energy of the system . An important feature of Hamilton ’ s equations is its symplectomorphism ( see Appendix B for a detailed overview ) . The symplectic structure underpinning our proposed network architecture draws inspirations from the original research of Tao ( 2016 ) in computational physics . In Tao ( 2016 ) , a generic , high-order , explicit and symplectic time integrator was proposed to solve ( 1 ) of an arbitrary separable and nonseparable HamiltonianH . This is implemented by considering an augmented Hamiltonian H ( q , p , x , y ) : = HA +HB + ωHC ( 2 ) with HA = H ( q , y ) , HB = H ( x , p ) , HC = 1 2 ( ‖q − x‖22 + ‖p− y‖22 ) ( 3 ) in an extended phase space with symplectic two form dq ∧ dp+ dx ∧ dy , where ω is a constant that controls the binding of the original system and the artificial restraint . Notice that the Hamilton ’ s equations forH dq dt = ∂H ∂p = ∂H ( x , p ) ∂p + ω ( p− y ) , dp dt = −∂H ∂q = −∂H ( q , y ) ∂q − ω ( q − x ) , dx dt = ∂H ∂y = ∂H ( q , y ) ∂y − ω ( p− y ) , dy dt = −∂H ∂x = −∂H ( x , p ) ∂x + ω ( q − x ) , ( 4 ) with the initial condition ( q , p , x , y ) |t=t0 = ( q0 , p0 , q0 , p0 ) have the same exact solution as ( 1 ) in the sense that ( q , p , x , y ) = ( q , p , q , p ) . Hence , we can get the solution of ( 1 ) by solving ( 4 ) . Furthermore , it is possible to construct high-order symplectic integrators forH in ( 4 ) with explicit updates . Our model aims to learn the dynamical evolution of ( q , p ) in ( 1 ) by embedding ( 4 ) into the framework of NeuralODE ( Chen et al. , 2018 ) . The coefficient ω acts as a regularizer , which stabilizes the numerical results ( see Section 4 ) .

Summary: This paper describes a deep learning approach for predicting Hamiltonian systems. The original paper enforces conservation in the loss function. Several of the follow-up papers embed a symplectic integrator instead, but these couldn't handle non-separable systems. This paper can both handle non-separable systems and use a symplectic integrator to enforce conservation. They demonstrate their system on quite a few examples and show lower error (both in prediction and the deviation in the Hamiltonian) than a NeuralODE or the original HNN paper. The final example, in Figure 5, shows a compelling visual improvement.