Deconstructing the Inductive Biases of Hamiltonian Neural Networks

1 INTRODUCTION . The inductive biases of convolutional neural networks , such as translation equivariance , locality , and parameter sharing , play a key role in their generalization properties . Other biases have similarly guided the development of deep models for other domains like graphs and point clouds . Yet since models often encode multiple biases simultaneously , it can be challenging to identify how each contributes to generalization in isolation . By understanding which inductive biases are essential , and which are merely ancillary , we can simplify our models , and improve performance . For example , Wu et al . ( 2019 ) demonstrate that Graph Convolutional Networks can be radically simplified to mere logistic regression by removing nonlinearities , leading to dramatic gains in computational efficiency , while retaining comparable accuracy . Hamiltonian Neural Networks ( HNNs ) ( Greydanus et al. , 2019 ) have emerged as a leading approach for modeling dynamical systems . These dynamics models encode physical priors and outperform alternative Neural ODE approaches ( Chen et al. , 2018 ) . In the spirit of Wu et al . ( 2019 ) , we seek to identify the critical components of HNNs . Since Lagrangian models ( LNNs ) ( Lutter et al. , 2019a ; Cranmer et al. , 2020 ) share the same structure and inductive biases as HNNs , we focus on HNNs where energy conservation and symplecticity are more explicit . HNNs encode a number of inductive biases that help model physical systems : 1 . ODE bias : HNNs model derivatives of the state rather than the states directly . 2 . Second-order ( SO ) bias : HNNs model changes in position through changes in velocity . 3 . Energy conservation bias : HNNs conserve their learned energy function . 4 . Symplectic bias : HNNs dynamics are symplectic : phase space areas are conserved , and the vector field has a Hamiltonian structure . In this paper we theoretically and empirically examine the role of these biases . In contrast to conventional wisdom , we find that the generalization benefit of HNNs is not explained by their symplectic or energy-conserving properties , but rather by their implicit second-order bias . We highlight these findings in Figure 1 . Abstracting and extracting out this second-order bias , we show how to improve the performance of Neural ODEs , empowering applications when HNNs assumptions are violated as is often the case . Code for our experiments can be found at : https : //github.com/ngruver/decon-hnn . 2 RELATED WORK . Physics-inspired models and energy conservation Since Greydanus et al . ( 2019 ) , many researchers have sought to extend the HNN approach to make it more general or applicable to systems that break energy conservation . Jin et al . ( 2020 ) , Li et al . ( 2020 ) , Tong et al . ( 2021 ) , and Xiong et al . ( 2020 ) propose methods for creating neural networks that preserve symplectic structure directly . Zhong et al . ( 2020 ) , and later Desai et al . ( 2021 ) , Li et al . ( 2021 ) , and Lee et al . ( 2021 ) propose models with additional capacity for changes to the system energy . In our analysis , we compare against approaches that add additional capacity for changes in energy , although our approach is fundamentally different — we attempt to remove unnecessary biases rather than add complexity . Physics-inspired models for control A large and rapidly expanding body of work explores how to use physics-based biases in dynamics models with controls or contacts ( Lutter et al. , 2019b ; Zhong et al. , 2019 ; Gupta et al. , 2019 ; 2020 ; Chen et al. , 2019 ; Hochlehnert et al. , 2021 ; Chen et al. , 2020 ; Zhong et al. , 2021b ) intended for application in model-based planning . Especially relevant to our evaluations , Alvarez et al . ( 2020 ) models MuJoCo ( Todorov et al. , 2012 ) trajectories with a numerically integrated neural network but does not explore other physics-inspired inductive biases . Analyzing physics-based inductive biases Karniadakis et al . ( 2021 ) and Liu et al . ( 2021 ) propose conceptual frameworks for deep learning with physics-based inductive biases , but present minimal empirical analysis of common design decisions . Zhong et al . ( 2021a ) compare many approaches to physics-inspired deep learning , with results that parallel some findings here . The contribution of our work , however , is not simply benchmarking but also an actionable theory of HNNs ’ success . In spirit , our work is similar to Botev et al . ( 2021 ) , which also critically examines the role of physicsbased priors in dynamics models , though their focus is on learning latent space dynamics , while we focus on temporal models in observation space . 3 BACKGROUND . We consider dynamical systems described by ordinary differential equations ( ODEs ) which determine how the system evolves over time . Even with high order derivatives , these systems can be arranged into the form dzdt = F ( z , t ) for z ∈ R n. If the dynamics F are time-independent , then the ODE can be understood as a vector field , specifying at each point where the next state will be . Neural ODEs ( NODEs ) ( Chen et al. , 2018 ) parametrize a vector field with a neural network and learn the dynamics directly from observed trajectories . A NODE dynamics model F̂θ is rolled out from the initial condition z0 with ODE integration , ẑt = ODESolve ( z0 , F̂θ , t ) , and fit to trajectory data by minimizing an L2 loss L ( θ ) = ∑T t=1 ‖ẑt−zt‖2 between the predicted and observed trajectories , ẑt and zt . Like NODEs , HNNs also model dynamical systems as a parameterized vector field , dz dt = J∇Ĥθ with J = [ 0 I −I 0 ] , ( 1 ) where Ĥθ is a neural network with scalar output ( Greydanus et al. , 2019 ) . This differential equation expresses Hamiltonian dynamics where p and q are the canonical positions and momenta , z = [ q p ] and d dt [ q p ] = [ ∂Ĥ ∂p −∂Ĥ∂q . ] . ( 2 ) It is common practice to also explicitly define H = T + V , where T and V are the kinetic and potential energy ( Zhong et al. , 2019 ; Gupta et al. , 2019 ; Finzi et al. , 2020 ) . The assumption that the Hamiltonian is separable holds for mechanical systems and allows for further simplification , Ĥθ ( q , p ) = 1 2 pTM−1θ ( q ) p+ Vθ ( q ) , ( 3 ) where positive definite mass matrix M and scalar V are outputs of the neural network . 4 BREAKING DOWN HNN PERFORMANCE . In the following section we investigate to what extent commonly held beliefs about HNN properties actually explain the ability of HNNs to generalize . To separate out the different properties of these models , we select synthetic environments from Finzi et al . ( 2020 ) and Finzi et al . ( 2021 ) that are derived from a time independent Hamiltonian , where energy is preserved exactly . We use kChainPendulum , a k link pendulum in angular coordinates , and kSpringPendulum , a pendulum connected with k spring links in Cartesian coordinates . We compute relative error between predicted states , ẑ , and ground truth , z , as ‖ẑ − z‖2/‖ẑ‖2‖z‖2 and between the predicted and ground truth Hamiltonian as |Ĥ−H|/|Ĥ||H| . Following Finzi et al . ( 2020 ) , we evaluate the performance of the model by computing the geometric mean of the relative error of the state over rollouts of length 20 times the size used at training , which more faithfully predicts downstream performance compared to other metrics like MSE ( Finzi et al. , 2020 ) . The geometric mean of a function f over time T is given by exp ( 1T ∫ T t=0 log f ( t ) dt ) . 4.1 ENERGY CONSERVATION . HNNs are commonly believed to be superior for energy conservation than comparable models ( Greydanus et al. , 2019 ; Cranmer et al. , 2020 ) . While there is some empirical evidence to support this claim , a precise mathematical explanation for why this should be the case has not been established . Surprisingly , we find that conditioned on the trajectory reconstruction error , HNNs are no better at conserving the true energy of the system than unconstrained NeuralODE models . Up to the numerical accuracy of an ODE solver , HNNs do conserve their own learned energy function Ĥ ( different from the true energy of the system H ) , due to basic properties of Hamiltonian mechanics , dĤ ( ẑ ) dt = ∇Ĥ > dẑ dt = ∇Ĥ > J∇Ĥ = 0 , ( 4 ) where the last equality follows from the antisymmetry of J . If the HNN has fit the data well , we might expect Ĥ to be close to the true Hamiltonian H , and so if Ĥ is conserved then it seems that H should be too . As we show in Appendix B.2 , the problem with this argument is that we have no guarantees that Ĥ and H are close even if we have fit the data well with low error in the rollouts or the dynamics . Since the dynamics error and the training rollout error depend only on gradients ∇Ĥ , while the gradients may be close , the differences between the two scalar functions can grow arbitrarily large . In Figure 2 ( left ) we show that the degree of energy conservation is highly correlated with the rollout error of the model , regardless of the choice of architecture . While HNNs have better rollout performance than NeuralODEs , they are on the same regression line with Rollout Error ∝ Energy Violation . The differences in energy violation are best predicted by the rollout error and not the architecture . In Appendix B.1 , we derive a bound that helps explain this behaviour . Given that the trajectories are in a bounded region of the phase space and that there is a fixed amount of error in the dynamics model , the energy violation grows at most linearly in time and the dynamics error . In Figure 2 ( right ) we demonstrate empirically that energy is not conserved as time progresses for HNNs.1 In fact , the energy error of both NODE and HNN models grow linearly as our bound suggests . Although energy conservation may be helpful for generalization , the evidence does not indicate that HNNs are inherently better at conserving energy than NODEs , suggesting that the superior generalization of HNNs can not be attributed to superior energy conservation . 4.2 SYMPLECTIC VECTOR FIELDS . A defining property of Hamiltonian mechanics is the fact that the dynamics are symplectic . If energy conservation does not explain the effectiveness of HNNs , the symplectic property of HNN dynamics may be the cause . Informally , one of the consequences of symplectic dynamics is that every part of the state space is equally attractive and repulsive . There are no sources where all nearby trajectories flow out from , or sinks where all nearby trajectories flow into , only saddle points and centers where the inflow and outflow is balanced . Symplectic integrators ( Leimkuhler and Skeel , 1994 ) make use of this property for more stable integration of Hamiltonian systems over very long timespans , and it is intuitive that enforcing this property in learned dynamics would have benefits also , at the very least for reducing the size of the hypothesis space . More formally , symplecticity is the property that the J matrix ( the symplectic form ) is preserved by the dynamics ( Equation 1 ) . This condition can be expressed as a constraint on the Jacobian DF of the vector field F ( z ) = dzdt ( here the derivative D maps from a function to its Jacobian ) . In terms of the Jacobian , symplecticity is the condition DF > J + JDF = 0 or equivalently ( JDF ) T = JDF since J > = −J . The significance of this condition is that areas occupied by states in phase space ( which have units of energy ) are preserved by symplectic transformations . One consequence is that a volume of solutions in phase space will continue to occupy the same volume over time and will not be compressed or expanded . In other words , the vector field has 0 divergence : Tr ( DF ) = 0 , which one can derive from the above expression . 1The energy violation is considerably larger than merely the numerical error associated with the solver . On Rn , it can be shown that all symplectic vector fields can be expressed as the dynamics of some Hamiltonian system , and vice versa : F = J∇H ⇐⇒ ( JDF ) T = JDF ( 5 ) To show the forward direction , one can simply substitute in Hamiltonian dynamics . It is clear that the symplecticity property is satisfied : using J2 = −I , JDF = JD ( J∇H ) = J2∇2H = −∇2H , and −∇2H is a symmetric matrix . The reverse direction is less obvious ; however , by Poincaré ’ s lemma if a vector field F satisfies ( JDF ) T = JDF on Rn , then there exists a Hamiltonian function H such that F = J∇H , which is shown in subsection B.3 . The equivalence of Hamiltonian dynamics and symplecticity allows us separate the unique properties of HNNs from other inductive biases that result indirectly from modeling F throughH . Following Ghosh et al . ( 2020 ) , we can create a regularizer SymplecticError ( F ) = ‖ ( JDF ) T − JDF‖2 that directly measures the degree to which the symplectic property is violated . By parametrizing a NeuralODE and regularizing the symplectic error , we can enforce Hamiltonian structure while still directly modeling dzdt = F rather than H . Alongside an unregularized NeuralODE , we can isolate and evaluate the benefit of this Hamiltonian structure bias with a direct comparison . Surprisingly , we find that the Hamiltonian structure bias , as enforced by the symplectic regularizer , provides no real benefit to the model ’ s ability to generalize over the long test rollouts ( Figure 3 left ) . The achieved symplectic error Figure 3 ( right ) is not positively correlated with the final test rollout error of the model , and in some cases is even negatively correlated . Even when the symplectic error is very low and the symplecticity condition is enforced , there is no consistent improvement on the rollout generalization .

Summary: This is a well-written analysis of Hamiltonian Neural Networks (HNNs), a class of physics-inspired deep neural networks. The work is motivated by a desire to apply HNNs to non-toy datasets as well as to gain an understanding of the key inductive biases that explain the majority of their performance. Through controlled experiments on synthetic trajectory data, they explore energy conservation, a symplectic bias, complexity of state representation, and second-order structure bias. The key finding is that the second-order structure in HNNs is the main explainer of its performance. A simpler model that combines a Neural ODE with second-order structure is introduced, which can be seen as a distilled HNN. The simpler model achieves stronger performance on Mujoco rollout prediction compared to the HNN.