Deconstructing the Inductive Biases of Hamiltonian Neural Networks

1 INTRODUCTION . The inductive biases of convolutional neural networks , such as translation equivariance , locality , and parameter sharing , play a key role in their generalization properties . Other biases have similarly guided the development of deep models for other domains like graphs and point clouds . Yet since models often encode multiple biases simultaneously , it can be challenging to identify how each contributes to generalization in isolation . By understanding which inductive biases are essential , and which are merely ancillary , we can simplify our models , and improve performance . For example , Wu et al . ( 2019 ) demonstrate that Graph Convolutional Networks can be radically simplified to mere logistic regression by removing nonlinearities , leading to dramatic gains in computational efficiency , while retaining comparable accuracy . Hamiltonian Neural Networks ( HNNs ) ( Greydanus et al. , 2019 ) have emerged as a leading approach for modeling dynamical systems . These dynamics models encode physical priors and outperform alternative Neural ODE approaches ( Chen et al. , 2018 ) . In the spirit of Wu et al . ( 2019 ) , we seek to identify the critical components of HNNs . Since Lagrangian models ( LNNs ) ( Lutter et al. , 2019a ; Cranmer et al. , 2020 ) share the same structure and inductive biases as HNNs , we focus on HNNs where energy conservation and symplecticity are more explicit . HNNs encode a number of inductive biases that help model physical systems : 1 . ODE bias : HNNs model derivatives of the state rather than the states directly . 2 . Second-order ( SO ) bias : HNNs model changes in position through changes in velocity . 3 . Energy conservation bias : HNNs conserve their learned energy function . 4 . Symplectic bias : HNNs dynamics are symplectic : phase space areas are conserved , and the vector field has a Hamiltonian structure . In this paper we theoretically and empirically examine the role of these biases . In contrast to conventional wisdom , we find that the generalization benefit of HNNs is not explained by their symplectic or energy-conserving properties , but rather by their implicit second-order bias . We highlight these findings in Figure 1 . Abstracting and extracting out this second-order bias , we show how to improve the performance of Neural ODEs , empowering applications when HNNs assumptions are violated as is often the case . Code for our experiments can be found at : https : //github.com/ngruver/decon-hnn . 2 RELATED WORK . Physics-inspired models and energy conservation Since Greydanus et al . ( 2019 ) , many researchers have sought to extend the HNN approach to make it more general or applicable to systems that break energy conservation . Jin et al . ( 2020 ) , Li et al . ( 2020 ) , Tong et al . ( 2021 ) , and Xiong et al . ( 2020 ) propose methods for creating neural networks that preserve symplectic structure directly . Zhong et al . ( 2020 ) , and later Desai et al . ( 2021 ) , Li et al . ( 2021 ) , and Lee et al . ( 2021 ) propose models with additional capacity for changes to the system energy . In our analysis , we compare against approaches that add additional capacity for changes in energy , although our approach is fundamentally different — we attempt to remove unnecessary biases rather than add complexity . Physics-inspired models for control A large and rapidly expanding body of work explores how to use physics-based biases in dynamics models with controls or contacts ( Lutter et al. , 2019b ; Zhong et al. , 2019 ; Gupta et al. , 2019 ; 2020 ; Chen et al. , 2019 ; Hochlehnert et al. , 2021 ; Chen et al. , 2020 ; Zhong et al. , 2021b ) intended for application in model-based planning . Especially relevant to our evaluations , Alvarez et al . ( 2020 ) models MuJoCo ( Todorov et al. , 2012 ) trajectories with a numerically integrated neural network but does not explore other physics-inspired inductive biases . Analyzing physics-based inductive biases Karniadakis et al . ( 2021 ) and Liu et al . ( 2021 ) propose conceptual frameworks for deep learning with physics-based inductive biases , but present minimal empirical analysis of common design decisions . Zhong et al . ( 2021a ) compare many approaches to physics-inspired deep learning , with results that parallel some findings here . The contribution of our work , however , is not simply benchmarking but also an actionable theory of HNNs ’ success . In spirit , our work is similar to Botev et al . ( 2021 ) , which also critically examines the role of physicsbased priors in dynamics models , though their focus is on learning latent space dynamics , while we focus on temporal models in observation space . 3 BACKGROUND . We consider dynamical systems described by ordinary differential equations ( ODEs ) which determine how the system evolves over time . Even with high order derivatives , these systems can be arranged into the form dzdt = F ( z , t ) for z ∈ R n. If the dynamics F are time-independent , then the ODE can be understood as a vector field , specifying at each point where the next state will be . Neural ODEs ( NODEs ) ( Chen et al. , 2018 ) parametrize a vector field with a neural network and learn the dynamics directly from observed trajectories . A NODE dynamics model F̂θ is rolled out from the initial condition z0 with ODE integration , ẑt = ODESolve ( z0 , F̂θ , t ) , and fit to trajectory data by minimizing an L2 loss L ( θ ) = ∑T t=1 ‖ẑt−zt‖2 between the predicted and observed trajectories , ẑt and zt . Like NODEs , HNNs also model dynamical systems as a parameterized vector field , dz dt = J∇Ĥθ with J = [ 0 I −I 0 ] , ( 1 ) where Ĥθ is a neural network with scalar output ( Greydanus et al. , 2019 ) . This differential equation expresses Hamiltonian dynamics where p and q are the canonical positions and momenta , z = [ q p ] and d dt [ q p ] = [ ∂Ĥ ∂p −∂Ĥ∂q . ] . ( 2 ) It is common practice to also explicitly define H = T + V , where T and V are the kinetic and potential energy ( Zhong et al. , 2019 ; Gupta et al. , 2019 ; Finzi et al. , 2020 ) . The assumption that the Hamiltonian is separable holds for mechanical systems and allows for further simplification , Ĥθ ( q , p ) = 1 2 pTM−1θ ( q ) p+ Vθ ( q ) , ( 3 ) where positive definite mass matrix M and scalar V are outputs of the neural network . 4 BREAKING DOWN HNN PERFORMANCE . In the following section we investigate to what extent commonly held beliefs about HNN properties actually explain the ability of HNNs to generalize . To separate out the different properties of these models , we select synthetic environments from Finzi et al . ( 2020 ) and Finzi et al . ( 2021 ) that are derived from a time independent Hamiltonian , where energy is preserved exactly . We use kChainPendulum , a k link pendulum in angular coordinates , and kSpringPendulum , a pendulum connected with k spring links in Cartesian coordinates . We compute relative error between predicted states , ẑ , and ground truth , z , as ‖ẑ − z‖2/‖ẑ‖2‖z‖2 and between the predicted and ground truth Hamiltonian as |Ĥ−H|/|Ĥ||H| . Following Finzi et al . ( 2020 ) , we evaluate the performance of the model by computing the geometric mean of the relative error of the state over rollouts of length 20 times the size used at training , which more faithfully predicts downstream performance compared to other metrics like MSE ( Finzi et al. , 2020 ) . The geometric mean of a function f over time T is given by exp ( 1T ∫ T t=0 log f ( t ) dt ) . 4.1 ENERGY CONSERVATION . HNNs are commonly believed to be superior for energy conservation than comparable models ( Greydanus et al. , 2019 ; Cranmer et al. , 2020 ) . While there is some empirical evidence to support this claim , a precise mathematical explanation for why this should be the case has not been established . Surprisingly , we find that conditioned on the trajectory reconstruction error , HNNs are no better at conserving the true energy of the system than unconstrained NeuralODE models . Up to the numerical accuracy of an ODE solver , HNNs do conserve their own learned energy function Ĥ ( different from the true energy of the system H ) , due to basic properties of Hamiltonian mechanics , dĤ ( ẑ ) dt = ∇Ĥ > dẑ dt = ∇Ĥ > J∇Ĥ = 0 , ( 4 ) where the last equality follows from the antisymmetry of J . If the HNN has fit the data well , we might expect Ĥ to be close to the true Hamiltonian H , and so if Ĥ is conserved then it seems that H should be too . As we show in Appendix B.2 , the problem with this argument is that we have no guarantees that Ĥ and H are close even if we have fit the data well with low error in the rollouts or the dynamics . Since the dynamics error and the training rollout error depend only on gradients ∇Ĥ , while the gradients may be close , the differences between the two scalar functions can grow arbitrarily large . In Figure 2 ( left ) we show that the degree of energy conservation is highly correlated with the rollout error of the model , regardless of the choice of architecture . While HNNs have better rollout performance than NeuralODEs , they are on the same regression line with Rollout Error ∝ Energy Violation . The differences in energy violation are best predicted by the rollout error and not the architecture . In Appendix B.1 , we derive a bound that helps explain this behaviour . Given that the trajectories are in a bounded region of the phase space and that there is a fixed amount of error in the dynamics model , the energy violation grows at most linearly in time and the dynamics error . In Figure 2 ( right ) we demonstrate empirically that energy is not conserved as time progresses for HNNs.1 In fact , the energy error of both NODE and HNN models grow linearly as our bound suggests . Although energy conservation may be helpful for generalization , the evidence does not indicate that HNNs are inherently better at conserving energy than NODEs , suggesting that the superior generalization of HNNs can not be attributed to superior energy conservation . 4.2 SYMPLECTIC VECTOR FIELDS . A defining property of Hamiltonian mechanics is the fact that the dynamics are symplectic . If energy conservation does not explain the effectiveness of HNNs , the symplectic property of HNN dynamics may be the cause . Informally , one of the consequences of symplectic dynamics is that every part of the state space is equally attractive and repulsive . There are no sources where all nearby trajectories flow out from , or sinks where all nearby trajectories flow into , only saddle points and centers where the inflow and outflow is balanced . Symplectic integrators ( Leimkuhler and Skeel , 1994 ) make use of this property for more stable integration of Hamiltonian systems over very long timespans , and it is intuitive that enforcing this property in learned dynamics would have benefits also , at the very least for reducing the size of the hypothesis space . More formally , symplecticity is the property that the J matrix ( the symplectic form ) is preserved by the dynamics ( Equation 1 ) . This condition can be expressed as a constraint on the Jacobian DF of the vector field F ( z ) = dzdt ( here the derivative D maps from a function to its Jacobian ) . In terms of the Jacobian , symplecticity is the condition DF > J + JDF = 0 or equivalently ( JDF ) T = JDF since J > = −J . The significance of this condition is that areas occupied by states in phase space ( which have units of energy ) are preserved by symplectic transformations . One consequence is that a volume of solutions in phase space will continue to occupy the same volume over time and will not be compressed or expanded . In other words , the vector field has 0 divergence : Tr ( DF ) = 0 , which one can derive from the above expression . 1The energy violation is considerably larger than merely the numerical error associated with the solver . On Rn , it can be shown that all symplectic vector fields can be expressed as the dynamics of some Hamiltonian system , and vice versa : F = J∇H ⇐⇒ ( JDF ) T = JDF ( 5 ) To show the forward direction , one can simply substitute in Hamiltonian dynamics . It is clear that the symplecticity property is satisfied : using J2 = −I , JDF = JD ( J∇H ) = J2∇2H = −∇2H , and −∇2H is a symmetric matrix . The reverse direction is less obvious ; however , by Poincaré ’ s lemma if a vector field F satisfies ( JDF ) T = JDF on Rn , then there exists a Hamiltonian function H such that F = J∇H , which is shown in subsection B.3 . The equivalence of Hamiltonian dynamics and symplecticity allows us separate the unique properties of HNNs from other inductive biases that result indirectly from modeling F throughH . Following Ghosh et al . ( 2020 ) , we can create a regularizer SymplecticError ( F ) = ‖ ( JDF ) T − JDF‖2 that directly measures the degree to which the symplectic property is violated . By parametrizing a NeuralODE and regularizing the symplectic error , we can enforce Hamiltonian structure while still directly modeling dzdt = F rather than H . Alongside an unregularized NeuralODE , we can isolate and evaluate the benefit of this Hamiltonian structure bias with a direct comparison . Surprisingly , we find that the Hamiltonian structure bias , as enforced by the symplectic regularizer , provides no real benefit to the model ’ s ability to generalize over the long test rollouts ( Figure 3 left ) . The achieved symplectic error Figure 3 ( right ) is not positively correlated with the final test rollout error of the model , and in some cases is even negatively correlated . Even when the symplectic error is very low and the symplecticity condition is enforced , there is no consistent improvement on the rollout generalization .

Summary: Incorporating physics-informed inductive bias, especially Hamiltonian and Lagrangian dynamics, into deep neural networks has been the focus of a fast-growing body of work over the last few years. This paper has carried out a formal analysis of these methods to identify which specific aspect(s) of these energy-conserving networks contribute(s) the most to their superior performance in prediction and generalization. Through theoretical and empirical analyses, this paper concludes that incorporating the second-order structure (i.e., using a neural network to model the acceleration) and reducing functional complexity through a change of coordinates, not energy conservation or the symplectic structure, play the most critical role in improving the performance. The experiments are thorough and emphasize the message precisely.