Symmetry Control Neural Networks

1 Introduction . Building in a bias to neural networks has been a key mechanism to achieve extra-ordinary performance in tasks such as classification . A standard example is to utilise translation invariance in convolutional neural networks Krizhevsky et al . ( 2012 ) and by now building in equivariance to other symmetries such as rotational symmetries has proven to be very successful ( e.g . Cohen & Welling ( 2016 ) ) . Possible motions are constrained due to symmetries of the system . In technical terms , motion takes place on a subspace of phase space . Energy conversation – related to invariance under time translation – has been utilised in the context of Hamiltonian Neural Networks ( HNNs ) Greydanus et al . ( 2019 ) where the energy functional , i.e . the Hamiltonian is inferred from data . This approach has seen large improvements in predicting the dynamics over baseline neural networks which simply try to predict the change of phase-space coordinates in time . Here we extend this approach by learning and incorporating additional constraints due to further symmetries of the system . Coarsely speaking , finding symmetries corresponds to finding good coordinates . In classical mechanics this is achieved by performing canonical transformations and identifying cyclic coordinates which reveal conserved quantities . The aim of this paper is to demonstrate that multiple conserved quantities can indeed be automatically found in this way which has not been demonstrated beforehand . Similar in spirit to learning the Hamiltonian , we formulate loss functions which enforce a representation in terms of cyclic coordinates and use them as the input for our Hamiltonian , differing from previous flow-based approaches searching for these coordinates ( Bondesan & Lamacraft , 2019 ; Li et al. , 2020 ) . We experimentally find as a proof of principle that this mechanism identifies the underlying conserved quantities such as angular momentum , momentum , the splitting into decoupled subsystems , and can find the number of conserved quantities . 0.6 0.8 1.0 1.2 1.4 m1 0.6 0.8 1.0 1.2 1.4 m 2 40.0 900.0500.0 200.0 80.0 80 .0 5.0 30 .0 5.0 30.0 Figure 1 : Effect of additional loss components : Loss-contours for the HNN-loss ( 6 ) ( shown in gray ) and the Poisson loss ( 7 ) ( red ) arising from the angular momentum ( ‖ { L , H } ‖2 ) in the 2-body Hamiltonian ( 10 ) with respect to two model parameters m1 and m2 . The data model corresponds to m1 = m2 = g = 1 ( indicated with a star ) and we evaluate the loss over our training set . The analytic constraint m1 = m2 which arises from evaluating the Poisson bracket { L , H } ∼ ( m1 − m2 ) is clearly visible and provides additional constraints on the model parameter space . We demonstrate significant improvement in the predictions of the underlying Hamiltonian and subsequently the dynamics of the system . From our trained networks , we can find analytic expressions for the conserved quantities and determine the number of conserved quantities . 2 Theory . We briefly describe the standard techniques in Hamiltonian mechanics which our network utilises.1 We consider a classical system with N particles in d spatial dimensions . Such a system can be described by the variables ( q , p ) , where q = ( q1 , ... , qN ·d ) are typically the positions for each dimension of the objects and p = ( p1 , ... , pN ·d ) are the corresponding momenta . This is the input to our network and we are interested in predicting the timeevolution of this system , i.e . ( q , p ) at later time steps . This pair ( q , p ) is an element of phase space in which every point corresponds to a state the system can take . The time evolution is governed by the Hamiltonian H ( q , p ) 2 and the associated Hamiltonian equations : dq dt = ∂H dp = { q , H } , dp dt = −∂H dq = { p , H } , ( 1 ) where { • , • } are the Poisson bracket . They are defined as { f , g } : = N ·d∑ i=1 ∂f ∂qi ∂g ∂pi − ∂f ∂pi ∂g ∂qi . ( 2 ) The Poisson bracket does not only arise for the time evolution of the canonical coordinates ( q , p ) but also for any function of these coordinates g ( q , p ) which does not explicitly depend on time : dg ( q , p ) dt = N ·d∑ i=1 ∂g ∂qi dqi dt − ∂g ∂pi dpi dt = { g , H } , ( 3 ) where we have used the Hamiltonian equations ( 1 ) in the last step . From this expression we see that the Poisson bracket of a conserved quantity with the Hamiltonian H ( q , p ) vanishes and that the Hamiltonian itself is a conserved quantity . The physics of this system is invariant under diffeomorphic coordinate transformations on the canonical coordinates and we will use coordinate transformations to reveal the constants of motions and hence the symmetries of the system . We use a particular type of diffeomorphic transformations T , namely canonical transformations which are transformations that leave the structure of the Hamiltonian equations ( 1 ) and in particular the Poisson bracket unchanged : T : ( q , p ) 7→ ( Q ( q , p ) , P ( q , p ) ) , { f , g } p , q = { f , g } P , Q , H ( p , q ) = H̃ ( P ( p , q ) , Q ( p , q ) ) . ( 4 ) 1We refer the reader for more details to standard textbooks such as Landau & Lifshitz ( 1982 ) . 2We focus on time-independent Hamiltonians for simplicity . • • Learning Symmetries Training the neural network 14 { Pi , Pj } = { Qi , Qj } = 0 { Qi , Pj } = δij ·Pi = − ∂ℋ∂Qi ·Qi = ∂ℋ∂Pi ·Pi = 0 L = ∥ ∂ℋ∂p − ·qtarget∥2 +∥ ∂ℋ∂q + ·ptarget∥2 + . . . We are interested in finding transformations T such that at least one coordinate satisfies 0 = Ṗi = − ∂H ∂Qi = { Pi , H } . ( 5 ) Such a coordinate Pi is conserved in the system and the Hamiltonian does not depend on the associated Qi , i.e . it depends on fewer degrees of freedom and the motion in phase space is restricted to a lower dimensional manifold . Put differently , the cyclic coordinates provide via constraints of the type ( 5 ) additional restrictions on the allowed Hamiltonian function which we learn with our symmetry control neural networks ( cf . Figure 1 shows explicit constraint from angular momentum conservation in a 2-body example in addition to constraints arising from satisying the Hamiltonian equations of motion ) . Symmetry Control Neural Networks : In a first step we search for such cyclic coordinates with a network Tψ and use them as input for our Hamiltonian Hφ . The structure of our two trainable networks is shown in Figure 2 . In order to find cyclic coordinates and to identify a Hamiltonian , our loss function contains several components : 1 . The first loss ensures that our Hamiltonian satisfies Hamiltonian equations ( 1 ) , which we can ensure as follows : LHNN = N ·d∑ i=1 ∥∥∥∥∂Hφ ( P , Q ) ∂pi − dqidt ∥∥∥∥ 2 + ∥∥∥∥∂Hφ ( P , Q ) ∂qi + dpidt ∥∥∥∥ 2 . ( 6 ) The time derivatives are provided by the data and the derivatives of the Hamiltonian with respect to the input variables can be obtained using auto-differentiation . This is the same loss as introduced in Greydanus et al . ( 2019 ) . 2 . To ensure that our transformation Tψ are of the type we are interested in ( cf . Eq . ( 4 ) ) , i.e . our new coordinates fullfil the Poisson algebra , we enforce the following loss : LPoisson = N ·d∑ i , j=1 ‖ { Qi , Pj } − δij‖2 + N ·d∑ i , j > i ‖ { Pi , Pj } ‖2 + ‖ { Qi , Qj } ‖2 , ( 7 ) where in some practical applications we only enforce this loss on n cyclic coordinate pairs . The first part of this loss ensures that a vanishing solution is not allowed . 3 . Hamilton ’ s equations have still to be satisfied with respect to the new coordinates . For the cyclic coordinates we have enforced by our architecture that Hφ is independent of Qi . However , to ensure that Pi is actually conserved , we require the following additional loss : L ( n ) HQP = n∑ i=1 ∥∥∥∥dPidt ∥∥∥∥ 2 + ∥∥∥∥dQidt − ∂Hφ ( P , Q ) ∂Pi ∥∥∥∥ 2 + β N ·d∑ i=n+1 ∥∥∥∥dPidt + ∂Hφ ( P , Q ) ∂Qi ∥∥∥∥ 2 + ∥∥∥∥dQidt − ∂Hφ ( P , Q ) ∂Pi ∥∥∥∥ 2 , ( 8 ) where n denotes the number of cyclic variables we are imposing and β denotes a hyperparameter for our experiments . The time derivatives can be calculated using either expressions in ( 3 ) . Our total loss is a weighted sum of these three components : L = LHNN + α1LPoisson + α2L ( n ) HQP , ( 9 ) where the weights αi are tuned . For integrating the solutions in time from our respective symmetry control network we use a fourth order Runge-Kutta integrator as in Greydanus et al . ( 2019 ) which unlike symplectic integrators allows for a comparison with neural network approaches directly predicting the dynamics of a system.3 3 Experiments . Our experiments are designed with the following goals in mind : 1 . SCNN-base : We want to compare the performance of symmetry control neural networks with HNNs and baseline neural networks which directly predict ( q̇ , ṗ ) . In these experiments we generally use the maximal number of conserved quantities . To evaluate the performance , we use the accuracy of the prediction and whether physically conserved quantities are actually conserved . 2 . SCNN-constraint : We explore whether imposing domain knowledge about symmetries improves the performance . This is motivated by the fact that we often know about the existence of certain conserved quantities . 3 . Analytic formulae : We evaluate whether our networks actually use conserved quantities in close proximity to known physically conserved quantities . To do this we fit analytical formulae on the conserved quantities and check how accurately we can recover known formulae . Throughout , we run a hyperparameter analysis on how the different loss components influence our results to provide a first numerical analysis of these constraints . We test our method on synthetic examples and freely available experimental data , several of which have already been used in Greydanus et al . ( 2019 ) . 3.1 Physical Systems and Objective of Experiments . We consider the two-body problem in two dimensions which is governed by the following Hamiltonian H = p 2 x1 2m1 + p2y1 2m1 + p2x2 2m2 + p2y2 2m2 − g ‖q1 − q2‖2 . ( 10 ) To simplify the problem we set m1 = m2 = g = 1 . We generate datasets with near circular orbits where the whole system has a small velocity in an arbitrary direction to avoid accidental symmetries ( see Appendix B for more details ) . Besides the Hamiltonian itself , conserved quantities are the total momentum in x and y direction , and the angular momentum : Px = px1 + px2 , Py = py1 + py2 , L = ( px1 − px2 ) ( qy1 − qy2 ) − ( py1 − py2 ) ( qx1 − qx2 ) . ( 11 ) The second system is the 3-body system which is governed by the following Hamiltonian : H = 3∑ i=1 p2xi 2mi + p2yi 2mi − 1 2 3∑ i=1 , j 6=i g ‖qi − qj‖2 , ( 12 ) 3We find that the main numerical inaccuracy in the prediction arises from inaccuracies of the Hamiltonian Hφ rather than the choice of this integrator when comparing it with standard symplectic integrators ( Rein & Liu , 2012 ) . Here in this paper , our focus is on predicting the Hamiltonian most accurately rather than making most accurate predictions with a Hamiltonian . where we take a univeral mass and coupling mi = g = 1 . As in the 2-body example , we consider systems which have a small velocity in an arbitrary direction ( cf . Appendix B ) . Again , the overall momentum in both directions and the angular momentum are conserved quantities : Px = px1 + px2 + px3 , Py = py1 + py2 + py3 , L = ( 2qx1 − qx2 − qx3 ) py1 − ( 2qy1 − qy2 − qy3 ) px1 + even permutations . ( 13 ) The third system which we are interested in is the n-dimensional coupled oscillator which is described by the following Hamiltonian H = n∑ i=1 p2i 2m + n∑ i , j=1 qiAijqj , ( 14 ) where we use m = 1/2 and the coupling matrix Aij is a symmetric n × n−matrix with positive eigenvalues . As this matrix is not diagonal , the potential energy of each oscillator in these coordinates is influenced by the position of the other oscillators . By diagonalization the system can be described by decoupled harmonic oscillators . The conserved quantities are the energies of these subsystems : Ei = p̃ 2 i + λiq̃ 2 i , ( 15 ) where p̃ and q̃ are the canonical coordinates associated to the respective eigenvalue λi of the coupling matrix Aij . Note that these systems include spring-like interactions . To include a model with few conserved quantities and to show that the SCNN can be used to determine the number of conserved quantities , we consider the double pendulum , a chaotic system , where only energy is conserved . We test the performance of SCNNs on a charged massive particle in a background magnetic field where the kinetic and potential energy are not decoupled in the Hamiltonian . To estimate the feasibility of our approach with experimental data where energy is not conserved , we test SCNNs on data from two experiments presented in ( Schmidt & Lipson ( 2009 ) ) : two-masses-three-springs is a coupled two-dimensional harmonic oscillator , and we combine data from the physical pendulum to a real spherical pendulum . Details about these systems can be found in Appendix A and B .

Summary: This paper presents the results of a NN trained to learn symmetries in physics, specifically, to learn and preserve quantities that are preserved (e.g., energy, angular momentum). The input is a sequence generated from a Hamiltonian dynamics. Results of experiments on 2 and 3 body problems and a harmonic oscillator are presented. The training networks are small, shallow feedforward networks. There is some customization of the training networks to incorporate "cyclic" coordinates. Results indicated empirical conservation up to small error of physically conserved quantities. The paper is fairly easy to read, with much relevant background provided.