Symmetry Control Neural Networks
Authors: Marc Syvaeri
Sven Krippendorf
Venue:  (2021)
Keywords: Inductive (symmetry) Bias
Predictive Models
Hamiltonian Dynamics
Physics

Abstract: This paper continues the quest for designing the optimal physics bias for neural networks predicting the dynamics of systems when the underlying dynamics shall be inferred from the data directly. The description of physical systems is greatly simplified when the underlying symmetries of the system are taken into account. In classical systems described via Hamiltonian dynamics this is achieved by using appropriate coordinates, so-called cyclic coordinates, which reveal conserved quantities directly. Without changing the Hamiltonian, these coordinates can be obtained via canonical transformations. We show that such coordinates can be searched for automatically with appropriate loss functions which naturally arise from Hamiltonian dynamics. As a proof of principle, we test our method on standard classical physics systems using synthetic and experimental data where our network identifies the conserved quantities in an unsupervised way and find improved performance on predicting the dynamics of the system compared to networks biasing just to the Hamiltonian. Effectively, these new coordinates guarantee that motion takes place on symmetry orbits in phase space, i.e.~appropriate lower dimensional sub-spaces of phase space. By fitting analytic formulae we recover that our networks are utilising conserved quantities such as (angular) momentum.

Under review conference paper at ICLR 2021
SYMMETRY CONTROL NEURAL NETWORKS
A Anonymous authors
Paper under double-blind review
ABSTRACT
This paper continues the quest for designing the optimal physics bias for neural networks predicting the dynamics of systems when the underlying dynamics shall be inferred from the data directly. The description of physi- cal systems is greatly simplified when the underlying symmetries of the sys- tem are taken into account. In classical systems described via Hamiltonian dynamics this is achieved by using appropriate coordinates, so-called cyclic coordinates, which reveal conserved quantities directly. Without changing the Hamiltonian, these coordinates can be obtained via canonical transfor- mations. We show that such coordinates can be searched for automatically with appropriate loss functions which naturally arise from Hamiltonian dy- namics. As a proof of principle, test our method on standard classical physics systems using synthetic and experimental data where our network identifies the conserved quantities in unsupervised way and find im- proved performance on predicting the dynamics of the system compared networks biasing just to the Hamillonian. Effectively, these new coor- dinates guarantee that motion takes place on symmetry orbits in phase space, ie, appropriate lower dimensional sub-spaces of phase space. By fit- ting analytic formulae we recover that our networks are utilising conserved quantitics such (angular) momentum.
INTRODUCTION
Building in a bias to neural networks has been a key mechanism achieve extra-ordinary performance in tasks such as classification. A standard example is utilise translation invariance in convolutional neural networks Krizhevsky et al.] (2012) and by now building in equivariance to other symmetries such as rotational symmetries has proven to be very successful (e.g. Cohen & Welling (2016)).
Possible motions are constrained due to symmetrics of the system. In technical terms, motion takes place on a subspace of phase space. Energy conversation related to invariance under time translation has been utilised in the context of Hamiltonian Neural Networks (HNNs) Greydanus et al. (2019) where the energy functional, i.e. the Hamiltonian is inferred from data. This approach has seen large improvements in predicting the dynamics over baseline neural networks which simply try to predict the change of phase-space coordinates in time. Here we extend this approach by learning and incorporating additional constraints due to further symmetries of the system.
Coarsely speaking, finding symmetrics corresponds to finding good coordinates. In classical mechanics this is achieved by performing canonical transformations and identifying cyclic coordinates which reveal conserved quantities. The aim of this paper to demonstrate that multiple conserved quantities can indeed be aulomatically found in this way which has not been demonstrated beforehand.
Similar in spirit to learning the Hamiltonian, we formulate loss functions which enforce representation in terms of cyclic coordinates and use them as the input for our Hamiltonian, differing from previous flow-bascd approaches scarching for these coordinates (Bondesan & Lamacraft, 2019 Li et al.] 2020). We experimentally find as a proof of principle that this mechanism identilies the underlying conserved quantities such angular momentum, momentum, the splitting into decoupled subsystems, and can find the number of conserved quantities.

---

Under review conference paper at ICLR 2021
Figure Effect of additional loss components: Loss-contours for the HNN-loss
G (shown in gray)
1.4
and the Poisson loss 7 (red) arising from the angu- lar momentum (IKL, 17 H22) in the 2-body Hamilto- nian (10) with respect to two model parameters mi and m2. The data model corresponds to my ma g = (indicated with a star) and we evaluate the loss over our training set. The analytic constrriit mi = m2 which arises from evaluating the Poisson bracket H, (m1 - m2) is clearly visible and provides additional constraints on the model param- ctcr space.
5100
500.9
200.0
m2
46.0
1.0
3019
60.0
0.6
0.6
0.8
1.0
1.2
1.4
m
We demonstrate significant improvement in the predictions of the underlying Hamiltonian and subsequently the dynamics of the system. From our trained networks, can find analytic expressions for the conserved quantities and determine the number of conserved quantities.
2 THEORY
We briefty describe the standard tochniques in Hamiltonian mechanics which our network utilises !
We consider classical system with N particles in d spatial dimensions. Such system can be described by the variables (q,p), p), where q = (g1,,.,,qv QN-a) are typically the positions for each dimension of the objects and p (Pi...PPN-d) are the corresponding momenta. This is the imput to our network and we are interested in predicting the time- evolution of this system, ie. (q.p) at later time steps.
This pair (q. p) is an element of phase space in which every point corresponds to state the system can take. The time evolution is governed by the Hamiltonian H(q, p p and the associated Hamiltonian equations:
dq 0H Hg
dp
OH
(p.
(1)
di dp
di dq
where )) are the Poisson bracket. They are defined as N.d 0f 0g
0f 0g
(f.g) = Σ
(2)
i=1 0y, 0p: Sp, 0q.
The Poisson bracket does not only arise for the time evolution of the canonical coordinates (q, p) but also for any function of these coordinates g(q, p) which does not explicitly depend on time:
dg(q. dt -
N.d 0g dq. 0g dp: = {g. 7(
dq. dt Op, dt
where we have used the Hamiltonian equations I in the last step. From this expression we that the Poisson bracket of a conserved quantity with the Hamiltonian H(q. p) vanishes and that the Hamiltonian itself is conserved quantity.
The physics of this system is invariant under diffeomorphic coordinate transformations on the canonical coordinates and we will use coordinate transformations to reveal the constants of motions and hence the symmetries of the system. We use a particular type of diffeomor- phic transformations T, namely cononical transformations which are transformations that leave the structure of the Hamiltonian equations [1] and in particular the Poisson bracket. unchanged:
T: (q.p) - (Qq.))
= 4( H((PPp q). Q(p.g))
(4)
We refer the reader for more details to standard textbooks such as Landau & Lifshitz (1822 We focus on time- indeeeddeen Hamiltonians for simplicity.
2

---

Under review conference paper at ICLR 2021
Cyylic = const
p. 4 (Input)
(P. q): Canonical
P Pather Qathrr
# S(P eyelle- P other Qother
Transformation Network
Hamiltonian Network
dp
(Output)
Qeyetic
Figure 2: Structure of symmetry control networks: The network To translorms the input coordinates of phase space (p, q) to canonical coordinates where some coordinates are forced to be cyclic. These coordinates are used as input the Hamiltonian network Hg. The output, the time derivatives of our initial coordinates, is calculated from the Hamiltonian using autoodifferentiation.
We are interested in finding transformations 7 such that at least one coordinate satisfies
0H
0= P:
(P,r H).
aQ,
Such a coordinate P. is conserved in the system and the Hamiltonian does not depend on the associated Qi i.c. it depends on fower degrees of freedom and the motion in phase space is restricted to a lower dimensional manifold. Put differently, the eyclic coordinates provide via constraints
the type (5) additional restrictions on the allowed Hamiltonian function which we learn with our symmetry control neural networks (ef. Figure [ shows explicit constraint from angular momentum conservation in a 2-body example in addition to constraints arising from satisying the Hamiltonian equations of motion).
Symmetry Control Neural Networks: In a first step we search for such cyclic coordi- nates with a network T. and use them as input for our Hamiltonian Hg. The structure of our two trainable networks shown in Figure 2 In order to find cyclic coordinates and to identify Hamiltonian, loss function contains several components:
1. The first loss ensures that our Hamiltonian satisfies Hamiltonian equations which can ensure as follows:
N.d
(H dq. dt
OHo Q)
dp:
CHNN
(6)
Op:
0q: dt
The time derivatives are provided by the data and the derivatives of the Hamiltonian with respect to the inpul variables can be obtained using mnto-diiferentiation. This is the same loss as introduced in Greydanus et al. (2019).
2. To ensure that transformation Ty are of the type
interested in (cf. Eq. (4),
i.e. our new coordinates fullfil the Poisson algebra, we enforce the following loss:
Nid
Nid
Prosssn
Σ I1Q,, - 60ll2 *-=1
Σ EIP., + I1Q QiHle 1,32
(7)
where in some practical applications we only enforce this loss cyclic coordinate pairs. The first part this loss ensures that a vanishing solution is not allowed.
3. Hamilton's equations have still be satisfied with respect to the new coordinates. For the cyclic coordinates we have enforced by our architecture that Hg is independent of
Q, However, ensure that P, is actually conserved, we require the following additional loss:
[(n)
dP:
dQ:
(HPP 8H
HOP
dt
+
dt
OP:
2
2
N-d
dP:
OH (P. Q)
dQ: dl.
87 (P.Q) Q
3
Σ
+
(8)
dt
aQ.
aP:
i=++1
where denotes the number of eyclic variables we imposing and B denotes a hy- perparameter for our experiments. The time derivatives can be calculated using either expressions in 33)
3

---

Under review conference paper at ICLR 2021
Our total loss is a weighted sum of these three components:
L= LHNN + on Proisson + 0r2 022
(n)
where the weights a; are tuned.
For integrating the solutions in time from respective symmetry control network
fourth order Runge-Kutta integrator as in Greydanus et al. (2019) which unlike symplectic integrators allows for comparison with neural network approaches directly predicting the dynamics of a system
3 EXPERIMENTS
Our experiments are designed with the following goals in mind:
1. SCNN-base: We want lo compare the performance of symmetry control neural net- works with HNNs and baseline neural networks which directly predict (q, p). In these experiments we generally
the maximal number of conserved quantities. To eval- wate the performance, we use the accuracy of the prediction and whether physically conserved quantities actually conserved.
2. SCNN-constraint: We explore whether imposing domain knowledge about symme- tries improves the performance. This is motivated by the fact that we often know about the existence of certain conserved quantities.
3. Analytic formulae: We evaluate whether our networks actually use conserved quan- tities in close proximity to known physically conserved quantities. To do this we fit analytical formulae on the conserved quantitics and check how accurately we can re- cover known formulae.
hyperparameter analysis how the different loss components in- fluence our results to provide first numerical analysis of these constraints. We test our method on synthetic examples and freely available experimental data, several of which have already been used in Greydanus et al. (2019).
Throughout,
3.1 PHYSICAL SYSTEMS AND OBJECTIVE OF EXPERIMENTS
We consider the two-body problem in two dimensions which governed by the following Hamiltonian
P. 2m
F
P2
9
H
(10)
2m
2112
2mz
||q
q2ll2
To simplify the problem set m1 m2 === We generate datasets with near circular orbits where the whole system has a small velocity in an arbitrary direction to avoid acciden- tal symmetries (sce Appendix B for more details). Besides the Hamiltonian itself, conserved quantitics are the total momentum in x and direction, and the angular momentum:
Pj = Pxt + Pa2
P. = Py1 + Py2
L (Px1 Pa2) (g/1 9y2) (Py1 Pya) (q.1 9x2).
(11)
The second system is the 3-body system which is governed by the following Hamiltonian:
H
yi
Σ
9
(12)
2m;
2m;
2
q
9/11
1=1,441
We find that the main numerical inaccuracy in the prediction arises from inaccuracies of the Hamiltonian H.s rather than the choice of this integrator when comparing it with standard symplec- tic integrators (Rein & Liu 2012). Here in this paper, focus predicting the Hamiltonian most accurately rather than making most accurate predictions with Hamiltonian.
4

---

Under review conference paper at ICLR 2021
where we take a univeral mass and coupling my = 9 = As in the 2-body example, we consider systems which have small velocity in an arbitrary direction (cf. Appendix B Again, the overall momentum in both directions and the angular momentum are conserved quantities:
P. Pat P22 + Px3
Py Pyt + Py2 + Py3
= (2 - qx2 4u3) py1 - (2qg1 - 9,2 (y3) Pel + even permutations
(13)
The third system which we are interested in is the n-dimensional coupled oscillator which is described by the following Hamiltonian
n
H
(14)
2m
where we 1/2 and the coupling matrix Ag is symmetric n n-matrix with positive eigenvalues. As this matrix is not diagonal, the potential energy of each oscillator in those coordinates is influenced by the position of the other oscillators. By dingonalization the system can be described by decoupled harmonic oscillators. The conserved quantitics are the energies of these subsystems:
E. E +
(15)
where p and the canonical coordinates associated to the respective eigenvalue A of the coupling matrix Aij. Note that these systems include spring-like interactions.
To include a model with few conserved quantities and lo show that the SCNN can be used to determine the number of conserved quantities, we consider the double pendulum, a chaotic system, where only energy is conserved. We test the performance of SCNNs a charged massive particle in a background magnetic field where the kinetic and potential energy are not decoupled in the Hamiltonian. To estimate the foasibility of our approach with experimental data where energy is not conserved, we test SCNNs on data from two experiments presented in (Schmidt & Lipson| (2009)): two-masses- thenasssss-rre---rrinnggs is coupled two-dimensional harmonic oscillator, and we combine data from the physical pendulum to real spherical pendulum. Details about these systems can be found in Appendix and
3.2
RESULTS
3.2.1
2-BODY PROBLEM
We have trained our networks with data from 800 trajectories with 50 datapoints each and
checked the performance on 200 different trajectories with 50 datapoints (cf. Appendix B
for more details). We tested a large set of different hyperparameters (ef. Appendix C
where the large bulk of the models converges and reveals the conserved quantities. The
implementation of our experiments is submitted alongside with this paper. We compare the following networks which all converge to a reasonable accuracy and our large training set.
ensures that overfitting does not pose problem here:
The baseline architecture has two hidden dense layers with 200 hidden units and tanh activation. The output is directly (q. p).
HNN: We use the original architecture with two hidden dense layers, both networks using tanh activations and 200 hidden units.
we use a transformation network architecture with two hidden dense layers (tanh activation) and a Hamiltonian neural network with two hidden dense layers as above. We enforce four cyclic coordinates (the maximum possible number). As described below, we vary the respective loss-weights and find that loss with 01,2 0.001 works best.
SCNN-base:
SCNN-constraint:
we impose domain knowledge about conserved momenta (Pa, Py) and angular momentum (L) respectively. The Hamiltonian is calculated with our
5

---

Under review conference paper at ICLR 2021
2-body problem
3-body problem
Deviation from True Trajectory
Devation from True Trajectory
Ground Truth
Baseline aeiiiii
Ground Truth
HNN
Time Step
SCNN-coral.
(F Fg
Pp.. Angular Momentum
BCNN--comi. (3
SCNN--cora (''. Energy
Time Step
Energy
Angular Mornenturn
Time Step
Time Stop
Time Step
Time Step
Kinetic Energy
Potential Energy
Kinetic Energy
Potential Energy
50"
201
Time Step
Time Stop
Time Stop
Time Step
Figure 3: Performance overview of the two-body and three- body experiments. Top line includes sample trajectories for the ground truth, the baseline neural network, a HNN, and our symmetry control networks. The colour coding is such that the same colour is used throughout all plots in this figure. The top line also shows the deviation of the predicted from the true trajectory. The solid line denotes the mean deviation over 50 trajectories and the one sigma band is shown shaded around it. Bottom two lines: We show mean squared deviation from conserved quantities where again show the mean deviation over 50 trajectorics and and the associated one sigma band.
6

---

Under review as conference paper at ICLR 2021
standard two hidden dense layers from the input (Q.)). Beyond the HNNN-oss, we only impose the loss components ensuring {H, P.) = 0 and {H, LJ 0 respectively (cf. Figure for a visualization). Improve on loss description, if we want to keep it.
For all networks we analyse the time evolution predicted from our trained network by deter- mining the trajectory, the deviation from the ground truth and the evolution of the conserved quantities. A sample for the predicted trajectories is shown for each network in Figure To quantify the accuracy of the respective networks we show the mean-squared deviation from the ground truth trajectories which we obtain over 50 different trajectories on our trained networks. We see a modest improvement with our SCNN-basc architecture and an order of magnitude improvement when we include domain knowlodge about conserved quantitics for SCNN-constraind networks. Finally the bottom line of plots in Figure 3 shows the devi- ation of conserved quantities predicted from our networks comparison lo the underlying conserved quantity where we find an improvement compared to the HNN network.
To analyse the conserved quantities, we fit polynomials in (q, p) to the output of the cyclic coordinates. Picking the solution with the lowest degree and good accuracy we recover known conserved quantitics of this system in the SCNN-bass-model:
Pa - 4.21 P=1 - 4.21 P=2 1.26 Py1 1.29 P22
(0.03).
Pe = 0.93 P=t - 0.92 P=2 - 3.23 Pyt - 3.22 p22
(0.03)
L 1.07 fu1Pp1 0.88 q.12,2 0.93 9x2Py1 1.03
9x2Py2
+1.01 9y1Ps1 - 0.89 9y1Px2 0.92 9p2Px1 + 0.99 9y2Px2 (0.10)
(16)
where the remaining terms are smaller than 0.1 and the number in brackets denotes the respective mean-squared error of the fit. The first two are clear linear combinations of the conserved momenta in Equation (II) and the third one is corresponding to angular momentum. To find these analytic fits we have simply used a polynomial ansatz and checked that there are significant changes when allowing for higher powers in the analytic ansatz (cf. Appendix
3.2.2 THREF-RODY PROBLEM
Apart from adapting the input and output dimensions appropriately, we keep the baseline and HNN architecture unchanged to the two-body problem. For our symmetry control networks we report results for SCNN-base architectures with six cyclic coordinates and SCNN-constraint architectures where we use our domain knowledge and enforce as in the two-body examples the conservation of momenta and angular momentum.
We train our networks starting again at near circular orbits (cf. Appendix Bp. However, after an initial circular motion for around 10 time steps the motion changes drastically when two are not able to capture these
particles come close each other. In our experiments, dynamics and we do not aim to predict the dynamics after this initial drastic deviation. We observe in our experiments that our symmetry control networks are capable to predict this initial drastic change correctly. This is in contrast to the HNN and baseline networks whose predictions for the trajectories are worse than for the symmetry control networks and do not capture this change. We show an example of the trajectories in Figure 3 alongside with the mean deviation of the trajectory, and the conservation of angular momentum, energy and the prediction of kinetic and potential energy. Instead of predicting the deviation from circular orbits, we find that the HNN networks predict more or less circular orbits, which explains a relatively good conservation of angular momentum while predicting trajectories which are worse with respect to the true trajectory.
3.2.3 COUPLED HARMONIC OSCILLATOR
Again we keep the HNN architecture and the architectures in comparison to our previous experiments unchanged apart from adapting the input and output dimensions. As these experiments do not posc a large challenge for cither HNNs and our symmetry control neural
'We check that there is significant change in the fit accuracy when including higher order polynomials.
7

---

Under review as conference paper at ICLR 2021
Figure 4: Comparing the test loss for HNN (blue), SCNN with one (red) and two (green) cyclic coordinates over the training time. We see that too many conserved quantitios lead to a dramatic drop in performance, while choos- ing the correct number of conserved quanti- ties results in a similar performance than the HNN. We averaged over 100 time steps to get a smooth curve.
Test Loss CHNN
SCNN with one cons. quantity SCNN with two cons. quantities
1.2
1.0-
0.8
0.6
0.4
0.2
400C0
Steps
networks, we would like to focus on fitting analytic expressions for the conserved quantities and the Hamiltonian. For instance, in the case of n 2, we find that the conserved quantities show explicitly the decoupling of the two sub-systems in our symmetry control networks. We find for the conserved quantities the underlying energies of the subsystem, e.g.:
Pi 0.67 of - 3.6 9142 + 8.23 o + 0.46 v 1.78 P1P2 + 4.2 p3 (0.29).
P2 4.53 9 1.74 9192 0.9 q3 4.5 pî 1.88 P1P2 0.57 p (0.26),
(17)
where the largest of the neglected terms arises with a factor smaller than 0.05 and the MSE of the fit is shown in the brackets. The entire Hamiltonian is also fitted very accurately as
H = 1.02 oî - 0.41 91 1.89 o + 0.97 p + 0.99 p2 (0.01),
(18)
where the largest term which is neglected has factor smaller than 0.05 and the MSE error is shown in the brackets. For these experiments we have used a SCNN-base with two cyclic coordinates. Details regarding the data have used for training and testing can be found in Appendix B
3.2.4 DOUBLE PENDULUM
The double pendulum is a good example to see what happens when looking at more con- served quantities than there exist. We tested the SCNN with one and two conserved quan- tities while setting the values of = to increase the effect of our symmetry ensuring loss components (B = for comparability). Comparing with the HNN, we find that the the test loss (CHNN) of the SCNN is very similar to the loss of the HNN for one conserved quantity, while for two conserved quantitics the convergence is much slower and after 50000 steps the difference is still a factor three over the complete test set (see Figure 4 We are not able to find an analytical formular of the conserved quantity but this comes as no surprise due to the complicated structure of the Hamiltonian (cf. Appendix A).
3.2.5 FURTHER EXPERIMENTS AND SUMMARY PERFORMANCE
We have analysed the effect of varying the loss weights on our physical systems and find no large effects when varying the weights in the range 10-3 < < 1 (ef. Appendix D for detailed results). A summary of the performance of our SCNNs in comparison to HNNs can be found in T To estimate the performance conservation we have integrated the mean square deviation from the true trajectory and the energy of the system respectively. Whereas the performance on the test loss is comparable to the HNN and baseline (a notable exception being the real harmonic oscillator and the double pendulum), the trajectories and energy are predicted much more accurately. An exception is the harmonic oscillator which already is very well captured with HNN.
the trajectories and the overall energy
4 RELATED WORK
There are several related directions where symmetries have been learned and utilised with neural networks. Also the use of physics bias not just via the Hamiltonian framework is playing large role in related works. We brielly comment several of them:
8

---

Under review conference paper at ICLR 2021
Test loss
Trajectory
Sum- mary of quantita- tive results across tasks. All quanti- 77.1e2 25.6622
Energy
Table 1:
Task
SCNN (HNN)
SCNN (HNN)
SCNN (HNN)
Two-body
5.25 0.15
82c2 + 1lc2
14.79 = 2.32
(x109)
0.044)
(114c3 = 9c3)
(19.12 1.92)
Three-body
6.85
1..662=3.422
ties are multiplied with 10 if not otherwise stated.
(x109)
(32.52+
(.0.0)2=8.862)
(13.5+2.4)
HO (x10%)
1.19
1.15c3 0.9662
32.39 + 9.24 9.21
(8.49
(2.29e2 1.39c2)
(8.75 1875+2.00 2.88)
Double Pend.
24.46
13.2e2 = 5.0e2
18.6 = 5.6
(47.01 3.97)
(18.0c2 = 2.3ee2
(229.3 169.4)
Real Sph. Pend.
6,84 0.78
(6.61 0.75)
Real HO(x109)
1.85 + 0.33
(6.10 1 1.83)
The baseline approach to predict dynamics has been studied in many other approaches previously (e.g. Grzeszezuk al. (1998) for early work on this topic).
An alternative to describing dynamics via Hamiltonian is via Lagrangian. The equiva- lent approach to HNNSs in terms of learning Lagrangians is discussed in (Cranmer et al. 2020
Gupta et al. 2019 Lutter et al. 2019
Canonical transformations to cyclic coordinates have been utilised in the context of newwnetwwrks. [oodeeaa & Lamacraft (2019) aim at finding the canonical transforma- tion to cyclic coordinates but they do not study the improved predictions using those coordinates and their loss uses finite difference as opposed to an auto-differentiation loss utilise here. [i et al. (2020) requires all coordinates to be cyclic (i.e. the system is integrable) which is not the case in our approach.
Predicting the coordinates (q. p) from images has been pursued in Grryydanus et al. 2019 Toth et al.) 2019). Here this would add an additional network before our input with the target output (q, p) as explicitly demonstrated to work in (Toth et al. 2019
Very accurate prediction of dynamics can be performed using interaction networks (Battaglia et al., 2016; WWttters et al.) 2017). Biasing the local interactions via the Hamilionian in graph networks has been discussed in SSanche-Gouzalee el al. (2019) It would be very interesting to apply our approach in this Tramework analyse the performance at larger systems.
Obtaining analytic formulae for the network output on physical systems has been pur-
sued in (Cranmer et al. 2019 Sahoo et al. 2018 Wetzel et al. 2020
DISCUSSION
One great challenge in physics the search for conserved quantities and underlying symme- tries. Our architecture provides approach on how to find these symmetries using neural networks automatically without prior knowledge of them. We find interesting performance simple toy systems where the symmetries well known. We are able to identify these symmetrics and by using them we can improve on performance with Hamiltonian noural networks.
will be extremely interesting to extend on this proof of concept analysis by applying it to state of the art predictive networks such in Shanche--Gozzaler al., 2019) which as of now only utilise the Hamiltonian and not all of the symmetries.
A second line of investigation in the realm of physics. An example for area of future application is in integrable systems which for classical systems is satisfied if all coordinates are cyclic. Determining whether a system is integrable is generally speaking unknown and our method provides an automated route to search for the required structures.
We hope to report on progress in these directions at future conferences.

---

Under review conference paper at ICLR 2021
REFERENCES
Peter Battaglia, Razvan Pascanu, Matthew Lai, Danilo Jimenez Rezende, et al. Interac- tion networks for learning about objects, relations and physies. In Advances in neural information processing systems, pp. 4502 4510, 2016.
Roberto Bondesan and Austen Lamacraft. Learning symmetries of classical integrable sys- tems. arXiv preprint AXX::1906.04643 2019.
Taco Cohen and Max Welling. Group equivariant convolutional networks. In International conference on machine learning, pp. 2990-2999, 2016.
Miles Cranmer, Sam Greydanus, Stephan Hoyer, Peter Battaglia, David Spergel, and Shirley Ho. Lagrangian neural networks. In ICLR 2020 Workshop on Integralion of Deep Newral Models and Differential Equations, 2020.
Miles D Cranmer, Rui Xu, Peter Battaglia, and Shirley Ho. Learning symbolic physics with graph networks. arxiv preprint or Xiw:1909.05862 2019.
Sam Greydanus, Misko Dzamba, and Jason Yosinski. Hamiltonian neural networks. CoRR, abs/1106.01563, 2019. URL http:///axiiti wwt.:oor/////aoeoeeoo http:://aaxxx..orr/a/////00000 01563
Radek Grzeszczuk, Demetri Terzopoulos, and Geoffrey Hinton. Neuroanimator: Fast neural network emulation and control of physics- based models. In Proceedings of the 25th annual conference Computer graphics and interactive techniques, pp. 9-20, 1998.
Jayesh K. Gupta, Kunal Menda, Zachary Manchester, and Mykel J. Kochenderfer. A general framework for structured learning of mechanical systems, 2019.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems, pp- 1097-1105, 2012.
T.D. Landau and F.M. Lifshitz. Mechanics: Volume 1. Number v. 1. Elsevier Science 1982. ISBN 9780050503479. URL https://books. wowpl/ooooo.ooooeee.ooooooooooo.
Shuo-Hui Li, Chen-Xiao Dong, Linfeng Zhang, and Lei Wang. Neural canonical transfor- mation with symplectic flows. Physical Review X, 10(2):021020, 2020.
Michael Lutter, Christian Ritter, and Jan Peters. Deep lagrangian networks: Using physics model prior for doep learning. arXw preprint WTV1:10070444 2019.
F. Podregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Griscl, M. Blon- del, P. Prettenhofer, R. Weiss, V. Dubourg, Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrol, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825 2830, 2011.
H. Rein and S. F. Liu. REBOUND: open-source multi-purpose N-body code for collisional dynamics. AAP, 537:A128, January 2012. doi: 00.1051/0004-638 6361/2011110885
Subham Sahoo, Christoph Lampert, and Georg Martius. Learning equations for extrapola- tion and control. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th Inter- national Conference on Machine Learning, volume 80 Proceedings of Machine Learning Research, pp. 14500 Stockholmsmässan, Stockholm Sweden, 10-15 Jul 2018. PMLR. URL http://procceedings wttp::/ceeediiinmm......mmmm mlr, wwwpp//800s/aoooo http:/ceeeeaag......reeee//////a
Ba. htttmommm....mll1
Alvaro Gonzalz,G Victor Bapst, Kyle Cranmer, and Peter Battaglia. Hamiltonian graph networks with ode integrators. Xiv preprint arXiv: 11t::199...2900, 2019.
Michael D. Schmidt and Hod Lipson. Distilling free-form natural laws from experimental data. Science, 324:81 85, 2009.
10

---

Under review conference paper at ICLR 2021
Peter Toth, Danilo Jimenez Rezende, Andrew Jaegle, Sébastien Racaniere, Aleksan- dar Bolev, and Irina Higgins. Hamiltonian generalive nelworks. aiv preprial arXiv: 1909. 18789, 2019.
Nicholas Watters, Daniel Zoran, Theophane Weber, Peter Battaglia, Razvan Pascanu, and Andrea Tacchelti. Visual interaction networks: Learning physics simulator from video. In Aduances in neural informalion processing systems, pp. 1539-4547, 2017.
Sebastian J. Wetzel, Roger G. Melko, Joseph Scott, Maysum Panju, and Vijay Ganesh. Discovering symmetry invariants and conserved quantities by interpreting siamese neural networks, 2020.
A ADDITIONAL SYSTEMS
This appendix presents det ails about some of the physical systems we have tested our SCNNs on.
A.1 SPHERICAL PENDULUM
The spherical pendulum is governed by the following Hamiltonian:
1 + P
H
m g v 02
(19)
2m
where we take a univeral mass m = 1/ coupling
and pendulum length / = 1. Our data
generation is described in Appendix B is conserved:
In this system, the angular momentum in z-direction
L JaPy qyPx
(20)
Additionally, one can find the different Hamiltonians for the subsystems due to the fact that the spherical pendulum can be seen as a two-dimensional harmonic oscillator with degenerate cigenvalues.
We use the same architecture as for the harmonic oscillator, and focus on fitting conserved quantities. We enforce two cyelic coordinates (the maximal possible number) and scan over 01 02 E {10", 10-1, 10-2, 10-3), and are able extract the conserved quantities (using 10-3):
Pi = 1.19 oh + 0.4 4,4y + 0.72 by + 1.77 p2 + 0.51 P2P, + 1.09 p" - 0.83 JxPy + 0.84 9yPx
P2 -0.65 3 0.58 9-4y 1.38 d3 0.96 2h + 0.77 PaPy 1.92 p 1.18 gxPy + 1.18 qyPa
(21)
with MSE of 0.003 and 0.002 and the omitted terms are below 0.05. One can see here the degenerated energies. The fitted Hamiltonian is:
H = 0.68 q2 + 0.71 of + 1.02 pi + 1.02 p (0.03).
(22)
The true Hamiltonian (19 and the found one differr only by factor of 0.005, and, therefore, linear regression finds a harmonic oscillator like structure.
A.2 CHARGED PARTICLE IN A MAGNETIC FIELD
The charged particle in a magnetic field, where we choose circular B-field plus an electric field, is governed by the following Hamiltonian:
H
1 _ 2m
(Ip.
9u B)² (p, a d. B)²) + A (1 +
(23)
where take univeral mass m 1,2 electric field k 1, charge q and B-field B 1. Here, the angular momentum in z-direction is conserved
L quPy JyPx
(24)
11

---

Under review conference paper at ICLR 2021
We use the same neural network architecture as the previous example, and enforce cyclic coordinate conditions on the maximum number of coordinates, but in this case we used a1 = a2 = 1002 due to the better performance when fitting the analytic quantities. Using linear regression, we find the following analytic expressions for the conserved quantities and for Hamiltonian:
Pi = 3.59 E +358 1f +180 1/ 1.79 p2 2.74 4oPy + 2.75 qyPx (0.43):
P2 = 2.33 of 2.32 13 - 1.17 1/ - p + 2.97 (aPy - 2.96 qylr (0.32),
H. 2.00 o + 2.00 of 1.00 p + 1.00 p -2.01 quPy 2.01 qyPx (0.01),
(25)
where we omitted terms smaller than 0.05 and the numbers in the bracket indicate the MSE of the fit.
A.3 REAL HARMONIC OSCILLATOR
As in the previous cases, we use the SCNN architecture to predict the motion of the particle. We are limited to the given data points from the datasets, and therefore only have one given trajectory. Therefore, the SCNN is not able to find the subsystems of the two dimensional harmonic oscillator, but is able to act on the test set with a good accuracy and performs much better than the HNN (see Table D].
B DATA
B.1
TWO-BODY PROBLEM
As described in the main text we set both masses and the coupling to unily. We sample 1000 trajectories with 50 data points each over time span of 20 which corresponds of at least two third of a full circle. We uniformly sample the starting point for the first body qı from the box [0.5, 1.5]². The starting position for the second body is at q2 qq
We choose the velocity of both bodies such a way that the bodies describe near circular orbits and give the entire system an overall momentum Prom = (2)
P1 (1 + <) Peire
Peom
P2 PP + Prom
(26)
where and 612 are sampled from a normal distribution with 0.05 and 0.1 respectively. We take 80 percent of the data for training and use the rest for testing.
B.2 THREE-BODY PROBLEM
The setup relatively similar to the two-body problem as we consider three body near circular motions and give the overall system an overall momentum. We initialise the three particles on the corners of an equilateral triangle. The velocity of the first particle is given as follows
pi = (1 + é) Peire
Prom
(27)
where the remaining velocities are chosen accordingly as in the two-body problem and where the center of mass motion and the deviation from the cireular motion are exactly generated as in the two-body example. We use 5000 trajectories with 20 data points for training and testing of our networks. The time span is 5 which corresponds to motion which is mostly circular. When using longer time spans, we find that convergence of the networks becomes problematic.
B.3 R--DIMENIONALL HARMONIC OSCILLATOR
The underlying Hamiltonian 14 is generated as follows. The coupling matrix is generated as the product of orthogonal matrix S and diagonal matrix D Ag ST.
12

---

Under review as conference paper at ICLR 2021
The entries of the diagonal matrix are random entries sampled uniformly in the interval [1- v which are then rounded to one digit to ensure that the values are sufficiently rational, i.c. that the motion is periodic. For instance, if we choose an entry in Dd equal to 2 the motion not periodic whereas if we choose 1.96 the motion is periodic. The matrix $ is obtained by generating random normal distributed entries (mean 0 and standard deviation 0) and we then orthonormalize the vectors.
The initial conditions (q(1 = 0), p(( 0)) are 2n-dlimensional vector which we generate as follows. We generale a vector with random entries (uniform sample) in the interval (0 - 2). We then normalize the vector to unit length and then re-scale it uniformly to a vector of length in the interval (0.2,2.0).
For our dataset utilise 100 initial conditions and generate 500 points on the trajectory. These points are taken from a time span of 10 which corresponds to a time where multiple periodic motions are included. We split it into equal size training and test set.
B.4 SPHERICAL PENDULUM
As described above we fix all parameters, and sample the starting conditions (q (t (t p pt 0)) from uniform distribution from the interval (-1,1). We ensure, that the energy is well-defined and negative.
For our dataset we utilise 100 initial conditions and generate 500 points on the trajectory. These points are take from a time span of 10 which corresponds to a time where multiple periodic motions are included. We use 80:20 split for the training and test set.
B.5 CHARGED PARTICLE IN MAGNETIC FIELD
As described above we fix all parameters, and sample the starting conditions (q (t: 0), p (t=t00 = (0) from a uniform distribution from the interval (-1,1), We then nor- malize the vector to unit length and then re-scale it uniformly interval 1.0). For our dataset we utilise 100 initial conditions and generate 500 points on the trajectory. These points are take from a time span of 10 which corresponds to a time where multiple periodic motions are included. We use 80:20 split for the training and test set.
a vector of length in the
B.6 DOUBLE PENDULUM
As described in the main text we fix all parameters, and sample the starting conditions (q = 0). (t=00 0) = 0) from a uniform distribution from the interval (0, 2T). Note, that 0, and 0g are angles and therefore, we have to ensure that they always lie in the inter- val (0, 2x). For our dataset we utilise 100 intial conditions and generate 500 points on the trajectory. These points are take from time span of 10 which corresponds to time where multiple periodic motions are included. We use a 80:20 split for the training and test set.
B.7 REAL HARMONIC OSCILLATOR
In this real world example we use data from the paper Schmidt & Lipson (2009), namely the example which is called wero-masss--urre-prnnnne use system with two fixed boundary points, connected by system with three springs and two masses. As this corresponds to two real harmonic oscillators we refer to this system as real harmonic oscillator. The dataset consists of one trajectory from real world system which is loosing energy. To make it comparable to the paper Greydanuu et al. (2019), we used the same split (taking the first 80 percent as training set., and the remaining part as test sct). Therefore we have a different distribution in the training set and the test set. Because the data consists of only a single trajectory of data points we are technically not able to find a conserved quantity. Therefore, we just test the performance and check how well the network generalizes to the test set.
To measure the data points, they
13

---

Under review conference paper at ICLR 2021
Two-body MSE Trajectory
0/2
100
10-1
10-2
10-3
01
10
7.41 + .66 x 10
9.87 + 1.16 10
6.06 + 0.61 10
3.30
0.70 10
10-1
2.88 0.23 10!
10¹
1.34 x 10
1.83
0.32 x 10
10-2
3.00 0.48
x 100
2.25
x 10
8.39 x 10
4.14 0.28 10
10-3
4.49 + 0.30 10
5.94 2.51
10
1.67 x 10 1.01
x 10
Table Loss-ypeerprrmntee scan for the two-body problem. We show the integrated MSE of the predicted and true trajectories.
B.8 REAL SPHERICAL PENDULUM
In this real world example we use data from the paper Schmidt & Lipson (2009), namely the example of the real pendulum. Due to the fact that we want to have more than one degree of freedom (otherwise the only conserved quantity the energy), we use always a pair of two data points from the one-dimensional pendulum to get two dimensional movement which is physically equivalent to spherical pendulum. We use the first 80 percent the data points as training set, and the remaining ones as test set. Because this is real-world system it looses some energy, and therefore, we have a slightly different distribution on the training set and the test set, but this difference is no limitation for the search of conserved quantitics.
C HYPERPARAMETERS
We have performed some hyper-parameter searches for our symmetry control networks which we give an overview here.
By varying the hidden layer size between 50-300, we find that there is a minimum size of layer size 100 to find convergence of our networks. We have varied the number hidden layers up to hidden layers. Two hidden layers are already sufficient (tested up to 5 hidden layers), hence we restricted ourselves on them. Coarsely speaking, we find that the precise architecture is less relevant in our current experiments.
More relevant are the fre-factors in the loss. Depending on the choice, we can either force our networks for better performance the predictions obtaining the conserved quantitics. For this tradc-off, we have optimized the experiments in this paper on the particle trajectories.
We find that pytorch's standard orthogonal initialization provides the best results out of the standard initializations. We have not observed large random seed dependence.
D Loss PREFACTORS
In our analysis of the loss factors we consider SCNNs with maximum number conserved quantities. This leaves us with two loss hyperparameters 0j and ag which we analyze here. We scanned them over the values {10", 101110111000 10 30-33 for the 2-body problem. The results in Table 2 below, where you used in the main text. As a comparison, the error of the HNN is significantly higher with 1.15 + 0.09 10-1
the average error of the trajectories as
The sccond measure we compare are the conserved quantitics. Thorefore, we look at the conservation of energy and angular momentum which is shown in Table and As comparison, the error of the energy for HNN is 1.91 + 0.19 x 10-5 and 1.23 + 0.10 X 10-3 for the angular momentum.
14

---

Under review conference paper at ICLR 2021
Two-body deviation energy
100
10-1
10-2
10-4
01
10
2.20 +: 0.29
10-1
2.04 1.38 10
4.83 4.74 10
2.01
0.77 x 10
10-1
2.88
0.23 10!
1.68
10
8.80 1.77 x 10
5.40
1.04 x 10
10-2
2,43 + 0.28 x 10-1
8.75
x 10
6.01 x 10
3.50 = 0.48 x 10
10-3
1.87 + 0.38 x 10-1
3.34 2.61 10-3
5.49 x 10 1.13 0.20 x 10
Table 3: Loss-hyperparameter sean for the two-body problem. We show the integrated deviation of the energy between the predicted and true trajectories.
Deviation of Angular Momentum for Two-body system
100
10-1
10-2
10-3
01
10"
1.69 0.21 10
0.15
10"
2.42
10-1
7.22 7 10 3
10-1
1.33 0.18 10a
1.08
10a
4.24 x 10-3
7.81 2.15 x 10-4
10-2
5.08 + 0.60 x 10a
x 100
4.94 x 10-1
1.64 0.31 x 10-1
10-3
+.15 10
4.15 3.00 10
3.74 10
2.97
0.63 x 10
Table 4: Loss-byperparameter scan for the two-body problem. We show the integrated deviation of the angular momentum between the predicted and true trajectories.
E POWER FOR MUSTINOMIAL-EXPASSION
In this appendix we elaborate how we decide which power for the multinomial expansion. For illustration purposes, we use the two-body example here.
We use the test set as the coordinates and fit multinomial function using Linear Regression and PolynomialFeatures implemented in sklearn (Pedregosa et al. 2011). When looking at the MSF-plot for the different degrees (ef. Figure 5) one can directly find a cut-off to determine the correct degree. Coordinate 1 and Coordinate 2 have degree one (both give us momentum conservation). while the fourth one can be fitted by a multinomial of order two (It is equivalent to the angular momentum). On the other hand, one can directly see, that Coordinate 3 has no multinomial expression, the error is small, but does not converge to specific value.
15

---

Under review conference paper at ICLR 2021
1.0
Coordinate
Coordinate 2
Coordinate 3
Coordinate
MSE
0.4
0.2
0.0
2
Degree of Multinomial Fit
Figure 5: Comparing the MSE depending on the degree of the used polynomials, We can see drastic drop of the eπror for a specific degree. When looking at the linear regression prefactors, we can identify the conserved quantity. The only distinct behaviour of the MSE is for coordinate 3. Here, we are not able to find suitable fit due to the problem, that we are not able fit the Hamiltonian function for the 2-body-problen with polynomials.
16


Reviews:
[{"review_id": "mfQO2UF7f0Q", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This submission explores the question of identifying conserved quantities in Hamiltonian dynamics for physical systems by attempting to learn canonical transformations. The approach closely resembles previous work on \"Hamiltonian neural networks\" but the loss is augmented with a term enforcing the invariance of the dynamics under the transformation and with a term that ensures the resulting transformed coordinates satisfy the constraints of the algebraic relations that emerge from the Poisson bracket. Together these two additions allow the authors to train a network that performs a change of coordinates which is subsequently optimized to bring it closer to a canonical transformation. Perhaps the main observation is that some of the cyclic coordinates identified by the network have a clear relation to the underlying conserved quantities. \n\nOn the whole I was not particularly impressed by the numerical results---despite imposing fixed numbers of conserved quantities, the performance is not meaningfully superior to the established HNN framework. I did not understand why the authors would use an integrator that is not symplectic---this seems like a clear route to bettering the performance over longer timescales. What is more, the weight of the additional terms in the loss was not systematically varied, so was not left with much sense of the extent to which it was affecting the optimization. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "_Egkrswxzn", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Summary.** The authors propose using a neural network to learn a canonical transformation of the data coordinates before learning a Hamiltonian. This is a novel contribution in that previous work has shown how to learn Hamiltonians with neural networks but it has not shown how to learn the proper canonical transformation. In the course of learning this canonical transformation, they show how to project out other symmetries (linear and angular momentum) and hence improve upon HNNs while also learning these other symmetries to a good approximation.\n\n**Strong points.** The authors have a strong grasp of Hamiltonian mechanics and they lay out the theory in an accurate and easy-to-follow manner. The core idea is a good one and it leads to promising (although entirely qualitative) results. The experimental setup is reasonable and the behavior of the trained models is visualized effectively. The symbolic regression applied to the learned models shows that they learned physically relevant quantities, and this is an excellent empirical validation of the authors’ approach, eg. Equation 16.\n\n**Weak points.** Some sections of the paper were very hard to follow. For example, the experimental setup which was introduced in the form of “Model 1, Model 2, Model 3…” was very confusing. First of all, the authors never gave intuition for what each of these models was specifically aiming to test. Here are my best guesses, although I would like to see the authors’ definitions as well:\n* Model 1: This appears to be the generic Symmetry Control NN. They enforce a Poisson loss in latent space and then train an HNN on it. Since they set \\beta to zero, H ends up being invariant to the transformed coordinates (P, Q) only implicitly, by way of the Poisson loss and the HNN loss. Since the fourth loss term is regularizing P_i to be constant, it appears that they are regularizing *all* P_i in the model to be constant. I don’t quite understand this. This presumes that all momenta are stationary, which is not the case for the 2-body system -- we only expect two of them to be constant. Is the idea that this regularization will hopefully force two to be constant while the other two are not?\n* Model 2: This enforces only half of the transformed coordinates to be cyclic. But the authors make another change: they force the learned canonical transformation to be linear by construction. So to summarize, they are changing two things: 1) now we're using a linear transformation for (p,q)->(P,Q) and 2) now we're using a different number of cyclical coordinates. A proper ablation of the experiment should just change one of these, and the authors should describe what they are trying to ablate.\n* Model 3: This resembles Model 1 more closely, but here we are explicitly defining the first two symmetries (x and y momentum conservation) but not the third (angular momentum)\n* Model 4: This is complementary to Model 3 in that they are fixing the angular momentum and learning the linear momenta\n* Model 5: They fix all the known symmetries (both angular and linear momenta). This corresponds to adding maximal domain knowledge to the model.\nThe remainder of the experimental results likewise need more explanations. There should be a table of quantitative results. There should be more discussion regarding the system of coupled oscillators. There should be quantitative measurements of error-of-fit for the symbolic regression models\n\tThe 3-body results are quite good and it was interesting to see that these models excelled in this context compared to baseline NNs and HNNs. The authors provide a nice, intuitive discussion of these results.\n\n**Recommendation.** 5 : Marginally below acceptance threshold\n\n**Reasoning.** This paper tackles a significant problem, presents a novel and useful method, and achieves promising empirical results. Its weaknesses were 1) that experimental methods and results were not sufficiently well explained and 2) there were not enough quantitative results. The paper cannot be accepted to ICLR as-is. I would consider changing my recommendation if these two core issues were addressed in a substantial way.\n\n**To improve the paper.** The authors should make their explanation of methods substantially clearer, with special attention paid to explaining why they design the experiments and models the way they did. The authors should provide qualitative results for all three tasks.", "extra_scores": {}, "extra_text": {}}, {"review_id": "S4dxT47QqT", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents the results of a NN trained to learn symmetries in physics, specifically, to learn and preserve quantities that are preserved (e.g., energy, angular momentum). The input is a sequence generated from a Hamiltonian dynamics. Results of experiments on 2 and 3 body problems and a harmonic oscillator are presented. The training networks are small, shallow feedforward networks. There is some customization of the training networks to incorporate \"cyclic\" coordinates. Results indicated empirical conservation up to small error of physically conserved quantities. The paper is fairly easy to read, with much relevant background provided.\n\nIn the early days of NNs, this might have been a very interesting paper. With today's advances, and NN finding success in almost every area with data, it is not clear what the contribution of this paper is. Perhaps the main innovation is the design of the networks. Unfortunately, there is little explanation provided of the experimental results.\n\n-- Why is this result interesting? Given that the output is a simple function of the input, why is this result surprising in the least?\n\n-- Given that the model for data is explicit and the training model is simple, can you say anthing rigorous to explain the results obtained empirically?\n\n-- the outputs are close but do not perfectly periodic. What parameter/model changes could possibly explain this? Would a different representation do better?\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "8FwPW5RgGnE", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a set of loss functions that can make neural networks learn Hamiltonian dynamics with conserved quantities.\nThe research builds on Hamiltonian networks and adds the search for conserved quantities as an auxiliary task.  The loss functions are a straight forward translation of equations describing Hamiltonian dynamics with conserved quantities.\n\nFor the machine learning audience, the paper could be written more didactically, such that it can be easier followed. In some parts, information is missing or misleading and the graphics need some improvements. Otherwise, the presentation is adequate.\n\nThe proposed method is novel, as far as I can tell. Although fascinating, its impact will probably not be very high, as real-world systems, for instance, robot dynamics are not a closed system and thus energy conservation is not given. Nevertheless, I would love to see more progress in this direction.\n\nMy main concern with the paper, as it is mostly empirical, does not provide enough evidence that the method would work in non-toy settings.\n\nConcrete problems:\n- the number of conserved coordinates has to be known\n    - here I think it would be possible to show what happens when the number of conserved quantities is scanned through. I guess a breakdown in performance would be observed when the number is too high. I think something like this would strengthen the paper\n    \n- the different models are confusing: If I understand correctly, then Model 1 is your method in its general form and is the only real model to be compared to. It only has the knowledge about how many conserved quantities are the system. The other models are some kind of ablation, an additional way to check which parts perform how well. I appreciate those models, but they should not be called model 1...5 but rather Sym-Net and Sym-Net + Oracle X Y Z, because they get quite a bit of extra information.\n\n- already for 3 bodies the new network does not perform much better (sometimes worse) than the baseline on its own terrain -- the conserved quantities:\n    - after 8 steps HNN is better or equally good on all conserved quantities (Fig 3). Still, the prediction is a bit better than the HNN, which puzzles me a bit. What is the reason for the better performance? Also, the qualitative predictions in Fig 3 are not really convincing.    \n- using data from a non-synthetic dataset that you are not generating yourself would strengthen the paper\n\nAs you are also after finding concise analytical expressions, your approach could be potentially nicely combined with the \"equation learning\" architecture [1]. In any case, it might also be a good baseline. It would be valuable to know whether the inductive bias of a functional form or the conserved quantities is more effective.\n\nDetails:\n - below Eq 3: unpack and explain a bit why the {conserved quantity,H} vanishes. (I think this will increase readability)\n - Eq 4: maybe Q and P could be introduced before they appear in EQ 4.\n - page 3: 2. state what you need from the Poisson algebra here: {p_i,p_j} = 0,  {q_i,q_j} = 0, {p_i,q_j} = \\delta_ij\n- p4 check last sentence\n- Fig 2 and 3 Font size is an order of magnitude to small. A proper legend would also help.\n\n[1] http://proceedings.mlr.press/v80/sahoo18a.html\n\n--- Post rebuttal update\n\nThe authors clarified and removed problems in the paper and did additional experiments. The changes to the paper are quite substantial and I cannot make a full review again.  (Upvote from 4->5)", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes to learn symmetries of a physical system jointly with its Hamiltonian from data by learning a canonical transformation that render some of the coordinates constant.
The Hamiltonian dynamics and "canonical" transformation are softly enforced via loss terms.
A few experiments are performed demonstrating that the idea works and can learn a few approximate invariants, as well as some improvements over baselines agnostic to the symmetries.
The idea is interesting, but the experiments are limited in scope. It is not clear how to extend this idea to more complex systems where we do not know the number of conserved quantities in advance. It is also not clear how good are the learned invariants, as the results showing errors in conserved quantities (Fig 3) suggest that it is not very precise beyond a few time steps.

Decision: Reject