Noether's Razor: Learning Conserved Quantities
Authors: Tycho F. A. van der Ouderaa
Mark van der Wilk
Pim De Haan
Venue: NeurIPS 2024 poster (2024)
Keywords: noether theorem razor bayesian model selection learning symmetry conserved quantities variational inference
Primary area: probabilistic_methods

Abstract: Symmetries have proven useful in machine learning models, improving generalisation and overall performance. At the same time, recent advancements in learning dynamical systems rely on modelling the underlying Hamiltonian to guarantee the conservation of energy.
These approaches can be connected via a seminal result in mathematical physics: Noether's theorem, which states that symmetries in a dynamical system correspond to conserved quantities.
This work uses Noether's theorem to parameterise symmetries as learnable conserved quantities. We then allow conserved quantities and associated symmetries to be learned directly from train data through approximate Bayesian model selection, jointly with the regular training procedure. As training objective, we derive a variational lower bound to the marginal likelihood. The objective automatically embodies an Occam's Razor effect that avoids collapse of conversation laws to the trivial constant, without the need to manually add and tune additional regularisers. We demonstrate a proof-of-principle on n-harmonic oscillators and n-body systems. We find that our method correctly identifies the correct conserved quantities and U(n) and SE(n) symmetry groups, improving overall performance and predictive accuracy on test data.

Noether's Razor: Learning Conserved Quantities
Tycho F. A. van der Ouderaa Imperial College London London, UK
Mark van der Wilk University of Oxford Oxford, UK
Pim de Haan
CuspAl
Amsterdam, NL
Abstract
Symmetries have proven useful in machine learning models, improving generalisa- tion and overall performance. At the same time, recent advancements in learning dynamical systems rely on modelling the underlying Hamiltonian to guarantee the be connected via seminal result
conservation of energy. These approaches in mathematical physics: Noether's s theorem, which states that symmetries in a dynamical system correspond to conserved quantities. This work uses Noether's theorem to parameterise symmetries as learnable conserved quantities. We then allow conserved quantities and associated symmetries be learned directly from train data through approximate Bayesian model selection, jointly with the regular training procedure. As training objective, derive a variational lower bound the marginal likelihood. The objective automatically embodies an Occam's 's Razor effect that avoids collapse of conservation laws the trivial constant, without the need to manually add and tune additional regularisers. We demonstrate a proof-of- principle on n-harmonic oscillators and n-boddy systems. We find that our method correctly identifies the correct conserved quantities and U(n) and SE(n) symmetry groups, improving overall performance and predictive accuracy on test data.
Introduction
Symmetries provide strong inductive biases, effectively reducing the volume of the hypothesis space. A celebrated example of this is the convolutional layer embedding translation equivariance in neural networks, which can be generalised other symmetry groups [Cohen and Welling 2016).
Meanwhile, physics-inntiie informed machine learning models [Greydanus et al. 2019 Cranmer et al. 2020 typically relying neural differential equations [Chen et al. 20188, embed constraints known from classical mechanics into model architectures to improve accuracy on physical dynamical systems.
Rather than strictly constraining a model to certain symmetries, recent works have explored whether invariance and equivariance symmetries machine learning models can also be automatically learned from data. This often relies on separate validation data Maile et al. 2022]. explicit regularisers [Finzi] jet al. 2021 or additional outer loops [Cubuk et al. 2018). Alternatively, we can take a Bayesian approach where we embed symmetries into the prior and empirically learn them through Bayesian model selection [yan der Wilk et al., 2018 Immer al. 2022 van der Ouderaa et al. 2022].
We propose to use Noether's 's theorem [Neether 1918) to parameterise symmetries in Hamiltonian machine learning models terms of their conserved quantities. To do so, we propose symmetrise a learnable Hamiltomian using set of learnable quadratic conserved quantitites, By choosing the conserved quantities be quadratic, we can find closed-form transformations that can be used to obtain an unbiased estimate of the symmetrised Hamiltonian.
Secondly, we phase symmetries implied by conserved quantities in the prior over Hamiltonians an leverage the Occam's 's razor effect of Bayesian model selection [Rasmussen and Ghahramani] 2000 van der Wilk et al.) 2018 learn conserved quantities and their implied symmetries directly from train data. We deniee practical lower bound using variational inference (Hoffman et al. 2013
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

resulting in single end-to-end training procedure capable of learning the Hamiltonian of a system jointl with its conserved quantities. As far as we know, this is the first case which Bayesian model selection with variational inference is successfully scaled to deep neural networks, an achievement in its own right, whereas most works so far have relied on Laplace approximations [Immer et al. 2022].
Experimentally, we evaluate our Noether's 's razor method on various dynamical systems, including n-simple harmonic oscillators and n-body systems. Our results suggest that our method is indeed capable of learning the conserved quantities that give rise to correct symmetry groups for the problem hand. Quantitatively, we find that our method that learns symmetries from data matches the performance of models with the correct symmetries built-in as oracle. We outperform vanilla training. resulting improved test generalisation and predictions that remain accurate over longer time periods.
2 Background
2.1 Hamiltonian mechanics
Hamiltonian mechanies a framework that deseribes dynamical systems in phase space, denoted
M = RM, with M even. Phase space elements (q.p) E .M follow Hamiltonian equations of motion:
0H Op,
aH
P:
0q:
where the Hamiltonian H .M- -> R is an observable which are smooth functions on the phase space, that corresponds to the energy of the system. It is often simpler to write x (q. so that we have: E = H and J = -- where I is the identity matrix, J is called the symplectic form and V = (xx) is the gradient of phase space coordinates.
Example: n-body problem in 3d. If we consider d-3 dimensional Euclidean space containing bodies, our position and velocity spaces are each H making up phase space M = 2R2-3n Our
Hamiltonian /l 3 R + R, which in this case is a separable function (g, p) K (q) + (p) of kinetic energy K (44) = >, m.] mm/lppll// 2// /2 and the potential energy P(p) = Σizs Gm,m,/ [|: 1
where m, is the mass of a body and G the gravitational constant.
2.2 Learning Hamiltonian mechanics from data
We can model the Hamiltonian from data Greydanus al. 2019 Ross and Heinonen 2023 Tanaka
et al. 2022 Zhong et al. (019). Concretely, we are interested in posterior over functions that the Hamiltonian can take p(H, D), conditioned on trajectory data D {x. a:) ] sampled from phase space at different time points (I, or time difference At=/' -t. Given a new data point a; we would like to make predictions p(æ; (xt He, D) over phase space trajectories into the future t'.
Hamiltonian neural networks [Greydanus et al.) 2019 Toth et 2019, Rezende et al., 2019 model the Hamiltonian H using a learnable Hamiltonian Ho M + Ie parameterised by Ti? With a straightforward Gaussian likelihood p( a2,,, 0) = -a laz, +
Hamiltonian neural networks
HoH 22 r) with small observation noise o a maximum likelihood fit can be found by minimising the negative liglliielhoood 0_= arg ming Σ, Σ, log p(z: ae |xt 0) minibatches of data using stochastic gradient descent. The mean of this likelihood represents single Euler
integration step (Sec. 2.1 of David and Méhats [2023). which bounds the possible accuracy of the fit the true Hamiltonian H. practice, we may replace this by more accurate differentiable numerical integrators Kidger 2022).
2.3 Noether's theorem
The theorem of [Noether] 1918), here presented in the Hamiltonian formalism [Bezz 2020 Amold 1989] links the concepts of an observable being conserved, to the Hamiltonian being invariant the symmetries generated by an observable.
'The term observable in classical mechanics should not be confused with the statistical notion of a variable being observed or not. In fact, we will model observables as latent variables that are not observed.

---

Conserved quantity Let O be the set of observables, which are smooth real-valued functions M - R on the phase space. Given a trajectory =(t) generated by the Hamiltonian H. we can compute the variation of an observable O E O in time via the chain rule and Hamilton's is equations of motion (Equation
dO
80 3g:
80 +
0H
80 0H
Σ
Q
Pi Op.
Σ
{0,
dt
3g. Öp: 0p: dq:
where the last equality defines the Poisson bracket 1 O + - 0 The Poisson bracket relates to the symplectic form via (O)((()) vO(x) JV (ar). An observable that does not change along any trajectory is called a conserved quantity. As we can see from Equation 2 an observable O is conserved if and only if {O, H]
From two conserved quantities 0,0' O, we can create new conserved quantity by linear combination aO + 33' E O with coefficients for a, 3 E R. which conserved because the Poisson bracket is linear in both arguments. Also, we can take the product OO' E O. with (0)')(z) = (x)),. which is conserved because the Poisson bracket satisfies Leibniz's law of differentiation {00', H {0, H + OfO', H). Finally, the Poisson bracket of the conserved quantities {0,0') E also conserved, because of the Jacobi identity.
Symmetries generated by observables Referring back as the Hamiltonian equations of motion in Equation ( note that these equations work not just for the Hamiltonian H E O of the system, but for any observable O O. So given any starting point zy. we can generate trajectory a=() satisfying
x: (0) Zo
((r) (VO(x(T))
We have used different symbol to not conflate the ODE time with regular time t of the trajectory generated by the Hamiltonian. Denote the flow associated to this ODE generated by observable O by % M + M, mapping 2y to (% 00th = 46 Φ. Hence, the observable O generates a one mineemnnsioona group Go. parametrized by T, that a subgroup of the group Diff (M) of diffeomorphisms M MM
=xr) Note that any ODE flow satisfies Tg id,M and
Theorem (Noether). The observable E O is a conserved quantity on the trajectories generated by Hamiltonian H E O if and only H is invariant to Go, meaning that all 7 E R. H o o = H.
Proof. By reasoning analogous to that in Equation (2), the value of the Hamiltonian changes under the flow generated by observable as dH dr = {H, O). Noting that the Poisson bracket is anti-symmetric,
we have that: O is conserved quantity = {0, H) to the flow generated by O.
(H, > H is invariant
2.4 Automatic symmetry discovery
Symmetries play an important role in machine learning models, most notably group invariance and equivariance constraints [Cohen and Welling, 2016]. Instead of having to define symmetries explicitly in advance, recent attempts have been made to learn symmetries automatically from data. Even if learnable symmetries can be differentiably parameterised, learning them can remain difficult as symmetries act as constraints on the functions a model can represent and are, therefore, not encouraged by objectives that solely optimise train data fit. As a result, even if symmetry would lead to better test generalisation, the training collapses into selecting no symmetry. Common ways to overcome this are designing explicit regularisers that encourage symmetry (Benton al. 2020 van der Ouderaa et al. [2022]. which often require tuning, or use of validation data [Alct et al. 2021 Maile et 2022 Zhou al.) 2020). Learning symmetries for integrable systems was proposed in [Bondesan and Lamacraft 2019]. whereas our framework works more generally also for non- integrable systems, such as the 3-body problem. Recent works have demonstrated effectivity of Bayesian model selection to learn symmetries directly from training data. This works by optimising the marginal likelihood, which embodies an Oceam's 's razor effect that trades off data fit and model complexity. For Gaussian processes, the quantity can often be computed in closed-form [van der Wilk et al., 2018), and can be scaled to neural networks through variational inference [van der Ouderaa and van der Wilk 2021 and linearised Laplace approximations [Immer et al. 2022
3

---

3 Symmetrising Hamiltonians with Conserved Quantities
Our method introduced in the next section will learn the Hamiltonian of a system together with set of conserved quantities. First, in this section we discuss how the learned conserved quantities will be parametrised, and how we can make the Hamiltonian invariant to the symmetry generated by conserved quantities.
3.1 Parameterising conserved quantities
In this work, we limit ourselves to modelling up to a fixed maximum number of к conserved quantities C,, C2 ..... C : +
IR are observables parameterised by symmetrisation parameters n, to distinguish them from the model parameters parameterising the Hamiltonian scalar field.
In this paper, we consider quadratic conserved quantities of the form Cz)(2) = x= Ax 2 + b" x + c. As use the conserved quantities only through their gradients, the constant is arbitrary and can be ignored. The learnable symmetrisation parameters are thus = [A,b) for a symmetric matrix A. A quadratic conserved quantity C generates symmetry transformation whose scalar field - = JVC(2) = JAr + Jb is affine, or linear on the homogeneous coordinates (æt, 1). Its flow can be analytically solved
(x)x)) exp (- JAA 0z
(7
JA
x i
using the matrix exponential exp(-) for which efficient numerical algorithms exist [Moler and Van Loan 2003]. This equation can be verified to have the correct scalar field and boundary condition, and thus forms the unique solution to the ODE in Equation (3).
3.2 Symmetrising observables
Given observable C E O, want to transform observable f into î that is invariant to the transformations generated by C. This means that a 2 f for all symmetry time 7 R. Via Noether's 's theorem, we know that this is equivalent to C' being conserved in the trajectories generated by f. and also equivalent to {C, f) = O. However, this equation does not prescribe how to obtain such f. Instead, we'll Il create by symmetrizing over the symmetry group generated by C. This is done by averaging over the orbit of the transformation
f(x) - f(06(=))M(T
with measure μ over symmetry time 7. This measure μ induces measure on the 1-dimensional subgroup Go of the group of diffeomorphisms M + M. If this measure on Gc uniform (specifically, a right-invariant measure [Halmos 1950)), then f is indeed invariant.
Instead of single symmetry generator, we can also have set C (C1. (C.....,Ca we want to make f invariant to all of these. Assume that this set spans a vector space of observables that is closed under the Poisson bracket (i.e. they form Lie subalgebra). In that case, the groups of transformations of the observables combined generate group Gc (Hall 2015 Thm. 5.20J. This group is parameterized by vector of symmetry times E IR The corresponding flow is & $ n, To make an observable f invariant to the symmetries of all conserverved quantities C, equivalently to the group Ge, we symmetrize
Ck) ) of observables and
f(x) f(z)= J f((T(=))
(5)
with some measure i over R As before, if this induces uniform measure over Gc, then this symmetrization indeed makes ff invariant to Ge-
However, probability measure μu(T) that gives a uniform distribution over Ge might not exist, for example when the group contains non-compact group translations. Even when such a measure does exist, it may be hard to construct, and the symmetrisation integral in Equation ( may be intractable to compute. So instead, in practice, we approximate this by choosing (uT) to unit normal distribution (0, Ix) uniform distribution. This results in a relaxed notion of symmetry

---

in f which can be interpreted as form of robustness to actions of the symmetry group implied by the conserved quantity, by smoothing the function this direction around data, in contrast to strict invariance by definition closed under group actions along the full orbit. Finally, we approximate the integral by an unbiased Monte Carlo estimate with samples.
4 Automatic Symmetry Discovery using Noether's Razor
Now that we have a way of parameterising symmetry differentiably conservation laws through Noether's theorem, we need an objective function that is capable of selecting the right symmetry. Unfortunately, regular training objectives that only rely on data fit can not necessarily distinguish the correct inductive bias, as noted prior work [van der Wilk et al., 2018 [mmer et al.) 2022 van der Ouderaa and van der Wilk, (02T]. This is because, even if train data originates from a symmetric distribution, there can be both non- non-symmetric and symmetric solutions that fit the train data equally well, given sufficiently flexible model. Consequently, the regular maximum likelihood objective that only measures train data fit will not necessarily favour a symmetric model, even if we expect this to generalise best on test data. Instead of having to resort to rross-validation to select the right symmetry inductive bias, we propose to use an approximate marginal likelihood on the train data. This has the additional benefit of being differentiable, allowing symmetrisation to be learned with back- pock-propagttion along with regular parameters in a single training procedure. In our case, we use Noether's 's theorem to parameterise symmetries in our prior through conserved quantities, which we can optimise with back- prekprropaaation using a differentiable lower bound on the marginal likelihood. This quantity, also known the 'evidence' differs distinctly from maximum likelihood in that it balances both train fit as well as model complexity. The Occam's 's razor effect encourages symmetry and leverages the symmetrisation process 'cut away' prior density over Hamiltonian functions that are not symmetric, if this does not result in a worse data fit. The resulting posterior predictions automatically becomes symmetric if observed data obeys a symmetry (high evidence for symmetry), but can become non-symmetric if this does not match the data (low evidence for symmetry). Hence, the name of our proposed method for automatic inductive bias selection Noether's 's razor.
4.1 Probabilistic model with symmetries embedded in the prior.
Fe
C
To be more explicit about our probabilistic model, we can introduce four variables, namely non-symmetrised observable Fe, a set of conserved quantities C.y. which induce a symmetrised Hamiltonian H generating the observed trajectory data X. We treat trajectory data an an observed variable, consider the conserved quantitics as part of empirical prior as we optimise over them, and integrate out the Hamiltonian as latent. The construction can be interpreted a placing sophisticated prior over the functions that the symmetrised Hamiltonian H can represent, which is the variable of primary in- terest. The underlying non-symmetrised Fa does not have direct physical meaning as H does, but defines a prior over neural networks to flexibly define density over a rich class of possible functions. The conserved quantities C. control the amount of symmetry in the effective prior over symmetrised Hamiltonians H. Empirically optimising C. through Bayesian model selection allows us to 'cut away' density in the prior over // that correspond to functions that are not symmetric as the symmetrisation averages functions in Fg that lie in the same orbit and thereby increases the relative density of symmetric functions H. We hypothesise that we will not over-fit conserved quantities as is relatively low-dimensional only representing quadratic functions, while we integrate out the high-dimensional neural network model parameters that parameterises the observable Fg. In future work, it would be interesting to explore a richer function classes for conserved quantities, such as neural networks, although we expect this to be more difficult and to require additional priors or regularisation techniques to avoid over-fitting.
H
X
Figure 1: Graphical proba- bilistic model. Trajectory data X depends on a symmetrised Hamiltonian H induced by non- non-symmetrised observable Fr and conservation laws C.
5

---

4.2 Bayesian model selection for symmetry discovery
To learn the right symmetry from data, we propose use Bayesian model selection through optimisa- tion of the marginal likelihood. In the previous sections, we have phrased symmetries parameterised by n as part of the prior over Hamiltonians. The symmetry parameters 1) parameterise the space of possible 'models' that we consider, whereas the model parameters 0 parameterise the weights of a single model. To perform Bayesian model selection on the symmetries, we are interested in computing the marginal likelihood:
p(zln) - ( p(z ( m)p(0)de
(6)
which requires integrating (marginalising) the likelihood over model parameters 0 weighted by the prior, and is sometimes referred to as the 'evidence' for a particular model. Unlike maximum likelihood, the marginal likelihood has an Occam's razor effect (Smith and Spiegelhalter 1980 Rasmussen and Ghahramani] 2000] that balancing both data fit and model complexity, allowing optimisation of symmetry parameters v). Although the marginal likelihood is typically intractable, certain approximate Bayesian inference techniques can provide differentiable estimates. In the next sections, we will use variational inference to derive tractable and differentiable lower bound to the marginal likelihood that can be used to find a posterior over 0 and optimise symmetries v7.
Why the marginal likelihood can learn symmetry To understand why the marginal likelihood objective is capable of learning the right symmetry (to learn 77), Sec. 3.2 [van der Wilk et al. 2018 proposed to decompose it through the product rule:
p( n) p(ar [m) (22p(222<)) m)p(ars 11:2; n)
II p( 11:0-11
which shows that the marginal likelihood measures how much parts of the dataset predict other parts of the data- measure of generalisation that does not require cross-validation. Given perfect data fit, the marginal likelihood will be higher when the right symmetry is selected, as parts of the dataset will result in better and more certain predictions the maximum likelihood, which is always maximised with perfect data fit, with or without the right symmetry. For some posterior approximations, such as lincarised Laplace approximations, it can be analytically shown that symmetry maximises the approximate marginal likelihood (App. G.2 of Immer et al. 20222). ). Our method is very similar, but uses more expressive variational inference which can optimise the posterior globally, rather than relying on local Taylor expansion.
other part of the data. This is unlike
4.3 Lower bounding the marginal likelihood
The marginal likelihood of Hamiltonian neural network is typically not tractable in closed-form. However, we can derive a lower bound to the marginal likelihood using variational inference (VI):
log p(æ n) 2 Eg [lag 0, n)] KL(gm.s(0) |I p(0))
(8)
Σn
N
log.Naat. Al (xt), Iaa D KL (0) p(0 0, a ppriorr I))
E
where (xt) Σ (e, (2) (æi)) and H is unbiased S-sample Monte Carlo estimator
of the symmetrised Hamiltonian. We write Eg = Eg and E,- = 11 H(T) for which we can obtain an unbiased estimate by taking Monte Carlo samples. The first inequality is the standard VI lower bound. The second inequality follows from applying Jensen's 's inequality (again) which uses the fact that the log likelihood is convex function. Similar lower bounds to invariant models that average over a symmetry group have recently appeared in prior work (van der Ouderaa and van der Wilk 2021 Schwöbel et al. 2022 Nabarro et al. 2022 Full derivation in Appendix A.
4.4 Improved variational inference for scalable Bayesian model selection
Variational inference is common tool to perform Bayesian inference on models with intractable marginal likelihoods, including neural networks. In deep learning literature, however, its use is typically limited to better predictive uncertainty estimation and rarely for Bayesian model selection. Meanwhile, linearised Laplace approximations have recently been successfully applied to Bayesian

---

model selection Immer et al. 2021] and symmetry learning in specific [Immer et al. 2022 van der Ouderaa al., 2024], with few reported cases model selection using VI only single neural network layers vaan der Ouderaa and van der Wilk 2021 Schwöbel et al. 2021]. Optimising Bayesian neural networks with variational inference much less established than training regular neural networks, for which many useful heuristics are available. This work, however, provides evidence that it is also possible to perform approximate Bayesian model selection using VI in deep neural networks, which we deem an interesting observation in its own right. To make sure the lower bound on the marginal likelihood is sufficiently tight, we employ series techniques, including a richer non-mean field family of matrix normal posteriors (Louizos and Welling 2016], and closed- form updates of the prior precision and output variances derived with expectation maximisation. Details how train Bayesian neural network using variational inference be found in Appendix D
Results
In this section, we will discuss how the learned symmetries are analysed and then list our experiments and results.
5.1 Analyzing learned symmetries
In experiments, will find set of K conserved quantities C M R. As consider quadratic conserved quantities in particular, we can equivalently analyze the resulting generators of the associated symmetries Ca(z)) = JVC,, which are affine and thus representable with a matrix GR (MM-I))(M+I) on homogeneous coordinates (x, 1). In Appendix B we list for each system ground truth conserved quantities generators Gj. The learned and ground truth generators can be stacked in to the matrices G E <M++1))) G RLx(M+1)" respectively. As we can identify the symmetries only up to linear combinations, we have learned the correct symmetries the learned generators span a linear subspace of R(M+1)² that coincides with the space spanned by the ground truth generators. To verify this, test two properties. First, show that the matrix G has L non-zero singular values. Secondly, for the first L right singular vectors v; E R (M)),, we decompose v; = of + of in a vector in ground truth subspace, and one orthogonal to it. The learned 8; is a correct conserved quantity ;; = O, or equivalently, because the singular vectors are normalized, if ||o_| || 1. We call this measure the "parallelness".
5.2 Simple Harmonic Oscillator
POW
Jaamed sormeci
Faod soldl crocle
Tus
We start with a demonstration on the simple har- monic oscillator. This text book example has a 2-dimensional phase space, making Icarned Hamiltonians amenable to visualisation. Fur- ther, it has a clear rotational symmetry SO(2), relating to the conserved phase. On finite set of generated train data, we model the Hamiltonian using vanilla HNN, our symmetry learning method, and a model with true symmetry built-in as reference oracle (experimental details in Ap- pendix B..). In Figure 2 find that our symmetry learning method results in rotationally invariant Hamiltonian that matches the fixed rotational SO(2) symmetry. Further away from the origin, the learned Hamiltonian differs from the ground truth Hamiltonian, as there is no data in that region. In Table I we find that the learned symmetry has a better ELBO on the train set and matches the improved predictive performance the model with the correct symmetry built-in. The symmetry learning method outperforms the vanilla model in terms of predictive performance on the test set.
Figure 2: Learned Hamiltonians on phase space of
simple harmonic oscillator by HNN models.
7

---

Table I:
Learning Hamiltonian dynamics of the simple harmonic oscillator. We compare a vanilla HNN, our symmetry learning method, and a model with the correct SO(2) symmetry built-in as reference oracle. Our method achieves reference oracle performance, indicating correct symmetry learning, and outperforms the vanilla model by improving predictive performance on the set.
Learedd dynamics:
Train data
Test data
simple harmonic oscillator
Train MSE
NLL/N
KL/N
-ELBO/N (L)
Test MSE (J)
HNN
0.005
0.3667
3314.374
3314.741
0.005
HNN learned symmetry
(ours)
0.002
22618
3304.754
3302.136
0.002
IINN fixed SO(2)
(reference oracle)
0.002
33,213
3298.357
3295.144
0.002
5.3 72- Harmonic Oscillators
Now, we consider n n-harrnniic oscillators. This sys- tem has as symmetry group the unitary Lie group U(n) of dimensionality of n² (see Appendix B.2). We sample random trajectories from phase space and train a HNN neural network without and with sym- metry learning using variational inference. Again, we find improved ELBO and test performance for learned symmetries Table 2 Following the protocol from Section 5.1 we analyze the learned symmetries.
08
GGnN spectiul!
Gencrebor Specttaa
Figure Singular value and paralleiness of the singular vectors of the learned generators, for n oscilators. U(n) correctly learned.
that for varying n, indeed find that the matrix of learned symmetries has n² nonzero singular values. Furthermore, the first n² singular vectors lie in the ground truth subspace
In Figure 3 (right),
of generators with measured parallelness || 0.99, as seen in Figure 3 (left). This shows that the U (n) symmetry is corectly learned.
Table 2: Learning Hamiltonian dynamics of 3 f-ld harmonic oscillators. We compare HNN with symmetry learning to a vanilla HNN without symmetry learning and to the correct U(3) symmetry built-in fixed reference oracle. We find that method discover the correct symmetry, achieves reference oracle performance, and outperforms vanilla training in both ELBO and test performance.
Learned dynamies:
Train data KL/N
Test data
simple harmonic oscillator
Train MSE
NLL/N
-ELBOIN (J)
Test MSE
0.00106
-12.04
5.27
-6.77
0.00002141
HNN + learned symmetry HNN fixed symmetry U(x)
(ours)
0.00102
-12.16
2.53
-9.63
0.00000994
(reference oracle)
0.00102
-12.15
2.21
-9.94
0.00000898
5.4 n--oddy System
To investigate performance our method on more interesting systems, consider learning the Hamil- tonian of an n-body system with gravitational inter- that
Sioguaa volues
Puralluiness 11
1.00-
0.75
0.50
action. We trajectories and generators remain easy to visualise. As the Hamiltonian depends only on the norm of the momenta and on positions via the relative distances of the bodies, the three dimensional group SE(2) of rototranslations is an invariance of the ground truth Hamiltonian. However, as explained in Ap- pendix [B.3] the Hamiltonian has four more quadratic conserved quantities. They generate a 7-dimensional Lie group of symmetries. This group has the same orbits the phase space SE(2). Therefore, function being invariant to SF(2) equivalent to being invariant to G. We'll find that Noether's razor discovers not just SE(2), but all seven generators of G.
bodies in 2 dimensions
0
Figure Singular value and paralleiness of the singular vectors of the learned generators for three body system in two dimensions. The 7-dimensional Lie group G of quadratic con- served quantities is correctly learned.

---

Generators associated leamed conserved quantibles and their singular value decompesttio.
Example rotational symmetry associated to generator
1x:
2.0
130
rwh2
Figure 5: Learned generators associated by conserved quantities and their singular value decomposi- tion. We find subspace spanned by the 7 linear generators that correspond to the correct symmetries (see Appendix B:3): (1) rotation of the center of mass RCOM (2) rotation around the origin RABS, (4+5) translation, (5+6+7) nomentum-deeendeenn
translations P. Q. (8+9+10) inactive (A < 0.05).
The first singular vectors lie the ground truth subspace of generators with measured parallelness ||!! > 0.95.
In Table we compare performance of a vanilla variational HNN with our symmetry learning approach and a model that has the appropriate SE(2) symmetry of rototranslations built-in as an reference oracle. We find that our method is able to automatically discover the conserved quantities and associated generators that span the symmetry group. The model achieves the same performance as the model with the symmetry built-in as reference oracle, but without having required the prior knowledge. Compared the vanilla baseline, our approach improves test accuracy on both in- distribution out-of-distribution test sets.
Table 3: Learning Hamiltonian dynamics of 2d 3-body system with variational Hamiltonian neural networks (HNN). We compare our symmetry learning method to vanilla model without symmetry learning and a model with the correct SE(2) symmetry built-in as a reference oracle. Our method capable of discoverying symmetry achieves the oracle performance, outperforming the vanilla method.
Learned dynamics:
Train data
Test data
Test data (moveed)
Test data (wider) Test (1)
3-body system
NLL/N
-ELBON (1)
(4)
HNN
0.0028
-13.87
13.31
-952
0.0016
0,0035
0.0016
HNN leumed symmotry
(ours)
0.0177
-20.99
7.28
-12.81
0J0006
0,0004
0.0006
HNN fised SF())
(reference oracle)
0.0019
-19.27
7.96
-11.32
0.0006
IL0006
0.00N66
After training, we analyse the learned conserved quantities and implied symmetries by inspecting their associated generators. In Figure 5) we plot these generators as well as their singular value decomposition. We see that our method correctly learns 7 singular values with X; > 0.05 and the associated singular vectors lie in the ground truth subspace with || > 0.95. This indicates that our method is in fact capable of inferring the right symmetries from train data, beyond merely improving generalisation by improving predictive performance on the test set.
6 Conclusion
In this work, we propose to use Noether's theorem to parameterise symmetries in machine learning models of dynamical systems in terms of conserved quantities. Secondly, we propose leverage the Occam's 's ruzor effect of Bayesian model selection by phrasing symmetries implied by conserved quantities in the prior and learning them by optimising an approximate marginal likelihood directly on train data, which does not require validation data or explicit regularisation of the conserved quantities. Our approach, dubbed Noether's razor, encourages symmetries by balancing both data fit and model complexity. We derive a variational lower bound on the marginal likelihood providing a concrete objective capable of jointly learning the neural network as well as the conserved quantities that symmetrise the Hamiltonian. As far as we know, this is also the first time differentiable Bayesian model selection using variational inference has been demonstrated on deep neural networks. We demonstrate our approach on n harmonic oscillators and n--body systems. We find that our method learns the correct conserved quantities by analysing the singular values and correctness of the subspace spanned by the generators implied by learned conserved quantitites. Further, we find that our method performs on-par with models with the true symmetries built-in explicitly and we outperform vanilla model, improving generalisation and predictive accuracies on test data.

---

References
Ferran Alet, Dylan Doblar, Allan Zhou, Josh Tenenbaum, Kenji Kawaguchi, and Chelsea Finn. Noether networks: meta-learning useful conserved quantities. Advances in Neural Information Processing Systems, 34, 2021.
Jean-Pierre Amiet and Stefan Weigert. Commensurate harmonic oscillators: Classical symmetries. J. Math. Phys., 43(8))4110-41222 August 2002. Utthttp::/ddx httpp:::/dxxdo...oooo doi. wtt....rg/...11//1111150000
V Ammodd. Mathematical methods of classical mechanics, 1989.
John C Baez. Getting to the bottom of noether's theorem, 2020. URL http:://axww..og//aabbb..a.
2006. 14741
Gregory Benton, Mare Finzi, Pavel Izmailov, and Andrew Gordon Wilson. Learning invariances in neural networks. arXiv preprint arXiv:2010.11883 2020.
Roberto Bondesan and Austen Lamacraft. Learning symmetries of classical integrable systems. arXiv preprint arXiv: wrXivv1196.000 999.0006555 2019.
Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural ordinary
differential equations. Advances in neural information processing systems, 31, 2018.
Taco Cohen and Max Welling. Group equivariant convolutional networks. In International conference on machine learning, pages 2990-2999, PMLR, 2016.
Miles Cranmer, Sam Greydanus, Stephan Hoyer, Peter Battaglia, David Spergel, and Shirley Ho. Lagrangian neural networks. arXiv preprint arXiv:2003.04630 2020.
Ekin D Cubuk, Barret Zoph. Dandelion Mane, Vijay Vasudevan, and Quoc V Le. Autoaugment: Learning augmentation policies from data. arXiv preprint 1 2018.
Marco David and Florian Méhats. Symplectic learning for hamiltonian neural networks. Journal of Computational Physics, 494: 12495, 2023.
Mare Finzi, Gregory Benton, and Andrew Wilson. Residual pathway priors for soft equivariance constraints. Advances in Neural Information Processing Systems, 34, 2021.
Samuel Greydanus, Misko Dzamba, and Jason Yosinski. Hamiltonian neural networks. Advances in neural information processing systems, 32, 2019.
Roger Grosse and James Martens. A kronecker-factored approximate fisher matrix for convolution layers. In International Conference on Machine Learning, pages 573-582, PMLR, 2016.
Brian C Hall. Lie Groups, Lie Algebras, and Representations. Springer International Publishing, 2015. Uttpsstss://1ink springer. wttp://eee...eeooo/000.0.00//000 0tt.0//00000000077777----------0
Paul R Halmos. Measure Theory. Springer New York, 1950. URL Sictps://1ink.sprtt
com/
book/10. 1007 11117799978---84-94400000
Irina Higgins, Loic Matthey, Arka Pal, Christopher P Burgess, Xavier Glorot, Matthew M Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. ICLR (Poster), 3, 2017.
Matthew D Hoffman, David M Blei, Chong Wang, and John Paisley. Stochastic variational inference. Journal of Machine Learning Research, 2013.
Alexander Immer, Matthias Bauer, Vincent Fortuin, Gunnar Rätsch, and Khan Mohammad Emtiyaz. Scalable marginal likelihood estimation for model selection in deep learning. In International Conference on Machine Learning, pages 4573-4573. PMLR, 2021.
Alexander Immer, Tycho F. A. van der Ouderaa, Vincent Fortuin, Gunnar Rätsch, and Mark van der Wilk. Invariance learning in deep neural networks with differentiable laplace approximations, 2022.
10

---

Patrick Kidger. On neural differential equations. arXiv preprint arXiv:2202.02435, 2022.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Christos Louizos and Max Welling. Structured and efficient variational deep learning with matrix gaussian posteriors, In International conference on machine learning, pages 1708-1716. PMLR, 2016.
Kaitlin Maile, Dennis George Wilson, and Patrick Forré. Equivariance- architectural optimiza- tion of neural networks. In The Eleventh International Conference on Learning Representations, 2022.
Cleve Moler and Charles Van Loan. Nineteen dubious ways compute the exponential of a matrix, twenty-five years later. SIAM review, 45(1):3 49, 2003.
Seth Nabarro, Stoil Ganev, Adrià Garriga-Alonso, Vincent Fortuin, Mark van der Wilk, and Laurence Aitchison. Data augmentation in bayesian neural networks and the cold posterior effect. In Uncertainty in Artificial Intelligence, pages 1434-1444. PMLR, 2022.
E. Noether. Invariante variationsprobleme.. Königlich Gesellschaft der Wissenschaften Göttingen Nachrichten Mathematik-physee Klasse, 1918.
Carl Rasmussen and Zoubin Ghahramani. Occam's 's razor. Advances in neural information processing systems, 13, 2000.
Danilo Jimenez Rezende, Sébastien Racanière, Irina Higgins, and Peter Toth. Equivariant hamiltonian flows. arXiv preprint 1909:119990133339 2019.
Magnus Ross and Markus Heinonen. Learning energy conserving dynamics efficiently with hamilto- nian gaussian processes. arXiv preprint arXiv:2303.01925, 2023.
Pola Schwöbel, Martin Jørgensen, Sebastian W Ober, and Mark van der Wilk. Last layer marginal likelihood for invariance learning. arXiv preprint arXiv:2106.07511 2021.
Pola Schwöbel, Martin Jørgensen, Sebastian W Ober, and Mark Van Der Wilk. Last layer marginal likelihood for invariance learning. In International Conference Artificial Intelligence and Statistics, pages 3542-3555. PMLR, 2022.
Adrian FM Smith and David J Spiegelhalter. Bayes factors and choice criteria for linear models. Journal of the Royal Statistical Society: Series B (Methodological), 22(2):213-220) 1980.
Yusuke Tanaka, Tomoharu Iwata, et al. Symplectic spectrum gaussian processes; Learning hamil- tonians from noisy and sparse data. Advances in Neural Information Processing Systems, 35: 20795-20808, 2022.
Peter Toth, Danilo Jimenez Rezende, Andrew Jaegle, Sébastien Racanière, Aleksandar Botev, and Irina Higgins. Hamiltonian generative networks. arXiv preprint 1909 2019.
Tycho van der Ouderaa, Alexander Immer, and Mark van der Wilk. Learning layer-wise equivariances automatically using gradients. Advances in Neural Information Processing Systems, 36, 2024.
Tycho FA van der Ouderaa and Mark van der Wilk. Learning invariant weights in neural networks. In Workshop in Uncertainty & Robustness in Deep Learning. ICML, 2021.
Tycho FA van der Ouderaa, David W Romero, and Mark van der Wilk. Relaxing equivariance constraints with non-stationary continuous filters. arXiv preprint arXiv:2204.07178. 2022.
Mark van der Wilk, Matthias Bauer, ST John, and James Hensman. Learning invariances using the
marginal likelihood. arXiv preprint arXiv: 180888888853.. 2018.
Yaofeng Desmond Zhong, Biswadip Dey, and Amit Chakraborty. Symplectic ode-net: Learning hamiltonian dynamics with control. arXiv preprint 1xXiv:1909.12011 2019.
Allan Zhou, Tom Knowles, and Chelsea Finn. Meta-learning symmetries by reparameterization. arXiv preprint arXiv:2007.02933. 2020.

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The paper proposes a new method that can automatically discover correct symmetries and improved predictive performance hold-out test data. This is justified by experiments that measure the correctness of symmetries and evaluate test performance.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The paper states modelling assumptions terms of used likelihood and prior. Further, the paper is restricted to fairly small scale experiments.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
12

---

Answer: [Yes]
Justification:
All theoretical results provides required assumptions and a complete proof.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Yes. For each dataset, we provide detailed description how data points were generated in Appendix C. For each experiment, we provide training details and important hyperparameters in Appendix C.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
13

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [NA]
Justification: Code will published upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cr
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ prijps.cc/ubblic/guiieee CoodeSumeeieeeeee ssi iessonPolll molyicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: Yes.
Justification:
See Appendix C.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [NA]
Justification: We report negative log likelihoods, ELBO, test predictive performance datasets after optimising the ELBO for thousands of epochs. Train and test scores are computed over the full dataset, but repeated for multiple seeds.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
14

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: All experiments were run on a single NVIDIA RTX 4090 GPU with 24GiB of GPU memory.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: pertpa://neuripae http://www.pp...pppp.c.ccccceeee
Answer: [Yes]
Justification:
The paper respects the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This paper is foundational paper not tied to particular application.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
15

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA].
Justification: This paper does pose such risk and experiments only consider very simple text book problems.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
Justification:
The paper does not use existing assets.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
16

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
batasets.papersstt
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The datasets used in this model are new, but details are clearly described in App. C.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA].
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA].
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
17

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
18

---

A Mathematical derivations
A.1 ELBO of Hamiltonian Neural Network
We can find a lower bound on the marginal likelihood log p(æ (7) through variational inference,
log p(
77) 2 Eo logg(( 0, m)] KL p(0))
(9)
= Ba Σ i=1
log log. Nas Ho,n( oauntt
KL sm))(()))))))) p(O | o oprie I))
(10)
Eg
Σnm iml
log log.N(s> Er, [ (
KL (1m.sss()) p(0 0,
o prior ())
(11)
2 E0
ET Σ
log.N
(xi) oasu x)] kL(qm,s (00) p(@ 0, o prior I))
(12)
M
log Na
(xi)), oama 1) - (l (0) p(0 Σ Ho. (T,,""x))) s
0, a prior I))
M
(13)
where we use M samples to obtain an unbiased estimate of Eg Em and single sample E, = BT-11 p(r) and use an S-sampled Monte Carlo estimate of the symmetrised Hamiltonian:
1 (x,) S Σ Ho., ( ( (x()) with samples 7(1), 7()) ((9) p(T)
(14)
with the fact that this yields an unbiased estimator the true symmetrised Hamiltonian E, (+)m(.) = (.), where we obtained an unbiased estimate of expectations through S sampled symmetry transformations and M sampled parameters. The first inequality is the standard VI lower bound. The second inequality follows from applying Jensen's 's inequality (again), using the fact that the log likelihood is convex. Similar bounds to symmetrisation by averaging over orbits have appeared in prior work van der Ouderaa and van der Wilk 2021
Schwöbel et al. 2022
Nabarro
al. 2022
B Ground truth conserved quantities
In this section, we'll discuss the conserved quantities present in the ground- truth Hamiltonians the systems we discuss.
As stated in Section products, and Poisson brackets. Thus we'll speak of the generating set of conserved quantities, which combine into all conserved quantities.
we can combine conserved quantities into ones by linear combinations,
B.1 Simple harmonic oscillator
For the simple harmonic oscillator, the phase space is R x R. The ground truth Hamiltonian is
1²
ky²
H(p.q)
2m
We choose m == so that H(p g) = (p² /2. Time evolution a rotation of phase space.
The Hamiltonian itself generates all conserved quantities.
19

---

B.2 n-Simple harmonic ascillators
The phase space is R We choose all lo m so that the Hamiltonian
Hp, q) = |2 /2+ 1/
Time evolution rotates each pair Pgi. The conserved quantities are generated by the following set, for i,j = n, à + j.
H. = (uî + +11//2
Raj = 9:P} 4yPs
Fij = 9o9j + p.P5
The conserved quantity H; rotates the pair (ts. Raj rotates both pairs (4i, q3) and (pi, and Fij rotates both pairs (g;)P) and P:).
Alternatively, we can interpret the phase space as C [Amold 1989 Sec. 41EJ, with the positions being the real part and the momenta the imaginary part. In that case, the symplectic form becomes simply the complex number -i. Then (x) x x/2. Time evolution is multiplication by the complex number e it Conserved quantities are H4 x7 22. which is real, and Ctj = x7 whose real part corresponds to Fij and imaginary part to Raj. The conserved quantities H; and Ci are quadratic and thus their symmetries are generated by linear matrices, which are all "kew-Hermittan In fact, all kew-Herrmitia matrices are spanned by these generators, This shows that the combined symmetry group fact (nn) [Amiet and Weigert 2002].
B.3 n-body
In D spatial dimensions, with bodies, the phase space is 2 and the Hamiltonian is
H(p q) = = ||p:11
Gm,m, mj
2m
11
4 12 +
143
with small to make it smooth.
The main conserved quantities are, for d, dt' I.....,de d'
Ta Σ Pid
RABS
Σ(e (q: pi)aa
COM R
(fCOM A accom)as
with 9com = Σ; m.q./ E; my and pOOM E. m.p./ E, my
These generate further conserved quantities of interest:
Pa T
Qud' T.T
RREL
Σ ((q (00M) A (p. - -PCOM)ds R 2RROOM
As Ta is linear, we can still learn Pa and Quæ as quadratic conserved quantities. As the RREL is a linear combination of other conserved quantities, we disregard it in our analysis of the learned symmetries.
The corresponding symmetries are: center of mass of all the bodies the plane dd', while preserving the positions relative to the center of mass. RABS rotates all bodies relative to the origin. RREL rotates all bodies relative to the center of mass. Pa translates in the direction proportional to its COM momentum. Qad translates in direction proportional to paoM,'' and vice versa.
Ta translates all bodies in the d direction. CCOM rotates the dd
20

---

These symmetries together generate a group we' call G. This group has as subgroup SE(D), which generated by T and RA The group has the same orbits SE(D), as each element in & can be seen rototranslation conditional on some property of phase space. Because the orbits are the same, for any observable f M R, we have that f invariant to true if and only f invariant to SH(2).
This system can have further conserved quantities, such as the Laplace-Runge- vector for However, these are not expressible as a quadratic polynomial. As far as we know, the conserved quantities listed above are all that are expressible quadratic polynomial.
C Experimental details
All experiments were run single NVIDIA RTX 4090 GPU with 24GiB of GPU memory.
C.1 Simple harmonic oscillator experiment
Data For the training data, we sampled initial conditions from unit Gaussian and simulated datapoints with At = 0.2 apart. For test data, we sampled 100 initial conditions from unit Gaussian and simulated 20 timesteps with At 0.2 from each initial condition.
Training We use an MLP with 2 hidden layers, each consisting of 200 hidden neurons and linear exponential unit activation function with a 2. For symmetrisation, we uniform measure for μuT)) We use 20 Euler steps for time integration. We use fixed output noise and closed-form prior variance (Appendix Dj. We optimise the ELBO in full batch with Adam [Kingma and Ba 2014 (31 = 0.9, 82 = 0.999) trained for 2000 epochs with learning rate of 0.001, cosine annealed to O.
200 samples from
C.2
n2 Simple harmonic oscillators experiment
Data For training data, we randomly sampled 200 initial conditions independently from unit normal. For each initial condition, we simulated a trajectory consisting of 50 data points at 0.3 time units apart.
Training We use an MLP with 3 hidden layers with 200 hidden units and exponential linear activation functions with a 1. We optimise the ELBO in mini batches of B = 20 trajectories, using 100 symmetrisation samples. 20 Euler steps for time integration, and M weight samples using Adam (Kingma and Ba 2014] (3) = 0.9, 32 = 0.999) for 2000 epochs with a learning rate starting from 0.001, cosine annealed
C.3 n--ody experiment
Data For training data, we randomly sampled 200 initial conditions by independently sampling positions from a unit normal, shifted by normal with standard deviation 3. From each initial condition, we simulated trajectories consisting of 50 data points 0.3 time units apart.
Training We use MLP with 4 hidden layers with 250 hidden unit units and exponential linear unit activation functions with a = 1. We optimise the ELBO batches of B 20 trajectories, S 100 symmetrisation samples, 20 Euler steps for time integration, and M 2 weight samples using Adam Kingmaa and Ba, 2014] (31 = 0.9, 82 0.999) for 2000 epochs with learning rate starting from 0.001, cosine annealed to 0.
21

---

D On training a neural network with variational inference
D.1 Matrix normal variational posterior
Naively, the covariance of a Gaussian posterior over weights grows quadratically with the number of parameters |0]2. It is therefore common to disregard all correlations between weights, resulting in a diagonal or mean-field posterior. Although the ELBO remains a lower bound for any choice of approximate family, more crude approximations can increase the slack in the bound, possibly making it harder to use estimates for Bayesian model selection. We, therefore, propose to use matrix normal posteriors [Louizos and Welling 2016 factorised per layer,
(()) II g(0)). with
(())) N(O: |, m,, Si A1)
(15)
where (0) 0.....,0 01) in terms of weight and bias matrices of each layer with in; input and out, output dimensions, ve) IW; bi]= Roou (1m we can equivalently write this posterior as a factorised matrix
denote the weights of each layer 1. we denote the parameters
normal distribution:
( II ((.),, with 1=1
((,)) -MN(W, bi] M. A.)
(16)
The variational parameters S; A1) provide the mean M2 well correlations between layer inputs and bias A E (1) and layer outputs S Rou For L hidden layers of width H, the number of variational parameters scales quadratically O(LH²) compared to the quartic number of variational parameters (LHH*) we would need to represent the full covariance. This strikes a practical balance between taking important correlations into account while avoiding having make a mean- mean-field assumption. Further, we note that the matrix gaussian posterior of [oouzzos and Welling, 2016] is the same approximate distribution as used in Kronecker-factore Laplace approximations [Grosse and Martens 2016). In Laplace approximations the covariance is the inverse Hessian, whereas in variational inference the covariance is optimised using the ELBO. Layer-factored matrix normal distributions have been succesfully applied perform approximate Bayesian model selection based on the Laplace approximation in [Immer et al. [021] 2022 van der Ouderaa et al. 2024). This work provides evidence that variational inference can also be used to obtain approximate posteriors of this form and obtain lower bound on the marginal likelihood that sufficiently tight to perform Bayesian model selection in deep neural networks.
D.2 Closed-form output variance
can be shown that the output variance that maximises the marginal likelihood 22 is the empirical variance of the output. We, therefore, either fixing the output variance 6 typically to very small number in noise-free settings, or setting the output variance to an empirical output variance. An exponentially weighted average of the empirical variance over mini-batches can be used.
On downscaling the KL term by a 3-scllrr Many deep learning papers that use variational inference down-scale the KL term by 3--prrameter [Higgins al. 2017]. We note that, for standard Gaussian likelihoods, scaling the output variance is equivalent to inversely scaling the KL term. We do advice against downscaling of the KL term, it makes it less clear that the resulting objective is still a lower bound to the marginal likelihood, and hides the fact that the lower bound corresponds to changed model with altered output variance. In MAP estimation under Gaussian likelihood, the output variance is arbitrary as it only scales the objective not effecting the optimum, and the objective is often simplified as the mean squared error. In variational inference, the output variance does play an important role of balancing the relative importance between the log likelihood (data fit) and KI. term (pull to prior). In this setting, using half mean squared error effectively corresponds an output variance of 2 = 1. In practice, this value is often too high because common machine learning datasets have little label noise. As a result, the log likelihood term too weak and the KL term too strong. We hypothesise that this has led to practitioners to down- weightttgg the KL term to obtain sensible posterior predictions, without necessarily realising that they were effectively altering the output variance of the model. Using automatic output variance, the optimal f can set to (a running estimate of) the inverse of the empirical variance, also known the empirical prior precision.
22

---

D.3 Closed-form prior variance minimising inverse KL
Consider the setting of D-dimensional Gaussian q. parameterised by mean m and covariance S, and zero mean Gaussian Pe with scalar variance v in each of the equally many dimensions:
(m,, S)
Pv (0,
where 0 denotes a zero vector and I an identity matrix. As the log likelihood does not depend on we can find e that optimises the marginal likelihood by finding the minimiser of the inverse KIL:
v
arg min KL l || po] arg min
log
D + Tr((el)-1
(0
' m)
2
S
arg min
1
log
D Tr(S)/v + m m///
2
S
arg min
[D log(u) + Tr(S)/u m m//]
Setting the derivative to zero:
0 Dlog(v) Tr((S) m m/u
0 dv
0 Du + Tr(S) mmm". m
v²
v. Tr(S) + m m
(17)
D
We found KL-minimising variance U, in closed- flosed-form as function of m and S. Verified numerically.
D.4 Plugging minimising variance into KL
Plugging e, back into the Gaussian Pr. and computing the KI::
I
[g [g || Po.]
log
+ Tr((u. (0 (v) m))
S
Jue I
DTr(S)
Dm m
V.
log
D + Tr(S) + m m
log
2
[S]
Tr(S) + m. m
2
S
Trr(S)
+ m m
D log
-log log |S|
D
shows that the resulting KL is only measuring the relative volume log ppl posterior, which can be further simplified as
between the prior and the
1
KI. |1 p...
2 log((T(()) m. m) m.mm)
[D
D Dog D) - -log |S|]
In practice, might reparameterise S LL" in terms of its triangular Cholesky factor L and
log S = log LL | = log L = 2 log L 2oo I L = 2 Σ log L
Tr(S) Tr(LT)) Σ D
i.j
This gives the final expression of the KL
KL kL 11 Po.]
1 2
D log
Σ1
Σ m
D log D) - >
log Lis
(18)
2.8
which implicitly uses the derived optimal variance u. Ei, 4;,+2; m
E
Code and Questions
The code is available at /tttps:///ithhbb.com http://www...oeeeeoo//o/eeeeeere For any questions, please contact the corresponding author, Tycho van der Ouderaa, by email.
23


Reviews:
[{"review_id": "CGBdm6bVYQ", "reviewer": "Reviewer_hvbw", "summary": "This work aims to model symmetries for strong inductive biases in machine learning models of dynamic systems. Instead of constraining models to certain symmetries, this work focuses on automatically learning them from data. It proposes to parameterize symmetries using conserved quantities by Noether's theorem. The conserved quantities are further incorporated into priors and the model is trained by optimizing a lower bound of the marginal likelihood, which is able to balance both data fit and model complexity. Empirical results on both n-harmonic oscillators and n-body system problems are presented.", "questions": "- See weakness.", "limitations": "Yes.", "rating": 9, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- The integration of Noether’s theorem with variational inference to learn conserved quantities for symmetries is a novel and significant contribution, providing a new perspective on symmetry learning.\n- Experiments show that the learned symmetries have the same rank as the ground truth symmetries, showing the effectiveness of the proposed learning method. Further from the empirical results, the model with learned symmetries has a similar performance to the one with built-in oracle symmetries and outperforms the ones without.\n- The work is overall well-written and has provided sufficient background for the authors to understand the proposed method as well as references to existing work.", "weaknesses": "- This work can be further improved by providing computational analysis on the computation overhead induced by symmetry learning. Maybe the authors can elaborate on this point.\n- From Section 5.1, it seems that the choice of hyperparameter K has to be greater than the dimension of ground-truth symmetry in order to capture the full symmetries. I wonder how to choose such hyperparameters in practice.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback and help to improve the paper. We appreciate that the reviewer rated our paper with a \"9: Very Strong Accept\", found our approach a novel and significant contribution, well presented and recognized the new perspective on symmetry learning as well as convincing experimental results.\n\nWe agree with the authors that including computational analysis can further strengthen the paper and will report time measurements in the final version. In terms of the choice of hyperparameter K, the singular values described in Sec. 5.2 can be used to validate a chosen setting of K. If there are many (close to) zero singular values, K can be decreased, and if there are none such values, K should be increased. Setting K too high will not hurt performance, but add some computational overhead. In our experiments, we typically set K=10 or K=20.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "a29JzbQQaf", "reviewer": "Reviewer_qjc1", "summary": "The authors propose to use the parameterized symmetries for learning correct Hamiltonian dynamics from data. It is based on the Noether’s theorem, which states that the continuous symmetries generated by observables O for Hamiltonian H yields the conservation of O, and vice versa. To do this, the authors parameterizes the observables as a quadratic form, and makes the learnable Hamiltonian function (neural network) to be invariance  with respect to the one-parameter group (flow) generated by O, which is analytically expressible. Invariance is achieved through techniques such as orbit pooling, which averages the outputs of the Hamiltonian over the group orbit. To learn the Hamiltonian with proper symmetries, the authors propose to use the Bayesian approach, i.e., learning the ELBO, thus the title of the paper is the “Noether’s razor” (similar to the Occam’s razor that regularize the model with proper weights). The authors validate their approach to classical examples like Harmonic oscillator and n-body systems, which showcase the method can model the appropriate dynamics as well as find a correct symmetries.", "questions": "The authors directly parameterize the model Hamiltonian to be invariant by averaging over the group orbit. Another way to achieve the symmetry is satisfying {C, H} = 0, according to the Noether’s theorem. Since the latter can be achieved easily by adding the regularizer like ||{C, H}||, I am curious about why the authors chose to use the method proposed in the paper instead of this regularizer, and how much of a performance difference there is between the two methods.\n\nSome typos:\n\n- page 2, line 70, $x_t + J \\nabla H_{\\theta} (x_t) \\Delta t$ \n\n- page 3, line 85, $\\nabla O^T(x) \\cdot J \\nabla H (x) $\n\n- page 3, line 120, also\n\n- page 4, line 156, the the", "limitations": "The authors do not explicitly mention the limitation in the main manuscript. Please refer and address Weaknesses.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The motivation of this paper is clear, and highly relevant to both the AI + Science and geometric machine learning communities. The paper is generally well-written and easy to follow. The method is based on well-known Noether’s theorem, thus is very principle. Overall, I believe this paper is worthy of publication and will provide valuable insights to researchers in the relevant field, even though it may not be particularly striking to a broader ML audience.", "weaknesses": "- This paper assumes that the symmetry arises from the symplectic flow of a quadratic form, which is a significant assumption that lacks a clear explanation. The authors should justify the use of a quadratic form, demonstrating that it is a reasonable assumption for the systems of greatest interest (not only for the benchmarks used in the paper).\n\n- The experiment consists only of low-dimensional examples modeling phase space trajectories directly, raising doubts about whether the proposed results can be applied to high-dimensional data. Please note that even the original Hamiltonian neural network [1] includes a task involving learning the underlying dynamics from images.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback and help to improve the paper. We thank the reviewer for finding the paper clear, well-written, easy to follow, and highly relevant to both the AI + Science and geometric machine learning communities. Further, we appreciate the reviewers deemed the paper very principled and 'worthy of publication'.\n\n> Quadratic form\n\nWe indeed consider conserved quantities of the quadratic form, but feel this is not an overly strong assumption. It restricts us to the symmetry group having an affine (linear+translation) action on the phase space, but it does not limit the structure of the symmetry group itself. As far as we know, basically any work on in/equivariant deep learning uses groups with an affine action. The model can in principle still learn any dynamical system, also those with non-quadratic symmetries, given a sufficiently flexible network. Compared to regular HNN models, which do not have any learnable symmetries built-in, we deem our model as an improvement, even in those cases, as it can improve generalisation by picking up on some quadratic conserved quantities which may be present. Please see our overall rebuttal for more details.\n\n> High-dimensional data\n\nIn our experiments, we focus on n-harmonic oscillators in higher-dimensions and chaotic n-body systems, both directly on phase space. We did find that our method remains to work well when increasing dimensionality. We discover a 25-dimensional symmetry group for n=5, which for groups is very high-dimensional.\n\nOur focus is demonstrating the principle of learning symmetrisation through conserved quantities. We therefore do not also consider learning models from images or only observing positional data. These tasks would imply a latent variable model which would significantly complexity the model description and distract from our primary research interest of demonstrating the Noether's razor effect. Regardless, we see no a-priori reason why Noether’s razor could not be applied to such extensions since they have been shown to work for regular HNNs in prior work. We consider this an interesting follow-up research direction, but out of scope for this paper.\n\n> Regularizing objective\n\nThis is an interesting suggestion. Instead of averaging the neural network along the group orbit, we indeed could also have considered regularising it in the same direction. In principle, this would also be a valid way of incorporating symmetry in the prior and is common for some other hyperparameters, such as the prior variance on weight magnitudes. We hypothesise that averaging generalises better as it smoothes the function globally, not only around train data, but refrain from making hard statements on this, since we did not try this yet. Although regularization could offer an interesting alternative to averaging to perform symmetrization, our use of averaging should not impact the main conclusions of our work, which was demonstrating Noether's razor. Nevertheless, we agree with the reviewer that it would be interesting to try this and intend to add it as an alternative baseline in the final version.\n\nTypos will be fixed, thanks for spotting them.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xaUnW4CB0x", "reviewer": "Reviewer_zC9J", "summary": "This paper proposes a Bayesian framework to learn conserved quantities (and, implicitly, symmetries) in the context of Hamiltonian systems. The idea is that the Hamiltonian system has specific parametric conserved quantity (given by a quadratic function, line 136), in that case the conserved quantity can be learned from data in a Bayesian setting (using the parameterization of the conserved quantity as a prior over the possible Hamiltonian systems) via ELBO. \n\nEven though the setting is a bit restricted, the idea is quite intriguing and conceptually very interesting. My main criticism of the paper is that the methodology is not explained in a clear, self-contained way, making the paper hard to follow for the NeurIPS audience. Since the main contribution of the paper is conceptual, I think this is a major drawback. The paper seems hastily written (it has many typos, including in the abstract \"conversation laws\" instead of \"conservation laws\", inconsistent notations, and incomplete definitions) and it could significantly benefit from a careful major revision.", "questions": "Is an explicit formula for $\\Phi$ needed in order to implement the optimization? Is that a limitation for the approach?\n\nWhat is the relevance of the distribution over $\\tau$? How do the results depend on it?\n\nCan we used this to learn the specific conservation law ($\\eta$) or is it only known implicitly in the solution $H(\\Phi)$?", "limitations": "The methodological limitations are not explicitly stated.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The topic is very interesting. Incorporating symmetries via conservation laws in machine learning models is an interesting idea in the context of physics informed machine learning, and machine learning models for physical systems. The combination of Noether's theorem with Bayesian modeling is novel. The empirical evaluation, though limited, show that these ideas can be promising.", "weaknesses": "In my opinion, the paper would benefit from a major revision focused on improving the exposition to clearly explain the methodology.\n\nThe paper considers a specific example, where the conservation law is parameterized by a quadratic function. Under this assumption we have an explicit formula for the flow $\\Phi$ that seems to be key for implementing the optimization. Without this assumption it is unclear to me that the problem is computationally tractable. I don't see this as a weakness necessarily (though it would make sense to state it as a limitation in the conclusion, if that's the case). However, it does make the main contribution of the paper to be mostly conceptual. Therefore this paper main contribution should be the explanation of the methodology. In my opinion this explanation is not sufficiently clear.\n\nIn particular it'd be nice to see an explanation of why one would choose to optimize the ELBO over direct optimization over the parameters that define the conservation laws and the Hamiltonian.\n\n\n**Comments:**\n- Line 85: The definition of the bracket depends on C which has not been defined before. Should it be O?\n- Please make explicit what is the domain and range of the functions considered in the manuscript. For example \n$x: \\mathbb R \\to \\mathcal M$\n- This sentence could be explained better: \"We have used a different symbol to not conflate the ODE time $\\tau$ with regular time $t$ of the trajectory generated by the Hamiltonian.\"? In particular in relation to referring $\\tau$ as \"symmetry time\".\n- Also, the difference on the parameters $\\eta$ that define the space of symmetries vs $\\theta$ that define the space of Hamiltonians should be made more explicit (section 3.1). In particular, this should be explained in section 4.\n- Section \"Why the marginal likelihood can learn symmetry\". What does it mean to \"learn symmetry\" mathematically in this context? Does it mean to learn $\\eta$?\n- A few typos: \"conversation laws\", \"train data\", \"generalisaiton\", \"als\".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback and help to improve the paper. We thank the reviewer for finding the paper 'intriguing' and 'conceptually very interesting' and that the empirical evaluation shows that the ideas can be promising\n\n> Q1. Non-quadratic conserved quantity\n\nFirst, we believe that the quadratic conserved quantity is not an overly strong assumption. It restricts us to the symmetry group having an affine (linear+translation) action on the phase space, but it does not limit the structure of the symmetry group itself. As far as we know, basically any work on in/equivariant deep learning uses groups with an affine action. Please see our overall rebuttal for more details.\n\nIf one wishes, our method can be generalized to non-quadratic conserved quantities whose flows have no explicit solution! To optimise our objective we require differentiable samples from our flow. In case of a non-quadratic conserved quantity, we can use the reparameterization-trick by sampling a particular symmetry time and solving the flow through a differentiable ODE solver. Our code, in fact, also includes an implementation of this in JAX (using the diffrax library), which we will release upon acceptance. We chose not to describe this in-depth in the paper, because we relied on quadratic conserved quantities for our main results.\n\n> Q2. Direct optimization of conserved quantities\n\nAs described in Sec. 4, we can not directly learn conserved quantities with data fit loss because no symmetry will always make fitting easier (even though it can hurt generalization). This is acknowledged by virtually every symmetry learning paper, which is why most methods rely on validation data or other forms of model selection. Unlike cross-validation, which requires validation data and can only evaluate certain settings of hyperparameters through expensive model retraining, approximate Bayesian model selection allows for gradient-based optimization of hyperparameters.\n\n> Q3. Relevance of distribution over \\tau\n\nWe only considered a zero-centred Gaussian distribution over \\tau with learning the variance and found this to work well across experiments. Although we are free to choose any distributional family, we expect that unimodal distributions around the origin are often the most sensible and least difficult to optimize. Further, we can argue that a Gaussian subsumes a regular HNN, which would be equivalent to having a \\delta peak (zero variance Gaussian) at the origin. Taken together, we think a Gaussian \\tau is sufficient in most cases, especially since the variance is learned, and will add some of these considerations in the main text.\n\n> Q4. Can we use this to learn the specific conservation law?\n\nYes! We can directly inspect the learned conservation laws as well as the associated symmetries, which was not possible in prior work. We find that we learn the correct conservation laws for our n-harmonic oscillator and n-body system experiments. With the quadratic conserved quantity, we can compute the symmetry action with a matrix exponential. With a free-form conserved quantity, this would require solving an ODE. This is a clear advantage of our method compared to prior work, and we will make this more clear from the main text.\n\n> Description of our methodology\n\nThe reviewer points out that the current methodology is not always clear from the main text. We agree that we might have focused too much on the conceptual idea of Noether's razor and will move more practical details and contributions from the appendices to the main text (e.g. App. D). This should give a more explicit description of how the forward-pass and objective calculation are implemented in practice.\n\n> Comments\n\nWe agree that the difference between parameters \\theta and symmetry parameters \\eta can be made more explicit early on. In our work, we have a neural network for F and a quadratic conserved quantity for C, so \\theta are all neural network parameters and \\eta are the matrices A and vectors b. With learning symmetries, we mean learning \\eta, which parameterise learnable conserved quantities and thus directly imply learnable symmetries by Noether's theorem. We hope this resolves the issue.\n\nTypos will be fixed, thanks for spotting them.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "30UQqysH59", "reviewer": "Reviewer_Pckk", "summary": "This paper proposes Noether's Lazor, a Bayesian framework incorporating learnable symmetries for Hamiltonian neural networks. It parametrizes a hidden symmetry as a flow (single-parameter group) derived from the system's conserved quantity. The flow is then applied to the Hamiltonian as it transforms the system's states while conserving the quantity. The objective function is given as the ELBO that averages over the transformation, forcing the Hamiltonian to be invariant, corresponding to the symmetry. Numerical experiments show that the proposed method finds true symmetries.", "questions": "1. I think we can think of two baselines: (a) Bayesian but not symmetry-adapted, i.e. the objective is given by E_\\theta[ loss of HNN ] + KL, and (b) symmetry-adapted but not Bayesian, i.e. the objective is Eq.(8) but no KL and no expectation w.r.t. \\theta. What would their performance be in the experiments?\n1. Why did you use Bayes instead of CV? What are the pros/cons?\n1. Can you measure the wall clock time of the vanilla HNN and the proposed method?\n\nMinor issues:\n1. C(x) first appears at line 85 without a formal definition.\n1. At line 245, H_{\\theta, \\eta}( \\Phi^{\\tau}(x) ) would be H_\\theta(\\Phi^{\\tau}_{\\eta}(x)) since the symmetrizing parameter \\eta is used in the time evolving operator \\Phi but not in H.", "limitations": "Limitations are not addressed.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. This study addresses the challenging but essential problem of finding symmetries in a Hamiltonian dynamical system. \n1. Extracted symmetries in the experiment (Fig 5) are interpretable and interesting.", "weaknesses": "1. It is not mentioned enough how the proposed method differs from previous studies. For example, [Alet et al., 2021] might be one of the most relevant works but only mentioned that it uses a validation set. Also, it says that `Similar lower bounds to invariant models that average over a symmetry group have recently appeared in prior work [van der Ouderaa and van der Wilk, 2021, Schwöbel et al., 2022, Nabarro et al., 2022]`, but the differences are not mentioned. \n1. The computational complexity is not presented. Since the computation of a Bayesian method is usually heavy, it would be nice to mention how it is compared to the vanilla HNN. \n1. The concepts of Noether's theorem and Occam's Razor do not sound tightly connected. As Theorem 1 says, Noether's theorem implies that a Hamiltonian H is G_O invariant, where G_O is the flow of an observable O, iff O is conserved under the system of H. It seems the authors use this notation as a prior of H to impose symmetries (symmetrization of H). However, it takes the integral over the transformation parameter \\tau (5). This integral is independent of what we do to obtain the marginal likelihood --- the latter takes an average over the parameter of H. So, I think we can use the symmetrization of H without the Bayesian manner, and I cannot find any special reason why we have to use Noether's theorem and Occam's Razor simultaneously. (If I say something wrong, please correct me.)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback and help to improve the paper. We thank the reviewer for finding the paper 'intriguing' and 'conceptually very interesting' and that the empirical evaluation shows that the ideas can be promising\n\n> Difference with prior work\n\nWe thank the reviewer for pointing this out and will clearly state differences with prior work.\n\nThe reviewer is correct that [Alet et al., 2021] is closely related. There are, however, important differences.\nThe most important difference is that we explicitly use the learned conserved quantities to infer the symmetries of the phase space. These symmetries then are used to symmetrize the Hamiltonian. In doing so, our work uses the essence of Noether’s theorem. In contrast, their work only learns conserved quantities, without inferring the resulting symmetries. Secondly, [Alet et al, 2021] models images with low dimensional phase spaces, whereas we consider complex dynamical systems with higher dimensional phase spaces. Lastly, while both works appreciate that maximum likelihood training will not allow learning of symmetries, we use approximate Bayesian model selection using VI whereas they rely on a meta-learning objective. Meta-learning can not readily be combined with an ODE solver (as that would require a 2nd order solver) without considerable memory cost, and requires additional hold-out validation data.\n\nThe works [van der Ouderaa and van der Wilk, 2021, Schwöbel et al., 2022, Nabarro et al., 2022] use a similar way of lower bounding learnable symmetries via the ELBO, but only consider simple affine invariances in image classification tasks, whereas we learn more complex symmetry groups in dynamical systems. A second difference is that we use Noether’s theorem to parameterize learnable symmetries in terms of their associated conserved quantities. Lastly, prior works that use VI for symmetry learning have so far only demonstrated this successfully using single-layer approaches (e.g. single layer / last-layer VI), whereas we manage to scale our work to deep neural networks (details in App. D.)\n\n> Connection between Noether's theorem and Occam's razor\n\nWhile it may not be necessary to learn Hamiltonians symmetrized by conserved quantities via Occam’s razor, we argue that it is very compelling to do so. Learning symmetries naively is plagued by learning trivial symmetries, or only some of the symmetries repeatedly. Occam’s razor via Bayesian model selection provides a single ELBO objective that is simple to optimize. For example, we have shown that our method is able to discover a 25-dimensional symmetry group in the 5-harmonic oscillator. Also, we found all 7 conserved quantities of a 2-body system, generating a much larger group than the 3-dimensional group SE(2) we (naively) expected to find. This shows our method is very successful in avoiding these trivial solutions and instead finds all the symmetries in a system. \n\n> Q1. Baselines\n\nTo be clear, we train both the HNN baseline and HNN + symmetry adaptation using the same variational inference scheme (so are both `Bayesian' in that sense) to allow fair comparison. As such, the improvements in test performance can therefore be attributed to the proposed symmetry learning through Noether's razor. We will make sure this is very clear from the main text. All works on symmetry learning that we know of acknowledge the fact that learning symmetries can not reliably be done through maximum likelihood only, and therefore rely on some form of validation data, regularisation, or marginal likelihood estimates to perform model selection. Regardless, we will also run a ('non-Bayesian') maximum likelihood baselines of both HNN and HNN + symmetry adaptation in the final version.\n\n> Q2. Bayes instead of cross-validation\n\nThis is a great question. The main benefits of Bayes compared to CV is that the objective is 1) differentiable and we can thus optimize hyperparameters with gradients and 2) we have a single training procedure that does not require any retraining. With CV, we can evaluate only a finite number of settings for the conserved quantities that would have to be specified in advance. Further, it requires retraining the entire model for each possible configuration to be evaluated on hold-out validation data. We thank the reviewer for raising this, as it is an important motivation behind our method which we will make more clear from the text.\n\n> Q3. Wall clock time\n\nWe will report the memory and computational cost of our method and baseline in the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5AatfMkMrV", "reviewer": "Reviewer_tVqw", "summary": "This work proposes a novel way to parametrise Hamiltonians that are approximately invariant to a set of learnable symmetries with the aim of improving existing methods of learning Hamiltonians of dynamical systems. It does so by forcing Hamiltonian neural nets (HNN) to adhere to symmetries, just as in reality, through Noether's theorem. Specifically, they parametrise conserved quantities as quadratic functions for which the flow can be analytically solved and approximately integrate a non-invariant parametrisation of the Hamiltonian (given by an existing HNN) over this flow. The result of this integration is then known to be approximately invariant to the symmetries represented by the quadratic conserved quantities. To optimise their model, which produces an intractable marginal likelihood, the authors resort to variational inference as was done in [1]. In effect, they apply the methodology of [1] to a learnable set of conserved quantities and show experimentally that it allows them to learn the underlying symmetries of different dynamical systems.\n\n[1] van der Ouderaa, Tycho FA, and Mark van der Wilk. \"Learning invariant weights in neural networks.\" Uncertainty in Artificial Intelligence. PMLR, 2022.", "questions": "1. Can you more clearly separate novelty from background? Right now, a large part of the paper (Section 4) is discussed as if it were a contribution, while it is more of an application of previous derivations. I would also suggest the authors to perhaps write the paper from a more experimental perspective if it turns out the theoretical contributions are not as extensive.\n\n2. Can you elaborate on the proposed evaluation metrics for symmetry comparisons? Moreover, can you explain why there are no other baselines, such as the Laplace approximations or any of the other methods discussed in Section 2.4? Are they not applicable? If so, accentuate why they are not applicable in the text.\n\n3. Can you provide mean and variability metrics in addition to the given values for Tables 1, 2 and 3? Given the lacking novelty and dependence on empirical results, this is a crucial oversight to me. The experiments should be repeated for multiple seeds.\n\n4. Smaller question out of curiosity: it is mentioned that in Section 5.2 (lines 301-302) that the learned Hamiltonian does not generalise to regions further away from the origin due to a lack of data. The symmetries do seem to improve the generalising behaviour, yet are still insufficient, are there any directions for future work to enforce further physical properties to improve out-of-distribution generalisation? If so, an outlook to such a direction could be interesting.", "limitations": "The authors briefly mention that the experiments are rather small scale, but only in the author checklist. It would be appreciated if this is mentioned in the main body of the paper as well. Additionally, the remark on lines 301-302 about generalisation shortcomings could also be highlighted. Finally, the lack of strong guarantees about the quality of the selected/optimised model due to the use of variational inference could also be mentioned, i.e. VI does not guarantee in any way that the variational posterior is in fact invariant or even close to invariance to the learned symmetries. Only experimental evidence can be given in this regard, which luckily shows positive results.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 1, "strengths": "1. The structure and writing of the paper is fairly good and makes for an easy-to-follow story. I appreciate the completeness of the preliminaries as it makes for a self-contained and clear paper. \n\n2. The goal and main research question that the paper tackles is also very clear from the start: How to learn a representation of the Hamiltonian of a dynamical system that also adheres to symmetry, just like real Hamiltonians. The motivation as to why this is important is also clearly illustrated by the experiments, showing how representations without symmetry (regular HNNs) fail to model good Hamiltonians.\n\n3. The evaluation of how well the learned conserved quantities relate to the true underlying symmetries is reasonably well done. Moreover, it does seem to indicate that the proposed method can learn good representations of symmetries despite the use of approximations.", "weaknesses": "1. While the paper is well-structured and self-contained, it is not very clear which parts are background and which parts are the novel contributions. In particular, going through the related work, it seems the only real novelty is the parametrisation of symmetries of H via quadratic conserved quantities (Section 3.1).\n\n2.  The main weakness seems to be the novelty of the paper. Apart from providing a way to parametrise conserved quantities, the rest of the utilised methodology is very close to [1]. To be specific, Section 2 consists of the necessary preliminaries while Section 3.2 explains a well-known procedure to approximately symmetrise a non-invariant function [1]. Section 3.1 is surely novel, but it could be made clearer why quadratic conserved quantities were chosen, i.e. because they lead to analytical flows that are necessary in the symmetrisation. Finally, most of Section 4 and the corresponding appendices are essentially equivalent to the derivations given in Sections 3.6 and 3.7 of [1]. Even the same notation is (wrongly) used in the proof given in Appendix A.1: $E_{\\tau} = E_{\\tau}\\sim E_{\\prod_{s = 1}^S p(\\tau)}$, which should probably be $E_{\\tau} = E_{\\prod_{s = 1}^S p(\\tau)}$.\n\n3. The proposed metrics to measure how similar the learned symmetries are to the ground truth symmetries make sense, but I have some doubts about the \"parallelness\" metric. Specifically, the use of the Euclidean norm can give a skewed perspective as it is not an ideal metric for higher-dimensional vectors [2]. Are there no established methods to measure closeness to symmetries? If not, then the proposed metrics are another contribution. If yes, then please cite the necessary literature.\n\n4. The experimental evaluation is lacking variability metrics in general, making it very hard to grasp whether the differences are actually statistically significant and meaningful. Please include either standard error, standard deviation or at least 25-75 percent quantiles over a repeated set of independent runs. Secondly, I wonder why there aren't more baselines for the experiments, especially since the authors mention a couple of competitors [3, 4]. If they are not applicable, then this should be made more explicit as it can also help strengthen the proposed method.\n\n5. I do not agree that the use of approximate Bayesian inference in the context of model selection for deep learning has not been studied/scaled to neural networks as stated in the conclusion (lines 359-360). I would argue that any application of variational inference to Bayesian neural networks (BNN) for example is always a form of model selection as it selects a simpler, yet performant approximation to the BNN within the provided variational family [5, 6].\n\n[1] van der Ouderaa, Tycho FA, and Mark van der Wilk. \"Learning invariant weights in neural networks.\" Uncertainty in Artificial Intelligence. PMLR, 2022.\n\n[2] Aggarwal, C. C., Hinneburg, A., & Keim, D. A. (2001). On the surprising behavior of distance metrics in high dimensional space. In Database theory—ICDT 2001: 8th international conference London, UK, January 4–6, 2001 proceedings 8 (pp. 420-434). Springer Berlin Heidelberg.\n\n[3] Immer, A., van der Ouderaa, T., Rätsch, G., Fortuin, V., & van der Wilk, M. (2022). Invariance learning in deep neural networks with differentiable laplace approximations. Advances in Neural Information Processing Systems, 35, 12449-12463.\n\n[4] Bondesan, R., & Lamacraft, A. (2019). Learning symmetries of classical integrable systems. arXiv preprint arXiv:1906.04645.\n\n[5] Chen, L., Tao, C., Zhang, R., Henao, R., & Duke, L. C. (2018, July). Variational inference and model selection with generalized evidence bounds. In International conference on machine learning (pp. 893-902). PMLR.\n\n[6] Graves, A. (2011). Practical variational inference for neural networks. Advances in neural information processing systems, 24.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback and help to improve the paper. We thank the reviewer for finding the paper well-structured and self-contained with a clear goal. The reviewer appreciated the empirical validation and experiments that demonstrate improved generalization of the proposed approach.\n\n> Q1. Novelty\n\nFirstly, most works on symmetry learning focus on relatively simple groups on static domains (e.g. affine invariances on images), whereas we tackle the more novel and open problem of learning symmetries in dynamical systems using train data. The Noether's theorem parameterization of symmetries as conserved quantities in Sec. 3.1 is indeed an important contribution to achieve this. Further, we took considerable steps to actually scale model selection with VI to DNNs, through non-mean-field posteriors and optimized prior/output variances (details in App. D). This differs greatly from typical mean-field and fixed variances commonly used in Bayesian NNs, which have not been shown to capable of performing model selection. Most importantly, we demonstrate successfully learning symmetries of dynamical systems, which are demonstrably correct, through conservation laws in dynamical systems. We deem this a very novel contribution.\n\n> Q2.a. Metrics\n\nWe thank the reviewer for pointing out potential nuances in measuring the learned symmetries. First, we’d like to point out that due to the assumption of quadratic conserved quantities, their dimensionality is only roughly quadratic in the dimensionality of the phase space, which is $(2 \\times 5)^2=100$ for the 5-harmonic oscillator, and less for the other experiments. Thus the spaces are not very high dimensional.\nThere is no standard way of evaluating learned symmetries. Our symmetry correctness measurement reduces to: measure the degree by which two linear subspaces, each generated by non-independent basis vectors, differ. The approach we proposed seemed like a simple way of measuring this, but welcome suggestions for alternatives.\n\n> Q2.b. Baselines\n\nMost recent work on Bayesian model selection that scales to deep learning has relied on the KFAC-Laplace approximation. Laplace is often easier than VI because the posterior covariance does not need to be optimized but can be obtained by estimating the local curvature through gradients. In our particular case, however, Laplace can not be used straightforwardly (we tried this first!). The technical reason for this is that KFAC requires gradients through our forward-pass which in our model already required gradients through the symplectic form. The required 2nd order gradients are not readily supported efficiently in most deep learning framework, not even in JAX/diffrax which is tailored for higher-order derivatives and differentiable solvers. We did take inspiration from Laplace-based approaches in our use of Kronecker-structured posteriors and optimizing variances (see App. D). We thank reviewers by bringing this up and will include this in the main text.\n\n> Q3. Seeds\n\nWe will run all experiments with multiple seeds and provide mean and standard error in the final version.\n\n> Q4. Generalization\n\nIn principle, generalisation properties are determined by the prior over functions (which includes choices in the neural network architecture). To obtain better performance regions far away from the data we therefore either need data in these regions, or improve the prior over Hamiltonians. This work provides important insights in automatic model selection to DNNs, and therefore provides a pathway to making this process automatic, without the need for retraining and manual cross-validation.\n\n> Limitations\n\nThank you for your suggestions, we will make sure these limitations are more clearly highlighted in the final version of the paper.\n\n> First successful model selection in DNNs\n\nWe follow what is common in Bayesian statistics and refer to the 'model' as the joint distribution (likelihood + prior). Hence, the variational parameters (\\mu, \\Sigma) are not considered to be part of the model and we therefore do not consider this to be a form of model selection. The hyperparameter \\eta parameterizes symmetries/conserved quantities and is part of the model, impacting the effective prior over Hamiltonians H. We show that we can successfully optimize \\eta with the ELBO, which demonstrates model selection in DNNs. Although VI can indeed be scaled to DNNs for uncertainty estimates [1], its use for model selection has not been successfully demonstrated yet, as also evidenced by the following citation from  [1]: “Empirically we found optimising the parameters of a prior (by taking derivatives of Eq.1) to not be useful, and yield worse results.”. We took considerable efforts to improve the tightness of the lower bound to make model selection work, using non-mean-field posterior and optimized variances (see App. D for details). So far, we are not aware of other works that have managed to do model selection in DNNs with VI, and therefore consider this to be significant. We hope this clarifies our claim, and will make sure that sufficient context is provided in the final version.\n\n\n[1] Blundell, et al. \"Weight uncertainty in neural network.\" ICML 2025", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers for the feedback and help to improve the paper. We are excited to have received a "9: Very Strong Accept", and are confident that the concerns raised by the lower score reviews are sufficiently addressed in this rebuttal. Overall, most reviewers found the paper very 'clearly written', 'principled', 'easy-to-follow' and 'highly relevant to both the AI + Science and geometric machine learning communities'. In particular, we are proud that reviewers found the paper 'intriguing and conceptually interesting’, and were positive about the empirical validation and deemed the results promising.

Most raised concerns are addressed in direct rebuttals. Since several reviewers raised questions about the use of the quadratic form, we also provide a more thorough discussion on this issue in this overall rebuttal:

> Quadratic form

We indeed consider learnable conserved quantities of the quadratic form. Regardless of the quadratic form, our model can in principle still learn any dynamical system, also those with non-quadratic symmetries, given a sufficiently flexible network. This is already a direct improvement over models without learnable invariance properties, such as regular HNNs, while also capturing commonly encountered symmetries that are currently hard-wired into architectures.

Regarding the quadratic form as a type of learnable symmetry, we feel that this is not an overly strong assumption. Quadratic conserved quantities can represent any symmetry that has an affine (linear+translation) action on the state space. This includes essentially all cases that are currently studied in geometric deep learning. Note that the quadratic constraint does not limit the shapes of the symmetry groups that we learn. As an example, in our experiments we find the Euclidean group, which is itself a curved manifold with non-trivial topology. Hence, we find this quadratic assumption to be not overly strong. However, we welcome suggestions for common examples in deep learning with equi/invariance to groups with a non-affine action, so that we can list these examples as limitations of our method.

The technical reason why quadratic conserved quantities can represent complex groups, is that the learned conserved quantities are a basis of the Lie algebra of the Lie group of symmetries, with the Poisson bracket being identical to the Lie bracket of the Lie algebra. The structure of the Poisson bracket between the conserved quantities determines the shape of the group, and this is not constrained by the conserved quantities being quadratic.

If one wishes, we can extend our framework to non-quadratic conserved quantities to model non-affine actions. This has the downside that the symmetrization needs to be done with an ODE-solving, as in Eq. (3), instead of with the matrix exponential. This slows down the implementation by a large amount. Furthermore, such free-form conserved quantities have many more parameters, which complicates the Bayesian model selection. We did implement this and will include it in the code that will be released upon acceptance. We have omitted the details on free-form conserved quantities from the present manuscript for space reasons, but will elaborate on it in the appendix of the final version.

Meta-review: This paper presents an innovative approach that integrates Noether’s theorem with Bayesian model selection to learn conserved quantities directly from data. This method enables the automatic discovery of symmetries in dynamical systems, which is both novel and impactful. I recommend accepting the paper for the following reasons. The primary strength of this work lies in its successful combination of Noether’s theorem with variational inference to learn conserved quantities, which, in turn, defines the symmetries of the system. This approach addresses a significant challenge in machine learning and physics-informed models, where incorporating symmetries can greatly enhance model performance and generalization. The paper's method is particularly powerful because it avoids the need for manually designed symmetries or additional regularizers, instead relying on the natural outcome of the Bayesian model selection process. The authors demonstrate the effectiveness of their approach through empirical results on classical systems like harmonic oscillators and n-body problems, where the learned symmetries match the ground truth, further validating the method. The paper is also well-written, clearly explaining the methodology and its implications for both AI and physics communities. The authors have effectively addressed reviewer concerns, particularly those related to the choice of quadratic forms and the scalability of their approach to more complex, high-dimensional systems. Their response to these concerns, including additional details in the appendix, strengthens the overall contribution. While some concerns were raised about the computational complexity and the specific assumptions made (e.g., quadratic forms for conserved quantities), these do not significantly detract from the overall quality of the work. The authors have provided sufficient justification and context, making a strong case for the utility and novelty of their approach. Given its strong theoretical foundation, practical relevance, and positive empirical results, this paper is a valuable contribution to the field and should be accepted.
Decision: Accept (poster)