Noether Networks: meta-learning useful conserved quantities
Authors: Ferran Alet
Dylan Doblar
Allan Zhou
Joshua B. Tenenbaum
Kenji Kawaguchi
Chelsea Finn
Venue: NeurIPS 2021 Poster (2021)
Keywords: meta-learning
sequential prediction
physics
video prediction

Abstract: Progress in machine learning (ML) stems from a combination of data availability, computational resources, and an appropriate encoding of inductive biases. Useful biases often exploit symmetries in the prediction problem, such as convolutional networks relying on translation equivariance. Automatically discovering these useful symmetries holds the potential to greatly improve the performance of ML systems, but still remains a challenge. In this work, we focus on sequential prediction problems and take inspiration from Noether's theorem to reduce the problem of finding inductive biases to meta-learning useful conserved quantities. We propose Noether Networks: a new type of architecture where a meta-learned conservation loss is optimized inside the prediction function. We show, theoretically and experimentally, that Noether Networks improve prediction quality, providing a general framework for discovering inductive biases in sequential problems.

Noether Networks: Meta-Learning Useful Conserved Quantities
Ferran Alet"!, Dylan Doblar" Allan Zhou², Joshua Tenenbaum', Kenji Kawaguchi", Chelsea Finn² MIT, 2tannfrd University, NNational University of Singapore {alet, ddoolar))mmmiidi tmit. edu
Abstract
Progress in machine learning (ML) stems from a combination of data availability, computational resources, and an appropriate encoding of inductive biases. Useful biases often exploit symmetries in the prediction problem, such as convolutional networks relying translation equivariance. Automatically discovering these useful symmetries holds the potential to greatly improve the performance of ML systems, but still remains challenge. In this work, focus sequential prediction problems and take inspiration from Noether's theorem to reduce the problem of finding inductive biases to meta-learning useful conserved quantities. We propose Noether Networks: a new type of architecture where a meta-learned conservation loss is optimized inside the prediction function. We show, theoretically and experimentally, that Noether Networks improve prediction quality, providing a general framework for discovering inductive biases in sequential problems.
Introduction
The clever use of inductive biases exploit symmetries has been the heart of many landmark achievements in machine learning, such translation invariance in CNN image classification [35], permutation invariance in Graph Neural Networks [52] for drug design [57], or roto-translational equivariance in SE3-transformers [25] for protein structure prediction [33]. However, for data distributions of interest, there may be exploitable symmetries that are either unknown or difficult to describe with code. Progress has been made in automatically discovering symmetries for finite groups [74], but meta-learning and exploiting general continuous symmetries has presented major challenge. part, this is because symmetries describe the effect of counterfactuals about perturbations to the data, which are not directly observable.
In this work, we propose to exploit symmetries in sequential prediction problems indirectly. We take inspiration from Noether's theorem [44], which loosely states the following:
For every continuous symmetry property of dynamical system, there is corresponding quantity whose value is conserved in time.
For example, consider system of planets interacting via gravity: this system is translation invariant in all three cardinal directions (i.e. translating the entire system in the x.y, axis conserves the laws of motion). Noether's 's theorem asserts there must be a conserved quantity for each of these symmetries; in this case, linear momentum. Similarly, the system has a time- invariance (i.c. the laws of motion are the same today as they will be tomorrow). In this case, the corresponding conserved quantity is the total energy of the system.
Equul contribution. Our code is publicly available at https://ls..caal..mi http://ii.caill.mm...c/hhhhhre edu/nocther.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

farcow
0222
ET
Final Prediction
Naccher (2p, Firr7 Ge)
Taller predictive model Ja to iiput in enforce consevaation encoded by
Iei (20)
Ground truth
Io (i1)
y T22
9o (TT)
Input
TT
Initial Prediction
Figure I: Noether Networks enforce conservation laws, meta-learned by 9é. in sequential predictions made by fe. which is tailored the input To to produce final predictions $1.7. Imposing these meta-learned inductive biases improves video prediction quality with objects sliding down ramp.
Inspired by this equivalence, we propose that approximate conservation laws are a powerful paradigm for meta-learning useful inductive biases in sequential prediction problems. Whereas symmetries difficult to discover because they are global properties linked to counterfactuals about unobserved perturbations of the data, conserved quantities can be directly observed in the true data. This provides an immediate signal for machine learning algorithms to exploit.
Our approach involves meta-learning a parametric conservation loss function which is useful to the prediction task. We leverage the tailoring framework [4], which proposes to encode inductive biases by fine-tuning neural networks with hand-designed unsupervised losses inside the prediction function. Whereas traditional auxiliary losses are added to the task loss during training, tailoring losses are optimized inside the prediction function both during training and testing, customizing the model to each individual query. In doing so, we ensure there no generalization gap for the conservation loss. We propose to meta-learn the self-supervised loss function and parameterize it in the form of conservation loss; i.e. L(an, ē::T; 9o) Σ (g I (ā.))) This conservation form encodes a meta-inductive bias (inductive bias over inductive biases) which narrows the search space exponentially (in T) and simplifies the parameterization. Figure demonstrates the Noether Network pipeline for a video prediction task.
The main contribution of this paper is the Noether Network, architecture class and algorithm that can automatically learn its own inductive biases in the form meta-learned conservation loss func- tions. We theoretically characterize the advantages of such conservation laws effective regularizers that constrain the learning to power-dimensions output manifolds, lowering the generalization gap. Empirically, first find that, when the meta- neta-learned conservation loss takes the form of synthesized program, Noether Networks recover known conservation laws from raw physical data. Second, we videos, modestly improving the
find that Noether Networks learn conservation losses generalization of a video prediction model, especially on longer-term predictions.
Theoretical Advantages of Enforcing Conservation Laws
In this section, we demonstrate principled advantages of enforcing conservation laws of the form {s(fo(x)) 9a (xx) by considering special case where preimages under dø form affine subspaces.
Let input and target be v,y E Rd, and let the Noether embedding be Ja IRd + P where P (x:)
E Rdd. We consider restricted class of models, parameterized by 0 E θ, of the form fz x + for Ua E gd such that for all the preimage of g6 is g Kgs (x) (or + Az z E R." ). Rdxm, Here, m S d is the dimensionality of the preimages gø- We denote by C' the smallest upper bound on the loss value as L(fo(T)))) S C' (for all xr y and 0)

---

Define v(()= Ez,y [f v), - [zf v), vs), with Lipschitz constant Ç. Therefore,
Ç 2 is the smallest real number such that, for all v and v' in V. (v(())) ((())+((')) S - 22 where
= {vo E IRd E @). Finally, we define R = supace |10l12. and the generalization gap by
G(0) (2x,(((((())))))
Σ (Lo(()))))))
Theorem shows that enforcing conservation laws of go go (J)())) = go(x) is advantageous when the dimension of the preimages under is less than the dimension of x; that is, when m < d.
Theorem 1. Let E N+. Then, for any d > O, with probability at least 1 å over an iid draw of examples ((zi, ys));- the following holds for all 8 E O:
{ EIn(max(VE. 1)) (l G(0) = CV
(2()(())/)) In(1/5) + 1 1)
2210 n
(1)
2n
where & = m if go = go(x) for any x E Re and 0 E 9, and & = d otherwise.
The proof is presented in Appendix A. In Theorem I, when we enforce the conservation laws, d is replaced by m, the dimension of the preimage. We now discuss various cases for the values of m: (Case of m d) Let consider an extreme scenario where the function Is maps all R to one single point. In this scenario, the dimensionality of preimages under go is maximized d. Accordingly, the bounds with and without enforcing the conservation laws become the same. Indeed, in this scenario, the conservation laws of go (x)) = (x(x) give us no information, because they always hold for all x E Rd even without imposing them.
(Case of m 0) Let us consider another extreme scenario where the function 9o is invertible. In this scenario, the dimensionality of preimages under Is is zero as m = O. Thus, imposing the condition of go (f(((=)) - (x(x) makes the bound in Theorem to be very small. Indeed, in this scenario, the condition of 9o (e())()) (xx implies that fa( = x: i.e., x is not moving, and thus it is easy generalize.
(Case of 0 < < d) From the above two cases, we can see that the benefit of enforcing conservation laws comes from more practical cases in- betweewwen these, with m d.
In Theorem 1, the function I& can differ from the true function 9 underlying the system. This is because we analyze a standard generalization gap: i.e., the difference between the expected loss and the training loss. The cost of not using the true 9 is captured in the training loss; i.e., the training loss can be large with the function Je that differs significantly from the true g Even in this case, the generalization gap can be small. For example, in the case of m = O, the generalization bound is small, whereas the training loss will be large unless Iz-1 = Fz. Therefore, our analysis gives the insight on the trade-off between the training loss and the dimensionality of preimages under Jg.
3 Noether Networks
Leveraging tailoring to encode inductive biases. We perform a prediction-time optimization to encourage outputs to follow conservation laws using the tailoring framework [4]. Tailoring encodes inductive biases in the form of unsupervised losses optimized inside the prediction function. In doing so, tailoring fine-tunes the model to each query to ensure that minimizes the unsupervised loss for that query. For example, we may optimize for energy conservation in physics prediction problem. In meta-tailoring, we train the model to do well on the task loss after the tailoring step has fine-tuned its parameters. In contrast to auxiliary losses, which would optimise the conservation loss only for the training points, tailoring allows us to ensure conservation at test time. Since we aim to build in the conservation in the architecture, advantage of tailoring losses is that they are casier to meta-learn. Auxiliary losses are pooled over all examples and training epochs and their effect is only known at validation/test time. We would need to use implicit gradients [39, 47] to know their eventual effect on the final weights at the end of training. With tailoring, we can directly measure the effect of the meta-learned update the same sample.
want to ensure it is also satisfied for unseen test samples. Another
limitation of tailoring framework is that the tailoring loss must be user-specified. This is acceptable in domains where the desired inductive bias is both known and easily encoded, but problematic in general - we address this issue with Noether Networks. Our approach can be seen as a generalization of tailoring where the unsupervised loss is meta-learned and takes the form of a conservation loss.
3

---

Noether Networks: discovering useful conservation laws. We build on the paradigm of conser- vation laws; i.c. quantities that are conserved over time. This has the following two challenges:
1. Real data often has noise, breaking any exact conservation law. Moreover, many conser- vation laws are only approximately satisfied given only partial information of the system. For example, conservation of energy is not fully satisfied in real dissipative systems and estimating conservation of mass from pixels is bound be inexact in the case of occlusions.
2. Optimizing for conservation alone can lead to trivial quantities, such as predicting constant value go go(z) C' independent of z.
We search for useful conservation laws, whose (approximate) enforcing brings us closer to the true data manifold for the current sequence. Note that useful conserved quantity doesn't need to be exactly conserved in the training data to improve prediction performance. We only need its conservation to be informative of the data manifold. This allows to discover conservation laws even in imperfect, noisy environments. Conversely, tailoring with a trivial conserved quantity cannot improve the final predictions (see App. B for detailed explanation). Finally, viewing conserved quantities as useful inductive biases aligns well with their use, since we often care about biases only insofar they help improve task performance.
The search for such useful conserved quantities can take many forms. In this work, we focus on two: first, in the beginning of section 4.1, we use combination of program synthesis and gradient descent to generate a large, but finite, number of candidate parametric formulas for physical system prediction. We then try meta-tailoring with each formula on the training set and pick the conservation formula with the best loss. Formulas can be a useful, interpretable description in scientific domains given the appropriate state-space. However, for general descriptions from raw data, we would like to describe the loss with a neural network, as we describe in the next section.
Meta-learning a neural loss function. We propose Noether Networks, an architecture class for sequential prediction that consists of a base predictor ʃa ã + T and a meta-learned tailoring loss, parameterized as the conservation of neural embedding ds 3t - R This embedding takes raw predictions as input (such as images in the case of video prediction). The conservation form induces meta-inductive bias over potential learned tailoring losses. The Noether loss is formulated as
LNoothorr (ro, (1:Ti9o)=
go(())) - go (5+)
1g (0r- 9o (a.))
(a)
(b)
where zo is the ground-truth input, the ã. f0( the model's predictions, and ão & zo- Expressions 2(a) and 2(b) are equivalent if we fully enforce the conservation law, but they differ if conservation is not fully enforced. When not fully conserving, 2(a) propagates information from ground truth more directly to the prediction, but 2(b) may be a useful approximation which better handles imperfectly conserved quantities, where the quantity should be Lipschitz but not exactly constant. In both cases, if we tailor 0 with a single gradient step; the gradient update takes the form
0(xo: o) = Ain VoLNoether (2o, it ē1.T (o)r(0)))))))
(3)
We compute final predictions as 3o ( frrr) back to o, which will be optimized so that the unsupervised adaptation 0 + O(0n d) helps lower Ltask.The The optimization requires second-order gradients to express how & affects Lrnsk through (x Φ). This is similar to MAML [23], as well as works meta-learning loss func- tions for few-shot learning [5] and group distribution shift [71]. Algorithm 1 provides pseudo-code.
(.+-1). (.t=)) We can now backpropagate from
For deep learning frameworks that allow per-example weights, such as JAX [9], the loop over sequences in Alg. 1 can be efficiently parallelized. To parallelize it for other frameworks we use the CNGRAD algorithm [4], which adapts only the Conditional Normalization (CN) layers in the inner loop. Similar to BN layers, CN only performs element- wise affine transformations: y = C Y + B, which can be efficiently parallelized in most deep learning frameworks even with per-example y,
Finally, even though we use the two-layer optimization typical of meta-learning, we are still in the classical single-task iigle-distribution setting. Noether Networks learn to impose their own inductive biases via learned loss and to leverage it via an adaptation of its parameters 0. This is useful as we often do not have access to meta- petatpaatitions that distinguish data between distributions or tasks.

---

Algorithm Prediction and training procedures for Noether Networks with neural conservation loss
Given: predictive model class f: embedding model class prediction horizon T; training dist. Durain: batch size N; learning rates Ama Aout, dembi task loss Ltaski Noether loss Noethhr
1: procedure PREDICTTSFFURNNCFCECC
2:
To, f- ID, F0
3:
It + fe(I vt E fI....,
b Initial predictions Inner step with Noether loss > Final prediction with tailored weights
4: O(zoi o) t Am VoCNoet 2 (1:1ig0)
5:
it t (a( (i Vi E (.,
6:
return F1TT
7:
procedure TRAIN
8:
Φ -aanomlyy initialized weights
Initialize weights for Noether embedding b Initialize weights for predictive model f
9:
<- randomly initialized weights while not done do
10:
11:
Sample batch 0: ................................ O.T Drrain for 0 n N do
(N)
1.T PREDICTSEQUENCHE"" 0)
14:
Φ t Aemb V En=u ( 1.7,, (Ir)
Outer step with task loss for embedding
15:
- 0 dout Vo En Crask 1,7, (17951177
Outer step for predictive model
16:
return o,
Experiments
Our experiments are designed to answer the following questions:
1. Can Noether Networks recover known conservation laws in scientific data?
2. Are Noether Networks useful for settings with controlled dynamics?
3. Can Noether Networks parameterize useful conserved quantities from pixel information?
4. How does the degree of conservation affect performance the prediction task?
4.1 Experimental domains
To answer the above questions, we consider a number of different experimental domains, which we overview in this subsection, before moving on to the evaluation and results.
Spring and pendulum from state coordinates. Greydanus et al. [28) propose the setting of an ideal spring and ideal pendulum, which will allow to understand the behavior of Noether Networks for scientific data where we know useful conserved quantity: the energy. They also provide data from a real pendulum from [53]. contrast to the ideal setups, here the data is noisy and the Hamiltonian is only approximately conserved, the system is dissipative. For the two pendulum, input r (p.4) E R² contains its angle q and momentum p. Given the parameters in [28] the energy + p", with p² 3 cos being simpler equivalent. For the spring, input I = (p. 4) E IR2 contains the displacement q and momentum and the energy H. = (9 + +P))
is H = 3(1
Thus, 2² + 1.0 p2 is a conserved quantity, where the coefficient 1.0 has appropriate units.
We build on their vanilla MLP baseline and discover conservation laws that, when used for meta- tailoring, improve predictions. Since the baseline predicts d. (ax)) rather than Ft+1, we apply the loss di to finite-difference approximation, i.e. LNocther (Ne, Fy. + da A.) = LNothhe (It, Be + fo(x))Ar).
di
Domain specific language for scientific formulas. To search over Hamiltonians, we program simple domain specific language (DSL) for physical formulas. Since formulas in the DSL have physical meaning, each sub-formula carries its own associated physical units and is checked for validity. This allows to significantly prune the exponential space, as done in Al-Feynman [63]. The vocabulary of the DSL the following: Input (i): returns the i-th input, Operation: one of (+, -, / sin, cos, x²) Parameter (u): trainable scalar with units [u].

---

Table I: Recovering H. for the ideal pendulum.
Table Recovering H. for the ideal spring.
Method
Description
RMSE
Method
Description
RMSE
Vanilla MLP
N/A
0.0563
Vanilla MLP
N/A
.0174
Noether Nets
p² 2.99 cos(4)
0.0423
Noether Nets
q² 1.002 p2
.0165
True H. [oracle]
p² 3.00 cg)
0.0422
True H [oracle]
q² 1.000 p²
.0166
Figure 2: Noether Networks can recover
MSE between coordinates
Change HNN-conserved quantity
the energy of real pendulum, even
0.020
Jased NN Voeefar ar Renwak
0.0
though it is not fully conserved. This
-0.1
is because they only look for quantities
0.015
0.2
Ground fruih
whose conservation helps improve pre- dictions. Moreover, by only softly en- couraging conservation, it better encodes imperfect conservations.
0.010
-0.3
Naether hetwarr
Draci too
0.005
-0.4
0.000
10 Prediction horizon
Prediction horizon
Pixel pendulum with controls. In settings such robotics we may be interested in action- conditioned video prediction [45]: predicting future frames given previous frames and an agent's s actions. We recorded videos of episodes in OpenAl Gym's pendulum swing up environment, using scripted policy. Each model receives four history frames and a sequence of 26 policy actions starting from the current timestep, and must predict 26 future frames. As this is visually simple environment we limit the training dataset to 5 episodes of 200 frames each, holding out 195 episodes for testing.
Video prediction with real-world data. To characterize the benefits provided by Noether Net- works in real-world video prediction setting. we evaluate the effect of adding a Noether conservation loss to the stochastic video generation (SG model [18] on the ramp scenario of the Physics 101 dataset [69]. The dataset contains videos of various real-world objects sliding down an incline and colliding with a second object. Each model is conditioned on the first two frames and must predict the subsequent 20 frames. To increase difficulty, we restrict our data to the 20-degree incline.
4.2 Evaluation
Can Noether Networks recover known conservation laws in scientific data? We first generate all valid formulas up to depth 7, removing operations between incompatible physical units and formu- las equivalent up to commutativity. We obtain 41, 11,460 and 72,372 candidate conserved formulas for the pendulum and spring, respectively, which need not have units of energy. These formulas are purely symbolic, with some trainable parameters still to be defined. We thus optimize their parameters via gra- dient descent to have low variance within sequences of true data. We then do the same for random se- quences; if conservation is two orders of magnitude smaller for the true data, accept it an approx- imate conservation. This measure is similar to others used before [38], but is not sufficient when the data is noisy, the minimisation is sub-optimal, or there are numerical issues. As result, obtain 210 and 219 candidates for approximately conserved quantities for the pendulum and spring, respectively.
starting from the pre-trained vanilla MLP, we fine-tune it for 100 epochs using meta-tailoring, with one inner step and a range of inner learning rates 10* for k E (-3, -2.5, -2.,...., .., 1}. We then evaluate the fine-tuned model on long-term predictions, keeping the expression with the best MSE loss in the training data. We observe that this process correctly singles out equations of the same form as the true Hamiltonian H. with almost the exact parameters. Using these as losses reaches equivalent performance with that of the oracle, which uses the true formula (see Tables 1 and 2).
Finally, for each potentially conserved quantity, we try it meta-tailoring loss:
Finally, we run the same process for real pendulum [53], where energy is not conserved. We use largely the same pipeline as for the ideal pendulum, the differences are explained in Appendix C. Noether Networks discover p² 2.39 cos(g), close to the (potentially sub-optimal) H = p2 2.4 cos(g) described in [28]. This Noether Network improves the baseline by more than one order of magnitude, matches the performance of hand-coding the conservation loss, and improves over Hamil- tonian Neural Networks, which fully impose conservation of energy. Results can be seen in figure 2.

---

Figure 3: In controlled pendulum environment, the Noether Network has lower mean squared error (left) and slightly better structural similar- ity (right). Metrics are computed and plotted by 1 prediction timestep, where timestep 0 is given history frame. Note that simply training SVG for more steps does not increase performance.
MSE
SSIM
Souther Setwers
3.55
Prediction horizon
Prodiction horizon
Test LPIPS
Test MSE
Test PSNR
Test SSIM
In-4
Roacher Verwork: step! buseire
0.003
33
0.22
0.963
0.975
Prediction horizom
Prediction haricori
Prediction Fronizum
Predik
horizum
Figure 4: The Noether Network outperforms the baseline by a small margin in all four metrics in real- world video prediction, showing they can meta-learn useful conserved quantities from raw pixel data.
Are Noether Networks useful in settings with controlled dynamics? For the controlled pendulum experiments, begin with an SVG model but modify it (1) remove the prior sampling component since the environment is deterministic and (2) concatenate each action to the corresponding frame encoder output and feed the result to the LSTM. After training the SVG model for 50 epochs and fixing its weights, we run Algorithm for 20 epochs to learn an embedding network for conserving useful quantities. We found that Noether Networks perform better in this setting by directly adapting the latent LSTM activations against the conservation loss, rather than adapting any network parameters. Since Noether Networks take additional training steps starting from a pre-trained SVG model, we also continued training the SVG model for 20 further epochs, which we call "SVG (more steps)." Figure shows the results of each method on held out sequences. Noether Networks improve the overall mean squared error (MSH) and structural similarity (SSIM) over the base SVG.
Can Noether Networks parameterize useful conserved quantities from raw pixel information? We train on only 311 training sequences, which makes generalization difficult. In this setting, the base SVG model struggles with overfitting - novel test objects often morph into objects seen at train-time, and the object in motion often morphs with the target object (as shown in figure 1). Our Noether Network uses the inner loss formulation of Equation 2(b), where Io is a two-layer CNN receiving two consecutive frames followed by a fully-connected projection layer producing 64-dimensional embed- dings. We meta-tailor the embedding and the base model for 400 epochs. As seen in figure 4, taking a single inner tailoring step improves performance slightly over the baseline SVG model with respect to all four considered metrics: learned perceptual image patch similarity (LPIPS) [72], mean squared error (MSE), peak signal-to-nin -o-isee ratio (PSNR), and structural similarity (SSIM). As the prediction horizon increases, the Noether Network performs comparatively better, likely because conservations encourage the model to retain aspects of the ground-truth input. These results provide some evidence that Noether Networks are capable of learning useful inductive biases from raw video data.
To investigate whether the Noether embeddings learn relevant features, we use Gradient- Class Activation Mapping (Grad-CAM) [54] to compute importance maps for sequences in the test set. Since the Noether embedding has 64 dimensions, we perform principal component analysis (PCA) before Grad-CAt Grad-CAM to reduce the dimensionality of the embedding, and to sort dimensions by the percentage of variance they explain in the test set frames. Interestingly, we find that the first PCA dimension captures 83.6% of the variance, and the first four dimensions capture 99.9% of the variance.
Figure 5 shows Grad-CAM localization maps for two example test-set sequences, (a) depicts orange torus and (b) shows a blue brick, where warmer colors (red) indicate high importance and cooler colors (blue) indicate low importance. Our interpretations of Grad-CAN localization maps are consistent across examples, see Appendix F for additional ones. The first dimension, which explains the vast majority of the variance, primarily focuses on the sliding object in both examples. It also attends the object on the table, and to the edge of the ramp. The attention to the objects suggests that the Noether Network learns to conserve quantities related the objects' pixels and their motion (since it takes
7

---

(a) Orange object
(b) Blue object
Prediction horizon
Embedding dimension (after PCA)
Embedding dimension (after PCA)
Figure 5: Grad-CAM localization maps show that the Noether embeddings attend to relevant frame regions for two example sequences. Heatmaps for the first six dimensions of the embedding (in the PCA basis, sorted by decreasing amount of variance explained) are shown for three time steps.
two consecutive frames). We hypothesize that the attention to the table edge encodes information about the orientation of the sequence, since half of the training sequences are randomly flipped horizontally. While some the dimensions focus nearly uniformly on the entire frame, the fourth dimension focuses the orange sliding object in (a), and the human hand and table in (b). In all the test set examples, whenever the human hand is in the frame, is attended to by this dimension. We hypothesize that it learns present, and possibly picks up on similarly-colore Finally, the sixth dimension seems to track blue sliding objects.
Inner loss Inner step
200
150
123
100
25
conserve the hand in sequences where it is pixels as well.
step
J--7 Outer loss Vs. Inner step
How does the degree of conservation affect performance? All of the results presented in this work were achieved by Noether Networks trained and evaluated with single inner step. To char- acterize how the degree to which the conservations are imposed affects video prediction performance, we optimize the inner (Noether) loss for many inner steps and measure the outer (task) loss, shown in figure 6. Here, the inner loss is optimized by Adam for 500 steps, as opposed to the single step of SGD during training (both settings optimization, the outer loss improves for roughly 150 inner steps, which suggests that the approximate conservations learned by the Noether Network generalize to more exact conservation settings.
1.12
1.10
1.00
1.04
102
1000
100
200
200
400
530
Innor step
learning rate of 10 4). During the
Figure 6: Despite training with only one inner step, as lower the Noether loss to zero (top), the outer loss remains low (bottom).
5 Related Work
Unsupervised adaptation and meta-learning loss functions. Hand-designed unsupervised losses have been used to adapt to distribution shifts with rotation-prediction [59] or entropy- minimization [65], as well to encode inductive biases [4]. Unsupervised losses have also been meta- learned for learning to encode human demonstrations [70]. few-shot learning exploiting unsupervised information [41, 5], and learning to adapt to group distribution shifts [71]. In contrast, our unsuper- vised loss only takes the single query we care about, thus imposing no additional assumptions on top of standard prediction problems, and takes the form of a conservation law for predicted sequences.
Encoding of symmetries and physics-based based inductive biases in neural networks. We propose to encode symmetries in dynamical systems as meta-lcarned conserved quantities. This falls under the umbrella of geometric deep learning, which aims to encode symmetries in neural networks; Bronstein et al. [12] provide an excellent review of the field. Most relevant to the types of problems we focus on tronsaaaalaai equivariance [62. 66, 25, 51], applications of GNNs in physical settings [13, 6, 34, 3, 2, 49, 46, 50, 17] and the encoding of Lime-invariance in RNNs [22, 32. 30, 14, 60). Recent works have encoded Hamiltonian and Lagrangian mechanics into neural models [28, 40, 15, 24], with gains in data-efficiency in physical and robotics systems, including some modeling controlled or dis- sipative systems [73, 19]. In contrast to these works, we propose to encode biases via prediction-time fine-tuning following the tailoring framework [4], instead of architectural constraints. This allows
8

---

us to leverage the generality of loss functions to encode a class of inductive biases extending beyond Lagrangian mechanics, and handle raw video data. In this regard, Noether networks are closer to energy-based models (EBMs) [1, 42, 29, 37, 20], which also optimize loss function to make predic- tions. However, EBMs directly optimize the prediction, instead of tailoring the weights of a forward model, which can ease the learning problem. In particular, [42] show how EBMs can be a good model to enforce constraints, showing it for predicting two bouncing balls from state information.
Outside mechanics, Suh and Tedrake [58] highlight the difficulty of learning physical inductive biases with deep models from raw data and instead propose to encode mass conservation from pixels with a constrained linear model. Finally, Noether's 's theorem has been used [26, 61] to theoretically understand the optimization dynamics during learning; unrelated our goal of discovering inductive biases for sequential prediction.
Discovery of symmetries and conserved quantities. Noether networks are related to slow feature analysis [68, 43, 7, 27, 31], which aims at extracting the smoothest features from time series in an unsupervised way, to use it for dimensionality reduction, regression, or classification. In contrast to this subfield, Noether networks impose these approximate conservations at prediction-tins to improve predictions. There are other previous works which aim to learn conserved quantities in dynamical systems from data. The approach of Schmidt and Lipson [53] focuses on candidate equations that correctly predict dynamic relationships between parts of the system by measuring the agreement between the ratios of the predicted and observed partial (time) derivatives of various components of a system. A set of analytical expressions is maintained and updated as new candidates achieve sufficient accuracy. More recently, Liu and Tegmark [38] discover conservation laws of Hamiltonian systems by performing local Monte Carlo sampling followed by linear dimensionality estimation to predict the number of conserved quantities from an explained ratio diagram. Wetzel et al. [67] learn Siamese Network that learns to differentiate between trajectories. Contrary to both, Noether Networks do not need segmentations into trajectories with different conserved quantities and can deal with raw pixel inputs and outputs.
Other approaches learning symmetries from data include neuroevolution for learning connectiv- ity patterns [56), regularizers for learning data augmentation [8], and meta-learning equivariance- inducing weight sharing [74]. However, these approaches do not aim to learn conserved quantities in dynamical systems. Finally, the two-step pipeline of symbolic search with unit constraints followed by gradient descent to estimate parameters has many commonalities with approaches for inductive process modeling [36, 11, 10] for scientific discovery. However, Noether networks then leverage the discovered physical equations to improve the predictions of a neural network.
Neural networks to discover physical laws. There has been growing interest in leveraging the functional search power of neural networks for better scientific models. Multiple works train a neural network and then analyze it to get information, such as inferring missing objects via back- proppapttan descent [3], inspecting its gradients to guide a program synthesis approach [63], learning a blue-print but-regularized ancoeenscdders [64]. Others, such as DreamCoder [21], take the explicit approach with a neural guided synthesis over the space of formulas. Unlike these works, our method can readily be applied both to symbolic conservation laws and to neural network conservation losses applied to raw videos.
with GNN [16], leveraging
6 Discussion
Limitations. A disadvantage of our proposed approach over architectural constraints (in the cases when the latter can represent the same bias) is computation time. Noether Networks optimize the conservation loss inside the prediction function, which results in an overhead. This can be reduced by only fine-tuning the higher-level layers, avoiding pack-propagation back to the input. Another approach to reduce training time could be to warm-start the Noether embeddings with pre-trained contrastive embeddings, which have an aligned goal and have shown great results for video and audio classification [48]. It is also sometimes the case that, by encoding the inductive bias during training the trained but un-tailored Noether Network performs better than a comparable architecture without the bias during training. It is also worth noting that the theoretical guarantees apply to fully enforced conservation constraints, while in practice we only fine-tune our model to better satisfy them. However, we believe the theoretical result still conveys the potential benefits of Noether networks.

---

Broader impact. We propose new framework for meta-learning inductive biases in sequential settings, including from raw data. On the positive side, our framework can be used to discover conserved quantities in scientific applications, even when these are not fully conserved. By improving video prediction, we are also empowering predictive models for entities such as robots and self-driving cars, helping elder care and mobility. However, at the same time, it could be used for better predicting the movement and tracking of people from CCTV cameras, affecting privacy. It could also be used to improve the quality of deepfake video generation, which can be used for malicious purposes.
Overview and future work. We propose Noether Networks: a new framework for meta-learning inductive biases through conserved quantities imposed at prediction time. Originally motivated by applications in physics, our results indicate that Noether networks can recover known conservation laws in scientific data and also provide modest gains when applied to video prediction from raw pixel data. The generality of optimizing arbitrary unsupervised losses at prediction time provides an initial step towards broader applications. Finally, this work points at the usefulness of designing meta-learned inductive biases by putting priors on the biases instead of hard-coding biases directly.
Acknowledgements
We want to thank Leslie Pack Kaelbling, Tomás Lozano-Pérez and Maria Bauza for their useful and detailed feedback, as well as Yannic Kilcher, Max Welling, and Joel Zylberberg for insightful comments. We gratefully acknowledge support from GoodAI; from NSF grant 1723381; from AFOSR grant 1-99550-7-1-0116
from ONR grants N00014-18- 1-2847 and N00014-21-1-2682. from the Honda Research Institute, from MIT-IBM Watson Lab; from SUTD Temasek Laboratories; and from Google. Chelsea Finn is fellow in the CIFAR Learning in Machines and Brains program. We also acknowledge the MIT SuperCloud and Lincoln Laboratory Supercomputing Center for providing HPC resources that have contributed to the reported research results. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of our sponsors.
References
[1] D. H. Ackley, E. Hinton, and T. J. Sejnowski. A learning algorithm for boltzmann machines. Cognitive science, 91):147-169, 1985.
[2] F. Alet, A. K. Jeewajee, M. Bauza, A. Rodriguez, T. Lozano-Perez, and L. P. Kaelbling. Graph element networks: adaptive, structured computation and memory. Proceedings of the 36th International Conference on Machine Learning- Searning-Volume 97, 2019.
[3] F. Alet, E. Weng, T. Lozano-Perez, and L. Kaelbling. Neural relational inference with fast modular meta-learning In Advances in Neural Information Processing Systems (NeurIPS) 32, 2019.
[4] F. Alet, M. Bauza, K. Kawaguchi, N. G. Kuru, T. Lozano-Perez, and L. P. Kaelbling. Tailoring: encoding inductive biases by optimizing unsupervised objectives at prediction time. arXiv preprint arXiv:2009. 10623, 2020.
[5] A. Antoniou and A. J. Storkey. Learning to learn by self-critique. In Advances in Neural Information Processing Systems, pages 9940-9950, 2019.
[6] W. Battaglia, R. Pascanu, M. Lai, D. Rezende, and K. Kavukcuoglu. Interaction networks for learning about objects, relations and physics. arXiv preprint arXi::1612.00222. 2016.
[7] Y. Bengio and J. Bergstra. Slow, decorrelated features for pretraining complex cell-like networks. Advances in neural information processing systems, 22:99-107, 2009.
[8] G. Benton, M. Finzi, Izmailov, and A. G. Wilson. Learning invariances in neural networks. arXiv preprint arXiv:2010.1188) 2020.
[9] J. Bradbury, R. Frostig, P. Hawkins, Johnson, C. Leary, D. Maclaurin, G. Necula, A. Paszke,
J. VanderPlas, S. Wanderman-Milne and Q. Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL http:///gthhubb.cooooogglee//
10

---

[10] W. Bridewell and P. Langley. Two kinds of knowledge in scientific discovery. Topics in Cognitive Science, 2(1):36 2010.
[11] W. Bridewell, Langley, L. Todorovski, and S. Džeroski. Inductive process modeling. Machine learning, 71(1):1-32, 2008.
[12] M. M. Bronstein, Bruna, T. Cohen, and P. Veličković, Geometric deep learning: Grids, groups, graphs, geodesics, and gauges. arXiv preprint arXiv:2104.13478 2021.
[13] M. B. Chang, T. Ullman, A. Torralba, and J. B. Tenenbaum. A compositional abject-based approach to learning physical dynamics. arXiv preprint ww1vv:661..00344 2016.
[14] K. Cho, B. Van Merriënboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, and Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv: /406. 1078, 2014.
[15] M. Cranmer, S. Greydanus, S. Hoyer, P. Battaglia, D. Spergel, and S. Ho. Lagrangian neural networks. arXiv preprint arXiv:2003.04663 2020.
[16] M. Cranmer, A. Sanchez-Gonzaler P. Battaglia, R. Xu, K. Cranmer, D. Spergel, and S. Ho. Discovering symbolic models from deep learning with inductive biases. arXiv preprint arXiv:2006.11887 2020.
[17] F. de Avila Belbute- Peress D. Economon, and J. Zico Kolter. Combining differentiable pde solvers and graph neural networks for fluid flow prediction. arXiv e-prints, pages arXiv-2007, 2020.
[18] E. Denton and R. Fergus. Stochastic video generation with a learned prior. In International Conference on Machine Learning, pages 1174-1183, PMLR, 2018.
[19] S. A. Desai, M. Mattheakis, D. Sondak, P. Protopapas, and S. J. Roberts. Port-hamiltonian neural networks for learning explicit time-dependent dynamical systems. Physical Review E, 104(3):03431,, 2021.
[20] Y. Du and I. Mordatch. Implicit generation and generalization in energy-based models, arXiv preprint 193,0099,,,,8689 2019.
[21] K. Ellis, C. Wong, M. Nye. M. Sable-Meyer, L. Cary, L. Morales, L. Hewitt, A. Solar-Lezama, and J. B. Tenenbaum. Dreamcoder: Growing generalizable, interpretable knowledge with wake-sleep bayesian program learning. arXiv preprint arXiv:2006.088381, 2020.
[22]
Elman. Finding structure in time. Cognitive science, 14(2):1792211, 1990.
[23] C. Finn, P. Abbeel, and S. Levine. Model- aodesl-gnoostic meta-learning for fast adaptation of deep networks. arXiv preprint
2017.
[24] M. Finzi, K. A. Wang, and A. G. Wilson. Simplifying hamiltonian and lagrangian neural networks via explicit constraints. arXiv preprint arXiv:2010.13581, 2020.
[25] F. B. Fuchs, D. E. Worrall, V. Fischer, and M. Welling. Se (3attrafssrrm:r::: 3d roto-translation equivariant attention networks. arXiv preprint arXiv:2006. 10503, 2020.
[26] G. Gluch and R. Urbanke. Noether: The more things change, the more stay the same. arXiv preprint 114::2204400508 2021.
[27] R. Goroshin, J. Bruna, Tompson, D. Eigen, and Y. LeCun. Unsupervised learning of spatiotemporally coherent metries. In Proceedings of the IEEE international conference computer vision, pages 4086-4093, 2015.
[28) S. Greydanus, M. Dzamba, and J. Yosinski. Hamiltonian neural networks. In Advances in Neural Information Processing Systems, pages 15353-15363, 2019.
[29] G. Hinton, S. Osindero, M. Welling, and Y.-W. Teh. Unsupervised discovery of nonlinear structure using contrastive backpropagation. Cognitive science, 30(4):725-731, 2006.

---

[30] S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural computation, 9()):
1735-1780, 1997.
[31] D. Jayaraman and K. Grauman. Slow and steady feature analysis: higher order temporal coherence in video. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3852-3861, 2016.
[32] M. Jordan. Serial order: parallel distributed processing approach. In Advances in psychology. volume 121, pages 471-495. Elsevier, 1997.
[33] J. Jumper, R. Evans, A. Pritzel, T. Green, M. Figurnov, O. Ronneberger, K. Tunyasuvunakool, R. Bates, A. Žídek, A. Potapenko, et al. Highly accurate protein structure prediction with alphafold. Nature, 596(7873) 2021.
[34] Kipf, E. Fetaya, K.-C. Wang, M. Welling, and R. Zemel. Neural relational inference for 2018.
interacting systems. arXiv preprint arXiv:/802.0444
[35] A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. Advances neural information processing systems, 15: 2012.
[36] Langley, J. N. Sanchez, L. Todorovski, and S. Dzeroski. Inducing process models from continuous data. 2002.
[37] Y. LeCun, S. Chopra. R. Hadsell, M. Ranzato, and F. Huang. A tutorial on energy- learning. Predicting structured data, 1(0), 2006.
[38] Z. Liu and M. Tegmark. Ai poincaré: Machine learning conservation laws from trajectories. arXiv preprint arXiv:201I.04690, 2020.
[39] J. Lorraine, P. Vicol, and D. Duvenaud. Optimizing millions of hyperparameters by implicit differentiation. In International Conference on Artificial Intelligence and Statistics, pages 1550- PMLR, 2020.
[40] M. Lutter, C. Ritter, and J. Peters. Deep lagrangian networks: Using physics model prior for deep learning. arXiv preprint arXiv: 1907004990, 2019.
[41] L.. Metz, N. Maheswaranathan, B. Cheung, and J. Sohl-Dickstein. Meta-learning update rules for unsupervised representation learning. arXiv preprint arXiv: /80.002804.00222,, 2018
[42] A. Mnih and G. Hinton. Learning nonlinear constraints with contrastive backpropagation. In Proceedings. 2005 IEEE International Joint Conference on Neural Networks, 2005., volume 2, pages 1302-1307. IEEE, 2005.
[43] H. Mobahi, Collobert, and J. Weston. Deep learning from temporal coherence in video. In Proceedings of the 26th Annual International Conference on Machine Learning, pages 737-744, 2009.
[44] E. Noether. Invariante variatioonspoobleme. Nachrichten von der Gesellschaft der Wissenschaften zu Göttingen, Saathematisch--Pyiikkaiiii Klasse, 1918:235-257, 1918.
[45] J. Oh, X. Guo, H. Lee, R. Lewis, and S. Singh. cotioonlo video prediction using deep networks in atari games. arXiv preprint 10XXvv1507.087500 2015.
[46] T. Pfaff, M. Fortunato, A. Lunchez-Gonzalee and W. Battaglia. Learning mesh-base simulation with graph networks. arXiv preprint arXiv:2010.03409, 2020.
[47] A. Raghu, M. Raghu, S. Kornblith, D. Duvenaud, and G. Hinton. Teaching with commentaries. arXiv preprint arXiv:2011.03037 2020.
[48] A. Recasens, P. Lue, J.-B. Alayrac, L. Wang, F. Strub, C. Tallee, M. Malinowski, V. Patraucean, F. Altché, M. Valko, et al. Broaden your views for self-supervised video learning. arXiv preprint MrXiv:2103.16553 2021.
[49] E. Rossi, B. Chamberlain, F. Frasca, D. Eynard, Monti, and Bronstein. Temporal graph networks for deep learning on dynamic graphs. arXiv preprint arXiv: 10637, 2020.
12

---

[50] A. Sunchez-Gonzaler J. Godwin, T. Pfaff, R. Ying, J. Leskovec, and P. Battaglia. Learning to simulate complex physics with graph networks. In International Conference on Machine Learning, pages
PMLR, 2020.
[51] V. G. Satorras, E. Hoogeboom, and M. Welling. E (n) equivariant graph neural networks. arXiv preprint wtxiv2202.009844 2021.
[52] F. Scarselli, M. Gori, A. Tsoi, M. Hagenbuchner, and G. Monfardini. The graph neural network model. IEEE transactions on neural networks, 20(1):61-88)) 2008
[53] M. Schmidt and H. Lipson. Distilling free-form natural laws from experimental data, science, (24((923):8118)) 81, 2009.
[54] R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE international conference on computer vision, pages
2017.
[55] K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1556, 2014.
[56] K. O. Stanley, D. B. D' A'mrosio, and Gauci. A hypercube-basce based encoding for evolving large-scale neural networks, Artificial life, 15(2):185-21) 212, 2009.
[57] J. M. Stokes, K. Yang, K. Swanson, W. Jin, A. Cubillos-Ruii Ruiz, N. M. Donghia, C. R. MacNair, S. French, L. A. Carfrae, Z. Noom-Ackermann, et al. A deep learning approach to antibiotic discovery. Cell. 180(4))688-7022 2020.
[58] H. Suh and Tedrake. The surprising effectiveness of linear models for visual foresight in object pile manipulation. arXiv preprint 20XX::0002.09093,, 2020.
[59] Y. Sun, X. Wang, Z. Liu, J. Miller, A. A. Efros, and M. Hardt. Test-time training for out-of- distribution generalization. arXiv preprint
2019.
[60] C. Tallec and Y. Ollivier. Can recurrent neural networks warp time? arXiv preprint arXiv:1804.11 11 188, 2018.
[61] H. Tanaka and D. Kunin. Noether's 's learning dynamics: The role of kinetic symmetry breaking in deep learning. arXiv preprint arXiv:2105.02716 2021.
[62] N. Thomas, T. Smidt, S. Kearnes, L. Yang, L. Li, K. Kohlhoff, and Riley. Tensor field networks: Rotation-and and translation-equivn equivariiat neural networks for 3d point clouds. arXiv preprint arXiv: 1802.00 08219, 2018
[63] S.-M. Udreseu and M. Tegmark. Ai feynman: A sion. Science Advances, 6(16))ca 2020.
method for symbolic regres-
[64] S.-M. Udrescu and M. Tegmark. Symbolic pregression: Discovering physical laws from distorted video. Physical Review E. 103(4):043307, 2021.
[65] D. Wang, E. Shelhamer, S. Liu, B. Olshausen, and T. Darrell. Fully test-time adaptation by entropy minimization. arXiv preprint arXiv:2006. 10726, 2020.
[66] M. Weiler, M. Geiger, M. Welling, W. Boomsma, and T. Cohen. 3d steerable cnns: Learning rotationally equivariant features in volumetric data. arXiv preprint arXiv:1807.0233 2018.
[67] S. J. Wetzel, R. G. Melko, Scott, M. Panju, and Ganesh. Discovering symmetry invariants and conserved quantities by interpreting siamese neural networks. Physical Review Research, 2 (3):033499, 2020.
[68] L. Wiskott and T. Sejnowski. Slow feature analysis: Unsupervised learning of invariances. Neural computation, 14(4):715-770, 2002.
[69] Wu, J. J. Lim, H. Zhang, J. B. Tenenbaum, and W. Freeman. Physics 101: Learning physical object properties from unlabeled videos. In BMVC, volume 2, page 7, 2016.
13

---

[70] T. Yu, C. Finn, A. Xie, S. Dasari, T. Zhang, P. Abbeel, and S. Levine. One-shot imitation from observing humans via Comain-adapties meta-leaming. arXiv preprint arXiv:/802.01557, 2018.
[71] M. Zhang, H. Marklund, Dhawan, A. Gupta, S. Levine, and C. Finn. Adaptive risk min- imization: A meta-learning approach for tackling group distribution shift. arXiv preprint arXiv:2007.02931, 2020.
[72] R. Zhang. P. Isola, A. A. Efros, E. Shechtman, and O. Wang. The unreasonable effectiveness of deep features perceptual metric. In CVPR, 2018.
[73] Y. D. Zhong, B. Dey, and A. Chakraborty. Symplectic ode-net: Learning hamiltonian dynamics with control. arXiv preprint arXiv: wxXiv:199.12077) 2019.
[74] A. Zhou, T. Knowles, and C. Finn. Meta-learning symmetries by reparameterization. arXiv preprint arXiv:2007.02933, 2020.
14


Reviews:
[{"review_id": "jhUkW74MM6W", "reviewer": "Reviewer_eyYP", "summary": "\n**Summary and Contributions**\nThe paper proposes to approximately conserve desired quantities in sequential problems, such as when predicting the physical trajectory of an object. They argue intuitively that learning the right conserved quantities would be a useful inductive bias because symmetries (which are typical inductive biases) are often related to a conserved quantity through Noether Theorem. The proposed method meta-learns the quantity to be conserved from the data.\n\nOn the theoretical side, they study a specific setting where they prove an upper bound on the generalization gap which can be decreased when conservating some quantity. This suggests that learning models that conserve the right quantities could improve generalization.\n\nThey show in simple evaluation settings that the proposed method approximately learns the correct quantity to conserve and that conserving this quantity can help generalization.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**\n- **Nice and interesting idea that will be of interest to the community**.The idea and argument for learning the quantities to conserve instead of the symmetries is very compelling and will be of interest to the symmetry/physics/ML subcommunity. I find the explanation provided on lines 36-38 particularly compelling: “whereas symmetries [… ] perturbating the actual data, conserved quantities […] only the true data”.\n- **Impressive results: recovering the hamiltonian** I’m impressed by the fact that you essentially recover the mathematical formulation of the correct hamiltonian even in noisy environments. The proposed method that achieves this is interesting and a valuable contribution.\n- **Novelty** to my knowledge using conservation of a quantity as an inductive bias is surprisingly novel\n- all theoretical results seem correct to me.\n\n**Weaknesses**\n- **Somewhat misleading characterization of your theory in abstract and introduction** The theory sounds much stronger than it actually is. Line 10-11 (abstract): “We show theoretically […] that Noether Networks improve prediction quality”, and lines 55-56 (intro) “We theoretically characterize […] lowering the generalization gap”.  Both are a mischaracterization of the theoretical results which you provide for 2 reasons. First, the theory only holds in a constrained case where the preimages of the conserved law form parallel affine subspaces and the model is constrained to translating the input. Second, even under the aforementioned constrained, your results do not prove anything about the generalization gap, only upper bounds which could be very loose. If you wanted to use bounds to prove that models that conserve quantities have smaller generalization gaps than those which don’t, you would need to provide a LOWER bound for theorem 1 and show that this lower bound is larger than the upper bound from Theorem 2. To be clear, I do not think that section 2 is problematic, but the language in the abstract and intro should not suggest that you prove any strong theoretical results (use words such as “suggest”, “could”, “restricted setting”).\n- **Could be more clear** I find that many parts of the paper were not very clear (at least not until reading all the paper). E.g,\n    - Figure 1 is probably your most important figure in that most people will try to understand your method from this. I did not understand it at all before reading the paper, and even now I find it hard to understand. I would especially suggest (1) making it clear what is the difference between the top and the bottom sequences; and (2) using the caption to explain in more detail what is seen in the figure.\n    - The first paragraph on page 3 is very hard to understand due to having to parse all that notation at once. I would suggest trying to explain with English words what is happening. For example, the fact that you are assuming that input space can be partitioned in subspaces in which the desired quantity is conserved (i.e., the preimages) and that these partitions are parallel affine subspaces of dimension m. Diluting the notation with some intuition / English explanations would really help the reader.\n    - I find the explanation of (meta)-tailoring in the text hard to understand. Maybe having a figure would help. Your algorithm does somewhat help but it doesn't show what happens at test time. Part of the issue is that you use the language of meta-learning but say it's different because you use it in a single task setting. Maybe you could stick with standard meta-learning and say that you treat every sequence as a single \"subdataset\". This is for example what people do when talking about neural processes.\n    - when reading the intro and abstract I originally thought that you would use the conserved quantities to learn the symmetries. But the entire paper has nothing to do with symmetries: you simply hypothesize (using the intuition behind Noether theorem) that conserving quantities is a useful inductive bias because that's related to symmetries which have been shown to be a useful inductive bias. I think it would be useful to make it explicit right after your statement of Noether's theorem that you do not do anything with symmetries.\n- **Experimental results are hard to judge** besides for the first experiment, the experimental results are hard to judge as (1) the only baseline doesn't even try to add some inductive bias; (2) there are no standard benchmarks for these types of results; (3) on the provided line plots the results seem quite similar and the lack of standard errors make it hard to judge if these results are significant.\n\n**Questions**\n- I do not understand how you avoid learning a trivial/constant value to conserve (i.e., problem 2 line 121). In particular, as you are performing training and meta-training on the training set, I do not understand what is the advantage of learning a useful conservation law (I agree that it would be the case if you used a validation set). Can you please clarify?\n- To obtain symbolic formulas of the Hamiltonian, I don’t really understand what you do once you have 20 candidates. Do you train all of the 20, and then select the best one on the training set?\n\n**Suggestions / specific issues**\n- Please remove the main text from the supplementary material.\n- In the future, please do not remove numbered lines for reviewing (the first paragraph of page 3 has no numbers)\n- For theorem 1 and theorem 2, the loss will often be the cross-entropy and so $C= \\infty$, right? If so, it should at least be pointed out that the bounds are vacuous for log loss.\n- Please be consistent with your notation. For example, $\\phi$ in section 2 is a function, but a parameter for the (same?) function $g_\\phi$ in section 3. As another example you are using $f_{theta}(x)$ in section 2, $f_{theta}(x,t)$ in section 3.3 and then back to $f_{theta}(x)$ in Algorithm 1.\n- In section 4.2. please mention the incurred computational overhead. In particular, it seems to be very large for finding symbolic formulas, right?\n- Please add in the main text that you had to remove some datasets from Greydanus because the DSL couldn’t find the form of the symbolic formula.  \n- Lines 515-517 of the appendices show an important advantage of your method, which I would briefly mention in the main text.\n- In the proof of theorem 2, I would suggest stopping after line 501 and simply saying “the rest of the proof is the same as for theorem 1 but uses $\\mathcal{}(r,\\mathcal{Z}) \\leq (2R \\sqrt{m}/r)^m$\". The redundancy of the current proof forces the reader to reread that part of the proof to understand how the result is different.\n\n**Minor Suggestions**\n- Line 30 / 33 / 114: i.e -> i.e.,\n- Line 59: improve -> improving\n- Top Paragraph of page 3: In the first occurrence of $i$ and $n$ please explain that it refers to the dataset/samples.\n- Line 100: ensure satisfies -> ensure that it satisfies\n- Line 100: bias -> unsupervised loss\n- Line 136: described -> describe\n- Line 242: cos q -> cos(q)\n- Appcs line 457: remove “at”\n- Appcs line 460: t he -> the\n- Appcs equation 19: this is exactly the same as equation 15, I’m not sure what you wanted to add.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations and societal impact are adequately discussed.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8 hours"}, "extra_text": {}}, {"review_id": "fb-wCZCEoHJ", "reviewer": "Reviewer_3378", "summary": "**Summary**: \nThis paper is concerned with the problem of automatic symmetry discovery, by meta-learning conserved quantities that are useful for predictions. They propose Noether Networks, where the learned symmetries take the form of a meta-learned conservative loss optimised inside the prediction function during test time. They demonstrate experimentally that the proposed approach can improve prediction quality and discover useful inductive biases in sequential problems.\n\n**Contributions**:\n- This paper proposes a new algorithm that automatically learns its inductive biases in the form of a meta-learned conservative loss optimised inside the prediction function.\n- (In a simplified setting) They theoretically show when and why enforcing conservative laws brings benefits.\n- They demonstrate through experiments that the proposed algorithm can recover known physical laws, and improve prediction performance for video data.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Originality**: \nFirstly, meta-learning symmetries is a relatively new and under-explored problem, and this paper offers a simple yet effective framework. Secondly, the idea of meta-learning conservation loss optimised inside the prediction function during test time has not been explored before, and the authors show empirically that this is actually a viable direction. Thirdly, this paper demonstrates the success of meta-learning continuous symmetries and learning symmetries that are useful even in noisy or partial observation settings, which I think is a big obstacle for wider applications of symmetries.\n\n**Quality**:\nAlthough under a simplified setting, the theoretical analysis in section 2 still offers some insight on when and why enforcing conservation laws is beneficial. The conclusion is sensible and expected, but I did not check the mathematical statements rigorously.\n\nThe methodology is simple and clean. I’d be curious to hear the following because I think it is less clear in the manuscript:\n- When a conservation loss is learned, does it always correspond to certain continuous symmetries? (the opposite of Noether’s theorem) How does it compare to learning a regulariser in general?\n- To enforce the conservation laws, why did the authors choose tailoring and optimise conversion loss during test time. What is the motivation behind this choice?\n\nThe experiments are all sensible and well-conducted. But I would be more convinced if the authors can show on video data what would happen when the unsupervised loss does not take the form of conservation loss. Is it obviously inferior compared to a conservation loss?\n\n**Clarity**:\nThe paper is well written and it is enjoyable to read. The methodology and theoretical analysis are clean and well-paced.\n\n**Significance**:\nThis paper presents a simple yet effective framework to discover continuous symmetries in the form of a meta-learned conservation loss. It has been demonstrated to be effective for discovering continuous symmetries, and can be applied to challenging settings where the conservation law is not exact due to noise, or only partial information is available. I believe it is an interesting direction for future exploration.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "**Limitations**:\nI think the authors already address some of their limitations. One suggestion would be: A theoretical or empirical analysis of why the learned conservation loss must correspond to symmetries rather than a generic form of regularisation.\n\n**Societal Impact**:\nI think the authors already have a good discussion of potential societal impact.\n", "review_text": "", "extra_scores": {"ethical_concerns": "I have no ethical concerns regarding this work.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "OrJ35o6cM1x", "reviewer": "Reviewer_HJHv", "summary": "The authors tackle the challenge of automatically discovering symmetries in dynamical systems. As continuous symmetries are equivalently described by conserved quantities--as stated by Noether's theorem--they introduce a method for learning (approximately) conserved quantities. They do so by building on the `meta-tailoring' framework which allows to fine-tune the model according to the learned conservation laws at test time. For problems with few and interpretable coordinates (e.g. position, momentum, etc) they propose a symbolic approach that generate many valid (i.e. with compatible physical units) expressions that are parametrized with learnable weights. For raw-pixel data, they parametrize and learn the function to be conserved with a neural network. In both cases, the learned conservation function is used to guide the model towards physically 'realistic' solutions. They empirically assess their method on pendulum and spring systems, but also on real data from the Physics 101 dataset.", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\n## Strengths\nThe proposed algorithm is of relevance to the community as incorporating physical inductive bias by hand is not always possible. Empirical assessment shows that the model is able to recover the true conserved quantity and that generalization performance is indeed improved. Authors also show that enforcing a `soft conservation' is useful as dissipative systems would only approximately conserve energy.\nThe theoretical evidence from Section 2 strengthen the motivation of the proposed approach.\n\n## Weaknesses\nThe two main weaknesses that I can identify are;\n1/ clarity, as developed later.\n2/ the symbolic approach to learn the conserved expression. It is an interesting approach and it seems to work well in practice, but it has a bit of a heuristic flavour and has a non marginal computational overhead. Also, I am not entirely convinced that there exist problems of interest where there is a closed-form conserved quantity that we wouldn't know about. I may be wrong though.\nAlso, I am not really convinced by the Physics 101 experiment as the improvement seems marginal (hard to tell without confidence intervals) and there is no visualisation of the dataset, predictions or learned embeddings.\n\n## Clarity\nGenerally, I feel that even though the paper is fairly well written some of the core ideas could and should be better conveyed. For instance, readers that aren't familiar with `meta-tailoring may struggle at Section 3.1. Since this is a core aspect of the proposed method, it would be worth adding an equation or a diagram guiding the reader.\nAlso, regarding Section 2, the first paragraph is quite packed with notations. I am not certain that it is necessary to introduce general notations as a special case is eventually considered. Also, a simple diagram could be useful to illustrate this setting and give intuition to the reader on the improved generalization gap. If I understand correctly, the main idea of this section is that, if the solution space is an affine subspace, and this is enforced, the the generalization gap bound grows w.r.t. the dimension of the subspace instead of the embedding space. I think that if this is indeed the core message, it should be stressed more.\nAlgorithm 1 is extremely welcomed as it is really helpful to understand the proposed approach.\n\n## Reproducibility\nNo code has been provided in the supplementary materials, yet there seems to be quite some experimental details in Section 4. I am not certain that this is enough to reproduce the results.\n\n## Additional feedback\n- Links are broken.\n- Lack of standard errors / confidence intervals.\n- 53: \"are\"?\n- Section 2: Is this setting motivated by a physical problem? Or because it is simple enough so that such analysis can be done.\n- 65-66: What is the difference between $\\phi$ and $G$?\n- Equations 1 and 2: It should be stressed that the key difference is that $d$ is replaced by $m$, which is great if $m \\le d$ (as stated on line 67).\n- 76: It may not be immediate for the reader to understand what are the dimensions $d$ and $m$. A drawing may help?\n- Figure 2 & Figure 3: Looks a bit blurry and the font is quite small.\n- 246-161: It would be quite helpful to see what the 'Physics 101 ramp data' look like. Would there be a way also to visualize the learnt embeddings? Am whether it is correlated to the `true' conserved quantities (i.e. energy) in this setting?\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations are discussed.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "AoUoydV0n6", "reviewer": "Reviewer_A5RG", "summary": "The authors present a meta-learning approach for identifying conserved quantities in time and leveraging them in time series predictions. They demonstrate the utility of their approach on a series of physical systems and real-world settings.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The idea of formulating a loss based on time conservation is very interesting in the context of time series, as it allows to account for additional symmetries in a data driven fashion. Identifying those symmetries directly can be a significantly harder task using the original data.\n\nThe experiments conducted show that their approach can successfully identify existing conserved quantities and improve predictive performance in physical setups and video prediction tasks. Although the improvements in the latter tasks are minor, they still serve as a demonstration of the algorithms utility, since in these cases the functional form of the conservation law was learned in a completely data driven fashion. As such, an analysis of the learned function (e.g. the filters of the CNN) would be of high interest to the reader, which is currently missing from the text. Another point which would warrant a more detailed discussion is how learned symmetries can be applied to the initial model. Currently this is done via a single gradient update of the weights, however this might not be the best strategy in all scenarios (e.g. due to the strong non-linear dependence of a function on the weights). Evidence for this can be seen in the pendulum control experiments, where it proved to be more beneficial to adjust the activations.\n\nThe algorithm, experiments and results are described in a clear fashion and theoretical background is provided on when it can be beneficial to account for conservation laws.\n\nOn the whole, the approach presented in the paper can be considered an interesting step towards learning conserved quantities and has the potential to serve as a conceptual basis for future developments, although a more in-depth analysis would be desirable.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Potential limitations and societal impact have been sufficiently addressed.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers were happy in the end to accept this paper: I vote to accept. The biggest changes the reviewers wanted to see in the camera ready include: (i) improving the clarity, (ii) additional visualizations/experiments. Making these changes as well as others suggested by the reviewers will make a good paper a great one and a welcome addition to the conference.
Decision: Accept (Poster)