Noether Networks:
Meta-Learning Useful Conserved Quantities

Ferran Alet∗1 , Dylan Doblar∗1 , Allan Zhou2 ,Joshua Tenenbaum1 , Kenji Kawaguchi3 , Chelsea Finn2
1
MIT, 2 Stanford University, 3 National University of Singapore{alet,ddoblar}@mit.edu

Abstract
Progress in machine learning (ML) stems from a combination of data availability,
computational resources, and an appropriate encoding of inductive biases. Usefulbiases often exploit symmetries in the prediction problem, such as convolutional
networks relying on translation equivariance. Automatically discovering theseuseful symmetries holds the potential to greatly improve the performance of ML
systems, but still remains a challenge. In this work, we focus on sequential
prediction problems and take inspiration from Noether’s theorem to reduce theproblem of finding inductive biases to meta-learning useful conserved quantities.
We propose Noether Networks: a new type of architecture where a meta-learned
conservation loss is optimized inside the prediction function. We show, theoreticallyand experimentally, that Noether Networks improve prediction quality, providing ageneral framework for discovering inductive biases in sequential problems.

1

Introduction

The clever use of inductive biases to exploit symmetries has been at the heart of many landmark
achievements in machine learning, such as translation invariance in CNN image classification [25],permutation invariance in Graph Neural Networks [34] for drug design [38], and roto-translational
equivariance in SE3-transformers [19] for protein structure prediction. However, there may be
exploitable symmetries that are either unknown or difficult to describe with code. In this work, we
aim to find symmetries in sequential prediction problems, inspired by Noether’s theorem [29], which
loosely states the following:
For every continuous symmetry property of a dynamical system,
there is a corresponding quantity whose value is conserved in time.
Motivated by this equivalence, we propose that conservation laws are a powerful paradigm for metalearning useful inductive biases in sequential problems. Whereas symmetries are global properties
linked to counter-factuals about perturbing the data, conserved quantities can be evaluated using only
the true data. This provides an immediate signal for machine learning algorithms to exploit.
Our approach involves meta-learning a parametric conservation loss function which is useful tothe prediction task. We leverage the tailoring framework [3], which proposes to encode inductive
biases by fine-tuning neural networks with hand-designed unsupervised losses inside the prediction
function. In contrast to traditional auxiliary losses, which are added to the task loss during training,
tailoring optimizes both during training and testing, customizing the model to each individual queryto ensure there is no generalization gap for the self-supervised loss. We propose to meta-learn
the self-supervised loss function and parameterize it in the form of a conservation loss over time∗

Equal contribution. Our code is publicly available at https://lis.csail.mit.edu/noether.

Fourth Workshop on Machine Learning and the Physical Sciences (NeurIPS 2021).

...

...
Figure 1: Noether Networks enforce conservation laws, meta-learned by gφ , in sequential predictions
made by fθ , which is tailored to the input x0 to produce final predictions x̂1:T . Imposing thesemeta-learned inductive biases improves video prediction quality with objects sliding down a ramp.

PT
steps t; i.e. L(x0 , x̃1:T , φ) = t=1 |gφ (x0 ) − gφ (x̃t )|2 . By doing so, we allow the network to
discover its own inductive biases. The conservation form encodes a meta-inductive bias (inductive
bias over inductive biases) which narrows the search space exponentially (in prediction horizonT ) and simplifies the parameterization. We show experimentally how this framework allows us torecover known conservations and use them for better generalization in scientific data, as well as learn
useful conservation laws from raw pixels.

2

Theoretical advantages of enforcing conservation laws

In this section, we demonstrate principled advantages of enforcing conservation laws of the formgφ (fθ (x)) = gφ (x) by considering a special case where preimages under gφ form affine subspaces.
Let input x and target y be x, y ∈ Rd , and let the Noether embedding be gφ : Rd → P whereP = {gφ (x) : x ∈ Rd }. We consider a restricted class of models, parameterized by θ ∈ Θ, ofthe form fθ (x) = x + vθ for vθ ∈ Rd such that for all x, the preimage of gφ is gφ−1 [{gφ (x)}] ={x + Az : z ∈ Rm }, A ∈ Rd×m . Here, m ≤ d is the dimensionality of the preimages of gφ . We
denote by C the smallest upper bound P
on the loss value as L(fθ (x), y) ≤ C (for all x, y and θ).
nDefine ψ(v) = Ex,y [L(f (x, v), y)] − n1 i=1 L(f (xi , v), yi ), with Lipschitz constant ζ. Therefore,ζ ≥ 0 is the smallest real number such that, for all v and v 0 in V, |ψ(v) − ψ(v 0 )| ≤ ζkv − v 0 k2 ,where V = {vθ ∈ Rd : θ ∈ Θ}. P
Finally, we define R = supθ∈Θ kvθ k2 , and the generalization gap bynG(θ) = Ex,y [L(fθ (x), y)] − n1 i=1 L(fθ (xi ), yi ). Theorem 1 shows that enforcing conservation
laws of gφ (fθ (x)) = gφ (x) is advantageous when the dimension of the preimages under gφ is lessthan the dimension of x; that is, when m < d.
Theorem 1. Let ρ ∈ N+ . Then, for any δ > 0, with probability at least 1 − δ over an iid draw of n
examples ((xi , yi ))ni=1 , the following holds for all θ ∈ Θ:rG(θ) ≤ Cr√√ξ ln(max( ξ, 1)) + ξ ln(2R(ζ 1−1/ρ ) n) + ln(1/δ)ζ 2/ρ
+ 1{ξ ≥ 1}
.
2n
n

(1)

where ξ = m if gφ (fθ (x)) = gφ (x) for any x ∈ Rd and θ ∈ Θ, and ξ = d otherwise.
The proof is presented in Appendix A. In Theorem 1, when we enforce the conservation laws, d is
replaced by m, the dimension of the preimage. So, enforcing conservations improves the bound on
generalization error for m < d.
2

Algorithm 1 Prediction and training procedures for Noether Networks with neural conservation loss
Given: predictive model class f ; embedding model class g; prediction horizon T ; training
dist. Dtrain ; batch size N ; learning rates λin , λout , λemb ; task loss Ltask ; Noether loss LNoether
1: procedure P REDICT S EQUENCE(x0 ; θ, φ)2:x̃0 , x̂0 ← x0 , x0
3:x̃t ← fθ (x̃t−1 ) ∀t ∈ {1, . . . , T }4:
θ(x0 ; φ) ← θ − λin ∇θ LNoether (x0 , x̃1:T ; gφ )5:x̂t ← fθ(x0 ;φ) (x̂t−1 ) ∀t ∈ {1, . . . , T }6:
return x̂1:T. Initial predictions. Inner step with Noether loss
. Final prediction with tailored weights

7: procedure T RAIN
8:
φ ← randomly initialized weights. Initialize weights for Noether embedding g
9:θ ← randomly initialized weights. Initialize weights for predictive model f
10:
while not done do
(0)
(N )
11:
Sample batch x0:T , . . . , x0:T ∼ Dtrain
12:
for 0 ≤ n ≤ N do
(n)
(n)
13:
x̂1:T ← P REDICT S EQUENCE(x0 ; θ, φ)
14:
15:
16:

3

PN
(n)
(n)
φ ← φ − λemb ∇φ n=0 Ltask (x̂1:T , x1:T ) . Outer step with task loss for embedding
PN
(n)
(n)
θ ← θ − λout ∇θ n=0 Ltask (x̂1:T , x1:T ). Outer step for predictive model
return φ, θNoether Networks

Leveraging tailoring to encode inductive biases. We perform a prediction-time optimization to
encourage outputs to follow conservation laws using the tailoring and meta-tailoring frameworks [3].
Tailoring encodes inductive biases in the form of unsupervised losses optimized inside the prediction
function. In doing so, tailoring fine-tunes the model to each query to ensure that it satisfies theunsupervised loss for that query. For example, we may optimize for energy conservation in a physics
prediction problem. In meta-tailoring, we train the model to do well on the task loss after the tailoring
step has fine-tuned its parameters. This allows us to impose inductive biases on the test queries notknown at training time, as in transductive learning [46]. A limitation of tailoring is that the tailoring
loss must be user-specified. This is acceptable in domains where the desired inductive bias is both
known and easily encoded, but problematic in general — we address this issue with Noether Networks.
Meta-learning a neural loss function. We propose Noether Networks, an architecture class forsequential prediction that consists of a base predictor fθ : x̃t 7→ x̃t+1 and a meta-learned tailoring
loss, parameterized as the conservation of a neural embedding gφ : x̃t 7→ Rk . This embedding takes
raw predictions as input (such as images in the case of video prediction). The conservation form
induces a meta-inductive bias over potential learned tailoring losses. The Noether loss is formulated as
LNoether (x0 , x̃1:T ; gφ ) =TX

2|gφ (x0 ) − gφ (x̃t )| ≈

t=1

|

T
X

2

|gφ (x̃t−1 ) − gφ (x̃t )|

(2)

t=1

{z

(a)

}

|

{z

(b)

}

where x0 is the ground-truth input, the x̃t = fθ (x̃t−1 ) are the model’s predictions, and x̃0 , x0 .
Expressions 2(a) and 2(b) are equivalent if we fully enforce the conservation law, but they differ
if conservation is not fully enforced. When not fully conserving, 2(a) propagates information fromground truth more directly to the prediction, but 2(b) may be a useful approximation which better
handles imperfectly conserved quantities, where the quantity should be Lipschitz but not exactly
constant. In both cases, if we tailor θ with a single gradient step; the gradient update takes the form
θ(x0 ; φ) = θ − λin ∇θ LNoether (x0 , x̃1:T (θ); gφ ).
(3)
We compute final predictions as x̂t = fθ(x0 ;φ) (x̂t−1 ). We can now backpropagate from
Ltask (x1:T , x̂1:T ) back to φ, which will be optimized so that the unsupervised adaptation θ 7→
θ(x0 ; φ) helps lower Ltask .The optimization requires second-order gradients to express how φ affects
Ltask through θ(x0 ; φ). This is similar to MAML [17], as well as works on meta-learning loss functions for few-shot learning [4] and group distribution shift [52]. Algorithm 1 provides pseudo-code.
3

Table 1: Recovering H for the ideal pendulum.

Table 2: Recovering H for the ideal spring.

MethodDescription

RMSE

MethodDescription

RMSE

Vanilla MLP
Noether Nets

N/A
p2 − 2.99 cos(q)0.05630.0423

Vanilla MLP
Noether Nets

N/A
q 2 + 1.002 p2

.0174
.0165

True H [oracle]p2 − 3.00 cos(q)0.0422

True H [oracle]q 2 + 1.000 p2.0166

Figure 2: Noether Networks can recover
the energy of a real pendulum, eventhough it is not fully conserved. Thisis because they only look for quantities
whose conservation helps improve predictions. Moreover, by only softly encouraging conservation, it better encodes
imperfect conservations.

4

Experiments

Can Noether Networks recover known conservation laws in scientific data? We first evaluate
the capabilities of Noether networks on a set of symbolic regression tasks involving simulated and real
physical systems. The ideal spring and ideal pendulum settings of Greydanus et al. [21] allow us to
examine the behavior of Noether Networks for scientific data where we know a useful conserved quantity: the energy. We use their MLP baseline and discover conservation laws that, when used for metatailoring, can improve prediction performance. Since the baseline does not predict xt+1 but ratherddt (xt ), we apply the loss to a finite-difference approximation, i.e. LNoether (xt , xt + fθ (xt )∆t ).
For our pipelined approach, we program a simple domain specific language (DSL) for physicalformulas. In contrast to most program synthesis approaches, our setting has continuous parameterswhich can be trained via gradient descent. We first search all valid symbolic formulas up to depth
7 and then train their parameters on the true data to be approximately conserved by minimizing
variance of the formula on the training set using SGD. Then, we verify whether the formula indeed isapproximately conserved by comparing its variance on true vs. random data.Finally, we try using eachconserved expression as a Noether loss in the meta-tailoring setting (trying several learning rates foreach), starting from the baseline MLP weights. The formula with the best performance on the training
data is selected and evaluated on the test set. See Appendix C for additional experimental details.
For the ideal pendulum and spring, our method recovers the true Hamiltonian H almost perfectly.
Using these expressions as losses reaches equivalent performance with that of the oracle, which
fine-tunes using the true formula. Results are shown in Tables 1 and 2.
Finally, we run the same process for data coming from a real pendulum [35], where the Hamiltonian
is not conserved. We use largely the same pipeline as for the ideal pendulum, the differences are
explained in Appendix C. Noether Networks end up discovering p2 − 2.39 cos(q), compared to the(potentially sub-optimal) H = p2 − 2.4 cos(q) described in [21]. Using the discovered loss improves
the baseline MLP by more than one order of magnitude, matches the performance of hand-coding
the conservation loss, and improves over Hamiltonian Neural Networks, which fully impose theconservation of energy. Results can be seen in Figure 2.
Can Noether Networks parameterize useful conserved quantities from raw pixel information?
To characterize the benefits provided by Noether Networks in a real-world video prediction setting, we
compare a Noether Network version of the SVG video prediction model — a variational autoencoderbased method [13] — to a standard SVG baseline on the 20 degree ramp scenario of the Physics 101
dataset [50]. The dataset contains videos of various real-world objects sliding down an incline andcolliding with a second object. Each model is conditioned on the first two frames of a given sequenceand must predict the subsequent 20 frames.
4Figure 3: The Noether Network outperforms the baseline by a small margin in all four metrics in realworld video prediction, showing they can meta-learn useful conserved quantities from raw pixel data.
In our experiments, we train on only 311 training sequences, which makes generalization to the test
set difficult. In this low-data setting, the baseline SVG model struggles with overfitting — predicted
frames often morph objects in the test set into objects seen at train-time, and the object in motion
often morphs with the target object (see the prediction x̃T in Figure 1, for example). Our Noether
Network uses the inner loss formulation of Equation 2(b), where gφ is parameterized as a two-layer
CNN with a final fully connected projection layer that produces 64-dimensional embeddings. We
meta-tailor the embedding and the model from scratch for 400 epochs. As seen in Figure 3, taking
a single inner meta-tailoring step improves performance marginally over the baseline SVG modelon the test set w.r.t. MSE, PSNR, and SSIM. These results provide some evidence that Noether
Networks are capable of learning useful inductive biases from raw video data.

5

Conclusion

We propose Noether Networks: a new framework for meta-learning inductive biases through approximately conserved quantities imposed at prediction time. Originally motivated by applications
in physics, our results indicate that Noether networks can recover known conservation laws inscientific data and also provide modest gains when applied to video prediction from raw pixel
data. The generality of optimizing arbitrary unsupervised losses at prediction time provides aninitial step towards broader applications. Finally, this work points at the usefulness of designingmeta-learned inductive biases by putting priors on the biases instead of hard-coding biases directly.

Acknowledgements
We want to thank Leslie Kaelbling, Tomás Lozano-Pérez and Maria Bauza for insightful feedback. We
gratefully acknowledge support from GoodAI; from NSF grant 1723381; from AFOSR grant FA955017-1-0165; from ONR grants N00014-18-1-2847 and N00014-21-1-2685; from the Honda ResearchInstitute, from MIT-IBM Watson Lab; from SUTD Temasek Laboratories; and from Google. Chelsea
Finn is a fellow in the CIFAR Learning in Machines and Brains program. We also acknowledge
the MIT SuperCloud and Lincoln Laboratory Supercomputing Center for providing HPC resourcesthat have contributed to the reported research results. Any opinions, findings, and conclusions or
recommendations expressed in this material are those of the authors and do not necessarily reflect theviews of our sponsors.

References[1] F. Alet, A. K. Jeewajee, M. Bauza, A. Rodriguez, T. Lozano-Perez, and L. P. Kaelbling. Graphelement networks: adaptive, structured computation and memory. Proceedings of the 36th
International Conference on Machine Learning-Volume 97, 2019.
[2] F. Alet, E. Weng, T. Lozano-Perez, and L. Kaelbling. Neural relational inference with fast
modular meta-learning. In Advances in Neural Information Processing Systems (NeurIPS) 32,
2019.
[3] F. Alet, M. Bauza, K. Kawaguchi, N. G. Kuru, T. Lozano-Perez, and L. P. Kaelbling. Tailoring:encoding inductive biases by optimizing unsupervised objectives at prediction time. arXiv
preprint arXiv:2009.10623, 2020.
5

[4] A. Antoniou and A. J. Storkey. Learning to learn by self-critique. In Advances in Neural
Information Processing Systems, pages 9940–9950, 2019.
[5] P. W. Battaglia, R. Pascanu, M. Lai, D. Rezende, and K. Kavukcuoglu. Interaction networks forlearning about objects, relations and physics. arXiv preprint arXiv:1612.00222, 2016.
[6] G. Benton, M. Finzi, P. Izmailov, and A. G. Wilson. Learning invariances in neural networks.
arXiv preprint arXiv:2010.11882, 2020.
[7] M. M. Bronstein, J. Bruna, T. Cohen, and P. Veličković. Geometric deep learning: Grids, groups,
graphs, geodesics, and gauges. arXiv preprint arXiv:2104.13478, 2021.
[8] M. B. Chang, T. Ullman, A. Torralba, and J. B. Tenenbaum. A compositional object-based
approach to learning physical dynamics. arXiv preprint arXiv:1612.00341, 2016.
[9] K. Cho, B. Van Merriënboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, and
Y. Bengio. Learning phrase representations using rnn encoder-decoder for statistical machinetranslation. arXiv preprint arXiv:1406.1078, 2014.
[10] M. Cranmer, S. Greydanus, S. Hoyer, P. Battaglia, D. Spergel, and S. Ho. Lagrangian neuralnetworks. arXiv preprint arXiv:2003.04630, 2020.
[11] M. Cranmer, A. Sanchez-Gonzalez, P. Battaglia, R. Xu, K. Cranmer, D. Spergel, and S. Ho.
Discovering symbolic models from deep learning with inductive biases. arXiv preprint
arXiv:2006.11287, 2020.
[12] F. de Avila Belbute-Peres, T. D. Economon, and J. Zico Kolter. Combining differentiable pde
solvers and graph neural networks for fluid flow prediction. arXiv e-prints, pages arXiv–2007,2020.
[13] E. Denton and R. Fergus. Stochastic video generation with a learned prior. In International
Conference on Machine Learning, pages 1174–1183. PMLR, 2018.
[14] S. A. Desai, M. Mattheakis, D. Sondak, P. Protopapas, and S. J. Roberts. Port-hamiltonianneural networks for learning explicit time-dependent dynamical systems. Physical Review E,
104(3):034312, 2021.
[15] K. Ellis, C. Wong, M. Nye, M. Sable-Meyer, L. Cary, L. Morales, L. Hewitt, A. Solar-Lezama,and J. B. Tenenbaum. Dreamcoder: Growing generalizable, interpretable knowledge withwake-sleep bayesian program learning. arXiv preprint arXiv:2006.08381, 2020.
[16] J. L. Elman. Finding structure in time. Cognitive science, 14(2):179–211, 1990.
[17] C. Finn, P. Abbeel, and S. Levine. Model-agnostic meta-learning for fast adaptation of deep
networks. arXiv preprint arXiv:1703.03400, 2017.
[18] M. Finzi, K. A. Wang, and A. G. Wilson. Simplifying hamiltonian and lagrangian neuralnetworks via explicit constraints. arXiv preprint arXiv:2010.13581, 2020.
[19] F. B. Fuchs, D. E. Worrall, V. Fischer, and M. Welling. Se (3)-transformers: 3d roto-translationequivariant attention networks. arXiv preprint arXiv:2006.10503, 2020.
[20] G. Głuch and R. Urbanke. Noether: The more things change, the more stay the same. arXiv
preprint arXiv:2104.05508, 2021.
[21] S. Greydanus, M. Dzamba, and J. Yosinski. Hamiltonian neural networks. In Advances in
Neural Information Processing Systems, pages 15353–15363, 2019.
[22] S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural computation, 9(8):1735–1780, 1997.
[23] M. I. Jordan. Serial order: A parallel distributed processing approach. In Advances in psychology,
volume 121, pages 471–495. Elsevier, 1997.
6

[24] T. Kipf, E. Fetaya, K.-C. Wang, M. Welling, and R. Zemel. Neural relational inference forinteracting systems. arXiv preprint arXiv:1802.04687, 2018.
[25] A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional
neural networks. Advances in neural information processing systems, 25:1097–1105, 2012.
[26] Z. Liu and M. Tegmark. Ai poincaré: Machine learning conservation laws from trajectories.
arXiv preprint arXiv:2011.04698, 2020.
[27] M. Lutter, C. Ritter, and J. Peters. Deep lagrangian networks: Using physics as model prior for
deep learning. arXiv preprint arXiv:1907.04490, 2019.
[28] L. Metz, N. Maheswaranathan, B. Cheung, and J. Sohl-Dickstein. Meta-learning update rules
for unsupervised representation learning. arXiv preprint arXiv:1804.00222, 2018.
[29] E. Noether. Invariante variationsprobleme. Nachrichten von der Gesellschaft der Wissenschaften
zu Göttingen, Mathematisch-Physikalische Klasse, 1918:235–257, 1918.
[30] T. Pfaff, M. Fortunato, A. Sanchez-Gonzalez, and P. W. Battaglia. Learning mesh-based
simulation with graph networks. arXiv preprint arXiv:2010.03409, 2020.
[31] E. Rossi, B. Chamberlain, F. Frasca, D. Eynard, F. Monti, and M. Bronstein. Temporal graphnetworks for deep learning on dynamic graphs. arXiv preprint arXiv:2006.10637, 2020.
[32] A. Sanchez-Gonzalez, J. Godwin, T. Pfaff, R. Ying, J. Leskovec, and P. Battaglia. Learning
to simulate complex physics with graph networks. In International Conference on Machine
Learning, pages 8459–8468. PMLR, 2020.
[33] V. G. Satorras, E. Hoogeboom, and M. Welling. E (n) equivariant graph neural networks. arXiv
preprint arXiv:2102.09844, 2021.
[34] F. Scarselli, M. Gori, A. C. Tsoi, M. Hagenbuchner, and G. Monfardini. The graph neuralnetwork model. IEEE transactions on neural networks, 20(1):61–80, 2008.
[35] M. Schmidt and H. Lipson. Distilling free-form natural laws from experimental data. science,
324(5923):81–85, 2009.
[36] K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image
recognition. arXiv preprint arXiv:1409.1556, 2014.
[37] K. O. Stanley, D. B. D’Ambrosio, and J. Gauci. A hypercube-based encoding for evolving
large-scale neural networks. Artificial life, 15(2):185–212, 2009.
[38] J. M. Stokes, K. Yang, K. Swanson, W. Jin, A. Cubillos-Ruiz, N. M. Donghia, C. R. MacNair,
S. French, L. A. Carfrae, Z. Bloom-Ackermann, et al. A deep learning approach to antibiotic
discovery. Cell, 180(4):688–702, 2020.
[39] H. Suh and R. Tedrake. The surprising effectiveness of linear models for visual foresight inobject pile manipulation. arXiv preprint arXiv:2002.09093, 2020.
[40] Y. Sun, X. Wang, Z. Liu, J. Miller, A. A. Efros, and M. Hardt. Test-time training for out-ofdistribution generalization. arXiv preprint arXiv:1909.13231, 2019.
[41] C. Tallec and Y. Ollivier.
arXiv:1804.11188, 2018.

Can recurrent neural networks warp time?

arXiv preprint[42] H. Tanaka and D. Kunin. Noether’s learning dynamics: The role of kinetic symmetry breaking
in deep learning. arXiv preprint arXiv:2105.02716, 2021.
[43] N. Thomas, T. Smidt, S. Kearnes, L. Yang, L. Li, K. Kohlhoff, and P. Riley. Tensor field
networks: Rotation-and translation-equivariant neural networks for 3d point clouds. arXiv
preprint arXiv:1802.08219, 2018.
[44] S.-M. Udrescu and M. Tegmark. Ai feynman: A physics-inspired method for symbolic regression. Science Advances, 6(16):eaay2631, 2020.
7

[45] S.-M. Udrescu and M. Tegmark. Symbolic pregression: Discovering physical laws fromdistorted video. Physical Review E, 103(4):043307, 2021.
[46] V. Vapnik. The nature of statistical learning theory. Springer science & business media, 1995.
[47] D. Wang, E. Shelhamer, S. Liu, B. Olshausen, and T. Darrell. Fully test-time adaptation byentropy minimization. arXiv preprint arXiv:2006.10726, 2020.
[48] M. Weiler, M. Geiger, M. Welling, W. Boomsma, and T. Cohen. 3d steerable cnns: Learning
rotationally equivariant features in volumetric data. arXiv preprint arXiv:1807.02547, 2018.
[49] S. J. Wetzel, R. G. Melko, J. Scott, M. Panju, and V. Ganesh. Discovering symmetry invariants
and conserved quantities by interpreting siamese neural networks. Physical Review Research, 2(3):033499, 2020.
[50] J. Wu, J. J. Lim, H. Zhang, J. B. Tenenbaum, and W. T. Freeman. Physics 101: Learning
physical object properties from unlabeled videos. In BMVC, volume 2, page 7, 2016.
[51] T. Yu, C. Finn, A. Xie, S. Dasari, T. Zhang, P. Abbeel, and S. Levine. One-shot imitation fromobserving humans via domain-adaptive meta-learning. arXiv preprint arXiv:1802.01557, 2018.
[52] M. Zhang, H. Marklund, N. Dhawan, A. Gupta, S. Levine, and C. Finn. Adaptive risk minimization: A meta-learning approach for tackling group distribution shift. arXiv preprint
arXiv:2007.02931, 2020.
[53] Y. D. Zhong, B. Dey, and A. Chakraborty. Symplectic ode-net: Learning hamiltonian dynamics
with control. arXiv preprint arXiv:1909.12077, 2019.
[54] A. Zhou, T. Knowles, and C. Finn. Meta-learning symmetries by reparameterization. arXiv
preprint arXiv:2007.02933, 2020.

8

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope? [Yes](b) Did you describe the limitations of your work? [Yes] In section 5(c) Did you discuss any potential negative societal impacts of your work? [Yes] In section 5(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem? [Yes]2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes](b) Did you include complete proofs of all theoretical results? [Yes] In appendix A
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)? [No] We plan,
however, to open-source our code-based once cleaned.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they
were chosen)? [Yes] Specified in the appendix.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)? [Yes] We plot SEM for the video prediction results(d) Did you include the total amount of compute and the type of resources used (e.g.,
type of GPUs, internal cluster, or cloud provider)? [Yes] We detail them in eachcorresponding appendix.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] We do so in the
appendix.
(b) Did you mention the license of the assets? [Yes] We do so in the appendix.
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A](d) Did you discuss whether and how consent was obtained from people whose data you’reusing/curating? [N/A](e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable? [N/A](b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable? [N/A](c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation? [N/A]9

A

Proofs

We utilize the following lemma in the proof of Theorem 1.
Lemma 1. Fix v ∈ Rd . Then, for any δ > 0, with probability at least 1 − δ over an i.i.d. draw of n
examples ((xi , yi ))ni=1 , the following holds:rn1Xln(1/δ)Ex,y [L(f (x, v), y)] −L(f (xi , v), yi ) ≤ C
.
n i=1
2n
Proof of Lemma 1. By using Hoeffding’s inequality, we have that
!


n
1X
2nt2
Pr Ex,y [L(f (x), y)] −L(f (xi , v), yi ) ≥ t ≤ exp − 2 ,
n i=1
C


2
where t ≥ 0. Solving δ = exp − 2nt
for t ≥ 0, we have that for any δ > 0, with probability at
C2
least 1 − δ, the following holds:rnln(1/δ)1X
Ex,y [L(f (x), y)] −L(f (xi , v), yi ) ≤ C
.
n i=1
2n

A.1
A.1.1

Proof of Theorem 1
Preparation

In this subsection, we focus on the case of gφ (fθ (x)) 6= gφ (x) as a preparation for the more generalcase in the next subsection. The (closed) ball of radius r centered at c is denoted by Br [c] = {v ∈Rd : kv − ck2 ≤ r}. Fix r > 0 and C(r, V) ∈ argminC {|C| : C ⊆ Rd , V ⊆ ∪c∈C Br [c]}. Let
N (r, V) = |C(r, V)|; i.e. the minimum number of balls of radius r needed to cover the a set ofvectors V.
The statement of theorem 1 vacuously holds if R is unbounded. Thus, we focus on the case of R < ∞in the rest of the proof. For any θ ∈ Θ, there exists v ∈ V such that
Ex,y [L(fθ (x), y)] −

n
n
1X
1X
L(fθ (xi ), yi ) = Ex,y [L(f (x, v), y)] −L(f (xi , v), yi ) = ψ(v).
n i=1
n i=1
(4)

Moreover, for any v ∈ V, the following holds: for any c ∈ C(r, V),ψ(v) = ψ(c) + (ψ(v) − ψ(c)).

(5)

For the first term in the right-hand side of (5), by using Lemma 1 with δ → δ/N (r, V) and taking
union bounds, we have that for any δ > 0, with probability at least 1 − δ, the following holds for allc ∈ C(r, V):rln(N (r, V)/δ)ψ(c) ≤ C.
(6)2n
By combining equations (5) and (6), we have that for any δ > 0, with probability at least 1 − δ, thefollowing holds for any v ∈ V and all c ∈ C(r, V):rln(N (r, V)/δ)ψ(v) ≤ C+ (ψ(v) − ψ(c)).
(7)
2nThis implies that for any δ > 0, with probability at least 1 − δ, the following holds for any v ∈ V:rln(N (r, V)/δ)ψ(v) ≤ C+ min |ψ(v) − ψ(c)|.
(8)
2nc∈C(r,V)
10

For the second term in the right-hand side of (8), we have that for any v ∈ V,
min |ψ(v) − ψ(c)| ≤ ζ min kv − ck2 ≤ ζr.

c∈C(r,V)c∈C(r,V)(9)q
Thus, by using r = ζ 1/ρ−1 n1 , we have that for any δ > 0, with probability at least 1 − δ, thefollowing holds for all v ∈ V:rrln(N (r, V)/δ)ζ 2/ρ
ψ(v) ≤ C
+
.
(10)
2n
n
Using equation (4), this implies that for any δ > 0, with probability at least 1 − δ, the following holds
for all θ ∈ Θ:rrn
1Xln(N (r, V)/δ)ζ 2/ρ
Ex,y [L(fθ (x), y)] −L(fθ (xi ), yi ) ≤ C
+
,
(11)
n i=1
2n
n
q
√
√
where r = ζ 1/ρ−1 n1 . Since N (r, V) ≤ (2R d/r)d = (2R(ζ 1−1/ρ ) nd)d ,n1X
L(fθ (xi ), yi )
n i=1
r
r
ln(N (r, V)/δ)
ζ 2/ρ
+
≤C
2n
n
r
r
ln(N (r, V)) + ln(1/δ)
ζ 2/ρ
=C
+
,
2n
n
s
r
√
ln((2R(ζ 1−1/ρ ) nd)d ) + ln(1/δ)
ζ 2/ρ
=C
+
,
2n
n
s
r
√
√
d ln( d) + d ln(2R(ζ 1−1/ρ ) n) + ln(1/δ)
ζ 2/ρ
=C
+
.
2n
n

Ex,y [L(fθ (x), y)] −A.1.2

Putting results together

In this subsection, we now generalize the proof of the previous subsection to the case of gφ (fθ (x)) =gφ (x). The condition of gφ (fθ (x)) = gφ (x) implies thatgφ−1 [{gφ (fθ (x))}] = gφ−1 [{gφ (x)}].

(12)

Since gφ−1 [{gφ (x)}] = {x + Az : z ∈ Rm } and fθ (x) = x + vθ , the right-hand side of equation (12)can be simplified as
gφ−1 [{gφ (fθ (x))}] = {fθ (x) + Az : z ∈ Rm } = {x + vθ + Az : z ∈ Rm }.
Substituting this into equation (12) yields{x + vθ + Az : z ∈ Rm } = gφ−1 [{gφ (x)}] = {x + Az : z ∈ Rm },(13)

where the last equality uses the fact that gφ−1 [{gφ (x)}] = {x + Az : z ∈ Rm }. Equation (13) implies
thatvθ ∈ Col(A) ⊆ Rd ,(14)

where Col(A) is the column space of the matrix A ∈ Rd×m . Let Ā ∈ Rd×m be a semi-orthogonal
matrix such that Ā> Ā = Im (i.e., the identity matrix of size m by m) and Col(Ā) = Col(A). Then,equation (14) implies that for any θ ∈ Θ, there exists z ∈ Rm such thatvθ = Āz.

(15)

We can further refine this statement by using the following observation. Since Ā> Ā = Im , we havethat
R ≥ kvθ k2 = kĀzk2 = kzk2 ,
11

(16)

and
|ψ(Āz) − ψ(Āz 0 )| ≤ ζkĀ(z − z 0 )k2 = ζkz − z 0 k2 .

(17)

Define Z = {z ∈ Rm : kzk2 ≤ R}. Then, equations (14) and (16) together imply that for anyθ ∈ Θ, there exists z ∈ Z such thatvθ = Āz.

(18)

Whereas the previous subsection defines the ball in the space of v ∈ Rd , we can now define the ballin the space of z ∈ Rm to cover the space of v ∈ Rd through equation (18). That is, we re-define
Br [c] = {z ∈ Rm : kz − ck2 ≤ r}. Fix r > 0 and C(r, Z) ∈ argminC {|C| : C ⊆ Rm , Z ⊆∪c∈C Br [c]}. Let N (r, Z) = |C(r, Z)|. Using (18), for any θ ∈ Θ, there exists z ∈ Z such that
Ex,y [L(fθ (x), y)] −

n
n
1X
1X
L(fθ (xi ), yi ) = Ex,y [L(f (x, Āz), y)] −L(f (xi , Āz), yi )
n i=1
n i=1

= ψ(Āz),(19)

Moreover, for any z ∈ Z, the following holds: for any c ∈ C(r, Z),ψ(Āz) = ψ(Āc) + (ψ(Āz) − ψ(Āc)).

(20)

For the first term in the right-hand side of (20), by using Lemma 1 with δ → δ/N (r, Z) and taking
union bounds, we have that for any δ > 0, with probability at least 1 − δ, the following holds for allc ∈ C(r, Z):rln(N (r, Z)/δ)ψ(Āc) ≤ C
.
(21)
2n
By combining equations (20) and (21), we have that for any δ > 0, with probability at least 1 − δ,
the following holds for any z ∈ Z and all c ∈ C(r, Z):rln(N (r, Z)/δ)ψ(Āz) ≤ C+ (ψ(Āz) − ψ(Āc)).
(22)2nThis implies that for any δ > 0, with probability at least 1 − δ, the following holds for any z ∈ Z:rln(N (r, Z)/δ)+ min |ψ(Āz) − ψ(Āc)|.
(23)
ψ(Āz) ≤ C
2nc∈C(r,Z)
For the second term in the right-hand side of (23), by using equation (17), we have that for any z ∈ Z,
min |ψ(Āz) − ψ(Āc)| ≤ ζ min kz − ck2 ≤ ζr.

c∈C(r,Z)c∈C(r,V)

If m = 0, then since N (r, Z) = 1 with r = 0,n

1XEx,y [L(fθ (x), y)] −L(fθ (xi ), yi ) ≤ C
n i=1

r

ln(1/δ).
2nThis proves the desired statement
q for m = 0. Thus, we focus on the case of m ≥ 1 in the rest of
the proof. By using r = ζ 1/ρ−1following holds for all z ∈ Z:

1
n , we have that for any δ > 0, with probability at least 1 − δ, therψ(Āz) ≤ C

ln(N (r, Z)/δ)
+
2n

r

ζ 2/ρ
.
n

(24)

Using equation (19), this implies that for any δ > 0, with probability at least 1 − δ, the following
holds for all θ ∈ Θ:rrn
1Xln(N (r, Z)/δ)ζ 2/ρ
Ex,y [L(fθ (x), y)] −L(fθ (xi ), yi ) ≤ C
+
,
(25)
n i=1
2n
n
12

where r = ζ 1/ρ−1

q

√
√
1
m
1−1/ρ
) nm)m ,n . Since N (r, Z) ≤ (2R m/r) = (2R(ζn1X
L(fθ (xi ), yi )
n i=1
r
r
ln(N (r, Z)/δ)
ζ 2/ρ
+
≤C
2n
n
r
r
ln(N (r, Z)) + ln(1/δ)
ζ 2/ρ
=C
+
,
2n
n
r
r
√
ln((2R(ζ 1−1/ρ ) nm)m ) + ln(1/δ)
ζ 2/ρ
+
,
=C
2n
n
r
r
√
√
m ln( m) + m ln(2R(ζ 1−1/ρ ) n) + ln(1/δ)
ζ 2/ρ
=C
+
.
2n
n

Ex,y [L(fθ (x), y)] −A.2

Implications of Theorem 1

We now discuss various cases for the values of m, the dimension of the preimages of gφ :• (Case of m = d) Let us consider an extreme scenario where the function gφ maps all x ∈ Rd
to one single point. In this scenario, the dimensionality of preimages under gφ is maximized
as m = d. Accordingly, the bounds with and without enforcing the conservation laws
become the same. Indeed, in this scenario, the conservation laws of gφ (fθ (x)) = gφ (x) give
us no information, because they always hold for all x ∈ Rd , even without imposing them.
• (Case of m = 0) Let us consider another extreme scenario where the function gφ is invertible.
In this scenario, the dimensionality of preimages under gφ is zero as m = 0. Thus, imposing
the condition of gφ (fθ (x)) = gφ (x) makes the bound in Theorem 1 to be very small. Indeed,in this scenario, the condition of gφ (fθ (x)) = gφ (x) implies that fθ (x) = x: i.e., x is notmoving, and thus it is easy to generalize.
• (Case of 0 < m < d) From the above two cases, we can see that the benefit of enforcing
conservation laws comes from more practical cases in-between these, with 0 < m < d.
In Theorem 1, the function gφ can differ from the true function gφ∗ underlying the system. This isbecause we analyze a standard generalization gap: i.e., the difference between the expected loss andthe training loss. The cost of not using the true gφ∗ is captured in the training loss; i.e., the training
loss can be large with the function gφ that differs significantly from the true gφ∗ . Even in this case,the generalization gap can be small. For example, in the case of m = 0, the generalization bound issmall, whereas the training loss will be large unless xt+1 = xt . Therefore, our analysis gives us theinsight on the trade-off between the training loss and the dimensionality of preimages under gφ .

B

Related Work

Unsupervised adaptation and meta-learning loss functions. Hand-designed unsupervised
losses have been used to adapt to distribution shifts with rotation-prediction [40] or entropyminimization [47], as well as to encode inductive biases [3]. Unsupervised losses have also been metalearned for learning to encode human demonstrations [51], few-shot learning exploiting unsupervised
information [28, 4], and learning to adapt to group distribution shifts [52]. In contrast, our unsupervised loss only takes the single query we care about, thus imposing no additional assumptions on topof standard prediction problems, and takes the form of a conservation law for predicted sequences.
Encoding of symmetries and physics-based inductive biases in neural networks. We propose
to encode symmetries in dynamical systems as meta-learned conserved quantities. This falls under theumbrella of geometric deep learning, which aims to encode symmetries in neural networks; Bronstein
et al. [7] provide an excellent review of the field. Most relevant to the types of problems we focus
on is roto-translational equivariance [43, 48, 19, 33], applications of GNNs in physical settings [8,135, 24, 2, 1, 31, 30, 32, 12] and the encoding of time-invariance in RNNs [16, 23, 22, 9, 41]. Recent
works have encoded Hamiltonian and Lagrangian mechanics into neural models [21, 27, 10, 18],
with gains in data-efficiency in physical and robotics systems, including some modeling controlled ordissipative systems [53, 14]. In contrast to these works, we propose to encode biases via predictiontime fine-tuning following the tailoring framework [3], instead of architectural constraints. This
allows us to leverage the generality of loss functions to encode a class of inductive biases extending
beyond Lagrangian mechanics, and handle raw video data. Outside mechanics, Suh and Tedrake [39]highlight the difficulty of learning physical inductive biases with deep models from raw data andinstead propose to encode mass conservation from pixels with a constrained linear model. Finally,
Noether’s theorem has been used [20, 42] to theoretically understand the optimization dynamics
during learning; unrelated to our goal of discovering inductive biases for sequential prediction.
Discovery of symmetries and conservation laws. There are several previous works which aim tolearn conserved quantities in dynamical systems from data. The approach of Schmidt and Lipson[35] focuses on candidate equations that correctly predict dynamic relationships between parts of thesystem by measuring the agreement between the ratios of the predicted and observed partial (time)
derivatives of various components of a system. A set of analytical expressions is maintained andupdated as new candidates achieve sufficient accuracy. More recently, Liu and Tegmark [26] discoverconservation laws of Hamiltonian systems by performing local Monte Carlo sampling followed bylinear dimensionality estimation to predict the number of conserved quantities from an explained ratio
diagram. Wetzel et al. [49] learn a Siamese Network that learns to differentiate between trajectories.
Contrary to both, Noether Networks do not need segmentations into trajectories with different
conserved quantities and can deal with raw pixel inputs and outputs. Other approaches to learning
symmetries from data include neuroevolution for learning connectivity patterns [37], regularizers
for learning data augmentation [6], and meta-learning equivariance-inducing weight sharing [54].
However, these approaches do not aim to learn conserved quantities in dynamical systems.
Neural networks to discover physical laws. There has been a growing interest in leveraging thefunctional search power of neural networks for better scientific models. Multiple works train a neuralnetwork and then analyze it to get information, such as inferring missing objects via back-propagation
descent [2], inspecting its gradients to guide a program synthesis approach [44], learning a blue-print
with a GNN [11], or leveraging overparametrized-but-regularized auto-encoders [45]. Others, such
as DreamCoder [15], take the explicit approach with a neural guided synthesis over the space of
formulas. Unlike these works, our method can readily be applied both to symbolic conservation laws
and to neural network conservation losses applied to raw videos.

C

Experimental details for scientific data

We use three different experiments from [21]: the ideal pendulum, the ideal spring, and the real
pendulum, under an Apache license. The first two come from simulated ODEs without noise, thelatter comes from a real data from Schmidt and Lipson [35]. It is worth noting that other datasetsfrom Greydanus et al. [21] (such as a planetary system) could not be included because the DSL
required too much depth to reach the conserved energy quantity. A better search, such as using
evolution, or a better DSL (such as those derived from many scientific formulas in DreamCoder [15])
could remedy this. Finally, note that an approach that searched over the same DSL encoding the loss
as a generic f (x0 , xt ) loss, not as a conservation |g(x0 ) − g(xt )| would require more than twice thedepth, thus not being able to cover the evaluated datasets.
For the ideal pendulum, input x = (p, q) ∈ R2 contains the angle q and momentum p of the
2 2ppendulum. The formula for the energy is H = 2mgl(1 − cos q) + l2m. Greydanus et al. [21] set
m = 21 , l = 1, g = 3, resulting in H = 3(1 − cos q) + p2 . Notice that a simpler conserved quantity
is p2 − 3 · cos q.
For the spring, input x = (p, q) ∈ R2 contains the displacement q and momentum p, and the system’sp2
energy is given by H = 12 kq 2 + 2m
. Greydanus et al. [21] set k = q = m = 1, resulting in
H = 12 (q 2 + p2 ), where units are omitted. Thus, q 2 + 1 · p2 is a conserved quantity, with 1 having
the appropriate units.
14

Since formulas in the DSL have physical meaning, each sub-formula carries its own associatedphysical units. When applying an operation, there is a check for validity (we can only input degrees
to a sinusoid, we can only add quantities of the same units, etc.). This allows us to significantly prune
the exponential space, as done in AI-Feynman [44]. The vocabulary of the DSL is the following:
Input(i): returns the i-th input, Operation: one of {+, −, ·, /, sin, cos, x2 }, Parameter(u):trainable scalar with units [u].
The number of formulas increases exponentially with the number of terms involved, leading to vast
search spaces. To save computation, we perform a pipelined approach that first generates all valid
formulas up to depth 7. This removes formulas whose operations take in incompatible physical units
and prunes equivalences of commutativity for the appropriate operations. We obtain 41,460 and
72,372 formulas for the pendulum and spring, respectively. Note that we do not force the conserved
quantity to have units of energy since in general not all conserved quantities are energies.
The resulting formulas are purely symbolic, with some trainable parameters still to be defined. We
check that they are approximately conserved by optimizing their parameters via gradient descentto have low variance within sequences of true data. We then do the same for random sequences; ifconservation is two orders of magnitude smaller for the true data, we accept it as an approximate
conservation. This measure is similar to others used before for conserved quantities [26], but it has
problems when the data is noisy, the minimisation is sub-optimal, or there are numerical issues. As
a result, we obtain 210 and 219 approximately conserved quantities for the pendulum and spring,
respectively.
Finally, we evaluate candidates expression for their usefulness as Noether losses. For each candidate
we try one inner step with a range of inner learning rates 10k for k ∈ {−3, −2.5, −2, . . . , 1}. To
save compute, we first obtain the top 20 candidate equations by the train performance when tailoring
the vanilla MLP with each loss. For those 20 candidates, we try them as meta-tailoring losses:
starting from the pre-trained vanilla MLP, we fine-tune it for 100 epochs using meta-tailoring witheach candidate Noether loss. We then evaluate the MSE using the fine-tuned model for long-term
predictions, keep the expression with the best loss in the training data, and report the long-term
prediction on the test data.
It is worth noting that we use a pipeline approach: first discovering approximately conserved
quantities, then pruning a subset of 20 that are most useful for tailoring and then finally pick the bestloss when used with meta-tailoring. This pipeline, along with the concrete numbers of 20 candidates
and 2 orders of magnitude comparing variance in real data vs. random data, were heuristically chosento speed up the search, without trying other hyper-parameters. For the real pendulum we foundthat two losses that were among the top 20 for tailoring diverged when used for meta-tailoring. We
conjecture this is the case because of the noise in the real data
For the ideal pendulum, in addition to the true formula, our method also finds equivalent losses thatthe initial pruning did not detect, such using a sum with a negative parameter: p2 + (−2.99) · cos(q)instead of p2 − 2.99 · cos(q).
For the real pendulum, it is worth noting that the energy candidate p2 − 2.4 cos(q) was proposed byhumans to fit the data, since the data is experimental. It could thus be slightly sub-optimal, which
could explain why Noether Networks improves its predictions. We also consider the possibility of it
being due to fluctuations in noisy data and it being a small dataset.
Experiments were performed on an NVIDIA Tesla V-100 GPU and 10 CPU cores, taking around 4
hours to run.

D

Experimental Details for Physics 101

We use a subset of the Physics 101 dataset [50] with videos from the ramp scenario, where various
objects are let go by a human hand at the top of a ramp. We could not find its associated license. There
are two ramp settings: 10 degrees and 20 degrees. To ensure the prediction problem is interesting andnontrivial, we only take examples with the 20 degree ramp since the object does not slide down theramp in many of the 10 degree examples. Additionally, we only take sequences with length at least 2
seconds, and use a frame rate of 15 frames per second when extracting frames to pass to the models.
We use videos recorded with the Kinect_RGB_1 sensor. Video frames are center-cropped at fullheight (1080 pixels) and downsample to obtain 128 × 128 images, and perform random horizontal
15

flipping for sequences. This preprocessing results in 389 possible sequences, and we take a random
80/20 train/test split. In the prediction task, we condition the model on two frames, and the model
must predict the subsequent 20 frames. To ensure that we extract segments of videos where the object
is moving, we take the middle 22 frames of each video clip. We use a mini-batch size of 2 for theseexperiments.
Our baseline model is the publicly-available implementation of SVG-LP [13], with the batch normalization layers replaced with layer normalization layers. This change is made because of the small
mini-batch size used in our experiments. As in the original implementation, the encoder and decoder
are based on VGG16 [36], the latent frame predictor is a 2-layer LSTM with hidden size of 128, andboth the prior and posterior are single-layer LSTMs that produce latent variables zt of dimension 64.
The learning rate used during the training of the baseline 0.0001. Teacher forcing is used during thetraining of both the SVG baseline — as done by Denton and Fergus [13] — as well as the Noether
Network’s embedding.
The meta-learned inner loss of the Noether Network is defined in Equation 2, with the learned
embedding gφ parameterized as a 2-layer convolutional network where each layer consists of a 5 × 5
convolution with 32 filters in the first layer and 64 filters in the second layer, a ReLU non-linearity,and 2 × 2 max pooling. There is a final linear layer which projects onto a 64-dimensional embedding
space, in which the MSE is computed for the inner loss.
The CNG RAD algorithm of Alet et al. [3] is used to perform meta-tailoring. In the Noether Network,
conditional normalization (CN) layers are inserted after each layer normalization layer in both theencoder and the decoder. The CN parameters are initialized to the identity transformation, andare adapted with a single inner step at prediction-time using an inner learning rate of 0.0001. The
embedding is trained in the outer loop with an outer learning rate of 0.0001, along with the rest of themodel parameters, excluding CN parameters. We report results obtained with meta-tailoring, where
all weights are randomly initialized. In the reported results, all models are trained for 400 epochs, atwhich point training has converged.
We ran all experiments with the Physics 101 dataset on an NVIDIA Tesla V100 GPU and 20 CPU
cores. Training the baseline until convergence took approximately 13 hours, and training the Noether
Network took approximately 2 days, 23 hours.

16