Towards fully covariant machine learning
Authors: Soledad Villar
David W Hogg
Weichi Yao
George A Kevrekidis
Bernhard Schölkopf
Venue: Accepted by TMLR ()

Abstract: Any representation of data involves arbitrary investigator choices. Because those choices are external to the data-generating process, each choice leads to an exact symmetry, corresponding to the group of transformations that takes one possible representation to another. These are the passive symmetries; they include coordinate freedom, gauge symmetry, and units covariance, all of which have led to important results in physics. In machine learning, the most visible passive symmetry is the relabeling or permutation symmetry of graphs. The active symmetries are those that must be established by observation and experiment. They include, for instance, translations invariances or rotation invariances of physical law. These symmetries are the subject of most of the equivariant machine learning literature. Our goal, in this conceptual contribution, is to understand the implications for machine learning of the many passive and active symmetries in play.
We discuss dos and don'ts for machine learning practice if passive symmetries are to be respected. We discuss links to causal modeling and argue that the implementation of passive symmetries is particularly valuable when the goal of the learning problem is to generalize out of sample. We conjecture that the implementation of passive symmetries might help machine learning in the same ways that it transformed physics in the twentieth century.

Published in Transactions on Machine Learning Research (01/2024)
Towards fully covariant machine learning
Soledad Villar"
soledad. villar edu
Deportment of Applied Mathematics ond Statistics, Johns Hopkins University Mathematical Institute for Data Science, Johns Hopkins University Flatiron Institute, a division of the Simons Poundation
David W. Hogg'
dawid.hogg Smmg..
Center for Cosmology and Particle Physics, Department of Physics, New York University Maa Planck Institute for Astronomy, Heidelbery Flatiron Institute, a division of the Simons Foundation
Weichi Yao
weichiy@umich. edu
Michigan Institute for Data Science, University of Michigan
George A. Kevrekidis
gkeurek1 @ifuu edu
Deportment of Applied Mathematics and Statistics, Johns Hopkins University Los Alamos National Laboratory
Bernhard Schölkopf
bs @ww.bbbngn..pp....
Maz. Planck Institute for Intelligent Systems and ELLIS Institute, Tübbingen
Reviewed on OpenReview: https: // oppnrevvee. net/ forum? id= g lUmp YuXg
Abstract
Any representation of data involves arbitrary investigator choices. Because those choices are external to the lata-generating process, each choice leads to exact symmetry, corresponding to the group of transformations that takes possible representation to another. These are the passive symmelries, they include coordinate freedom, gauge symmetry, and unils covariance, all of which have led to important results in physics. In machine learning, the most visible passive symmetry is the relabeling or permutation symmetry of graphs. The aclive symmelries are those that must be established by observation and experiment. They include, for instance, translations invariances or rotation invariances of physical law. These symmetries are the subject of most of the equivariant machine learning literature. Our goal, in this conceptual contribution, is to understand the implications for machine learning of the many passive and active symmetries in play. We discuss dos and don'ls for machine learning practice if passive symmetrics are to be respected. We discuss links to causal modeling and argue that the implementation of passive symmetries is particularly valuable when the goal of the learning problem is to generalize out of sample. We conjecture that the implementation of passive symmetries might help machine learning in the same ways that it transformed physies in the twentieth century.
1 Introduction
Many important ideas in machine learning (ML) have come from been inspired by mathematical physics. These include the kernel trick (Courant & Hilbert, 1953; Schölkopf & Smola, 2002) and the use of statistical mechanics techniques to solve probabilistic problems (Hastings, 1970; Gelfand, 2000). Here we suggest another connection between physics and ML, which relates to the representation of observables: When features and labels are represented in a mathematical form that involves investigator choices, methods of MI (or any
"joint first author

---

Published in Transactions on Machine Learning Research (01/2024)
relevant model, relationship, method, or function) ought to be written in form that is exactly equivariant to changes in those investigator choices. These ideas first appear in the physics literature in the 1910s (most famously in Einstein 1915). They are given in the introduction of The Classical Groups (Weyl, 1946) as a motivation to study group theory. Literally the first sentences of Modern Classical Physics (Thorne & Blandford, 2017) are
[..] central theme will be a Geometric Principle: The laws of physics must all be expressible as geometric (coordinate-indeeedeen) and reference- wwwteeerrrm--ieeeedeeeeee relationships between geometric objects (scalars, vectors, tensors, ...) that represent physical entitics.
This Geometric Principle leads to the important physical symmetries of coordinate freedom and gauge symmetrry a small generalization would include what we will refer to as units covariance (see the glossary in Appendix A). Each of these symmetries has led to fundamental results in physics. Some of these ideas are also exploited in ML, in particular in the geometric deep learning literature (Bronstein et al., 2021; Weiler et al., 2021a). We argue in this purely concept ual contribution that analogs of these symmetries could have an impact on ML, and thus increase the scope of group-equivariant methods in ML.
In natural science, there are two types of symmetrics (sce, for example, Section 4.1 of Rovelli & Gaul 2000). The first kind is passive, arising from the arbitrariness of the mathematical represenlation described above. An example familiar in ML is the equivariance of functions on graphs to the relabeling of the graph nodes. This is an exact, passive symmetry; graph neural network architectures (GNNs) build this passive symmetry in by design (Bruna al., 2013; Duvenaud al., 2015; Gilmer et al., 2017; Hamilton, 2020). See Huang et al. (2023) for a recent work describing passive and active symmetries in GNNs. An example familiar to physicists is what we call units covariance, which is the requirement that any correct description of the world has inputs and outputs with the correct units (and dimensions; see Appendix A for definitions). This symmetry is extremely powerful (see Section 3).
The second kind of symmetries is active. These are the ones that must be established by observations and experiments. The fundamental laws of physics do not seem (at current precision) to depend on position, orientation, or time, which in turn imply conservation of momentum, angular momentum, and energy (the celebratod thcorem of Nocther 1918). Active symmetrics like these are empirical and could (in principle) be falsified by experimental lests. Both active and passive symmetries can be expressed in termss of group groupoid actions and equivariances, but their epistemological content and range of applicability are very different.
In this contribution, we argue that passive symmetries apply lo essentially all data analysis problems. They have implications for how we structure ML methods. Although we provide some examples, most of our contributions are conceptual.
Our contributions:
We introduce the concept of passive and active symmetries to ML. We give a formal definition of passive and active symmetries in terms of group actions and explain how passive symmetries are always in play in problems using real-world data.
We provide guidance on how to structure ML models so that they respect the passive symmetries. We call out some current standard practices that can prevent models from obeying symmetries. We give particularly detailed guidance in the context of data normalization.
We illustrate with toy examples how enforcing passive symmetries in model structure and in normalization can improve regressions.
We demonstrate that imposing passive symmetries can lead to the discovery of important hidden objects in data problem. We show that a passively equivariant problem can often be made actively equivariant by the introduction of latent variables with appropriate group- theoethe properties.
2

---

Published in Transactions on Machine Learning Research (01/2024)
We draw connections with causal inference. One is that all causal graphs and mechanistic models are constrained to be consistent with the passive symmetries. Another is that the determination that a data problem has all the inputs necessary to express the symmetry exactly looks like a causal inference. We also explain how active symmetrics can be expressed in terms of interventions.
We provide glossary (in Appendix A) that can be used to translate terminology and relate ideas between physics and ML.
2 Passive symmetries
Passive symmetries arise from redundancies free parameters investigator choices in the representation of data. dhey.They are to be contrasted with the active symmetries, which arisc from observed or empirical invariances of the laws of physics with respect to parameters, like position, velocity, particle labeling, or angle. Passive symmetries can be established with no need of observations, as they arise solely from the principle that the physical world independent of the mathematical choices we make to describe it. The groups involved in coordinate freedom can be large and complicated (for example, groups of reparameterizations)
In contrast, big part of the literature equivariant ML is implicitly or explicitly looking at active symmetries. This is possibly because in most problems the coordinate system is fixed before the problem is posed, and both training and test data are expressed in those fixed coordinates. If data set is made with a fixed coordinate system, but still exhibits an observable invariance or equivariance with respect to (say) rotations, then that represents an active symmetry. However, cases of exact active symmetries are rare; they only really appear in natural-science contexts like protein folding or cosmology. For example, in a protein folding problem, the way the protein folds may not depend on its orientation in space (rendering the problem actively 0(3) equivariant). This finding relies the (empirical) observation that the local gravitational field (on Earth), for example, does not affect the folding. This may be approximately true assumedd or experimentally established; it is an active symmetry. In contrast, the fact that the protein folds in a way that doesn't it depend on the coordinate system chosen to describe it is absolutely and always true; it is not experimentally establishod; it is a passive symmetry.
The relationship between active and passive symmetries is reflected the relationship between what are sometimes called active and passive transformations, or alibi and alias transformations, depicted in Figure An active or alibi transformation is one in which the objects of study are moved (rotated, translated, interchanged, etc.). A passive or alias transformation is one in which the coordinate system in which the objects are represented is changed (rotated, translated, relabeled, etc.). Mathematically, the two seem very similar: For example, how do we know whether we rotated all the vectors in our problem by 30 deg, or else rotated the coordinate system used by -30 deg? The answer is that if you rotated absolutely all the vectors (and tensors) your problem, including possibly many latent physical vectors, then there would be no mathematical difference. However, this not possible in practice. In real problems, where some vectors can't be actively rotated (think, for example of the local gravitational-fiel vector, or the vector pointing towards the Sun), or some may not be known or measurable, the two kinds of transformations
different.
This problem that a passive symmetry only becomes an active symmetry when it is possible to transform every relevant thing correspondingly seaespoondiglyy-uugeeessss that it might be hard to implement or enforce an exact passive symmetry in a real anta--naalsii problem. It requires us to incorporate all relevant contextual information. How do we know if all relevant fcatures are part of our data set? We could perform the protcin-folding experiment in a closed, isolated environment to make sure no external forces are in play? This is impossible for many practical applications, and furthermore, there could still exist fundamental constants that are not part of our model or knowledge (see Section 5). Another approach is to perform the experiment multiple times after actively putting the molecules into different orientations. If the protein folds differently, we learn that the problem is not symmetric with respect to the 3d coordinates of the molecule, and therefore when a rotation is performed there must be at least one more vector that needs to be rotated as well (for instance, the gravity vector or the eigenvectors of some stress tensor, say). This identification of all necessary inputs to establish the passive symmetry is similar to the problem of performing interventions to learn the existence of confounding factors in causal inference (see Section 6).
3

---

Published in Transactions on Machine Learning Research (01/2024)
2
Z
X
X
x
Figure 1: Figure depicting the difference between active and passive transformations. (Left panel): The passive alias transformation corresponding to rotation of the coordinates through an angle 0 in the xy-plane. The equivariance of the dynamics with respect to this transformation is a passive symmetry. (Right pancl): The active or alibi transformation corresponding to a rotation of the double pendulum state through an angle -0 in the xy-plane. Equivariance with respect to this transformation is an active symmetry.
Once a passive symmetry-ann and all relevant contextual information-is identified, we want to write the data analysis problem or learned function such that it is exactly equivariant with respect to the relevant group: If the coordinate system of the inputs is changed, the coordinate system of the output should change correspondingly. ML methods that are not constrained to respect passive symmetries are doomed to make certain kinds of mistakes. We will provide some examples in Section and Scction 9.
The most restrictive form of the Geometric Principle quoted in Section 1 states that physical law must be written in terms of vectors, tensors, and (coordinate-invate invariant) scalars. These objects can only be combined by rules set out in the Ricci calculus (Ricci & Levi-Civita 1900; somctimes called Einstcin summation notation, Einstein 1916). This calculus was introduced to make objects equivariant to coordinate diffeomorphisms on curved manifolds, but it applies to 0(3) and Lorentz symmetry as well. In the Ricci calculus, objects are written in index notation (a scalar has no indices, a vector has one index, and a k-tensor has indices), outer products are formed, and only certain kinds of sums over pairs of indices are permitted. When the inputs function are scalars, vectors, and tensors, and the function conforms to the rules of the Rieci calculus, the function will produce geometric output (a scalar, vector, or tensor, depending on the number of unsummed indices), and the function will be precisely covariant to rotations and reflections of the coordinate system. This is how a large class of passive symmetries is enforced in physics contexts.
There are many other passive symmetrios, including coordinate diffcomorphisms, reparameterizations (includ- ing canonical transformatious in Lagrangian and Hamiltonian systems), units covariance (see Section 3), and gauge freedom. Some are easy to implement in ML contexts and some are difficult; not all group equivariances have practical implementations useful for ML methods available at present. Convolutional neural networks (LeCun et al. 1989) are very effective even when strict translation symmetry is not active; this effectiveness might be related to the connection to the passive translation symmetry of the image pixel grid.
Sometimes it is difficult to tell whether a symmetry is active or passive. For example, the law of gravity is explicitly (3) invariant: If you rotate all of the position vectors, you rotate all of the forces correspondingly too. Tf you rotate all of the initial conditions, you rotate all of the gravitational trajectories correspondingly too. But this symmetry also a passive symmetry: The law of gravity does not depend on the orientation (or handedness) of the coordinate system.
If an active symmetry is in play, the physical law can often be written in terms of invariants such that the symmetry is enforced by construction or definition, thereby making it also a passive symmetry. The discovery of general relativity (Einstein, 1915) can be seen as the discovery that active symmetries -the speed of light is the same for all observers, and the gravitational mass equals the inertial mass can be converted
"It should be noted here that with the word "vector" and "tensor" here we are making specific tochnical reference to true vectors and tensors in 3-space, subject to passive 0(3) symmetrics, like (physical) velocities, accelerations, and stress tensors, We are not including arbitrary lists or tables of data or coefficients, which are sometimes called "vectors" and "tensors" in ML contexts, See Appendix A for more detail.
4

---

Published in Transactions on Machine Learning Research (01/2024)
into passive symmetries: When laws are written in a form consistent with the passive symmetries, they automatically enforce the corresponding active symmetries. Indeed, the equations of general relativity were found by looking at every differential equation (to some degree) consistent with the passive symmetry of coordinate diffeomorphisms on curved spacetime manifolds until one was found that reduced to Newtonian gravity in the weak-field limit.
We stress that these kinds of insights had a big impact on physics (Earman & Glymour, 1978); only few years prior to 1915, such considerations would have seemed highly umusual; now this form of argument is canon in theoretical physics (see, for example, Zee 2016). That evolution motivates this contribution; passive symmetrics do not feature in most of today's ML practico-if the development of physics is any indication, their polential could be siguificant. Some valuable ML research has started recently along these directions (Weiler et al., 2021a; Bronstein et al., 2021).
3 Example: Units covariance
Perhaps the most universal passive symmetry is units covariance- the behavior of a system does not depend on the units system in which we write the measured quantities. It is a passive symmetry with extremely useful consequences.
Consider mass m near the surface of the Earth, close enough to the surface such that the gravitational field can be considered to be determined by a constant (not spatially varying) vector with magnitude g and direction downwards. Question 1: If this mass m is dropped (released at rest) from a height h from above the ground, how much time T' does it take to fall to the ground? Question 2: If this mass m is launched from the surface at a velocity of magnitude u at an angle 0 to the horizontal, how much horizontal distance L will it fly before it hits the surface again? Assume that only m, enters the solution; 1 In particular, assume that, the height h and the velocity are both small enough so that air resistance, say, can be ignored.
The answers to these questions are almost completely determined by dimensional (or covaricovariance) arguments (see Appendix A for definitions). The mass m has units of kg, the gravitational acceleration magnitude g has units of ms-2 the velocity magnitude v has units of ms-1, the time T has units of and the lengths/ h and L have units of m. The angle 0 is dimensionless. The only possible combination of m,g, h that has units of time is a / /h/g, where a is a dimensionless constant, which doesn't 't depend on any of the inputs. The only possible combination of m,g, 9, v, that has units of length is B(0) v²/g, where 3(0) is a dimensionless function of a single dimensionless input. That is, both Question 1 and Question can be answered up to a dimensionless prefactor without any training data. And both of those answers don't 't depend in any way on the input mass (which is the fundamental observation that leads to general relativity; Einstein 1915).
This shows that a function-and even a fundamental physical relationship-cat sometimes be inferred from units covariance only, that is, from a purely passive symmetry, combined with the empirical knowledge that the solution is independent of all other observables (which itself is a causal assumption). Unit covariance is widely used in numerical methods, and it has been discussed in ML previously (Villar et al., 2023; Bakarji et al., 2022; Xie et al., 2022). It can help with training, predictive accuracy, and out-of-sample generalization.
4 Formal definition
Consider L' to be space of all possible physical states of a specific system (for instance x E Z could be the positions, velocities, masses, spins, and charges of a set of particles, at a time or possibly at a set of times). We consider maps {0;; 2 + H) Tes where H is the space of encodings (or representations) of the values of those positions, velocities, masses, and so on, in some units system and some coordinate system. That is, any element E H will be a list of real values of vector and tensor components and scalars. Different maps i will have (in general) different coordinate origins, different axis orientations, and different units of measurement.
We will return to this seemingly innocuous point below. We note that it is an empirical statement, which will turn out to have rather significant implications.
5

---

Published in Transactions on Machine Learning Research (01/2024)
Provided that every i, records all of the information necessary to describe the state 2 (or, equivalently, every element = E H contains all of the information necessary to describe the state), for any two encodings e and by there is an invertible morphism B3 H + Ht that makes the diagram (1) commute.
id
2
(1)
H
H
The passive symmetries comprise the group P of invertible morphisms P] that makes the diagram commute. The group P consists of all the possible changes of units or coordinates or automorphisms between encodings or observables.
For example, take L to be the space of states of a protein molecule. Each map & could encode the positions of each of its atoms in some coordinate system. The passive symmetries include reordering of the atoms, changes of the coordinate system by any invertible morphism, and changes to the units in which positions (lengths) are measured.
Active symmetries, on the other hand, can be thought of as transformations of the world that preserve an obsorvable property. They involve interventions in the physical system, and therefore they are typically empirical and approximate. Not every passive symmetry corresponds to an active symmetry, nor vice versa.
To define the active symmetries we fix + 2 H an encoding as above; function F H + 3 where F is function that delivers possible observable prediction or system property; and group G' that acts the H and y via actions 8 and respectively. An element E 3 contains some prediction observable of interest in the system, such as the energy or the future time evolution (trajectory). Given a group element E G we might draw this commutative diagram:
F
Φ
H
[(g) T H
B(g)
p(g)
(2)
F
We say that the tuple (G, P, represents active symmetry of L. if the diagram (2) commutes.
To continue the example in which 2 is the space of states of protein molccule: F computes the total electrostatic energy of the protein, there an active symmetry corresponding to the group G = 0(3) of all the position
rotations and reflections, which (gg) the standard matrix representation of g applied vectors, and p(g) the identity operator for all g G. This symmetry is active in the sense that it says how something changes (the energy, and it doesn't) when the molecule's 's state is changed (it is rotated in space).
The alias alibi distinction (Section 2) maps onto the correspondence of related passive and active symmetries. The active symmetries correspond to interventions in the system, while passive symmetries are purely in the realm of representation. Most of the equivariant ML literature is focused on active symmetries. Projects begin with the question: What active symmetries are in play in this system? In what follows, we reserve the word "cquivariance" for active symmotries and use the word "covariance" for passive symmetrics, consistent. with the use of the word "covariance" in physics contexts (see Appendix A).
5 Experiments and examples
Black-body radiation: An important moment in the history of physics was the discovery that the electromagnetic radiation intensity Bx (energy per time per area per solid angle per wavelength) of thermal black-body radiation can be described with simple equation (Planck, 1901)
BA(A) = 2h
he exp X TT
(3)
6

---

Published in Transactions on Machine Learning Research (01/2024)
where h is Planck's constant, is the speed of light, À is the wavelength of the electromagnetic radiation, kt is Boltzmann's 's constant, and T is the temperature. In finding this formula, Planck had to posit the existence (and units) of the constant h = 6.62607015 x 10-34 kg kg m²s- (Planck's original value was presented with less precision and in ergs, which are different units but the same dimensions). Prior to the introduction of h, the only dimensionally acceptable expression for the black-body radiation intensity was (A) = c T/X*, which is the long- wng-wwvelength (infrared) or high-temperature limit of (3). Planck's discovery solved the "ultraviolet catastrophe" of classical physies. This is the problem that, classically, the black-body spectrum, or any thermal object, ought to contain infinite numbers of excited modes at short wavelengths, or high frequencies, and thus infinite energy density. Planck's solution seeded the development of quantum mechanics, which governs the behavior of all matter at small scales, and which cuts off the ultraviolet modes through quantization of energy.
Planck's problem can be solved almost directly with the passive symmetry of units covariance. That is, the exponential cut-off of the intensity appears at a wavelength set by the temperature and a new constant, that must have units of action (or action times c, or action divided by k, or one equivalent in terms of the lattice of dimensional features, see Appendix B and Villar et al. 2023).
In Figure 2 (left) we perform the following toy experiment: We generate noisy samples of intensities as a function of wavelength and temperature according to (3), and the learning task is to predict the intensity for different values of wavelengths and temperatures. We perform three experiments, described in Appendix B (A) a units-covariant regression (employing the approach of Villar et al. 2023) using only X, X, k; (B) a units covariant regression with an extra dimensional constant found by cross-validation; and (C) a standard multi-layer perceptron regression (MLP) with units constraints. Our results show that no units-covariant regression for the intensity as function of X, T, c, k can reproduce accurately the intensity Bx. However when the regression is permitted to introduce new dimensional constant (but enforce exact units-covariance given the new constant), it finds a constant with units that is consistent with h (or la times a combination of e and k). The units-covariant model with an extra constant outperforms the bascline MLP. Naïvely this suggests that passive symmetry brings new capabilities.
Springy double pendulum: The double pendulum connocted by springs is a toy example often used in equivariant ML demonstrations (Finzi et al., 2021; Yao el al., 2021; Villar el al., 2023). The final conditions (position and velocities of both masses after elapsed time T') are related to the initial conditions (position and velocities of the masses at the initial time), and the dynamics is classically chaotic. This means that prediction accuracy, in the end, must be bounded by considerations of the fundamental mathematical properties of dynamical systems.
The system is subject to a passive 0(3) symmetry (equivariance with respect to orthogonal coordinate transformations), an active O(2) symmetry (equivariance with respect to rotations and reflections in the 2D plane normal to the gravity), and an active time-translation symmetry, arising from the fact that the total energy is conserved. The 0(3) symmetry is passive, because it is guaranteed by the fact that all vectors must be described in a coordinate system; nothing physical can change as the vectors undergo passive transformations because of coordinate-systen changes. The O(2) symmetry is active, because it is an experimental fact that if the initial conditions are changed by an active or alibi rotation in the plane perpendicular to gravity, the dynamics and final state rotate accordingly. The O(2) active symnetty corresponds to the set of transformations in 0(3) that fix the gravity vector.
The passive 0(3) symmetry requires that the coordinates of all relevant vectors are transformed identically, the positions and momenta of both masses and the gravily vector. If the model doesn't 't have access to all relevant vectors as inputs then the predictions will not necessarily be 0(3) equivariant. We perform an experiment in which we predict the dynamics of the double pendulum using -3uivequu models. The symmetries are implemented by converting the network inputs (scalars and components of vectors) into invariant scalar quantities according to the Ricci calculus (which explicitly encodes O(3)), building the model in the space of the invariant scalars (as per Villar et al. 2021). These -i3)-invariant scalars are used to parameterize an 0(3) -33)-invari Hamiltonian, and the -quiviaaaar dynamics are predicted by integrating the Hamiltonian using a symplectic integrator, as proposed by Sanchez- Sanchez-Gonzalez et al. (2019) and implemented in Yao et al. (2019). More details are given in Appendix C.
7

---

Published in Transactions on Machine Learning Research (01/2024)
units covariant without extra dimensional constaat,, RMSE-6.35
units covariant with extra dimensional constant, RM5E-0.39
5000
standard MLP (no covariance). RMSE=0.64
10
-04
-46
-1
68
-08
9
.52
-24
4
1026
intenskl
$10
10
-0.
10"
-10
-2.
6.
-1.
0
0.8
-18
.32
10a
Groun with
Keowing
....
Leomecii
g
Learued-g
g
10-4
10-1
10
10 10-4
State. RelErr
0002 + 0.0004
0034 + 0.0011
.3100 0014
wavelength
Figure 2: (Left panel) We predict the intensity of black body radiation as a function of wavelength and temperature. For all experiments, we use an MLP consisting layers with 20 hidden units each. The standard MLP uses wavelength and temperature as features and docsn't require the output be dimensionally correct. The wnils covariand without extra constand learns scaling of the only dimensionally correct object one can construct with inputs X,T,c, C, & (see description main in text). The units covariant with extra dimensional constant incorporates a constant with units kkg, m, K] E (-2. 0, as an input feature, it performs a units covariant regression with the original features X, T, c, k, and the extra constant. It then selects a constant with a low validation error and reports the results on the test set. The constant learned for the depicted plot is 1.61e52 kg l -a------pp --- which is the same units and similar magnitude to the valid physical combination 222 (Right panel) Performance of learning the dynamics of the springy double pendulum. We consider the three models (described in the main text): (Known-g) an (qu)-aqquva model where the gravity is input to the model, (No-g) -33iciuvivariant model where the gravity is not given, and (learned-g) (3))eqquivariant model that uses the position and momenta well as an unknown vector that the model learns. The results show that -uuieequue permits the learning of the gravity vector from data with only minimal impact on performance. See Appendix C for more detailed description the experiment.
The models considered here are (Known-g) -qqiivaran model that has the positions, momenta, and gravity vector all as features (similar to the models in Villar et al. 2021; Yao et al. 2021): (No-g>an an O(3)- equivariant model that is missing the gravity vector as an input feature; and (Learned-g) an (33)eqquivariant model that has the position, momenta and an extra unknown vector as features. The latter model optimizes model weights along with the unknown vector. In the right panel of Figure 2 we show the performance of the three models. We remark that in (Learned-g), the learned vector in the performed experiments was nearly parallel to the true (but unknown) gravity vector g; the angle between the learned and true gravity vector ended up at 0.00016 radians.
Non-covariant data normalization In section 9 we discuss covariant vs non-covariant data normalization procedures. In appendix C we provide experiments (in the springy double pendulum setting) that empirically show that non-covariant data normalization can hurt the performance significantly, whereas covariant data normalization does not.
6 Connections with causality
There is nothing statistical about the notion passive symmetries, and thus everything we have said above also applies to causal models (Peters et al., 2017). There are, however, a few comments specific to causality.
The passive symmetry discussed in Section and indeed all passive symmetries pertaining to the (hard problem of) inference of causal structure: treating g constant, structural causal model with the following vertices: (a) an initial value of (b) value of m, chosen
also deliver information construct
8

---

Published in Transactions on Machine Learning Research (01/2024)
independently, and (c) final value of L, affected by noise term 8. Time ordering implies that possible causal arrows are from v, 0 to L. As argued above, dimensional analysis rules out the arrow m + T,, leaving us with the non-trivial result that in the causal graph, only u, 0 cause L. As in Section 3, this conclusion can be reached without any training data or interventions.
That said, dimensional analysis makes a strong assumption, which is that all relevant quantities for predicting L have been specified in the list m, 9, v, O. For example, if the projectile is large enough or the speed e is high enough, air resistance will come into play, and the size of the object and the density of air will enter, bringing new variables and new combinations of variables that matter to the answer. This difficulty is related to the problem in causal inference of knowing or specifying all possible confounding variables.
This can also be linked to the notion of experimental interventions. Suppose we assumee that only certain quantities come into the solution (say, m, g, h). How would we confirm this in practice? In essence, this is not a probabilistic statement, but one about the behavior of system under interventions. A set of experiments can indicate that a certain outcome (or effect variable) depends on a certain set of input (cause) variables but is independent of certain other potential cause variables. In this case, the physical law is not inferred from dimensional arguments alone, but from combination of dimensional and causal arguments.
Even if interventions are not available (for for example), physicists trying to infer a law will not do so based (purely) on input-output data: they will have prior knowledge from related problems informing them as to which variables are relevant. For example, we may know from having previously solved a related problem that we expect a problem to depend on g. This is a form of qualitative transfer that we expect will also become relevant for model transfer in ML (Rojas-Cardl) Carulla et al., 2018).
Finally, we remark that causal language appeared above in Section 2 where we implicitly contrast the active symmetries with the passive symmetries in terms of interventions: The active symmetries are those that make predictions for experiments in which interventions have been made (the molecule has been rotated with respect to the gravitational-filld vector, for example).
7 Connections to current ML practice
Most present-day ML implementations don't impose exact symmetries. Sometimes they approximate equivari- ances by means data augmentation (Chen et al., 2020; Huang et al., 2022). In this work we mostly focus on exact symmetrics: Given data spaces X and Y and group G acting on X and Y, equivariant ML restricts the function space to those satisfying flg x) = g f(x) for all f E 8, ge approaches to perform optimization in the space of equivariant functions:
E G, x E X. There are three main
Parameterizing the space of equivariant functions by extending the notion of group convolutions or weight sharing (Kondor & Trivedi, 2018; Cohen & Welling, 2016; Weiler et al., 2021b).
Explicitly parameterizing the space of equivariant functions via equivariant linear layers (via irreducible representations (Thomas et al., 2018; Geiger & Smidt, 2022; Kondor, 2018), or otherwise (Maron et al., 2018; Finzi et al., 2020; 2021).
Finding a set of invariant features and expressing invariant/equuivriiae functions in terms of those features. Related ideas have been studied in differential geometry and invariant theory (Fels & Olver, 1998), and recently have been used in machine learning (Villar et al., 2021; Blum-Smith & Villar, September 2023; Cahill et al., 2022; Dym & Gortler, 2022; Kaba et al., 2023). This can also be done by imposing symmetry as a soft constraint (Shakerinava el al., 2022; Gupla el al., 2023).
The three approaches achieve similar goals, but their practical implementation may be dramatically different. For example, efficiently computing a complete generating set of invariant/equivaiinn features may be prohibitive, especially for large finite groups. On the other hand, in some cases, one needs to go to high-order tensors for the linear layers be expressive enough, and this may be hard to construct. More often, it may be possible to construct such a famíly, but may lack proof that they are universal approximators of all invariant/ cquivariant functions within some well-defined context, even in limiting sense.
9

---

Published in Transactions on Machine Learning Research (01/2024)
Even the non-trivial diffeomorphism symmetries of general relativity have been considered for ML (Weiler et al., 2021a). And aside from the previously mentioned results in Convolutional/Can another example of successful exact universal parametrization of a family of functions is the implementation of symploctic networks, which exactly preserve a differential 2-form, the symplectic form, on the associated manifolds (Jin et al., 2020; Burby et al., 2020). Their use most relevant the study of Hamiltonian systems.
Neural Networks,
Overall, equivariant ML models can predict the properties and behaviour of physical systems (see Cheng et al. 2019), and have plenty of scientific applications (Batzner et al., 2022; Musaclian et al., 2022; Stärk et al., 2022; Yu et al., 2021; Wang et al., 2022). Interesting theoretical developments analyze the implicit bias, generalization error, and sample complexity of equivariant ML models (Petrache & Trivedi, 2023; Tahmascbi & Jegelka, 2023; Lawrence et al., 2021; Bietti et al., 2021; Elesedy & Zaidi, 2021; Elesedy, 2021; Huang et al., 2023; Mei et al., 2021).
8
Dos and Don'ts
MacKay famously wrote (see Muldoon 2021)
Principal Component Analysis is dimensionally invalid method that gives people delusion that they are doing something useful with their data. If you change the units that one of the variables is measured in, it will change all the "principal components"
This comment is aligned with our mission, but also misleading: If a rectangular data set contains only data with identical units (that is, all features of all records have the same units), then PCA does exactly the right thing. That said, if a rectangular data sot has features with different units (for example, if every record contains a position, a temperature, a voltage, and a few intensities), then indeed the output of PCA will be extremely sensitive to the units system in which the features are recorded, and thus not invariant to the units choices.
Consider a kernel function with inputs that are lists of features with different units. If the kernel function involves, say, an exponential of a sum of squares of differences of the input features, the output of the kernel function cannot obey the passive symmetry of units covariance. Quantities with different units cannot be summed, and dimensional quantities cannot be exponentiated. On the other hand, if a kernel function can be chosen that is units covariant, then the result of kernel algorithm can in principle be covariant. These considerations are relevant for the maximum margin hyperplane in kernel support-vector machines (Boser et al., 1992), eigenvectors in kernel PCA (Schölkopf & Smola, 2002), or Gaussian processes (Williams & Rasmussen, 2006).
Learning involves optimization. Optimization is of a scalar cost function. If passive geometric groups are in play, like 0(3), the parameters that are explicitly or implicitly components of vectors can only be combined into the scalar objective through the Euclidean norm. Otherwise the scalar objective isn't scalar in the geometrie sense of "invariant to O(3",, and the optimization won't return result that is invariant (or equivariant) to 0(3). Similarly, if the components of the vector are normalized differently before they are summed in quadrature, the objective won't 't be invariant to 0(3). And similarly, if all the different contributions to the objective aren't converted to the same units before being combined, the model won't be units covariant. The common practices of making objectives with functional forms other than Euclidean norm, normalizing features with data ranges, and combining features with different units, all make common ML. methods, by construction, inconsistent with the passive symmetrics in play. We say more about normalization below in Section 9.
Neural nets, in their current form, violate many rules. For example: Transcendental functions like exp () and arctanh () and most other nonlincar functions can only be applied to scalars-thal is, not components of vectors or lensors but only scalars and only dimensionless. That means that the nonlinearities in neural networks are (or should be) implicitly predicated on the weights removing the units of the input features, and the linear combinations performing some kind of dot products on the inputs. That, in turn, mcans that the internal weights in the bottom and top layers of a neural network implicitly have geometric properties and units. They have geometric properties and units such that the latent variables passed into the noulinear
10

---

Published in Transactions on Machine Learning Research (01/2024)
functions are dimensionless scalars. Because they have these properties, trained neural network cannot be covariant in the end, unless the inputs and outputs are already covariant scalars.
There are exceptions to the restrictions on nonlinear functions: If nonlinearities are mathematically homoge- neous, as it is for a pure monomial, or for the RELU function, dimensional scalars (but not vector or tensor components) can be taken as inputs. Tt is interesting to ask whether the success of RELU in MT. might be related to its homogeneity.
Above we said that the weights in neural network implicitly have geometric properties and units. What are these? Imagine, say, at the input layer, that three of the inputs are the components (V1, U2, 02) of a velocity vector, and, at the next layer, these have been multiplied by weights, summed, subtracted from a threshold, and passed into a nonlinear function (such as a sigmoid). Given that a nonlinearity is in play, implicitly the output of the multiplication by weights and summation a dimensionless scalar is inconsistent with the passive symmctrics of ((3) covariance and units covariance for a nonlincarity to be applicd to anything other than & dimensionless scalar. This condition will only be met if implicitly the three weights (Wi, W2, W3) that multiply these vector components are themselves the components of an (33)-covariant vector with units of inverse velocity. If the network has any nodes in the next layer that have graph connections (weights) to only one or two of the three components (V1, 22 02), that is, if the network is sparse in the wrong ways, these implicit conditions cannot be met. The passive symmetries thus also put conditions on network architecture or graph structure.
L1 and Ls norms are often inconsistent with the passive symmetries. This is because the sum of absolute values of input components, and the maximum of inputs, are rarely either geometrically, from a units perspective, covariant. There is rare exception if all features have the same units, and none of the features are components of geometric objects (they are all dimensionless scalars). Similarly, regularizers favoring flat loss minima (Hochreiter & Schmidhuber, 1997; Dinh et al., 2017; Petzka et al., 2021) are often not units covariant, changing their values under certain weight transformations that leave the overall function invariant. If reformulated as a regularizer that is a covariant function of the training points, this problem vanishes (von Luxburg al., 2004). Data normalization, batch normalization, and layer normalization are all generally brutal and often violate model equivariances (Aalto et al. 2022 and see Section 9).
Finally, we mention that passive symmetries play crucial role also when it comes to latent variable models, since unobserved latent factors usually come with a large class of allowed gauge transformations (permutations, rotations in the latent space, and coordinate-wise nonlinear transformations) which should be incorporated correctly when studying notions of identifiability (Khomakhem et al., 2020; Buchholz et al., 2022).
9 Example: Normalization
To make contemporary neural network models numerically stable, it is conventional to normalize the input data, and possibly also layers of the network with either layer normalization or batch normalization. This normalization usually involves shifting and scaling the features or latent variables to bring them closer to being zero mean and unit variance (or something akin to these).
For our purposes here, let's focus on data normalization and assume that it works as follows: The training data contains features X which are N x M, where N is the number of training data examples, and M is the number of features per data point. In the simplest possible form, the training data X are given shift and scaling as
o' (X-1),
(4)
where a is some scale or list of M scales derived from the features in the training data set, and μ is some shift. or list of M shifts derived from the features in X. It is not uncommon for a to be root-variance of the features (or mean absolute deviation other distribution-widtt. mcasure) and for μ to be mean median.
Naïve normalization like this will in general break the passive symmetries of geometry and units (Aalto et al., 2022). For one, different individual features in X will have different units. It does not make sense to add or average (nor add nor average the square) features with different units. For another, if subset of 3
11

---

Published in Transactions on Machine Learning Research (01/2024)
features in X are the components of 3-vector subject to 0(3) symmetry a subset of 9 features in X are the components of a tensor subject to ((3) symmetry, these components cannot be summed or medianed without violating 0(3) equivariance (and thus passive-symmetry covariance), nor can they be independently scaled without violating O(3).
What should be done instead? For one, the elements of the training data X that correspond to 3-vector components cannot be all treated monolithically in the computation of the shift μ and scale o. Instead, the vectors must be dotted into themselves each individually to construct scalar norms. Those scalar norms can subsequently be summed or medianed averaged or square rooted. Usefully, the root-mean-square of the components of a 3-vector can be seen as the square root of 1/3 of the dot product of the vector with itself. At application time, the three components of any 3-vector must be scaled identically, not independently, otherwise the vector is (in general) being arbitrarily rotated. In this 3-vector case, the shift μ cannot be a single number but instead, the shift must. itself be 3-vector which is an (3))-invivrriant function of the input vectors. That's natural in many but not all normalization methods in use at present.
For another, the features in X that correspond to the elements of tensors must not have their components treated equally in the computation of the shift μ and scale σ. Instead, the tensors must have their spectral norms taken (or have some other ((3)-cquivarian norms must be taken), with each tensor treated individually. Those norms can subsequently be summed or medianed or averaged or square-rooted. Unfortunately, taking spectral norms is more expensive than taking moments. Once again, at application time, the tensor components cannot be independently scaled with nine different scales; instead, one scale must be applied consistently to all nine components. Once again, the shift applied cannot be a single number applied to all components but instead the shift must itself be tensor. These vector and tensor considerations suggest that data normalization needs to be substantially modified to achieve covariance, that is, to accommodate the passive symmetries of vectors and tensors.
For yet another, clements of X with different units must be treated differently. For shifts, it makes sense to subtract from each quantity a shift μ that is computed from only those features that share units with the feature in question. For scales, it might make sense to find base-unit scales that make the range of scaled features as close as possible to having unit variance.
10 Discussion
In this conceptual contribution, we arguc that passive symmetrics are in play in essentially all ML or data-analysis (asks. They are exact, and true by definition, since they emerge from the redundancies or freedom in coordinate systems, units, or data representation. Enforcement of these symmetries should improve enormously the generalization capabilitics of MI. methods. We demonstrate this with toy examples.
In practice, implementation of the passive symmetries in an ML problem might be very difficult. One reason is that the symmetries are only exact when all relevant problem parameters (including often fundamental, unvaried constants) are known and included in the learning problem. If the problem has a passive symmetry by a group G, but there are missing elements K in the problem formulation (such as Planck's constant or the gravity vector in Section 5), then the active symmetry that is actually in play is the subgroup H of G that fixes K. Naively there should be no difference in the in-distribution performance between enforcing the symmetry by H, or including K to the inputs and enforcing the symmetry induced by G. However, using the full group equivariance is conceptually more elegant and it allows for out-of-distributin generalization (the model can generalize to sottings where K has changed). Of course, these unknown constants or features K are pieces of essential contextual information and can be difficult to find or learn. In our toy examples, we show that with sufficient knowledge of the problem (rich training data and knowledge of the group of passive symmetries) the relevant constant K can be learned from the data, including the Planck constant (for the blackbody-rddiation problem) and the gravitational acceleration vector (for the double-pendulum example). Identifiability issues may arise when more constants non-constanl features are missing.
Another difficulty is that some kinds of symmetries are hard to enforce. For example, complete coordinate diffcomorphisms and problem reparameterizations involve enormous groups which are hard to implement in any realistic MI. method. That said, many groups have been implemented usefully, including translations,
12

---

Published in Transactions on Machine Learning Research (01/2024)
rotations, permutations, changes of units, and some coordinate transformations (see Weiler et al. 2021a for review of the latter).
In addition to the exact (and true by definition) passive symmetriess, and the observed active symmetries, there are other kinds of approximate or weakly broken symmetries we might call observer symmetries. These arise from the point that the content of a data record (an image, say) is independent of the minor choices made by the observer in taking that data record (shooting the image, say). The details of the six-axis location and orientation of the camera, and of the exposure time and focus, can be changed without changing the semantic label content of the image. These symmetries are approximate, because these changes don't lead to invertible changes in the recorded data; there is no group or groupoid in the space of the data. However, the success of convolutional structure in image models might have to do with the importance of these observer symmetries. There is much more to do in this space.
One topic that we did not address in this note is the use of symmetries as design principle or inductive bias and their interaction with opt optimization. One feature of modern machine learning is that the models used to fit. the data are often overparameterized (Zhang et al., 2021). That means that the model has more parameters than training points, sometimes several orders of magnitude more. In particular, many possible functions fit the data perfectly, but not all of them generalize well to new data points (Hogg & Villar, 2021) (see the double descent, benign overfitting, and related ideas Belkin et al. 2019; Bartlett et al. 2020; Nakkiran et al. 2021; Huang et al. 2020; Sonthalia et al. 2023, among many others). Therefore, the design of the class of functions is very important for the performance of the models. Through the years, the classes of functions that have been empirically proven to be successful, incorporate symmetries in their design (for example, convolutional neural networks, graph neural networks, transformers, etc). It is not fully understood why these models are successful and whether the symmetries are a key aspect for their success.
Acknowledgement: It is a pleasure to thank Roger Blandford (Stanford), Ben Blum-Smith (JHU), Wilson Gregory (JHU), Nasim Rahaman (MPI-IS), and Shubhendu Trivedi for valuable comments and discussions. This project was started at the meeting Machine Learning for Science at Schloss Dagstuhl, 2022 September 18-23, 23. SV was partially supported by ONR N00014-22-1-2126,, the NSF-Simons Research Collaboration on the Mathematical and Scientific Foundations of Deep Learning (MoDL) (NSF DMS 2031985), NSF CISE 2212457, NSF CAREER 2339682, and an A12AT Amazon research award. This project made of open-source software, including Python, jax, objax, jupyter, numpy, matplotlib, scikit-learn. All code used in this project is available at repositorics https: //gi thub. com com/weichiyane wtw.:a/oooooaarruuluuuraaaaaae ainttM..
References
Max Shirokawa Aalto, Ekdeep Singh Lubana, and Hidenori Tanaka. Geometric considerations for normalization layers in equivariant neural networks. In AT for Accelerated Malerials Design NewrIPS 2022 Workshop, 2022. URL https::/
net//oorumiddyptKDiiss
Joseph Bakarji, Jared Callaham, Steven L Brunton, and J Nathan Kutz. Dimensionally consistent learning with buckingham pi. arXiv preprint orXi::2202.0400 2022.
Peter L Bartlett, Philip M Long, Gábor Lugosi, and Alexander Tsigler. Benign overfitting in linear regression. Proceedings of the National Academy of Sciences, (37444)30003330077) 2020.
Simon Batzner, Albert Musaelian, Lixin Sun, Mario Geiger, Jonathan P Mailoa, Mordechai Kornbluth, Nicola Molinari, Tess E Smidt, and Boris Kozinsky. E(3)-equivariant graph neural networks for data-efficient and accurate interatomic potentials. Nature communications, 13(1):1-11, 2022.
Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Sounik Mandal. Reconciling modern machine- machine-learning practice and the classical bias clas-vaiance trade-off. Proceedings of the National Academy of Sciences, 116(32): 15849 15854, 2019.
Alberto Bictti, Luca Venturi, and Joan Bruna. On the sample complexity of learning with geometric stability. arXiv:2106.07148 2021.
13

---

Published in Transactions on Machine Learning Research (01/2024)
Ben Clum-Smithe and Soledad Villar. Machine learning and invariant theory. Notices of the American Mathemalical Society, pp. 1205-120 1213, September 2023.
Bernhard E Boser, Isabelle M Guyon, and Vladimir N Vapnik. A training algorithm for optimal margin classifiers. In Proceedings of the fifth onnual workshop on Compulational learning theory, pp. 144-152, 1992.
Michael M Bronstein, Joan Bruna, Taco Cohen, and Petar Veličković. Geometric deep learning: Grids, groups, graphs, geodesies, and gauges. arXiv preprint urXic:2104.13488, 2021.
Joan Bruna, Wojeiech Zaremba, Arthur Szlam, and Yann LeCun. Spectral networks and locally connected networks on graphs. arXiv: 131.. 1312.6203, 2013.
S. Buchholz, M. Besserve, and B. Schölkopf. Function classes for identifiable nonlinear independent component analysis. In Advances in Neural Information Processing Systems 35 (NeurIPS 2022). Curran Associates, Inc., December 2022.
J. W. Burby, Q. Tang, and R. Maulik. Fast neural poincaré maps for toroidal magnetic fields. Plasma Physics and Controlled Pusion, 63(2), 12 2020. doi: 10.1088/ 111008813616687777 1161-33876/abbeee /abcbaa.
Jameson Cahill, Joseph W Tverson, Dustin G Mixon, and Daniel Packer. Group-invariant max filtering. arXiv preprint ar Xrc:22205 2022.
Shuxiao Chen, Edgar Dobriban, and Jane H Tee. group-theoretic framework for data augmentation. The Journal of Machine Learning Research, 21 21(1):8885 9955, 2020.
Miranda CN Cheng, Vassilis Anagiannis, Maurice Weiler, Pim de Haan, Taco S Cohen, and Max Welling. Covariance in physics and convolutional neural networks.
2019.
Taco Cohen and Max Welling. Group equivariant convolutional networks. In International conference machine learning, pp. 2990-2999. PMLR, 2016.
R. Courant and D. Hilbert. Methods of Mathematical Physics, volume 1. Interscience Publishers, Inc, New York, 1953.
Laurent Dinh, Razvan Pascanu, Samy Bengio, and Yoshua Bengio. Sharp minima can generalize for deep nets. arXiv, 1703.04933, 2017.
David Duvenaud, Dougal Maclaurin, Jorge Iparraguirre, Rafael Bombarell, Timothy Hirzel, Alán Aspuru-Guzit and Ryan P Adams. Convolutional networks on graphs for learning molecular fingerprints. In Neural Information Processing systems, pp. 2224-2233 2015.
Nadav Dym and Steven J Gortler. Low dimensional invariant embeddings for universal geometric learning. arXiv preprint arXiv:2205.0200 2022.
John Earman and Clark Glymour. Lost in the tensors: Einstein's struggles with covariance principles 1912-1916. Studies in History and Philosophy of Science Part A, ((4):251-278, 1978.
Albert Einstein. Die feldgleichungen der gravitation. Sitzungsberichle der Preussischen Akademie der Wissenschoften zu Bertin, pp. 844-847, 1915.
Albert Einstein. Die Grundlage der allgemeinen Relativitätstheriie. Annalen der Physsik 354(7):769 822, January 1916. doi: 1a.1002/aaddp1111333477777 19163540702.
Bryn Elesedy. Provably strict generalisation benefit for invariance in kernel methods. ar Xrxiv:2166.022 06.00446,22 2021. Bryn Elesedy and Sheheryar Zaidi. Provably strict generalisation benefit for equivariant models. ar 1rXi:22102.1033 2021.
Mark Fels and Peter Olver. Moving coframes: practical algorithm. Acta Applicandae Mathematica, 51: 161-213, 1998.
14

---

Published in Transactions on Machine Learning Research (01/2024)
Mare Finzi, Samuel Stanton, Pavel Izmailov, and Andrew Gordon Wilson. Generalizing convolutional neural networks for equivariance to lie groups on arbitrary continuous data. In International Conference Machine Learning, pp. 3165 3176. PMLR, 2020.
Mare Finzi, Max Welling, and Andrew Gordon Wilson. A practical method for constructing equivariant multilayer perceptrons for arbitrary matrix groups. arXiv:2104.0943 2021.
Mario Geiger and Tess Smidt. e3nn: Euclidean neural networks. arXi::2207.09// 2022.
Alan E. Gelfand. Gibbs sampling. Journal of the American Statistical Association, 96(45)):1300-330) 2000.
Justin Gilmer, Samuel Schoenholz, Patrick F. Riley, Oriol Vinyals, and George E. Dahl. Neural message passing for quantum chemistry. In Proceedings of the 34th International Conference on Machine Learning-Volle Volume 70, pp. 1263-1272. JMLR. org, 2017.
Sharut Gupta, Joshua Robinson, Derek Lim, Soledad Villar, and Stefanie Jegelka. Structuring representation geomelry with rotationally equivariant contrastive learning. arXiv preprinl urXiv:2306.1388 2023.
William L Hamilton. Graph representation learning. Synthesis Lectures on Artifical Intelligence and Machine Learning, 14(3):1-159, 2020.
W. K. Hastings. Monte carlo sampling methods using markov chains and their applications. Biometrika, 57 (1):97-109, 1970. ISSN 00063444.
Sepp Hochreiter and Jürgen Schmidhuber. Flat minima. Newral Compul., 9(1):1-42, 1997.
David W Hogg and Soledad Villar. Fitting very flexible models: Linear regression with large numbers of parameters. Publications of the Astronomical Society of the Pacific, 183(1027):093001, 2021.
Kevin H Huang, Peter Orbanz, and Morgane Austern. Quantifying the effects of data augmentation. arXiv preprint ar Xrr::22202.099 09134, 2022.
Ningyuan Huang, David W Hogg, and Soledad Villar. Dimensionality reduction, regularization, and genoralization in overparameterized regressions. arXiv:2011.11477, 2020.
Ningyuan Teresa Huang, Ron Levie, and Soledad Villar. Approximately equivariant graph networks. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
Pengzhan Jin, Zhen Zhang, Aiqing Zhu, Yifa Tang, and George Em Karniadakis. Sympnets: Intrinsic structure-preservingg symplectic networks for identifying haniltonian systems. Newral Nelworks, 132(C), 8 2020. doi: www.:/eet..eem.......mm 2020.08.017
Sékou-Oumar Kaba, Arnab Kumar Mondal, Yan Zhang, Yoshua. Bengio, and Siamak Ravanbakhsh. Equivariance with learned canonicalization functions. In Proceedings of the 40th International Conference Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 15546 15566. PMLR, 23-29 Jul 2023.
llyes Khemakhem, Ricardo Pio Monti, Diederik P. Kingma, and Aapo Hyvärinen. ICE-BeeM: Identifiable conditional energy-based deep models based nonlinear TCA. Tn Advances in Newral Information Processing Systems 33, 2020.
Risi Kondor. N-body networks: a covariant hierarchical neural network architecture for learning atomic potentials. crXic:1803.0155 015888 2018.
Risi Kondor and Shubhendu Trivedi. On the generalization of equivariance and convolution in neural networks to the action of compact groups. Proceedings of the 35th. International Conference on Machine Learning, 2018.
Hannah Lawrence, Kristian Georgiev, Andrew Dienes, and Bobak T Kiani. Implicit bias of linear equivariant networks. wrXiv:2110.06084 2021.
15

---

Published in Transactions on Machine Learning Research (01/2024)
Yann LeCun, Bernhard Boser, John S Denker, Donnie Henderson, Richard E Howard, Wayne Hubbard, and Lawrence D Jackel. Backpropagation applied to handwritten xip code recognition. Newral Compulation, (4):541 551, 1989.
Haggai Maron, Heli Ben-Hamu, Nadav Shamir, and Yaron Lipman. Invariant and equivariant graph networks. In International Conference on Learning Representations, 2018.
Song Mei, Theodor Misiakiewicz, and Andrea Montanari. Learning with invariances in random features and kernel models. arXiv:2102. 18219, 2021.
Conor Muldoon. DeepMind's EigenGame is not a game! medium, May 2021.
Albert Musaclian, Simon Batzner, Anders Johansson, Lixin Sun, Cameron J Owen, Mordechai Kornbluth, and Boris Kozinsky. Learning local equivariant representations for large-scale atomistic dynamics. arXiv:2204.05244 2022.
Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and llya Sutskever. Deep double descent: Where bigger models and more data hurt. Journal of Statistical Mechanics: Theory and Experiment, 2021 2021.
Emmm Noether. (Iranslated and republished as) Invariant variation problems. Transport Theory and Statistical Physics, 1(3):186 (07(6977, 1918.
J. Peters, D. Janzing, and B. Schölkopf. Elements of Causal Inference Foundations and Learning Algorithms. MIT Press, Cambridge, MA, USA, 2017. ISBN 978-0-262-03331-0.. trade-offs under (approximate) group
Mircea Petrache and Shubhendu Trivedi. equivariance. arXiv preprint 23Xi::2305.1700 17592, 2023.
Henning Petzka, Michael Kamp, Linara Adilova, Cristian Sminchisescu, and Mario Boley. Relative flatness and generalization. In Advances in Neural Information Processing Systems. Curran Associates, Inc., 2021.
Max Planck. On the law of the energy distribution in the normal spectrum. Ann. Phys, 1-553) 1901.
M. M. G. Ricci and T. Levi-Civita. Méthodes de caleul différentiel absolu et leurs applications. Mathematische Annalen, 54(1):125-201, 1900.
M. Rojas- Carulla, B. Schölkopf, R. Turner, and J. Peters. Invariant models for causal transfer learning. Journal of Machine Learning Research, 19(36):1 34, 2018.
Carlo Rovelli and Marcus Gaul. Loop quantum gravity and the meaning of diffcomorphism invariance. Towards quantum gravily, pp. 277 324. Springer, 2000.
Alvaro Sanchez- Gonzalez,,,,,,,,, Victor Bapst, Kyle Cranmer, and Peter Battaglia. Hamiltonian Graph Networks with ODE Integrators. 1rXiv:1909.10 12790, 2019.
B. Schölkopf and A. J. Smola. Learning with Kernels. MIT Press, Cambridge, MA, USA, 2002.
Mehran Shakerinava, Arnab Kumar Mondal, and Siamak Ravanbakhsh. Structuring representations using group invariants. In Advances in Neurol Information Processing Systems, volume 35, pp. Curran Associates, Inc., 2022.
Rishi Sonthalia, Xinyue Li, and Bochao Gu. Under paddee-parmeeterizzed double descent for ridge regularized least squares denoising of data on a line. arXiv preprint arXiv:2305. 14689, 2023.
Richard P. Stanley. Smith normal form in combinatorics. Journal of Combinatorial Theory, Series A, 144: 476 495, 2016.
16

---

Published in Transactions on Machine Learning Research (01/2024)
Hannes Stärk, Octavian Ganea, Lagnajit Pattanaik, Regina Barzilay, and Tommi Jaakkola. Equibind: Geometric deep learning for drug binding structure prediction. In International Conference on Machine Learning, pp. 20503 20521. PMLR, 2022.
Behrooz Tahmasebi and Stefanie Jegelka. The exact sample complexity gain from invariances for kernel regression. In Seirth-seventh Conference Neural Information Processing Systems, 2023.
Nathaniel Thomas, Tess Smidt, Steven Kearnes, Lusann Yang, Li Li, Kai Kohlhoff, and Patrick Riley. Tensor field networks: Rotation-and translation-cquivaiiant neural networks for 3d point clouds. arv: 14:::800.088 1802.08219 2018.
Kip S. Thorne and Roger D. Blandford. Modern Classicul Physics: Oplics, Fluids, Plasmas, Elasticily, Relativity, and Statistical Physics. Princeton University Press, 2017. ISBN 9781400848898.
Soledad Villar, David W Hogg, Kate Storey-Fisher, Weichi Yao, and Ben Blum-Smith. Scalars are universal: Equivariant machine learning, structured like classical physics. Advances in Neural Information Processing Systems, 34:28848 28863, 2021.
Solodad Villar, Weichi Yao, David W Hogg, Ben Blum-Smith, and Bianca Dumitrascu. Dimensionless machine learning: Imposing exact unils equivariance. Journal of Machine Leurning Research, 24(109):1 32, 2023.
Clrike von Luxburg, Olivier Bousquet, and Bernhard Schölkopf. compression approach to support vector model selection. J. Mach. Learn. Res.,
2004.
Rui Wang, Robin Walters, and Rose Yu. Approximately equivariant networks for imperfectly symmetric dynamics. orXiv:220.11199, 2022.
Maurice Weiler, Patrick Forré, Erik Verlinde, and Max Welling. Coordinate independent convolutional networks- - isometry and gauge equivariant convolutions on riemannian manifolds. arXiv, 2106.06020, 2021a.
Maurice Weiler, Patrick Forré, Erik Verlinde, and Max Welling, Coordinate independent convolutional networks-isometry and gange equivariant convolutions riemannian manifolds. arXiv:2166.000 06000000000 2021b.
Hermann Weyl. The Classical Groups. Princeton University Press, 1946.
Christopher K. I. Williams and Carl Edward Rasmussen. Gaussian processes for machine learning. MIT
press (Cambridge, MA), 2006.
Xiaoyu Xie, Arash Samaei, Jiachen Guo, Wing Kam Liu, and Zhengtao Gan. Data-driven discovery of dimensionless numbers and governing laws from scarce measurements. Nature Communications, 13(1): 7562, 2022.
Weichi Yao, Afonso S Bandeira, and Soledad Villar. Experimental performance of graph neural networks on random instances of max-cut. In Wavelets and Sparsity XVIII, volume 11138, pp. 111380S. International Society for Optics and Photonics, 2019.
Weichi Yao, Kale Storey-Fisher, David W. Hogg, and Soledad Villar. simple equivariant machine learning method for dynamics based on scalars. arXiv, 2110.03761, 2021.
Rose Yu, Paris Perdikaris, and Anuj Karpatne. Physics-guided ai for large-scale spatiotemporal data. Tn Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery Data Mining, pp. 4088 4089, 2021.
Anthony Zec Group theory in nutshell for physicists, volume 17. Princcton University Press, 2016.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking gencralization. Communications of the ACM, 64(3):107-115, 2021.
17

---

Published in Transactions on Machine Learning Research (01/2024)
A Glossary
We provide here glossary to translate terminologies among machine learning, mathematics, and physics.
active symmetry: symmetry is actice when an observed or empirical regularity of the laws of physics.
Examples include the observation that the fundamental laws don't depend on the location or time which the experiment takes place. We provide formal definition in Section 4.
conservation law:
We say that quantity obeys a conservation law if changes in that quantity (with time) inside some closed volume can are quantitatively explained by fluxes of that quantity through the surface of that volume. Active symunelries can lead to conservation laws in dynamical systems (when the dynamics is Lagrangian; Noether 1918).
coordinate freedom:
When physical quantities are measured, or represented in a computer, they must be expressed in some coordinate system. The redundancy of this representation. the fact that the invostigator had many choices for the coordinate system-leadss to the passive symmetry coordinate freedom: If the inputs to a physics problem are moved to a different coordinate system (because of a change the origin or orientation), the outputs of the problem must be correspondingly moved. Tn much of the literature "coordinate freedom" is only used in relationship general covariance, but it applies in all contexts (including non-physics contexts) in which coordinate system has been chosen.
covariance: When physical law is equivariant with respect to passive symmetry, then the law said to be covariant with respect to that symmetry.
dimensions: Dimensions the abstract generalization of units. Two quantities that can be given the same units (possibly with units change for one of them) have identical dimensions.
dimensional analysis:
The technique in physics of deducing scalings by consideration of units covariance is dimensional analysis.
equivariance: Let G be group that acts vector spaces X and Y as px : G - Sym(X) and PY : G SYm(Y) respectively. Namcly, px (and similarly py) maps each element of G' to a bijective function from X to itself satisfying some rules described in the definition of "group action". We say that a function f: X Y is equivariant if for any group element, E G and any possible input x, the function obeys ((x(()(x)) = (f))) This is typically expressed by saying that the following commutative diagram commutes for all g E G:
X S
px(1) X
pp(())
(5)
The actions of G in and Y induce an action on the space of maps from X to Y. f Maps X, Y) then we can define PXY G Sym(Maps(X,, Y)) such that (gf( pr f px -1 The equivariant maps the fixed points of this action.
While equivariance is mathematical property of map, in this contribution use the word "equivariance" mainly in the context of active symmetriee, and "covariance" in the context of passive symmetries.
gauge freedom: Some physical quantitics in field theorics (for example the vector potential in elcetromag- netism) have additional degrees of freedom that go beyond the choice of coordinate system and units. These freedoms lead to additional passive symmetries that are known as gauge freedom.
general covariance:
The covariance of relevance in general relativity (Einstein, 1916) is known as general covoriance. Because general relativity is a metric theory on an intrinsically curved spacetime of 3 + dimensions that invariant to arbitrary diffeomorphisms of the coordinate system, this is a very strong symmetry. General covariance sometimes called "coordinate freedom", but it special case thereof.
18

---

Published in Transactions on Machine Learning Research (01/2024)
group: A group G is set with binary operation satisfying the following: (1) Associativity: for all b, c E G we have (a.b) b) c a. (b: (2) Identity element: there exists an element e E G so that for all E G; (3) Inverse: for all a E G there exists a unique element b such that a- b. a = e; this clement is the inverse of a and it is typically denoted as a=!
e. a
group action: We consider group G, a space X, and map p G Sym(X), where Sym(X) denotes the bijective maps from X to itself. Lightly abusing notation, sometimes this map is expressed as p G x X + X. We say p is an action of G on X if it satisfies the following properties: (1) Identity: (x) = x for all x E X, where e is the group identity clement. (2) Compatibility: p(a)(((((=)) - p(a (x) for all a, b E G and a E X.
group representation: A represenlation of a group G is an action p G GL(V), where GL(V) is the space of invertible linear transformations of the vector space V. Sometimes it is said that V is a representation of G (and the action is implicitly known). When V is a finite dimensional vector space, the group representation allows us to map the multiplication of group elements to multiplication of matrices.
invariance:
An equivariance in which the action in the output space is trivial is called an ineariance. Physicists sometimes use the word invariant (gauge invariant, for example) for things we would call covariant.
passive symmetry: A symmetry is passive when it arises from choice in the representation of the data. Examples include coordinate freedom, gauge freedom, and units covariance. These symmetrics are exact and true by definition. We provide a formal definition in Section 4.
scalar: A number (with or without units), whose value does not depend the coordinate system in which it is represented, is scalar. Thus, say, the charge of particle is scalar, but the coordinate of its velocity is not a scalar.
symmetry:
Given a mathematical object X of any sort, (such as manifold, metric space, equation, etc), any intrinsic property of X which causes it to remain invariant under certain classes of transformations (such as rotation, reflection, inversion, other operations) is called a symmetry. For our purposes, the symmetries of interest can be expressed as equivariances or invariances, both defined above.
tensor:
A multi-linear function of k- vectors that outputs a vector, or a multi-linear function of vectors that outputs a scalar, is k-tensor. A rectangular array of data is not usually tensor according to this definition. A vector can be seen 1-tensor (the lincar function corresponding to the vector being the inner product with that vector), and a scalar can be seen as a O-tensor.
There is an alternative definition of tensor in terms of transformations with respect to the O(d) group, analogous to the primary definition of "vector" below.
units: All physical quantities are measured with a system of what we call units. quantity can be transformed from one unit system to another by multiplication with a dimensionless number. Almost all quantities including almost all scalars, vectors, and tensors-hawe units.
units covariance:
The left-hand side and the right-hand side of any equation must have the same units. This symmetry is called (by us) wnils covariance (contra Villar et al. 2023 where it is called "units equivariance").
vector:
An ordered list of numbers, all of which have the same units, that is subject to the passive O(d) symmetry corresponding to coordinate-system rotations, is a vector in d dimensions. These rotations are performed by the standard rotation (and reflection) matrix representation of the elements of O(d). See the definition of "tensor" for an alternative definition: The inner (or dot) product of two vectors produces a scalar; for this reason, a vector can be seen a 1-tensor. A generic ordered list of d features is not usually a vector according to this definition.
19

---

Published in Transactions on Machine Learning Research (01/2024)
B Units covariant regression: The black body experiment
For our first experiments, we consider the problem of black-body radiation, in which the intensity Bx():T) of radiation at wavelength À is a function of temperature T. The true relationship is given by (3), and we use it to construct a toy inference problem in which the units of the input quantities are non-trivial.
The training data are made by computing intensities (A; according to (3) at five temperatures (300, 1000, 3000, 10000, and 30000 K), and on a wavelength grid of 40 wavelengths on a logarithmically spaced grid from 10-8 to 10-1m m. To each evaluation of intensity we add a draw of zero-mean Gaussian noise with root-variance 0.1 times the true value of that intensity. The training data are cut on intensity, such that only data points with intensities above 10 in SI units are retained. The remaining data points so constructed comprise the training data, with features æ being the wavelengths and temperatures, and labels y being the noisy values of the intensity. In some cases we augment the features with copies of the speed of light c and the thermal (Boltzmann's 's) constant k. At each of the training data points, there are two or four fundamental features (the first 2, or all, of the list X, c, k), depending on the regression.
The test data are similarly generated except that they are generated at temperatures of 6000, 20000, and 60000 That is, the test data extend outside the temperature range of the training data, and the test and training data have no temperatures in common. Once the intensity cut is made, there are 126 data points in the training set and 108 in the test set.
We perform three regressions, the output of which are shown in Figure 2: In the first (dubbed "standard MLPP",, a 3-layer multi-layer perceptron from scikit-learn, with hidden-layer sizes of 20, 20, and 20 is trained on the logarithm of the first two training-set features (features T) and the logarithm of the training-set labels. The trained MLP is used to predict the test-set labels. This model incorporates explicit symmetries.
In the second regression (dubbed "units covariant without extra dimensional constant"), the same design of MLP is used, but instead of inputting the logarithms of the fundamental features, the input to the MLP is the set of logarithms of all the dimensionless combinations possible to make from the four features X, c, k, following the approach of Villar et al. (2023). This approach uses the Smith normal form (Stanley, 2016) to find products of integer powers of the input features that are dimensionless, and one product of integer powers of the inputs that has the same units as the label. The output of the MLP is multiplied by the combination of the input features with the correct units to be an intensity. In this case, there are non-trivial dimensionless features, so the MLP sets one amplitude only, and the predictions are pure power laws (as is visible in Figure 2).
In the third regression (dubbed "units covariant with extra dimensional constant"), the features are augmented with one dimensional quantity q, and then the same procedure is followed as for the previous units-covariant case. If the quantity q has units that are independent (in the appropriate sense) from the units of the four fundamental features, then new dimensionless constant becomes possible and the regression becomes non-trivial. The units of g are chosen from an exhaustive list of all possible units that involve powers between -2 and 2 of kg, m, s, and K. The value of is chosen, at each of these units choices, to be such that the range of dimensionless features in the training data has unit median. Again the MLP input is logarithms of dimensionless features and the output is multiplied by the Smith normal form returned combination with the correct units to be an intensity. Many choices for the units and value of q lead to good regression accuracy; all of them make q close to some combination of h, c, k, necessarily including h.
C Springy double pendulum
Setting. We consider the dissipationless spherical double pendulum with springs, with a pivot o and two masses connected by springs. The kinetic energy T and potential energy V of the system are given by
1
|221
KE-
+
(6)
2m1
2m²"
1 kalll - -qil qqil-22) -12)² - mm g. (q) - ()) - -12 g (q2 - -qo),
PE- ki (lq)
qol
-1)
2
(7)
20

---

Published in Transactions on Machine Learning Research (01/2024)
where q1, P1 are the position and momentum vectors for mass m1, similarly 92, P2 for mass m2. and position for the pivot. The springs have scalar spring constants K1, 22; and natural lengths 41, l2. The gravitational acceleration vector is g. In this work, we fix q with values (0, 0, in base length units and with (0, 0, -1) in base acceleration units, as well as (m1, m2, ki, 22, 12) set to (1,1,1,1,1,1,, where each element of the list. has appropriate base units.
The prediction task is to learn the positions and momenta over a of I later times given the initializations of the pendulum positions and momenta at to.
z(1) = (qi (1), (2) Pi(),,(()), LE (lo, (1...,,)))
(8)
The training imputs consist of N = 500 different initializations of the pendulum positions and momenta {z (t)) and the labels are the set of positions and momenta (t() = The model is evaluated a test data set with T = 150 and t0 0.
z0 ( with
Variants. For the same prediction task, we consider three different -33)_qquivariant models, fknwwn.g Leamed and [No., depending how the gravitational acceleration vector g is involved.
Known-g The model fKnown-z a function that predicts the dynamics:
fknown-g (R²)4 x R R x R - (
(9)
(z(0), c g. At) J ((At)
where g is known (0,0, 10,0,-1 in the base acceleration units and used with positions and momenta as input features. The model fearmedg is a function that predicts the dynamics:
Learned-g
fLearned- (R x R x R + (R*)²
(10)
(z(0),, qo. A + 2(Ai)
where g is unknown but set as a learnable variable and used with positions and momenta as input fcatures.
No-g The model fNO- is function that predicts the dynamics:
SNo-g (RR3) R R + (R3)4
(11)
(z), 9o At) + 2(At)
where g is unknown and not used as an input feature.
We evaluate the performance of the three predictive ive models based on the state relative error al a given time in terms of the positions and momenta of the masses,
Seate.RelErr()) (2(t) z(t)) (a(t) z(t))
EE (4.....,
(12)
V2(1) 2(t) + /zt)) z(t)
where a(t denotes the predicted positions and momenta at time and z(t) the ground truth.
Model. In all the variants of the experiment, the learned function f is implemented by a Hamiltonian Gonzalez et al., 2019). The model learns an scalar 0(3) -ivvaintt function H the input vectors, and uses a symplectic integrator to predict the dynamics:
neural network (Sanchez-Gonzalez
dq
0.H
dp.
d.H
= 1,2
(13)
di
Op.
di
aq.
We implement H to be 0(3) -ivaaiivar by simply restricting H to be a function of the inner products of the possible input vectors qı, q2, PI; PI, qo and possibly g, following the fundamental theorem for invariant.
21

---

Published in Transactions on Machine Learning Research (01/2024)
Training MSE
Test MSF
Test rollout error
No normalization
0.0000 + 0.0000
0.0028 + 0.0013
0.0029 + 0.0001
Non-equivariant normalization
0.0184 + 0.0006
0.2587 = 0.0233
0.1177 + 0.0043
Equivariant normalization
0.0000 + 0.0000
0.0070 + 0.0011
0.0053 + 0.0007
Table 1: Empirical performance of the different data normalization procedures.
theory for ((d) (see Weyl 1946 for the theory, and Villar et al. 2021 for discussion on machine learning implications). Namely,
variant (q,, 92, P1, P1, = h((v" W.y
(14)
where YKnown- = {qi, q Pt, P2. 0, 1)); Learned = {q) Q2, PI, P2; g) where g is a variable that is jointly optimized with the parameters that define :f; VNNo- = {qi 92. P1, P2). Note that q, 0 so all the corresponding inner products compuled as
and therefore dropped from the model. The model predictions are
f symplecticinnteegniinn Qq, 92, P1, P2), Az).
(15)
Note that this model is time-translationncgiivaaaan can be 03))cqquivariant (depending on the variant), but it isn't units- covarianv..
Our experiments show that the -quiiiaarrr variant with known gravity has the best performance, closely followed by the model with learned gravity. We note that the model learns vector very close to the ground-truth gravity vector, and therefore learns equivariant model. The non equivariant model with no gravity vector performs quite poorly, as expected. The results are presented in Figure 2.
Data normalization For the data normalization experiment, we consider the standard variant where the gravity vector is known. We normalize the inputs of H in (15) before computing the inner products. We consider the input data matrix with rows indexed by i = ,,...,N and columns of the form
( (q) (t), 02 ()).P))))))) (t), P2 (t), (0, (0)
10
10
(16)
Non-equivariant data normalization For the naive data normalization consider the data matrix with N rows and 3 x 4 x 150 columns (the first 4 colunns of (16)) and we normalize it by colunns. Namely, for each entry of the matrix we subtracts the mean of its column and divide by the standard deviation of the column. This operation does not respect any of the symmetrics of the problem, therefore when we food this normalized data to our standard model (15), the performance is bad, as shown Table 1.
Equivariant data normalization The equivariant data normalization takes all the vectors (q (t), 9 ht) and computes their 3 x 3 empirical covariance matrix Cq. Similarly it computes Cp. The normalized model uses the formulation (15) with
q
Q2
P
P2 00,0,
(0,0, -1)
y normalized
(17)
(LC VI/3tr(Ca)
(r
((13)
The performance is comparable to the one with no normalization, and several orders of margnitude better than the performance of the non-equivariant data normalization method.
22


Reviews:
[{"review_id": "69IJJAFV3w", "reviewer": "Reviewer_xD5g", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper is clear and well written;\n- The distinction between active and passive symmetries is potentially important and not widely appreciated in the community;\n- the historical perspective and links with physics and dimensional analysis are interesting and illuminating.\n\nWeaknesses:\n- The paper is somewhat philosophical in outlook. The authors are clearly aware of this and per se it's not a problem, but I think more effort should be made to explain the practical implications.\n- The main contribution is not entirely clear (see below).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper considers the problem of building predictive models that are equivariant with respect to passive symmetries, i.e. symmetries that arise from arbitrary choices such as coordinate systems. These are distinct from so called \"active\" symmetries which are (often approximate) empirically verified symmetries of systems. The paper contains an extensive introduction to the main concepts, some validation of a proposed strategy on two toy examples, and some dos and donts recommendations for practitioners.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- The procedure followed in the practical examples should be explained in detail, possibly providing pseudocode or access to notebooks. \n- A clearer explanation of the novelty (I guess the introduction of an additional feature vector to account for missing information which is learnt from data) should be given. \n- The method of Villar et al which is at the root of the proposed method should also be described, at least summarily, in order to make it clearer what are the main changes.", "broader_impact_concerns": "No ethical concerns here.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "eNWNwUvoBw", "reviewer": "Reviewer_VRS3", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is written at a conceptual level. The major weakness is the lack of concrete results (theory or experiments) which can be evaluated in the reviews. \n\nIn the abstract, the paper arguably translates between different languages of physics, mathematics, and machine learning. Although I see such traces in the glossary, these vocabularies are not fundamental in the later discussion.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper discusses ways of incorporating passive symmetries into ML frameworks at a conceptual level. This is different than the study of active symmetries as done in equivariant network literature.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "TMLR may not be best suited for such conceptual contributions so I'd like to recommend submitting elsewhere.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "bOtpbw2f0q", "reviewer": "Reviewer_JES3", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\nOverall, I found this paper to be very thought provoking and interesting to read. While invariance/equivariances have become quite popular in machine learning models, this paper demonstrates a very notable area of opportunity (passive symmetries) that have been largely ignored by the existing literature. Though this paper is more of a think piece rather than a traditional ML publication, I believe that the ideas it conveys will be of interest not only to those who study symmetry in ML but also to the larger ML community as a whole.\n\n**Clarity.** Overall the paper is very well written and easy to follow, even for someone who does not have a strong background in group theory. However, there is a bit of redudancy that could make the paper more streamlined (see Weaknesses).\n\n**Experiments.** The toy settings in Section 6 convey a promising illustration of the authors' main ideas.\n\n\n## Weaknesses\n\n- The experiments in Section 6 assume significant familiarity with prior work. The authors should include a more thorough description of the different methods, perhaps in the supplementary material, to assist those not familiar with the work.\n- Sections 9 and 10 felt a bit too speculative, especially after the toy experiments in Section 6 provide nice grounded evidence. It would be nice to include some specific toy experiments here to demonstrate the consequences of violating passive symmetries. Right now, these sections simply demonstrate a violation of principles.\n- The paper feels unnecessarily long, and (at least in the first 6 sections) somewhat redundant/repetative. I think that this paper could fit within the standard 12 page limits of a TMLR submission without needing to cut much content. For example:\n  - Get rid of the page break after the abstract\n  - Move the glossary to the appendix (it is very helpful, but disrupts the reading flow).\n  - Sections 3-5 seem to repeat many of the same ideas but with slight variation. It is possible that including the formal definitions of Section 5 before introducing the content in Section 3 could reduce the need to re-introduce passive and active symmetries multiple times.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper is a think-piece that argues that machine learning models should encode passive symmetries (i.e. natural equivariances due to the arbitrariness of the mathematical system used to encode phenomena). The paper first introduces notions of passive versus active symmetries, and discusses how much of the work in invariant/equivariant machine learning has targeted active symmetries but largely ignored passive symmetries. The authors use toy experiments to demonstrate that taking passive symmetries into account can improve data efficiency. The authors then comment on connections between passive symmetry and causal learning, and describe how our choices of nonlinear activation function and normalization layers lead to violations of passive symmetry.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "## Major (will not vote for acceptance without these changes)\n\n- Add Section 6 experimental details to the appendix\n\n## Very nice to have (but not absolutely necessary)\n\n- Can you include experiments that demonstrate how using nonlinearities/normalization that violate passive symmetries actually impacts performance? This section would become much stronger (and easier to understand) with some explicit experimental examples.\n- Shorten the paper and/or move some of the sections into supplementary material - ideally to fit within 12 pages\n\n## Extremely minor (typos)\n\n- Page 6 towards the bottom \"sometimes Einstein summation notation, Einstein 1916\" <- should this be \"sometimes *called* Einstein summation notation\"?\n- Page 8 \"The two approaches are theoretically eqiuvalent\" <- should be \"three\" approaches", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper adresses timely questions about symmetries in ML models and how taking "passive symmetries" into account may affect performance. The paper's scope is quite general and may catch the attention of a broad audience. It provides detailed numerical experiments backing up the proposed ideas and conjectures. The experiments are convincing and sufficiently detailed to be reproduced. However, as a general rule I recommend the authors to add a link to the code before the final submission.
Decision: Accept (poster)