Scalars are universal: Equivariant machine learning, structured like classical physics
Authors: Soledad Villar
David W Hogg
Kate Storey-Fisher
Weichi Yao
Ben Blum-Smith
Venue: NeurIPS 2021 Poster (2021)
Keywords: invariant theory
group theory
physics
regression
machine learning
symmetry
scalars

Abstract: There has been enormous progress in the last few years in designing  neural networks that respect the fundamental symmetries and coordinate freedoms of physical law. Some of these frameworks make use of irreducible representations, some make use of high-order tensor objects, and some apply symmetry-enforcing constraints. Different physical laws obey different combinations of fundamental symmetries, but a large fraction (possibly all) of classical physics is equivariant to translation, rotation, reflection (parity), boost (relativity), and permutations. Here we show that it is simple to parameterize universally approximating polynomial functions that are equivariant under these symmetries, or under the Euclidean, Lorentz, and Poincaré groups, at any dimensionality $d$. The key observation is that nonlinear O($d$)-equivariant (and related-group-equivariant) functions can be universally expressed in terms of a lightweight collection of scalars---scalar products and scalar contractions of the scalar, vector, and tensor inputs. We complement our theory with numerical examples that show that the scalar-based method is simple, efficient, and scalable. 

Scalars are universal: Equivariant machine learning, structured like classical physics
Soledad Villar
David W. Hogg Flatiron Institute
Department of Applied Mathematics and Statistics Johns Hopkins University
divison the Simons Foundation
Kate Storey-Fisher
Center for Cosmology and Partiele Physics Department of Physics, New York University
Weichi Yao
Ben Blum-Smith Center for Data Science New York University
Department of Technology, Operations, and Statistics Stern School of Business, New York University
Abstract
There has been enormous progress in the last few years in designing neural net- works that respect the fundamental symmetries and coordinate freedoms of phys- ical law. Some of these frameworks make use of irreducible representations, some make use of high-order tensor objects, and some apply symmetry-enforcingg con- straints. Different physical laws obey different combinations fundamental sym- metries, but large fraction (possibly all) of classical physics equivariant translation, rotation, reflection (parity), boost (relativity), and permutations. Here we show that is simple to parameterize universally approximating polynomial functions that are equivariant under these symmetries, or under the Euclidean, Lorentz, and Poincaré groups, at any dimensionality The key observation is that nonlinear O(d)-equivan -ediivviivrr (and related- group-equivant versally expressed terms of lightweight collection scalars- scalar products and scalar contractions of the scalar, vector, and tensor inputs. We complement our theory with numerical examples that show that the scalar-based method simple, efficient, and scalable.
functions can be uni-
Introduction
There is great deal of current interest in building machine-learning methods that respect exact or approximate symmetries, such as tramslation, rotation, and physical gauge symmetries [8 49
Some of this interest is inspired by the great success of convolutional neural networks (CNNs) [51]
which are naturally translation equivariant. The implementation of convolutional layers in CNNs has been given significant credit for the success of deep learning, in domain (natural images) in which the convolutional symmetry is only approximately valid. In many data-analysis problems in astronomy, physics, and chemistry there are exact symmetries that must be obeyed by any general- izable law or rule. Since the approximate symmetries introduced by convolutional networks help in natural-image domain, then we have high hopes for the value encoding exact symmetries for problems where these symmetries are known to hold exactly.
In detail, the symmetries of physics are legion. Translation symmetry (including conservation of linear momentum), rotation symmetry (including conservation of angular momentum), and time-
35th Conference on Neural Information Processing Systems (NeurIPS 2021).

---

translation symmetry (including conservation of energy) are the famous symmetries [69] But there are many more: there is a form of reflection symmetry symplectic symmetry that permits reinterpretations of positions and momenta; there are Lorentz and Poincaré symmetries (the fundamental symmetries of Galilean relativity and special relativity) that include velocity boosts; there is the generalization of these (general covariance) that applies in curved spacetime; there are symmetries associated with baryon number, lepton number, flavor, and color; and there are dimensional and units symmetries (not usually listed as symmetries, but they are) that restrict what kinds of quantities can be multiplied or added. If it were possible to param- eterize universal universally approximating function space that is explicitly equivariant under a large set of non-trivial symmetries, that function space would in some sense- contain within every possible law of physics. It would also provide basis for good new machine learning methods.
or CPT); there a
The most expressive approaches to equivariant machine learning make use of irreducible represen- tations of the relevant symmetry groups. Implementing these approaches requires a way to explic- itly decompose tensor products of known representations into their irreducible components. This is called the Clebsch-Gordan problem. The solution for SO(3) has been implemented, and there is recent exciting progress for other Lie groups
This is an area of current, active research.
Here we give an approach that bypasses the need for a solution to this problem. We find that, for a large class of problems relevant to classical physics, the space of equivariant functions can be constructed from functions only of a complete subset of the scalar products and scalar contractions of the input vectors and tensors. That is, invariant scalars are powerful objects, and point the way to building universally approximating functions that are constrained by exact, non-trivial symmetries.
Equivariance is the form of symmetry core to physical law: For an equivariant function, transform- ing the input results in an output representation transformed in the same way. An invariant function, on the other hand, produces the same output for transformed and non-transformed inputs.
Definition 1; Given a function f : X + y and group G acting on X' and in y as * (possibly the action is defined differently in X' and 3)). We say that f is:
G-invariant:
f(g
f(z) for all x X, ge
(1) (2)
G-equivariant:
x(xx))= - x f(x) for x E X, gEG.
Our contributions: In this work we provide a complete and computationally tractable characteriza- tion of all scalar functions f (I + R, and of all vector functions h : (Rd)) Rd that satisfy all of the symmetries of classical physics. The groups corresponding to these symmetries are given in Table I they act according to the rules in Table 2 The characterization we provide is physically principled: It is based on invariant scalars. It is also connected to the symmetries encoded in the Einstein summation rules, common notation in physics to write expressions compactly but that also allows only equivariant objects to be produced (see Appendix F
Our characterization is based on simple mathematical observations. The first is the First Fundamental Theorem of Invariant Theory for O(dy: a function of vector inputs returns an invariant scalar if and only if can be wrillen as a function only of the invariant scalar products of the inpul vectors [33 Section IIAA.9]. There is a similar statement for the Lorentz group O(1, d). The second observation is that a function of vector inputs returns an equivariant vector if and only if can be written as a linear combination of invariant scalar functions times the input vectors. In particular, if h (Rd)" + Rd of inputs v1.-.; Un O(d) or O(1, d'-equivarian,, then it can be expressed as:
a
h(m,, U2,--, mn) = Σ Se (155, M)M))=))) Ut
where St can be arbitrary functions, but if i is a polynomial function the fe can be chosen to be polynomials. In other words, the O(d) and O(1, di-equvvarian vector functions are generated as module over the ring of invariant scalar functions by the projections to each input vector. In this ex- pression, (--> denotes the invariant scalar product, which can be the usual Euclidean inner product, or the Minkowski inner product defined in terms of a metric A (see Table DD:
Euclidean: (v,, 0i) = wn Vj
Minkowski: (v,, m3) = v. A
Our mathematical observations lead to a simple characterization for very general class of equivari- ant functions, simpler than any based on irreducible representations or the imposition of symmetries

---

Orthogonal
((d) = (Q E Ruxd, Q QQ =
Rotation
SO(d) = (Q E Rudd QQQQ QQ' Ta. det(() =1
Translation
T(d) {w E R4)
Euclidean
B(d) T(d) O(d)
Lorentz
O d) = {Q E [(++) QQ A, A diag -1, -...., ....,,--1)) 1])}
Poincaré
I0(1, d) = T(d + 1) x O(1,d)
Permutation
S, (a [n] [n] bijective function)
Table The groups considered in this work.
Orthogonal: Lorentz Translation
(V1, un) (Q u1.... Qu,)
(Ul,.. Un) (e) + w,... Uk + w, U,+1s.., Un)
(where the first h vectors are position vectors)
Euclidean; Poincaré
(w, Q) (V1,.., Un) = (Q w..., U w, W.Q Uu+ vu)) (V1..., . v.) (Vo(1)... Vo(n))
Permutation
Table 2: The actions the groups on vectors. For the Euclidean group, the position vectors are positions of points; for the Poincaré group, the position vectors are positions of events.
through constraints (these methods are currently state of the art; see Section 2 This implies that very simple neural networks based on scalar products input vectors-thal enormously generalize those in [80] can universally approximate invariant and equivariant functions. This justifies the nu- merical success of the neural network model proposed in [8)] and mathematically shows that their method can be extended to a universal equivariant architecture with respect to more general group actions. We note that the formulation in B might superficially resemble an attention mechanism [7,, but it actually comes from the characterization of invariant functions of the group, in particular the picture for SO(d)-invariant functions a little different (see Proposition 5
In Section (and Appendix we describe the state of the art for encoding symmetries, the ex- pressive power of graph neural networks, and universal approximation of invariant and equivariant functions. In Section 5 we mathematically characterize the invariant and equivariant functions with respect to the groups in Table D In Section we present some examples of physically meaningful equivariant functions, and show how to express them in the parameterization developed in Section B In Section
5 and Appendix G we discuss which (of all possible) pairwise inner products one ought to provide, and in Section 6 discuss some limitations of our approach. We present numerical ex-
periments using our basedr-based approach compared to other methods in Section (see also (96)).
We also note that the symmetries considered in this work are all global symmetries, as they act on all points in the same way. Our characterization thus does not obviously generalize to all gauge symmetries, which are local symmetries that apply changes independently to points at different locations (see [1 (22) ). That said, we believe our model could be made general enough encompass gauge symmetries if we replace the global metric by any position-dependent metric Az. In this case, spatially separated vectors would need to be propagated the same point such that they can input to locally invariant functions, and this can be done with parallel transport. The parallel transport operations would also have to obey our invariance characterization, but we believe this is possible to do; will explore it in future work.
2 Related work
Group invariant and equivariant neural networks: Symmetries have been used successfully for learning functions images, sets, point clouds, and graphs. Neural networks can be designed to parameterize classes of functions satisfying different forms of symmetries, from the classical (ap- proximately) translation-invariant convolutional neural networks 152 as well new approaches that enforce additional symmetries (rotation, scale) on these networks 30 architectures that define permutation invariant and equivariant functions on point clouds pioneered
-to more recent
3

---

by deep sets and pointnets graph neural networks 33
99 73
74
to permutaaionn-equiaaiin functions on graphs expressed as
2 30
For instance, deep sets and pointnets parameterize functions on (Rd)) that are invariant or equivari- ant with respect to the group of permutations S,, acting as in Table 2 Invariant theory shows that all invariant and equivariant functions with respect to such actions can be approximated by easily char- acterized invariant polynomials [93]. However, the permutation group can act in significantly more complicated ways. For instance, graph neural networks are equivariant with respect to a different action by permutations (conjugation) that is much harder to characterize (see Appendix A
Since being introduced by pecially in computer vision 36
99 deep learning on point clouds has been extremely fruitful, es- Recently, new symmetries and invariances have been incorporated into the design of neural networks on point clouds; especially invariances and equivariances with
73
respect to rigid motions such translations and rotations. Many architectures have been proposed to satisfy those symmetries such as 355 based on irreducible representations (irreps), (50) 101 based on convolutions, [29] employing spherical harmonics and irreps. [02 using quaternions, and
(88 applying set of constraints to satisfy the symmetries. Most of the implementations of the ap-
proaches mentioned above (except for (X are limited to 2D or 3D point clouds. We provide an
overview of the main approaches below and in Appendix A
Recently 7 developed approach to enforcing permutation equivariance in neural network lay- ers using parameter sharing, which can then model other symmetry groups. The weight sharing ap- proach is significantly simpler to implement than the ones described above, and it has been proven to be very successful in practice, with several applications [89 (0), including autonomous driving (38 Characterizing the space of the functions this approach can express is an interesting open problem.
Universal approximation via linear invariant layers and irreducible representations: Univer- sally approximating invariant functions can be obtained by taking universal non-invariant func- tions and averaging them over the group orbits 197 68 However, this approach is not practical for large groups like S,, or infinite groups like O(d). A classical result in neural networks shows that ford-forward networks with non pol-polylo activations can universally approximate continu- ous functions [54] This arguably inspired the use of neural networks that are the composition of linear invariant or equivariant layers with compatible non-linear activation functions to create ex- pressive equivariant models 4 6 65) Linear G-equivariant functions can be written in terms of the irreducible representations of G (we explain this and refer the reader to related literature in Ap- pendix A.. However, the explicit parameterization of the linear maps is only known for few groups (for instance, S0(3)-equivariant linear maps are parameterized using the Clebsh-Gordan coefficients Moreover, very recent work 2 shows that the classes of functions defined in terms of neural networks over irreducible representations in (29 85 are universal. In particular, every con- tinous -qquiirquvva function can be approximated uniformly in compacts sets by those neural networks. Despite universality, there is a limitation to this approach: Although decompositions into irreps are broadly studied in mathematics (also as plethysms), the explicit transformation to the ir- reps is not known or possible for general groups. This is in fact an area of current, active research, but the implementation
29
3
where there has been recent exciting progress for other Lie groups is still limited.
Expressive power of (non-universal) neural networks on graphs: Graph neural networks express functions on graphs that are equivariant with respect to certain action by permutations. However, the architectures that are used in practice, typically based message passing (30 or graph convolu- tions 23 20 are not universal in general. Implementing a universally approximating graph neural network using the formulation from the previous section would be prohibitively expensive. There is work characterizing the expressive power of message passing neural networks, mainly in terms of the graph isomorphism problem [94 (7
X4 [4], and there research on the design of graph networks that are expressive enough to perform specific tasks, like solving specific combinatorial
16
59
optimization problems O
[K]
70
95.
Machine learning for physics with symmetry preservation: Machine learning has been applied extensively problems in physics. While many of these problems require certain symmetries- in many cases, exact symmetries-mos applied projects to date rely on the data to encode the
symmetry and hope that the model learns it. For instance, CNNs are commonly used to classify
galaxy images; data augmentation is used to teach the model rotational symmetry [39 B 22 BT] One welln example is the Kaggle Galaxy Challenge, a classification competition based on the

---

Galaxy Zoo project [58]: the winning model improved performance by concatenating features from transformed images of each galaxy before further training [2T]
There have been recent successes in enforcing physical symmetries in the architecture of the mod- els themselves 98, for instance, in weather and climate modeling [ and in modeling chaotic dynamical systems such as turbulence [89
In quantum many-body physics, recent work has shown that the symmetries of quantum energy levels on lattices can be enforced with gauge equiv- S8
90
60 78 [1. There is significant work on imposing permutation symmetry in jet assignment for high-energy particle collider experiments with self- attention networks [27 533. In molecular dynamics, rotationally invariant neural networks have been shown to better learn molecular properties [2 91 82), and Hamiltonian neural networks have been constructed to better preserve molecular conformations [55]. More broadly, Hamiltonian networks have been shown to improve physical characteristics, such as better conservation of energy, and [03]. and Lagrangian neural networks can also enforce conservation
ariant and invariant neural networks I7
J00
to better generalize 34 laws
19.
Invariant theory as a basis for enforcing symmetry in neural networks: We are aware of two lines of prior work that develop approaches related to the one taken here: 56 and 35 Y7
In 156 certain tasks in turbulence modeling and materials science are considered, which have built- in 0(3) and (in the latter case) octahedral symmetry. Each of these problems involves specific representation of the given symmetry group, for which an explicit generating set for the invariant algebra (there called an "integrity basis") is known. The authors construct and test models that learn an invariant function, built on these generating sets. In (57]. the turbulence example is taken up again, this time with the goal of learning an equivariant 2-tensor.
The idea of using the invariant algebra to enforce physical symmetries is also contemplated in 355 In this work, the authors are focused on developing the underlying invariant theory in the case of simultaneous Lorentz and permutation invariance. In the followup work [37] this idea is developed into three models, using different generating sets, which are tested against each other.
The present work shares with these works the idea of using invoriant-theoretic descriptions of invari- ant functions to hard-code physical symmetries into a neural network that models a physical system. We work in somewhat greater generality. While 56 57 focus on specific (small) representations of O(3) and finite extensions, and while 35 37 are focused on invariant functions and restricted to linearly reductive groups, we consider both invariant and equivariant functions for almost all groups relevant to physics, including groups (Euclidean and Poincaré) that are not reductive, although their invariant theory remains under control.
Inductive bias benefits of incorporating symmetries: The value of incorporating exact symme- tries in machine learning has been recently established empirically in several applications (see for instance (90).. Mathematical theory has been developed to explain how much one can improve in but many questions remain open.
terms of sample complexity and generalization 166 26
3 Equivariant maps
In this Section we provide simple characterization of all invariant and equivariant functions with respect to the actions in Table _ by the groups in Table [ In what follows, U1, v2, Un will be vectors in IR G will be group acting in E. and (IRd)"
2 f (IR-)" - E will be an invariant function with respect to the action, and h (Rdy" - Id will be an equivariant function with respect to the same action. V will denote a x n matrix whose columns are the 0/ vectors.
in Table
(dd and SO(d) invariance and equivariance: The following classical result in invariant theory (e.g.. 93 Section II.A.9) shows that O(d) invariant functions are functions of the scalars u/ vy-
Lemma (First Fundamental Theorem for O(d)): If f is an O(d)-invariant scalar function of vector inputs U1,.., Un E Rdd then f(e,, u vx) can be written function of only the scalar products of the U2. That is, there is a function g(.) such that
f(r,, U2 v.) = vVVVVV) = ((v o))E==))
(5)
5

---

Proof. Given M = VTV E I we can reconstruct v..... en modulo the orthogonal group ((d) by computing the Cholesky decomposition of M (see for instance [66] p. 174). Therefore, the function/ f is uniquely determined by the inner product matrix VTV.
The classical theorem also includes the fact that if is polynomial (in the entries of the Uz uy's), then can be taken to be polynomial. In Section S we observe that the function g can be determined by small subset of the scalars V/ Uj. There an analogous statement for SO(d) (again see [93 Section IIA::9]):
Lemma 2 (First Fundamental Theorem for SO(d)): If f is an SO(d)-invarian scalar function of vector inputs U1 U2.... Un E R", then f(11, u2..., Un)) can be written as a function of the scalar products of the U, and the d x d subdeterminants of the d x n matrix V.
Lemma 3: If is an O(d)-qquivariant vector function of n vector inputs 01; U2, Una then 1(11, U2, .n) must lie in the subspace spanned by the input vectors U1, U2,.., Un.
Rd be orthonormal basis of the orthogonal complement to span(e1. (11... k,). Then we can write h(v1,.., Un) = Σi= 244% + Σi= Bew, for some choice of or, St. We claim that the equivariance of h implies Bg = 0 for t = 1, I,.., Consider Q E Od) such that Q(v)= v for all span(u1, ... vn), and ((w) = for all in the orthogonal complement.
Proof. Let {wi,.., },}
Since (Qu1..., Qv.)) (U1, . Un) we have h(Qui,... Qu.) Σí= a + Σ Srw, while Q(h(vi..., )) = Ei O4224/ Ei= Bew. Therefore equivariance implies that all Be =
0. L
Note that Lemma 3 doesn't hold for SO(d): although when the codimension of span(U1, ... 2n) is or 2 2 a similar argument works, the situation dime span span(v).-.. (v)...., v.) = d - 1 breaks the proof, and there do exist equivariant vector functions that don't lie in the span when it has codimension 1. For instance, the cross product of two vectors RR an SO(3)-equivariant function that is not in the span of its inputs. Proposition 5 shows that generalized cross products are the only way that (O(d) -equivvquvvaiiatt vector functions can escape the span of the inputs. We further discuss this in Appendix F Proposition 4 below gives characterization of all -qqiivarri functions in terms of the scalars. We prove it in Appendix B
Proposition 4: If h is an O(d)-equivariant vector function of n vector inputs U1, ,,, Cn, then there are n (dd--invariant scalar functions fe(-) such that
hV1, - - un) Ei= (1 u2, Un) Uz-
Moreover, if h is polynomial function, the fc can be chosen to be polynomial.
The equivariant scalar functions form module over the ring of invariant scalar functions. Propo-
sition states that this module is generated by the projections (V1,..., 2n)) + vy. Proposition extends this result to SO(d) in terms of the generalized cross product. The definition of the general- ized cross product and the proof of Proposition are in Appendix B
Proposition 5: If h is an (O(d))-quivarr) characterization of Proposition holds, except when 01, U2, Un span (d -iimeemsiii space. In that case, there exist O(d) -nvariint scalar functions fe(-) such that
vector function of vector inputs 11. u,, then the
MV(1,, U2, vn) = Ei=1S((o U Un) U2 - [se() [s(U,, U2,.., vn)us,
us
where [n] := (1..... n). (d)) is the set of all (d- - 1) subsets of [n], and us represents the general- ized cross product of vectors vyy with j E S (taken in ascending order). Morcover, if is polynomial, the fi and fs can be taken be polynomial.
E(d) invariance and equivariance: When modelling point clouds, we may want to express func- tions that are translation invariant or equivariant with respect to a subset of the input vectors (for instance, position vectors are translation equivariant). To this end, we consider the group of transla- tions parametrized by w E Rdd acting on the vectors in (Rd) by translating the position vectors, and leaving everything else unchanged: w*((v,,.., Uk U,+ Un)) = (v) (w,+ , Uk+ 1k ). In this Section we characterize all functions that are translation and rotation invariant/equivariant In the exposition below we assume for simplicity that all vectors are position vectors, but the results generalize trivially to mix of vectors.

---

Lemma 6: Any translation-invann invariiant function f: (IR) th + IF' with inputs .,..., can be written uniquely as f(V1, u2.... , un) f(12 .,,.., Un 01), where f (124)=1 + Re is arbitrary function. If is polynomial, f' polymomial, and vice versa. If f equivariant for the action of any given subgroup G C GT,(n, R), then so is and vice versa.
The proof is given in Appendix There is nothing special about subtracting v1; there exist more natural choices to express translation invariant functions. For instance, one classical way in physics to express translation invariance is to take class representatives of the form (Vi,.., Un) where Ee = 0 (for example, subtracting the center of mass). Proposition 7 characterizes the space of eddivaivivaiiant functions that are also translation invariant or translation equivariant. The proof is in Appendix [
Proposition 7: An -quivvavirat function h (Rd)) + R that is translation invariant can be written as a where the St are O(d) and translation invariant and Ei= fe(11 02,-- vn) = 0. Similarly, if is translation-equivariant then we can choose >x fe(U1, U2,... Un) =
Lorentz symmetry: The Lorentz group acts on Minkowski spacetime as Lorentz transformations, which keep the metric tensor invariant. Lorentz tramsformations relate space and time between in- ertial reference frames, which move at constant relative velocity; spacetime intervals are invariant across frames. The group is made up of spatial rotations in the three space dimensions and linear velocity "boosts" along each dimension. This set of coameeriis-requaeee not O(4), but rather the non-compact group 0(1,3) defined in Table
for special relativity
The characterization of invariant and equivariant functions we obtained for the orthogonal group can be extended to the Lorentz group, obtaining a very similar result, summarized in Proposition g and proven in Appendix D
Proposition 8: A continuous function h (IR+1)" + Rd+1 is Lorentz-equivariant (with respect to the action in Table
2 if and only if
h(U1,.., un) =Z= fe(1i..., Un) Pt
(8)
where St 11 -> TR are Lorantz-invariant scalar functions. Moreover, the functions fa are uniquely determined by the pairwise Minkowski inner product (.,))
[u Mx) = Ie ((vs, (y)..==))
If h is polynomial, the ft and corresponding Or can be taken to be polynomial.
Poincaré symmetry: The Poincaré group combines translation symmetry with Lorentz symmetry. Together these complete the symmetries of special relativity, forming the full group of spacetime transformations that preserve the Minkowski metric. The generalization from Lorentz-equivarian formulation Poincaré-equivarian formulation is similar to the generalization from O(d) to E(d): The position vectors take on a special role, in which only differences of position can appear as vector inputs the functions; all other vectors act unchanged.
Proposition generalizzss the results above to the Poincaré group action. As before, we assume that all vectors position vectors for simplicity. The proof is analogous to the proof of Proposition
Proposition 9: A continuous function : (IRd+1)" -> IR++1 is Poinearé-equivariant (with respect to the action in Table 2 if and only if
h(u,,.., ux) St G.)
(10)
where the J+ (Rd+lyn + R are translation and Lorentz-invariant scalar functions, determined as in 2 by the pairwise Minkowski inner products 4 but also satisfying Ei=1 fc(v1, D2,... vn.) = 1.
Furthermore, if h is polynomial, the and corresponding 9t (as in can be taken to be polyno-
mial.
Permutation invariance and equivariance: Most physics problems are permutation-invarian in that once you know the masses, sizes, shapes, and of the objects in the problem, the physical predictions are invariant to labeling. In particle physics, this invariance raised to a fundamental symmetry; fundamental partieles (like electrons) are identical and exchangeable.
7

---

The characterization of permutation invarrinn functions with respect to the action in Table 2 is clas- sical [93] pp. 36-39]. Here we prove an extension that describes permutation-invarieee vector func- tions that are also Odd)-qquivrrian. The proof is in Appendix
Proposition 10: Let h (Re))" TR be -(d)-aquivariant (or continuous O(1, d - 1 ))-quivaariant and also permutation-invarian with respect to the action in Table Then can be written as
h(v v.) = Ei=1 f(ee, v1... 1/-1, 11-1,.., Un) Ut:
(11)
where f: (IR")" R is O(d)-invariant (or O(1, d 1-invariint) and permutation-invariaat with respect to the last n inputs.
Proposition 11 proven in Appendix
B eextenss the characterization above to permutation equivariant
functions (see also 35 48
Proposition 11: Let h : (Rd)" + (IE")" be O(d)-qquivariant (or continuous O(1, d -qquivarri))))) and also permutation-equivariiee
with respect to the action in Table 2
Then h can be written as
= (hi,... An)) where each h; (Idd)" -> Rdd is equi-iquvvar (or continuous O(1, 1)- equivariant) and
h. v.) = Ei 1() (vi..., (.) Ur:
(12)
where all the 1 (IR")" + R are O(d)-invariant (or O(1, d ((r -iiiaariian,,, and for all i, = I,.., and all a E Sn we have
f0 (y (oott)) =
(13)
4 Examples
Here we briefly state two classical physics expressions that obey all the symmetries, and show how to formulate them in terms of invariant scalars.
Total mechanical energy: In Newtonian gravity, the total mechanical energy T of n particles with scalar masses my vector positions r,, and vector velocities et is a scala function:
T Σ my |
Σ R
Gm;
(14)
In-ryl
where G is Newton's 'ss constant (a fundamental constant, and hence scalar). Since |a| = (a (aTa)' /2, this expression (14) is manifestly constructed from functions only of scalars m, and scalar products of vectors. It is also worthy of note that the positions T; only appear in differences of position.
Electromagnetic force law: The total electromagnetic force F acting on test particle of charge, 3-vector position, and 3-vector velocity (q, r, u) given set of other charges (g;, (e, Us) is a vector
function 41
F
n i== kqq |
Ei= k
vx (x(exx(r-re)) 1rrr-113
(15)
electrostatic force
magnetic force
where & an electromagnetic constant, cs the speed of light, and x b represents the cross product that produces a pseudo-vects vector perpendicular to vectors a and b according to the right-hand rule. This doesn't 't obviously obey our equivariance requirements, because cross products deliver parity- violating pseudo-vectors: these can't be written in -33)-equivariant form. However, cross of cross of vectors is vector, so this expression in fact -quivvaav,,,,, as are all forces (because forces must be O(3)-cquivariant vectors in order for the theory to be sell-consistent).
We can expand the vector triple product using the identity (b x e)
(a (aTc) (a b)
(r
ri)
(v (r T:)) u. (e (v) (r-r;)
F
k kgq
kqq
r:
c r r
i=1
(1- 1 T Di
(1- 2e
(r
(u (r r:))
kgg
kggs
(16)
'The energy is not scalar in special relativity or general relativity, but it is scalar in Newtonian physics.
8

---

where the quantity u oy is the scalar product of the velocities. All of the quantities are now straight- forwardly functions of invariant scalar products times the input vectors.
How many scalars are needed?
Our analysis in Section shows that the invariant and equivariant functions of interest (under actions in Table 2 from groups in Table D with input vectors U1,.., Un, can be expressed in terms of the This greatly simplifies the parameterization of such functions, but it signif- icantly increases the number of features if n > d. In Appendix G we remark that the scalars can be uniquely determined by a small subset of size approximately (d (d+1) n. This is related to the rigidity theory of Gram matrices (77 that answers when there exists unique set of vectors that realize partial set of distances, and it is closely related to the low rank matrix completion problem [83]. Furthermore, there is vast literature studying high probability robust reconstruction of all scalars from a random subset via convex optimization techniques [1]. Recently developed optimization techniques on Gram matrices could provide efficient algorithms to learn invariant and equivariant functions 42
scalars (ee "=)=
6 Limitations and caveats
Although the principal results presented here work for many groups, and work naturally at all spatial dimensions d (unlike methods based on irreps, for example). they do not solve all problems for all use cases of equivariant machine learning. For one, there are myriad groups-and especially discrete groups that apply to physical and chemical systems where invariant and equivariant functions do not have such nice characterization. Such is the case for the GNNs discussed in Appendix A
One example of a situation in which our formulation might not be practical is provided by multipole expansions (for example, those used in the fast multipole method (5] and n--ody networks ) In the fast multipole method, a hierarchical spatial graph is constructed, and high-order tensors used aggregate information from lower-level nodes into higher-level nodes. This aggregation is concise and linear when it is performed using high-order tensors; this aggregation is hard (or maybe impossible) when only scalars can be transmitted, without the use of the irreps of the relevant symmetry groups This is a research direction we are currently exploring.
Another example that suggests that universal functions might be cumbersome is the current forms in which classical theories-suue as electromagnetism and general relativity are written. For example, in Section E we showed that the electromagnetic force law can be written in the form of functions of scalars and scalar products times vectors, but that is not how the theory is tradi- tionally written. It is traditionally written in terms of the magnetic field (a pseudo-vector) or the electromagnetic tensor (an order-2 tensor). As another example, general relativity is traditionally written in terms of contractions of an order-4 curvature tensor. That is, although the theories can in principle be written in the forms we suggest, they will in general be much more concise or simple or clear in forms that make use of higher-order or non-equivariant forms.
Finally, although our results apply to many physically relevant groups, they do not encode all of the symmetries of classical physics. For example, one critical symmetry is the dimensional or units sym- metry: You cannot add or subtract terms that have different units (positions and forces, for example). This symmetry or consideration has implications for the construction of valid polynomials. It also implies that only dimensionless (unitless) scalar quantities can be the arguments of large classes of nonlinear functions, including exponentials sigmoids. These additional symmetries must be enforced at present with some additional considerations of network architecture or constraints.
We also note that in this work we have characterized global symmetries, which act each point in the same way. Our characterization does not obviously generalize to gauge symmetries, which local symmetries that apply changes independently to spatially separated points (see e.g. [II). That said, we believe our model could encompass general gauge symmetries if we replace the lo- cal metric by any position-depennent metric A,. Recent work has shown that equivariance under gauge symmetries is possible in the realm of convolutional neural networks by defining coordinate- independent kernels [92] In our case, we would have to propagate spatially separated vectors the same location in order to pass them to a locally invariant function, requiring the operation of parallel

---

MLP
101
104
MLP+Aug
EMLP (NLP-5015)
109
Scalors (I)
Test MSE
10-1
1
10-2
5013)
10-3
10
EMLP-
Scalars
10a
10a
10a Training Set Size
104
Training Set Size
Figure 1: Test error a function of training size for (Left) the O(5)-invariant task, and (Right) the O(3)- equivariant task. Scalars (I) denotes the MLP model using the scalars method for 0(5)-invariance construction, and Scalars (E+P) denotes MLP model using scalars method -()-evuvveee and permutation invariance construction. MLP denotes standard multilayer perceptron, and MLP+Aug denotes an MLP that been trained with data augmentation
given symmetry group. EMLP-G denotes EMLP models from A with different relevant symmetry groups G. For both tasks, the scalar method outperforms all other methods. The shaded regions depict 95% confidence intervals taken over runs.
transport. This operation would then also have obey our invariance characterization; believe this is possible and we will detail it in future work.
7 Numerical experiments
We demonstrate our approach using scalar- based multi-layer perceptrons (MLP) two toy learning
tasks from [28]: an O(S)-invriiant task and an equivvaraa task. Further numerical experiments with these methods applied to dynamical systems appear in (96). The code is available GitHut
and it reuses much of the functionality provided by EMLP (28]
0(S)-invarian task: Given observations of the form (zf a5.
=) where #f. TR and
an O(5)-invariant scalar function, we aim to learn f. In this case f is the example from 28
|
x x2
f(11, ( II)
11, E
(17)
2
|
|
We model f using Lemma ! namely f(z1, = g0 I1,
x1
22 x2) where g (R)
+ R
could be any function. We learn g by implementing as an MLP.
-uuiviquivr task: In this task, the data are n = 5 point masses and positions (m, x)) and the goal is to predict the matrix I = Σi m2 xz - -x,²] ). To this end we aim to learn
h: 3335 3x3
(18)
(m; F4 ) + I.
The function is -33)-qquivariant in positions, and Ss-invariant, namely, for all Q 0(3) and E S,, we have h( (1)) Q0 ) Qh((m;, x.) By Proposition 10 we know there exists functions for fr. and f2 such that
5 Σ
h((m,, x))_.))
folæT x4 mi, {oy 21, my, milu,es) x,pe
5
Σ (>1=
f1 xz, m., mj, 1 æı, my, mi mihaiss)) T.a + fallo, æj my m, m,)
Here the set notation means that the function f; is permutation-invariat with respect the inputs in the set. The function h is -uuiiqquia by construction (see Section 3). We model fo. fi and fa with MLPs on deep sets 100..
10

---

Acknowledgements: It is pleasure to thank Tim Carson (Google), Miles Cranmer (Princeton), Johannes Klicpera (TUM), Risi Kondor (Chicago), Lachlan Lancaster (Princeton), Yuri Tschinkel (NYU), Fedor Bogomolov (NYU), Gregor Kemper (TUM), and Rachel Ward (UT Austin) for valu- able comments and discussions. We especially thank Gerald Schwartz. (Brandeis) for indicating the line of the polynomial argument to and Mare Finzi (NYU) for their help with the EMLP code- base. SV was partially supported by NSF DMS 2044349, the NSF-Simons Research Collaboration on the Mathematical and Scientific Foundations of Deep Learning (MoDL) (NSF DMS 203 1985), and the TRIPODS Institute for the Foundations of Graph and Deep Learning at Johns Hopkins University. KSF was supported by the Future Investigators in NASA Earth and Space Science and Technology (FINESST) award number SONSSC20K1545.
References
[1] A. Alex, M. Kalus, A. Huckleberry, and von Delft. A numerical algorithm for the explicit calculation of SU(N) and SL(N, C) Clebsch-Gordan coefficients. Journal of Mathematical Physicx, 52(2):223507, 2011.
[2] B. Anderson, T. S. Hy, and R. Kondor. Cormorant: Covariant molecular neural networks. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Bil Alché-Buc, E. Fox, and R. Garnett, editors, Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[3] A. K. Aniyan and K. Thorat. Classifying radio galaxies with the convolutional neural network. The Astrophysical Journal Supplement Series, 230(2):20, Jun 2017.
[4] W. Azizian and M. Lelarge. Characterizing the expressive power of invariant and equivariant graph neural networks. arXiv:2006. 15646, 2020.
[5] R. Beatson and L. Greengard. A short course on fast multipole methods. Wavelets, Multilevel Methods and Elliptic PDEs, 1:1-37, 1997.
[6] Y. Bengio, A. Lodi, and A. Prouvost. Machine learning for combinatorial optimization: a methodological tour d d'horizo.. European Journal of Operational Research, 2020.
[7] A. Bietti, L. Venturi, and Bruna. On the sample complexity of learning with geometric stability. arXiv preprint arXiv:2106.0711 07144, 2021.
[8] A. Bogatskiy, B. Anderson, J. Offermann, M. Roussi, D. Miller, and Kondor. Lorentz group equivariant neural network for particle physics. In International Conference on Machine Learning, pages 992-1002. PMLR, 2020.
[9] G. Bouritsas, F. Frasca, S. Zafeiriou, and M. M. Bronstein. Improving graph neural network expressivity via subgraph isomorphism counting. arXiv:2006.0922 09252, 2020.
[10] D. Boyda, G. Kanwar, S. Racanière, D. J. Rezende, M. S. Albergo, K. Cranmer, D. C. Hack- ett, and P. E. Shanahan. Sampling using SU(N) gauge equivariant flows. Phys. Rev. D, 103(7):074504, 2021.
[1]] M. M. Bronstein, J. Bruna, T. Cohen, and P. Velivckovi'c. Geometric deep learning: Grids, groups, graphs, geodesics, and gauges. ArXiv, abs/2104.13479, 2021.
[12] E. J. Candes and Recht. Exact low-rank matrix completion via convex optimization. In 2008 46th Annual Allerton Conference on Communication, Control, and Computing, pages 806-812, IEEE, 2008
[13] Q Cappart, D. Chételat, E. Khalil, A. Lodi, C. Morris, and P. Veličković, Combinatorial optimization and reasoning with graph neural networks. arXiv:2102.09544 2021.
[14] 7. Chen, I.. Chen, S. Villar, and J. Bruna. Can graph neural networks count substructures? In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Neural Information Processing Systems, volume 33, pages 10383-10395. Curran Associates, Inc., 2020.
[15] Z. Chen, L. Li, and J. Bruna. Supervised community detection with line graph neural net- works. International Conference on Learning Representations, 2019.

---

[16] Z. Chen, S. Villar, L. Chen, and J. Bruna. On the equivalence between graph isomorphism testing and function approximation with gnns. In Neural Information Processing Systems, pages I 15868-15876, 2019.
[17] K. Choo, G. Carleo, N. Regnault, and T. Neupert, Symmetries and many-body excitations with neural-netwood quantum states. Physical Review Letters, 121:167204, Oct 2018.
[18] T. Cohen, M. Weiler, B. Kicanaoglu, and M. Welling. Gauge equivariant convolutional net- works and the icosahedral enn. In International Conference on Machine Learning, pages 1321-1330. PMLR, 2019.
[19] M. D. Cranmer, S. Greydanus. S. Hoyer, P. W. Battaglia, D. Spergel, and S. Ho. Lagrangian neural networks, arXiv:2003.04633 04630, 2020.
[20] M. Defferrard, X. Bresson, and Vandergheynst. Convolutional neural networks on graphs with fast localized spectral filtering. In neural information processing systems, pages 3844- 3852, 2016.
[21] S. Dieleman, K. W. Willett, and J. Dambre. Rotation-invariat convolutional neural net- works for galaxy morphology prediction. Monthly Notices of the Royal Astronomical Society, 450(2):441-1459, Apr 2015.
[22] H. Domínguez Sánchez, M. Huertas- -omppay, M. Bernardi, D. Tuccillo, and J. L. Fischer. Improving galaxy morphologies for sdss with deep learning. Monthly Notices of the Royal Astronomical Society, 47663):3661-3677 Feb 2018
[23] D. K. Duvenaud, D. Maclaurin, J. Iparraguirre, R. Bombarell, T. Hirzel, A. Aspuru-Guzik, and R. Adams. Convolutional networks on graphs for learning molecular fingerprints. In Neural Information Processing systems, pages 2224-2232, 2015.
[24] N. Dym and H. Maron. On the universality of rotation equivariant point cloud networks. arXiv:2010.02449, 2020.
[25] A. Einstein. Die Grundlage der allgemeinen Relativitätssheorie. Annalen der Physik, 354(7):769 822, Jan. 1916.
[26] B. Elesedy and S. Zaidi. Provably striet generalisation benefit for equivariant models. arXiv preprint arXiv:2102.1033: 2021.
[27] M. J. Fenton, A. Shmakov, T.-W. Ho. S.-C. Hsu, D. Whiteson, and P. Baldi. Permu- tationless Many-Jet Event Reconstruction with Symmetry Preserving Attention Networks. arXiv:2010.09206, 2020.
[28] M. Finzi, M. Welling, and A. G. Wilson. A practical method for constructing equivariant multilayer perceptrons for arbitrary matrix groups arXiv:2104.09459 2021.
[29] F. Fuchs, D. Worrall, V. Fischer, and M. Welling. Se (3)-transformers: 3d tronssaanllaion equivariant attention networks. Neural Information Processing Systems, 33, 2020.
[30] J. Gilmer, S. S. Schoenholz, P. F. Riley, O. Vinyals, and G. E. Dahl. Neural message passing for quantum chemistry. In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pages 1263-1272. JMLR. org, 2017.
[31] R. González, R. Muñoz, and C. Hernández. Galaxy detection and identification using deep learning and data augmentation. Astronomy and Computing, 25:103-109, 2018.
[32] R. Goodman and N. R. Wallach. Symmetry, representations, and invariants, volume 255. Springer, 2009.
[33] M. Gori, G. Monfardini, and F. Scarselli. A new model for learning in graph domains. In Pro- ceedings. 2005 IEEE International Joint Conference on Neural Networks, 2005., volume 2,
pages 729-734. IEBE, 2005.
12

---

[34] S. Greydanus, M. Dzamba, and J. Yosinski. Hamiltonian neural networks. In H. Wallach, Larochelle, A. Beygelzimer, F. d' Alché-BBuc E. Fox, and Garnett, editors, Neural Infor- mation Processing Systems, volume 32. Curran Associates, Inc., 2019.
[35] B. Gripaios, W. Haddadin, and C. G. Lester. Lorentz-and permutation-invariee of particles. Journal of Physics A: Mathemutical and Theoretical, 15(15):155201, 2021.
[36] Y. Guo, H. Wang, Q. Hu, H. Liu, L. Liu, and M. Bennamoun. Deep learning for 3d point clouds: A survey. IEEE Transactions on Pattern Analysis and machine intelligence, 2020.
[37] W. Haddadin. Invariant polynomials and machine learning. arXiv preprint arX1v:2104.12333,
2021.
[38] K. Huang. Traffic agent movement prediction using resnet- based model. In 2021 6th Interna- tional Conference on Intelligent Computing and Signal Processing (ICSP), pages 136-139. IEEE, 2021.
[39] M. Huertas- Huertas-Company, R. Gravet, G. Cabrera- Vives, P. G. Pérez-Gonzélez, S. Kartaltepe, G. Barro, M. Bernardi, S. Mei, F. Shankar, P. Dimauro, and et al. A catalog of visual-like mor- phologies in the 5 candels fields using deep learning. The Astrophysical Journal Supplement Seriex, 221(1):8, Oct 2015.
[40] A. Ibort, A. López Yela, and Moro. A new algorithm for computing branching rules and Clebsch-Gordan coefficients of unitary representations of compact groups. Journal of Math- ematical Physics, 58(10): 2017.
[41]
Jackson. Classical Electrodynamics; 2nd ed. Wiley, New York, NY, 1975.
[42] A. Jalali, M. Fazel, and Xiao. Variational gram functions: Convex analysis and optimiza- tion. SIAM Journal Optimization, 27(4):2634- 2017.
[43] C. K. Joshi, T. Laurent, and X. Bresson. An efficient graph convolutional network technique for the travelling salesman problem. arXiv: 190.:::906.01227, 2019.
[44] N. Karalias and A. Loukas. Erdos goes neural: an unsupervised learning framework for combinatorial optimization graphs. 1XXiv:2006.104433 2020.
[45] K. Kashinath, M. Mustafa, A. Albert, J. Wu, C. Jiang, S. Esmacilzadeh, K. Azizzadenesheli, R. Wang. A. Chattopadhyay, A. Singh, et al. Physics-informed machine learning: case stud- ies for weather and climate modelling. Philosophical Transactions of the Royal Society A, 379(2194):20200033, 2021.
[46] N. Keriven and G. Peyré. Universal invariant and equivariant graph neural networks. In Neural Information Processing Systems, pages 7092-7101, 2019.
[47] T. N. Kipf and M. Welling. Semi-supervised classification with graph convolutional networks. 2016.
arXiv:1609.028
[48] J. Klicpera, F. Becker, and S. Günnemann. Gemnet: Universal directional graph neural net- works for molecules. arXiv preprint arXiv:2106.088 200.000000,,903, 08903, 2021.
[49] R. Kondor. N-body networks: a covariant hierarchical neural network architecture for learning atomic potentials. arXiv:1803.0 015588 2018
[50] R. Kondor, H. T. Son, H. Pan, B. Anderson, and S. Trivedi. Covariant compositional networks for learning graphs. arXiv:1801.0214 2018.
[51] Y. LeCun, Y. Bengio, and G. Hinton. Deep learning. Nature, 521(7533):336 444, 2015.
[52] LeCun, B. Boser, S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and D. Jackel. Backpropagation applied to handwritten zip code recognition. Neural Computation, 1(4):541-55 1989.
[53] J. Lee, I. Park, I. J. Watson, and Yang. Permuutrnnto Jet- let-Parton Assignment using Self-Attentin Network. arXiv:2012.03342, 2020.
13

---

[54] M. Leshno, V. Y. Lin, A. Pinkus, and S. Schocken. Multilayer feedforward networks with nonpolynomial activation function can approximate any function. Neural Networks, 6(6):8118861) 1993.
[55] Z. Li, S. Yang, G. Song, and L. Cai. Hamnet: Conformation-guuieed molecular representation with hamiltonian neural networks, 2021.
[56] J. Ling, R. Jones, and J. Templeton. Machine learning strategies for systems with invariance properties. Journal of Computational Physics, 318, 05 2016.
[57] J. Ling, A. Kurzawski, and Templeton. Reynolds averaged turbulence modelling using deep neural networks with embedded invariance. Journal of Fluid Mechanics, 807:155-166, 2016.
[58] J. Lintott, K. Schawinski, A. Slosar, K. Land, S. Bamford, D. Thomas, M. Raddick, R. C. Nichol, A. Szalay, D. Andreescu, P. Murray, and J. Vandenberg. Galaxy Zoo: morphologies derived from visual inspection of galaxies from the Sloan Digital Sky Survey*. Monthly Notices of the Royal Astronomical Sociery, 389(3): 119-111991999 2008.
[59] A. Loukas. What graph neural networks cannot learn: depth vs width. In International Conference Learning Representationss 2019.
[60] D. Luo, G. Carleo, B. K. Clark, and Stokes. Gauge equivariant neural networks for quantum lattice gauge theories, 2020.
[61] D. Luo, Z. Chen, K. Hu, Z. Zhao, V. M. Hur, and B. K. Clark. Gauge Invariant Autoregressive Neural Networks for Quantum Lattice Models. arXiv:2101.07243 2021.
[62] M. Lutter, C. Ritter, and J. Peters. Deep lagrangian networks: Using physics as model prior for deep learning. In 7th International Conference on Learning Representations (ICLR). ICLR, May 2019.
[63] H. Maron, H. Ben-Hamu, N. Shamir, and Y. Lipman. Invariant and equivariant graph net- works. In International Conference Learning Representations, 2018.
[64] H. Maron, E. Fetaya, N. Segol, and Y. Lipman. On the universality of invariant networks. In International Conference on Machine Learning, pages 4363-4371, 2019.
[65] H. Maron, O. Litany, G. Chechik, and E. Fetaya, On learning sets of symmetric elements. In International Conference on Machine Learning, pages 6734-6744 6744. PMLR, 2020.
[66] S. Mei, T. Misiakiewiez, and A. Montanari. Learning with invariances in random features and kernel models. arXiv preprint arXiv:2102.132199 2021.
[67] C. Morris, M. Ritzert, M. Fey, W. L. Hamilton, J. E. Lenssen, G. Rattan, and M. Grohe. Weisfeiler and leman go neural: Higher-order graph neural networks. Association for the Advancement Artificial Intelligence, 2019.
[68] R. Murphy, B. Srinivasan, V. Rao, and B. Ribeiro. Relational pooling for graph representa- tions. In International Conference on Machine Learning, pages 4663-4673, 2019.
[69] E. Noether. Invariant variation problems. Transport Theory and Statistical Physics, 1(3):186- 207, 1971.
[70] A. Nowak, S. Villar, A. Bandeira, and J. Bruna. A note on learning algorithms for quadratic assignment with graph neural networks.
2017.
[71] D. L. Pimentel-Alareón, N. Boston, and R. D. Nowak. A characterization of deterministic sampling patterns for low-rank matrix completion. IEEE Journal of Selected Topics in Signal Processing, 10(4):623-633 666, 2016.
[72] V. L.. Popov and F. B. Vinberg. Invariant theory. In Algebraic geometry IV, pages 123-278. Springer, 1994.
14

---

[73] C. R. Su. K. Mo, and L. J. Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 652-660, 2017.
[74] R. Qi. L. Yi, H. Su, and L. J. Guibas. Pointnet++ deep hierarchical feature learning on point sets a metric space. In Neural Information Processing Systems, pages 5105-5114, 2017.
[75] S. Ravanbakhsh, J. Schneider, and B. Poczos. Equivariance through parameter-sharing. Pro- ceedings of the 34th International Conference on Machine Learning, 2017.
[76] M. M. G. Ricci and T. Levi-Civita. Méthodes de calcul différentiel absolu et leurs applica- tions. Mathematische Annalen, 54(1):125-201, 1900.
[77] B. Roth. Rigid and flexible frameworks. The American Mathematical Monthly, 88(1):6-21, 1981.
[78] C. Roth and A. H. MacDonald. Group Convolutional Neural Networks Improve Quantum State Accuracy. wiXiv2204.00088, 2021.
[79] A. Sanchez-Gonzalez, V. Bapst, K. Cranmer, and Battaglia. Hamiltonian Graph Networks with ODE Integrators. arXiv: 190.99299,799,,,, 2019.
[80] V. G. Satorras, E. Hoogeboom, and M. Welling. E(n) equivariant graph neural networks. arXiv:2102.09844, 2021.
[81] F. Scarselli, M. Gori, A. C. Tsoi, M. Hagenbuchner, and G. Monfardini. The graph neural network model. IEEE Transactions on Neural Networks, 20(1):61-80, 2008.
[82] T. Schütt, O. T. Unke, and M. Gastegger. Equivariant message passing for the prediction of tensorial properties and molecular spectra, 2021.
[83] A. Singer and M. Cucuringu. Uniqueness of low-rank matrix completion by rigidity theory. SIAM Journal on Matrix Analysis and Applications, 1221-166,, 2010.
[84] B. Tahmasebi and S. Jegelka. Counting substructures with higher-order graph neural net- works: Possibility and impossibility results. urXiv:2012.03177 2020.
[85] N. Thomas, T. Smidt, S. Kearnes, L. Yang, L. Li, K. Kohlhoff, and P. Riley. Tensor field networks: Rotation-and translation-equivarian neural networks for 3d point clouds. arXivv/802.088: 2018
[86] L. N. Trefethen and D. Bau III. Numerical linear algebra, volume 50. Siam, 1997.
[87] P. Veličković, G. Cucurull, A. Casanova, A. Romero, Liô, and Bengio. Graph attention networks. In International Conference on Learning Representations, 2018.
[88] T. Vieijra, C. Casert, J. Nys, W. De Neve, J. Haegeman, J. Ryckebusch, and F. Verstraete. Restricted boltzmann machines for quantum states with non-abelian or anyonic symmetries. Physical Review Letters, 124()), Mar 2020.
[89] R. Wang, K. Kashinath, M. Mustafa, A. Albert, and R. Yu. Towards physics-informed deep learning for turbulent flow prediction. In Proceedings the the 26th ACM SIGKDD International Conference Knowledge Discovery & Data Mining, pages 1457-1466, 2020.
[90] R. Wang, R. Walters, and R. Yu. Incorporating symmetry into deep dynamics models for improved generalization. In International Conference on Learning Representations, 2021.
[91] Z. Wang, C. Wang, S. Zhao, S. Du, Y. Xu, B.-L. Gu, and W. Duan. Symmetry-adapted graph neural networks for constructing molecular dynamics force fields, 2021.
[92] M. Weiler, Forré, E. Verlinde, and M. Welling. Coordinate independent convolutional networks isometry and gauge equivariant convolutions on riemannian manifolds, 2021.
[93] H. Weyl. The Classical Groups. Princeton University Press, 1946.
15

---

[94] K. Xu, W. Hu, J. Leskovec, and S. Jegelka. How powerful are graph neural networks? In International Conference on Learning Representations, 2018.
[95] W. Yao, A. S. Bandeira, and Villar. Experimental performance of graph neural networks on random instances of max-cut. In Wavelets and Sparsity XVIII, volume 11138, page 1 1380S. International Society for Optics and Photonics, 2019.
[96] W. Yao, K. Storey-Fisher, D. W. Hogg, and S. Villar. simple equivariant machine learning method for dynamics based on scalars, 2021.
[97] D. Yarotsky, Universal approximations of invariant maps by neural networks. arXiv: 10006 2018
[98] R. Yu, P. Perdikaris, and A. Karpatne. Physics-guides ai for large-scale spatiotemporal data. Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Min- ing, KDD "21, page 4088 4089, New York, NY, USA, 2021. Association for Computing Machinery.
[99] M. Zaheer, S. Kottur, S. Ravanbakhsh, B. Póczos, R. Salakhutdinov, and A. J. Smola. Deep sets. In Neural Information Processing Systems, pages 3394-3404, 2017.
[100] M. Zaheer, S. Kottur, S. Ravanbakhsh, B. Poezos, R. R. Salakhutdinov, and A. Smola. Deep sets. In Advances in Neural Information Processing Systems, volume 30, 2017.
[101] Z. Zhang, B.-S. Hua, D. W. Rosen, and S.-K. Yeung. Rotation invariant convolutions for 3d point clouds deep learning. 2019 International Conference on 3D Vision (3DV), pages 204-213. IEEE, 2019.
[102] Y. Zhao, T. Birdal, J. E. Lenssen, E. Menegatti, L. Guibas, and F. Tombari. Quaternion equivariant capsule networks for 3d point clouds. In European Conference on Computer Vision, pages 1-19. Springer, 2020.
[103] Y. D. Zhong, B. Dey, and A. Chakraborty. Symplectic ode-net: Learning hamiltonian dynam- ics with control. In 7th International Conference on Learning Representations (ICLR). ICLR, May 2020.
16


Reviews:
[{"review_id": "ikj5Jaf9APX", "reviewer": "Reviewer_jRKc", "summary": "In this paper, the authors study the characterisation of group invariant and equivariant functions in terms of a collection of group invariant scalars, such as the dot product between two vectors for O(n) invariance. These contrast with the mainstream approach in geometric deep learning of solving constraints on the parameter space that make the network architecture invariant/equivariant. The paper is purely theoretical and contain no experiments.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality:\nThe characterisation of invariant functions with respect to the orthogonal groups in terms of appropriate inner-products is well-known, however as far as I am aware, the characterisations of equivariant functions given here are new (the case of O(n) is probably known given its simplicity). While it is possible that similar results exist somewhere in the literature, presenting it in this fashion may be beneficial to machine learning researchers looking to construct simple parameterisations of invariant/equivariant functions without relying on the heavy machinery of group representation theory, which seems to be a common approach in the literature. This also means that it would make it more appealing if it were more approachable to a wider audience.\n\nQuality:\nThe proofs in the main text and appendix are rigorous and mathematically sound (although there are some presentation issues, see below). Experiments on simple datasets would have been nice to see however, which would have made a stronger case for the algorithm and enabled a discussion on the practicality/computational cost (try e.g. on the kaggle galaxy dataset that is mentioned in §2 and other simple physical models such as the pendulum). **Update:** The authors have addressed this point adequately and now have convincing experiments on synthetic and real data.\n\nClarity:\nThe writing in the main body is fairly easy to follow and the authors have done a good job at conveying the main ideas, however there are several misprints and some unclear points, which I list in the following.\n- The use of the term gauge equivariance in the title may be misleading. The paper only deals with global symmetries, whereas gauge symmetry in general refers to local symmetries. **Update:** The authors have agreed to remove the word \"gauge\" from the title to avoid confusion. \n- The authors put an emphasis on constructing equivariant polynomial functions at several points in the paper. I did not understand the purpose of this if the framework already contains a universal approximation of general equivariant nonlinear functions.\n- Conclusion is missing.\n- Line 29. There should be a right bracket \")\" after \"...linear momentum\"\n- Lemma 1. If this is a classic theory, is it necessary to include a proof?\n- Line 311-313. The line \"In particular invariant functions ... where $M$ is either $V^TV$ or $V^T \\Lambda V$\" seems to be incomplete.\n- Line 726-727. The line \"Isomorphically, $G_W$ is $SO(m)$ and its action on $W$ is $SO(m)$’s canonical action on $R^m$\" in the appendix is difficult to understand and needs improvement.\n- Line 737-738. For clarity, I would suggest changing \"Thus it and $v_1,...,v_n$ span $R^d$. So in all cases, i.e., for any tuple $(v_1,...,v_n)$, there exists a solution in the $a_t$ and $a_S$ to...\" to \"Thus, $v_1,...,v_n$ and its generalised cross product span $R^d$. So in all cases, i.e., for any tuple $(v_1,...,v_n)$, there exists $a_t$ and $a_S$ such that ... holds.\"\n- Line 316. I suggest changing \"We note that the results from [59] don’t assume the matrix M is positive semi-definite\" to \"We note that the results from [59] do not assume that the matrix M is positive semi-definite\"\n\nSignificance:\nThe work may be useful for incorporating physical inductive bias in neural networks without relying on tools from group representation theory. However the lack of experimental verification and seemingly little use case may make it less appealing to the general audience. **Update:** I have changed my view regarding the experiments in light of the rebuttal.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have explicitly acknowledged some limitations of this approach in the paper, including:\n- It only deals with the orthogonal family of transformations and the corresponding construction of invariant scalars for other types of groups is not clear cut.\n- Expressions are cumbersome when inputs are general tensors, which arises often in physics.\n\nIn addition to these, I would also point out the following:\n- Inability to deal with scalar fields and more general sections of vector bundles, which are natural objects in computer vision and in physical field theories (the technique in this work seems to only deal with point-mass-type systems). For example, group convolutions deal with group equivariance of scalar fields while steerable CNNs and gauge equivariant CNNs deal with equivariance on sections of general vector bundles. The approach in this paper do not include these settings.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "I14b1VYR_yl", "reviewer": "Reviewer_f6cj", "summary": "The paper proposes explicit constructions of invariant and equivariant functions for a range of symmetry groups and specific group actions that are relevant in classical physics. The symmetry groups under consideration, listed in Table 1, are the orthogonal groups $O(d)$, rotations $SO(d)$, translations $\\mathbb{R}^d$, Euclidean group $E(d)$, Lorentz group $O(1,d)$, Poincare group $IO(1,d)$ and permutation group $S_n$ (Table 1). Except for the permutation group, these groups act via their standard action on vectors in $\\mathbb{R}^d$ and leave scalars invariant (Table 2). The permutation group is assumed to act by permuting a set of vectors in $\\mathbb{R}^d$. The main contribution of the paper are theorems that characterize the spaces of equivariant vector-valued functions $f: (\\mathbb{R}^d)^n \\to \\mathbb{R}$ and invariant scalar-valued functions $h: (\\mathbb{R}^d)^n \\to \\mathbb{R}^d$. It is proven that these functions can generally be written in terms of functions of all $n^2$ pairwise inner products and subdeterminants of the $n$ input vectors. This result suggests an universal way of parameterizing equivariant/invariant neural networks. An important difference to prior work is that no constraints on the neural connectivity are required but just unconstrained functions of the inner products and subdeterminants.\n\nTwo classical examples from physics exemplify the results of the theorems.\nA potential issue of the proposed parametrizations is that the number $n^2$ of pairwise inner products is quadratic in the number $n$ of input vectors. The authors briefly discuss some results which suggest that the number of inner products that is required in practice is almost always of size approximately $(d+1)n$, which is of the same order as the original input ($dn$ neurons).\n\nNo empirical analysis of the parametrization and comparison with alternative methods is provided.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality:\n\nThe methods leverage classical results from invariant theory, and are therefore firmly grounded in the mathematical literature. In contrast to most related\twork, the method does not requite the user to implement irreducible representations or constraints on the network architecture. This is very useful as it greatly simplifies implementations. The ideas of the paper seem to have been explored before, for instance in Gripaios et al (2021). The related work section gives a comprehensive overview of the field.\n\nQuality:\n\nThe submission is mathematically sound and all claims are formally proven. The authors are honest about the about strengths and weaknesses of the work, which are in particular discussed in the limitations section 6. As a purely theoretical paper, the submission would be complete. However, the proposed method is not empirically evaluated and benchmarked against alternative approaches. This makes the paper as a whole incomplete, which justifies my final score.\n\nClarity:\n\nThe paper is well organized and the presentation of the theoretical results in section 3 is very clear. Section 4 provides some examples of these results in classical physics. The results in section 5 seem a bit suggestive and are not formulated in terms of a theorem.\n\nThe title of the paper refers to \"gauge-equivariant machine learning\", however, the relation to local gauge symmetries is unclear. The covered symmetry groups are rather global symmetries of Euclidean spaces or Minkowski spacetime, acting on all vectors in the same way. The authors should either adapt the title or clarify in how far the results relate to gauge symmetries.\n\nThe practical relevance of the method remains due to the missing empirical evaluation entirely unclear.\n\nSignificance:\n\nThe significance of the results can not be assessed without an implementation and comparison with alternative approaches. Such an empirical study is in my opinion indispensable as experience shows that theoretical results do often not directly translate into practice but that further tweaks are required. An implementation could furthermore support the claim from section 5 that not all $n^2$ inner products are required in practice, but only $(d+1)n$.\n\nOverall score:\n\nThe results of the paper look very promising and are of potentially high relevance to the community. However, without an empirical evaluation, at least on some toy problems, the submission remains incomplete. I would like to encourage the authors to add an experimental section to the submission. As it is currently still missing, I need to clearly reject the paper in its current state.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations of the methods are honestly discussed in section 6.", "review_text": "", "extra_scores": {"ethical_concerns": "There are no ethical issues with this paper.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "4FvpqublKz1", "reviewer": "Reviewer_xhVY", "summary": "The paper proves several simple characterizations for invariant and equivariant function for groups that are of interest to physics applications such as the orthogonal group/Lorentz group together with translations and permutations.  \n", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Positive points:\n-- the paper addresses a timely and interesting problem: encoding invariance and equivariance in deep learning models. While incorporating permutation symmetries in deep models is relatively well understood, our understanding of continuous symmetries like the ones addressed in this submission is still being developed. \n-- The paper is very well written and easy to follow. Great job!\n-- The characterizations discussed are simple and elegant and could serve as inspiration for future models\n\nNegative points: \n-- It is not obvious which of the theorems/propositions were known before and which were not. It is of course well known that the first fundamental theorem holds. What about its generalization to equivariant functions? And the permutation equivariance part?\n-- Although the characterization in the paper seems useful - it is not entirely clear how to leverage them in a deep learning framework, and I believe this can be an even more important contribution. There are two sections that try to address this issue: section 7 and the application of the message passing algorithm in the appendix. I would give a much larger room to this discussion and even implement the suggested method. To make room, Section 6 can be shortened. There is also no discussion of the relationship between the proposed algorithm and recent O(3) graph models.  \n\nOriginality: The paper leverages prior work from invariant theory and although I know some of the propositions are new, it is not clear to me which ones. This can be fixed in a revision.\n\nQuality: A high-quality paper addressing an interesting problem.\n\nClarity: Very clearly written.\n\nSignificance: This paper can pave the way for future research in the field. Implementing the suggested model would strengthen it.\n\nSummary: I like the paper, but the authors should address the above-mentioned issues, especially with regards to the new model. I'm open to raising my score as a result of the rebuttal.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The limitations are discussed.", "review_text": "", "extra_scores": {"ethical_concerns": "None.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "-W6KJI6CkM6", "reviewer": "Reviewer_StjN", "summary": "The paper characterizes scalar and vector functions that satisfy the symmetries of classical physics. Concretely, functions that are equivariant and invariant with respect to a list of (relatively) standard groups (such as $SO(d)$, $E(d)$, etc.) are characterized; the authors additionally show how to express a couple of physically meaningful equivariant functions in their framework. ", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n\n1. Overall, the paper is clear and well-written. The quality of the presentation is high; lemmas and propositions are well-formatted, derivations are well-justified. \n\n2. The paper presents a unified characterization of a fairly general class of equivariant functions, where the symmetries are specified via a canonical set of group actions oft-seen in physics.\n\nWeaknesses\n\n1. Although the paper characterizes a fairly general class of equivariant functions relevant in physics, arguably the implications of this characterization are not particularly significant. Namely, physicists are already keenly aware of the symmetries inherent to their problems and do a good job of incorporating these symmetries in their models [a,b,c]. Whether it be Lorentz group equivariance [a], or equivariance to arbitrary Lie groups [c], models are built to respect symmetries. I was interested in seeing more about what the authors had to say regarding the implications of their characterization to model-building, but this was only briefly mentioned (Section 7); no comparison was made to prior models and the paper did not point out any potential inefficiencies that could be remedied via their characterization. By itself, the characterization is not particularly interesting. As exhibited by the paper, the proofs are straightforward, methodologically not much original or interesting computation was presented.\n\n2. The paper fails to provide a sufficient comparison to prior work done in modeling physical symmetries. Section 7 gives a brief description in which particle position is predicted by taking into account various physical symmetries; how does this compare to work that already models these symmetries and does a similar prediction, say [c]? It seems the paper stops short of actually going through with building a model, or pointing out how prior work is inefficient/can be improved with the given characterization, or providing some convincing consequential implication of the work. This makes the paper seem unfinished, or perhaps preliminary.\n\n3. Narratively, I take umbrage with how the paper was presented. I believe the paper spends too many pages on characterizing symmetric functions, and too few pages on model-relevant considerations. How one builds the model in physics or in machine learning is frequently as, if not more, important as characterization of symmetries abstractly. See for example the discussion of model symmetries in [b], which describes how one builds SU(N) gauge equivariant flows (note the extensive interplay of symmetry and model considerations). Discussing the implications of the characterization in such a model-specific setting would help in demonstrating usefulness.\n\nVerdict\n\nAlthough the paper is well-written and presents a unified characterization of a fairly general class of equivariant functions relevant in physics, it is preliminary, in that it barely begins to address the implications of this characterization to modeling in physics. By itself, the theory is not particularly interesting and the proofs given are short and straightforward. As such, I recommend a rejection.\n\nMinor Corrections\n\nThe paper is, for the most part, well-written, and the ideas are clearly communicated. Some minor corrections are given below:\n\nL135: \"Implementating\" -> \"Implementing\"\n\nReferences\n\n[a] Bogatskiy, A., Anderson, B.M., Offermann, J., Roussi, M., Miller, D.W., & Kondor, R. (2020). Lorentz Group Equivariant Neural Network for Particle Physics. ArXiv, abs/2006.04780.\n\n[b] Boyda, D., Kanwar, G., Racanière, S., Rezende, D.J., Albergo, M.S., Cranmer, K., Hackett, D., & Shanahan, P. (2020). Sampling using SU(N) gauge equivariant flows. ArXiv, abs/2008.05456.\n\n[c] Finzi, M., Stanton, S., Izmailov, P., & Wilson, A.G. (2020). Generalizing Convolutional Neural Networks for Equivariance to Lie Groups on Arbitrary Continuous Data. ICML.\n\n\n### Post-rebuttal Update\n\nI have raised my score from a 4 to a 5.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have adequately addressed the limitations of the work. There is even a separate section, Section 6, that addresses limitations directly. The fact that the work does not characterize functions symmetric with respect to action by (most) discrete groups is stated explicitly, as well as the fact that dimensional (unit) symmetry is not addressed.", "review_text": "", "extra_scores": {"ethical_concerns": "This paper is reasonably theoretical in nature; I have no ethical concerns about its impact.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper originally got 1 "Clear reject", 2 "Ok but not good enough - rejection"s, and 1 "Marginally above the acceptance threshold", all with relatively high confidences. The author rebuttals were successful and all the four reviewers raised their scores, resulting 1 "Marginally below the acceptance threshold", 1 "Marginally above the acceptance threshold", and 2 "Good paper, accept"s. The AC found that the theory is valuable to the community, although it is a direct application of existing invariant/equivariant theories in mathematics. Thus the AC recommended acceptance. Nonetheless, the authors should remove "gauge" in the paper title and revise according to the reviewers' (especially Reviewer StjN's) concerns.
Decision: Accept (Poster)