Scalars are universal: Equivariant machine learning, structured like classical physics
Authors: Soledad Villar
David W Hogg
Kate Storey-Fisher
Weichi Yao
Ben Blum-Smith
Venue: NeurIPS 2021 Poster (2021)
Keywords: invariant theory
group theory
physics
regression
machine learning
symmetry
scalars

Abstract: There has been enormous progress in the last few years in designing  neural networks that respect the fundamental symmetries and coordinate freedoms of physical law. Some of these frameworks make use of irreducible representations, some make use of high-order tensor objects, and some apply symmetry-enforcing constraints. Different physical laws obey different combinations of fundamental symmetries, but a large fraction (possibly all) of classical physics is equivariant to translation, rotation, reflection (parity), boost (relativity), and permutations. Here we show that it is simple to parameterize universally approximating polynomial functions that are equivariant under these symmetries, or under the Euclidean, Lorentz, and Poincaré groups, at any dimensionality $d$. The key observation is that nonlinear O($d$)-equivariant (and related-group-equivariant) functions can be universally expressed in terms of a lightweight collection of scalars---scalar products and scalar contractions of the scalar, vector, and tensor inputs. We complement our theory with numerical examples that show that the scalar-based method is simple, efficient, and scalable. 

Scalars are universal: Equivariant machine learning, structured like classical physics
Soledad Villar
David W. Hogg Flatiron Institute
Department of Applied Mathematics and Statistics Johns Hopkins University
divison the Simons Foundation
Kate Storey-Fisher
Center for Cosmology and Partiele Physics Department of Physics, New York University
Weichi Yao
Ben Blum-Smith Center for Data Science New York University
Department of Technology, Operations, and Statistics Stern School of Business, New York University
Abstract
There has been enormous progress in the last few years in designing neural net- works that respect the fundamental symmetries and coordinate freedoms of phys- ical law. Some of these frameworks make use of irreducible representations, some make use of high-order tensor objects, and some apply symmetry-enforcingg con- straints. Different physical laws obey different combinations fundamental sym- metries, but large fraction (possibly all) of classical physics equivariant translation, rotation, reflection (parity), boost (relativity), and permutations. Here we show that is simple to parameterize universally approximating polynomial functions that are equivariant under these symmetries, or under the Euclidean, Lorentz, and Poincaré groups, at any dimensionality The key observation is that nonlinear O(d)-equivas -ediivviivrr (and related- group-equivant versally expressed terms of lightweight collection scalars scalar products and scalar contractions of the scalar, vector, and tensor inputs. We complement our theory with numerical examples that show that the scalar-based method simple, efficient, and scalable.
functions can be uni-
Introduction
There is great deal of current interest in building machine-learning methods that respect exact or approximate symmetries, such as tramslation, rotation, and physical gauge symmetries [8 49
Some of this interest is inspired by the great success of convolutional neural networks (CNNs) [51]
which are naturally translation equivariant. The implementation of convolutional layers in CNNs has been given significant credit for the success of deep learning, in domain (natural images) in which the convolutional symmetry is only approximately valid. In many data-analysis problems in astronomy, physics, and chemistry there are exact symmetries that must be obeyed by any general- izable law or rule. Since the approximate symmetries introduced by convolutional networks help in natural-image domain, then we have high hopes for the value encoding exact symmetries for problems where these symmetries are known to hold exactly.
In detail, the symmetries of physics are legion. Translation symmetry (including conservation of linear momentum), rotation symmetry (including conservation of angular momentum), and time-
35th Conference on Neural Information Processing Systems (NeurIPS 2021).

---

translation symmetry (including conservation of energy) are the famous symmetries [69] But there are many more: there is a form of reflection symmetry symplectic symmetry that permits reinterpretations of positions and momenta; there are Lorentz and Poincaré symmetries (the fundamental symmetries of Galilean relativity and special relativity) that include velocity boosts; there is the generalization of these (general covariance) that applies in curved spacetime; there are symmetries associated with baryon number, lepton number, flavor, and color; and there are dimensional and units symmetries (not usually listed as symmetries, but they are) that restrict what kinds of quantities can be multiplied or added. If it were possible to param- eterize universal universally approximating function space that is explicitly equivariant under a large set of non-trivial symmetries, that function space would in some sense- contain within every possible law of physics. It would also provide basis for good new machine learning methods.
or CPT); there a
The most expressive approaches to equivariant machine learning make use of irreducible represen- tations of the relevant symmetry groups. Implementing these approaches requires a way to explic- itly decompose tensor products of known representations into their irreducible components. This is called the Clebsch-Gordan problem. The solution for SO(3) has been implemented, and there is recent exciting progress for other Lie groups
This is an area of current, active research.
Here we give an approach that bypasses the need for a solution to this problem. We find that, for a large class of problems relevant to classical physics, the space of equivariant functions can be constructed from functions only of a complete subset of the scalar products and scalar contractions of the input vectors and tensors. That is, invariant scalars are powerful objects, and point the way to building universally approximating functions that are constrained by exact, non-trivial symmetries.
Equivariance is the form of symmetry core to physical law: For an equivariant function, transform- ing the input results in an output representation transformed in the same way. An invariant function, on the other hand, produces the same output for transformed and non-transformed inputs.
Definition 1; Given a function f : X + y and group G acting on X' and in y as * (possibly the action is defined differently in X' and 3)). We say that f is:
G-invariant:
f(g
f()) for all x X, ge
(1) (2)
G-equivariant:
x(xx))= - x f(x) for x E X, gEG.
Our contributions: In this work we provide a complete and computationally tractable characteriza- tion of all scalar functions f (I + R, and of all vector functions h : (Rd)) Rd that satisfy all of the symmetries of classical physics. The groups corresponding to these symmetries are given in Table I they act according to the rules in Table 2 The characterization we provide is physically principled: It is based on invariant scalars. It is also connected to the symmetries encoded in the Einstein summation rules, common notation in physics to write expressions compactly but that also allows only equivariant objects to be produced (see Appendix F
Our characterization is based on simple mathematical observations. The first is the First Fundamental Theorem of Invariant Theory for O(dy: a function of vector inputs returns an invariant scalar if and only if can be wrillen as a function only of the invariant scalar products of the inpul vectors [33 Section IIAA.9]. There is a similar statement for the Lorentz group O(1, d). The second observation is that a function of vector inputs returns an equivariant vector if and only if can be written as a linear combination of invariant scalar functions times the input vectors. In particular, if h (Rd)" + Rd of inputs v1.-.; Un O(d) or O(1, d'-equivarian,, then it can be expressed as:
a
h(m,, U2,--, mn) = Σ Se (155, M)M))=))) Ut
where St can be arbitrary functions, but if i is a polynomial function the fe can be chosen to be polynomials. In other words, the O(d) and O(1, di-equvvarian vector functions are generated as module over the ring of invariant scalar functions by the projections to each input vector. In this ex- pression, (--> denotes the invariant scalar product, which can be the usual Euclidean inner product, or the Minkowski inner product defined in terms of a metric A (see Table D]:
Euclidean: (v,, 0i) = wn Vj
Minkowski: (v,, m3) = v. A
Our mathematical observations lead to a simple characterization for very general class of equivari- ant functions, simpler than any based on irreducible representations or the imposition of symmetries

---

Orthogonal
((d) = (Q E Ruxd, Q QQ =
Rotation
SO(d) = (Q E Rudd QQQQ QQ' Ta. det(() =1
Translation
T(d) {w E R4)
Euclidean
B(d) T(d) O(d)
Lorentz
O d) = {Q E [(++) QQ A, A diag -1, -,,.., ....,,--1)) 1])}
Poincaré
I0(1, d) = T(d + 1) x O(1,d)
Permutation
S, (a [n] [n] bijective function)
Table The groups considered in this work.
Orthogonal: Lorentz Translation
(V1, un) (Q u1.... Qu,)
(Ul,.. Un) (e) + w,... Uk + w, U,+1s.., Un)
(where the first h vectors are position vectors)
Euclidean; Poincaré
(w, Q) (V1,.., Un) = (Q w..., U w, N.Q Uu+ vu)) (V1..., . v.) (Vo(1)... Vo(n))
Permutation
Table 2: The actions the groups on vectors. For the Euclidean group, the position vectors are positions of points; for the Poincaré group, the position vectors are positions of events.
through constraints (these methods are currently state of the art; see Section 2 This implies that very simple neural networks based on scalar products input vectors-thal enormously generalize those in [80] can universally approximate invariant and equivariant functions. This justifies the nu- merical success of the neural network model proposed in [8)] and mathematically shows that their method can be extended to a universal equivariant architecture with respect to more general group actions. We note that the formulation in B might superficially resemble an attention mechanism [7,, but it actually comes from the characterization of invariant functions of the group, in particular the picture for SO(d)-invariant functions a little different (see Proposition 5
In Section (and Appendix we describe the state of the art for encoding symmetries, the ex- pressive power of graph neural networks, and universal approximation of invariant and equivariant functions. In Section 5 we mathematically characterize the invariant and equivariant functions with respect to the groups in Table D In Section we present some examples of physically meaningful equivariant functions, and show how to express them in the parameterization developed in Section B In Section
5 and Appendix G we discuss which (of all possible) pairwise inner products one ought to provide, and in Section 6 discuss some limitations of our approach. We present numerical ex-
periments using our basedr-based approach compared to other methods in Section (see also (96)).
We also note that the symmetries considered in this work are all global symmetries, as they act on all points in the same way. Our characterization thus does not obviously generalize to all gauge symmetries, which are local symmetries that apply changes independently to points at different locations (see [1 (22) ). That said, we believe our model could be made general enough encompass gauge symmetries if we replace the global metric by any position-dependent metric Az. In this case, spatially separated vectors would need to be propagated the same point such that they can input to locally invariant functions, and this can be done with parallel transport. The parallel transport operations would also have to obey our invariance characterization, but we believe this is possible to do; will explore it in future work.
2 Related work
Group invariant and equivariant neural networks: Symmetries have been used successfully for learning functions images, sets, point clouds, and graphs. Neural networks can be designed to parameterize classes of functions satisfying different forms of symmetries, from the classical (ap- proximately) translation-invariant convolutional neural networks 152 as well new approaches that enforce additional symmetries (rotation, scale) on these networks 30 architectures that define permutation invariant and equivariant functions on point clouds pioneered
-to more recent
3

---

by deep sets and pointnets graph neural networks 33
99 73
74
to permutaaionn-equiaaiin functions on graphs expressed as
2 30
For instance, deep sets and pointnets parameterize functions on (Rd)) that are invariant or equivari- ant with respect to the group of permutations S,, acting as in Table 2 Invariant theory shows that all invariant and equivariant functions with respect to such actions can be approximated by easily char- acterized invariant polynomials [93]. However, the permutation group can act in significantly more complicated ways. For instance, graph neural networks are equivariant with respect to a different action by permutations (conjugation) that is much harder to characterize (see Appendix A
Since being introduced by pecially in computer vision 36
99 deep learning on point clouds has been extremely fruitful, es- Recently, new symmetries and invariances have been incorporated into the design of neural networks on point clouds; especially invariances and equivariances with
73
respect to rigid motions such translations and rotations. Many architectures have been proposed to satisfy those symmetries such as 355 based on irreducible representations (irreps), (50) 101 based on convolutions, [29] employing spherical harmonics and irreps. [02 using quaternions, and
(88 applying set of constraints to satisfy the symmetries. Most of the implementations of the ap-
proaches mentioned above (except for (X are limited to 2D or 3D point clouds. We provide an
overview of the main approaches below and in Appendix A
Recently 7 developed approach to enforcing permutation equivariance in neural network lay- ers using parameter sharing, which can then model other symmetry groups. The weight sharing ap- proach is significantly simpler to implement than the ones described above, and it has been proven to be very successful in practice, with several applications [89 (0), including autonomous driving (38 Characterizing the space of the functions this approach can express is an interesting open problem.
Universal approximation via linear invariant layers and irreducible representations: Univer- sally approximating invariant functions can be obtained by taking universal non-invariant func- tions and averaging them over the group orbits 197 68 However, this approach is not practical for large groups like S,, or infinite groups like O(d). A classical result in neural networks shows that ford-forward networks with non pol-polylo activations can universally approximate continu- ous functions [54] This arguably inspired the use of neural networks that are the composition of linear invariant or equivariant layers with compatible non-linear activation functions to create ex- pressive equivariant models 4 6 65) Linear G-equivariant functions can be written in terms of the irreducible representations of G (we explain this and refer the reader to related literature in Ap- pendix A.. However, the explicit parameterization of the linear maps is only known for few groups (for instance, S0(3)-equivariant linear maps are parameterized using the Clebsh-Gordan coefficients Moreover, very recent work 2 shows that the classes of functions defined in terms of neural networks over irreducible representations in (29 85 are universal. In particular, every con- tinous -qquiirquvva function can be approximated uniformly in compacts sets by those neural networks. Despite universality, there is a limitation to this approach: Although decompositions into irreps are broadly studied in mathematics (also as plethysms), the explicit transformation to the ir- reps is not known or possible for general groups. This is in fact an area of current, active research, but the implementation
29
3
where there has been recent exciting progress for other Lie groups is still limited.
Expressive power of (non-universal) neural networks on graphs: Graph neural networks express functions on graphs that are equivariant with respect to certain action by permutations. However, the architectures that are used in practice, typically based message passing (30 or graph convolu- tions 23 20 are not universal in general. Implementing a universally approximating graph neural network using the formulation from the previous section would be prohibitively expensive. There is work characterizing the expressive power of message passing neural networks, mainly in terms of the graph isomorphism problem [94 (7
X4 [4], and there research on the design of graph networks that are expressive enough to perform specific tasks, like solving specific combinatorial
16
59
optimization problems O
[K]
70
95.
Machine learning for physics with symmetry preservation: Machine learning has been applied extensively problems in physics. While many of these problems require certain symmetries- in many cases, exact symmetries-mos applied projects to date rely on the data to encode the
symmetry and hope that the model learns it. For instance, CNNs are commonly used to classify
galaxy images; data augmentation is used to teach the model rotational symmetry [39 B 22 BT] One welln example is the Kaggle Galaxy Challenge, a classification competition based on the

---

Galaxy Zoo project [58]: the winning model improved performance by concatenating features from transformed images of each galaxy before further training [2T]
There have been recent successes in enforcing physical symmetries in the architecture of the mod- els themselves 98, for instance, in weather and climate modeling [ and in modeling chaotic dynamical systems such as turbulence [89
In quantum many-body physics, recent work has shown that the symmetries of quantum energy levels on lattices can be enforced with gauge equiv- S8
90
60 78 [1. There is significant work on imposing permutation symmetry in jet assignment for high-energy particle collider experiments with self- attention networks [27 533. In molecular dynamics, rotationally invariant neural networks have been shown to better learn molecular properties [2 91 82), and Hamiltonian neural networks have been constructed to better preserve molecular conformations [55]. More broadly, Hamiltonian networks have been shown to improve physical characteristics, such as better conservation of energy, and [03]. and Lagrangian neural networks can also enforce conservation
ariant and invariant neural networks I7
J00
to better generalize 34 laws
19.
Invariant theory as a basis for enforcing symmetry in neural networks: We are aware of two lines of prior work that develop approaches related to the one taken here: 56 and 35 77
In 156 certain tasks in turbulence modeling and materials science are considered, which have built- in 0(3) and (in the latter case) octahedral symmetry. Each of these problems involves specific representation of the given symmetry group, for which an explicit generating set for the invariant algebra (there called an "integrity basis") is known. The authors construct and test models that learn an invariant function, built on these generating sets. In (57]. the turbulence example is taken up again, this time with the goal of learning an equivariant 2-tensor.
The idea of using the invariant algebra to enforce physical symmetries is also contemplated in 355 In this work, the authors are focused on developing the underlying invariant theory in the case of simultaneous Lorentz and permutation invariance. In the followup work [37] this idea developed into three models, using different generating sets, which are tested against each other.
The present work shares with these works the idea of using invoriant-theoretic descriptions of invari- ant functions to hard-code physical symmetries into a neural network that models a physical system. We work in somewhat greater generality. While 56 57 focus on specific (small) representations of O(3) and finite extensions, and while 35 37 are focused on invariant functions and restricted to linearly reductive groups, we consider both invariant and equivariant functions for almost all groups relevant to physics, including groups (Euclidean and Poincaré) that are not reductive, although their invariant theory remains under control.
Inductive bias benefits of incorporating symmetries: The value of incorporating exact symme- tries in machine learning has been recently established empirically in several applications (see for instance (90).. Mathematical theory has been developed to explain how much one can improve in but many questions remain open.
terms of sample complexity and generalization 166 26
3 Equivariant maps
In this Section we provide simple characterization of all invariant and equivariant functions with respect to the actions in Table _ by the groups in Table [ In what follows, U1, v2, Un will be vectors in IR G will be group acting in E. and (IRd)"
2 f (IR-)" - E will be an invariant function with respect to the action, and h (Rdy" - Id will be an equivariant function with respect to the same action. V will denote a x n matrix whose columns are the 0/ vectors.
in Table
(dd and SO(d) invariance and equivariance: The following classical result in invariant theory (e.g.. 93 Section II.A.9) shows that O(d) invariant functions are functions of the scalars u/ vy-
Lemma (First Fundamental Theorem for O(d)): If f is an O(d)-invariant scalar function of vector inputs U1,.., Un E Rdd then f(e,, u vx) can be written function of only the scalar products of the U2. That is, there is a function g(.) such that
f(r,, U2 v.) = vVVVVV) = ((v o))E==))
(5)
5

---

Proof. Given M = VTV E I we can reconstruct v..... en modulo the orthogonal group ((d) by computing the Cholesky decomposition of M (see for instance [66] p. 174). Therefore, the function/ f is uniquely determined by the inner product matrix VTV.
The classical theorem also includes the fact that if is polynomial (in the entries of the Uz uy's), then can be taken to be polynomial. In Section S we observe that the function g can be determined by small subset of the scalars V/ Uj. There an analogous statement for SO(d) (again see [93 Section IIA::9]):
Lemma 2 (First Fundamental Theorem for SO(d)): If f is an SO(d)-invarian scalar function of vector inputs U1 U2.... Un E R then f(11, u2..., Un)) can be written as a function of the scalar products of the U, and the d x d subdeterminants of the d x n matrix V.
Lemma 3: If is an O(d)-qquivariant vector function of n vector inputs 01; U2, Una then 1(11, U2, .n) must lie in the subspace spanned by the input vectors U1, U2,.., Un.
Rd be orthonormal basis of the orthogonal complement to span(e1. (11... k,). Then we can write h(v1,.., Un) = Σi= 244% + Σi= Bew, for some choice of or, St. We claim that the equivariance of h implies Bg = 0 for t = 1, I,.., Consider Q E Od) such that Q(v)= v for all span(u1, ... vn), and ((w) = for all in the orthogonal complement.
Proof. Let {wi,.., },}
Since (Qu1..., Qv.)) (U1, . Un) we have h(Qui,... Qu.) Σí= a + Σ Srw, while Q(h(vi..., )) = Ei O4224/ Ei= Bew. Therefore equivariance implies that all Be =
0. L
Note that Lemma 3 doesn't hold for SO(d): although when the codimension of span(U1, ... 2n) is or 2 2 a similar argument works, the situation dime span span(v).-.. (v)...., v.) = d - 1 breaks the proof, and there do exist equivariant vector functions that don't lie in the span when it has codimension 1. For instance, the cross product of two vectors RR an SO(3)-equivariant function that is not in the span of its inputs. Proposition 5 shows that generalized cross products are the only way that (O(d) -equivvquvaaiiatt vector functions can escape the span of the inputs. We further discuss this in Appendix F Proposition 4 below gives characterization of all -qqiivarri functions in terms of the scalars. We prove it in Appendix B
Proposition 4: If h is an O(d)-equivariant vector function of n vector inputs U1, ,,, Cn, then there are n (dd--invariant scalar functions fe(-) such that
hV1, - - un) Ei= (1 u2, Un) Uz-
Moreover, if h is polynomial function, the fc can be chosen to be polynomial.
The equivariant scalar functions form module over the ring of invariant scalar functions. Propo-
sition states that this module is generated by the projections (V1,..., 2n)) + vy. Proposition extends this result to SO(d) in terms of the generalized cross product. The definition of the general- ized cross product and the proof of Proposition are in Appendix B
Proposition 5: If h is an (O(d))-quivarr) characterization of Proposition holds, except when 01, U2, Un span (d -iimeemsiii space. In that case, there exist O(d) -nvariint scalar functions fe(-) such that
vector function of vector inputs 11. u,, then the
MV(1,, U2, vn) = Ei-1S((o U Un) U2 - [se() [s(U,, U2,.., vn)us,
us
where [n] := (1..... n). (d)) is the set of all (d- - 1) subsets of [n], and us represents the general- ized cross product of vectors vyy with j E S (taken in ascending order). Morcover, if is polynomial, the fi and fs can be taken to be polynomial.
E(d) invariance and equivariance: When modelling point clouds, we may want to express func- tions that are translation invariant or equivariant with respect to a subset of the input vectors (for instance, position vectors are translation equivariant). To this end, we consider the group of transla- tions parametrized by w E Rdd acting on the vectors in (Rd) by translating the position vectors, and leaving everything else unchanged: w*((v,,.., Uk U,+ Un)) = (v) (w,+ , Uk+ 1k ). In this Section we characterize all functions that are translation and rotation invariant/equivariant In the exposition below we assume for simplicity that all vectors are position vectors, but the results generalize trivially to mix of vectors.

---

Lemma 6: Any translation-invann function f: (IR) th + IF' with inputs .,..., can be written uniquely as f(V1, u2.... , un) f(12 .,,.., Un 01), where f (124)=1 + Re is arbitrary function. If is polynomial, f' polymomial, and vice versa. If f equivariant for the action of any given subgroup G C GT, R), then so is and vice versa.
The proof is given in Appendix There is nothing special about subtracting v1; there exist more natural choices to express translation invariant functions. For instance, one classical way in physics to express translation invariance is to take class representatives of the form (Vi,.., Un) where Ee = 0 (for example, subtracting the center of mass). Proposition 7 characterizes the space of eddivaivivaiiant functions that are also translation invariant or translation equivariant. The proof is in Appendix [
Proposition 7: An -quivvavirat function h (Rd)) + R that is translation invariant can be written as a where the St are O(d) and translation invariant and Ei= fe(11 02,-- vn) = 0. Similarly, if is translation-equivariant then we can choose >x fe(U1, U2,... Un) =
Lorentz symmetry: The Lorentz group acts on Minkowski spacetime as Lorentz transformations, which keep the metric tensor invariant. Lorentz tramsformations relate space and time between in- ertial reference frames, which move at constant relative velocity; spacetime intervals are invariant across frames. The group is made up of spatial rotations in the three space dimensions and linear velocity "boosts" along each dimension. This set of coameeriis-requaeee not O(4), but rather the non-compact group 0(1,3) defined in Table
for special relativity-
The characterization of invariant and equivariant functions we obtained for the orthogonal group can be extended to the Lorentz group, obtaining a very similar result, summarized in Proposition g and proven in Appendix D
Proposition 8: A continuous function h (Id+1)" + Rd+1 is Lorentz-equivariant (with respect to the action in Table
2 if and only if
h(U1,.., un) =Z= fe(1i..., Un) Pt
(8)
where St 11 -> TR are Lorantz-invariant scalar functions. Moreover, the functions fa are uniquely determined by the pairwise Minkowski inner product (.,))
[u Mx) = Ie ((vs, (y)..==))
If h is polynomial, the ft and corresponding Or can be taken to be polynomial.
Poincaré symmetry: The Poincaré group combines translation symmetry with Lorentz symmetry. Together these complete the symmetries of special relativity, forming the full group of spacetime transformations that preserve the Minkowski metric. The generalization from Lorentz-equivarian formulation Poincaré-equivarian formulation is similar to the generalization from O(d) to E(d): The position vectors take on a special role, in which only differences of position can appear as vector inputs the functions; all other vectors act unchanged.
Proposition generalizzss the results above to the Poincaré group action. As before, we assume that all vectors position vectors for simplicity. The proof is analogous to the proof of Proposition
Proposition 9: A continuous function : (IRd+1)" -> IR++1 is Poinearé-equivariant (with respect to the action in Table 2 if and only if
h(u,,.., (x) St v.)
(10)
where the J+ (Rd+lyn + R are translation and Lorentz-invariant scalar functions, determined as in 2 by the pairwise Minkowski inner products 4 but also satisfying Ei=1 fc(v1, D2,... vn.) = 1.
Furthermore, if h is polynomial, the and corresponding 9t (as in can be taken to be polyno-
mial.
Permutation invariance and equivariance: Most physics problems are permutation-invarian in that once you know the masses, sizes, shapes, and of the objects in the problem, the physical predictions are invariant to labeling. In particle physics, this invariance raised to a fundamental symmetry; fundamental partieles (like electrons) are identical and exchangeable.
7

---

The characterization of permutation invarrinn functions with respect to the action in Table 2 is clas- sical [93] pp. 36-39]. Here we prove an extension that describes permutation-invarieee vector func- tions that are also Odd)-qquivrrian. The proof is in Appendix
Proposition 10: Let h (Re))" TR be -(d)-aquivariant (or continuous O(1, d - 1 ))-quivaariant and also permutation-invarian with respect to the action in Table Then can be written as
h(v v.) = Ei=1 f(ee, v1... 1/-1, 11-1,.., Un) Ut:
(11)
where f: (IR")" R is O(d)-invariant (or O(1, d 1-invariint) and permutation-invariaat with respect to the last n inputs.
Proposition 11 proven in Appendix
B eextenss the characterization above to permutation equivariant
functions (see also 35 48
Proposition 11: Let h : (Rd)" + (IE")" be O(d)-qquivariant (or continuous O(1, d -qquivaari))))) and also permutation-equivariiee
with respect to the action in Table 2
Then h can be written as
= (hi,... An)) where each h; (Idd)" -> Rdd is equi-iquvvar (or continuous O(1, 1)- equivariant) and
h. v.) = Ei 1(( (vi..., (.) Ur:
(12)
where all the 1 (IR")" + R are O(d)-invariant (or O(1, d ((r -iiiaariian,,, and for all i, = I,.., and all a E Sn we have
f0 (y (Pott)) =
(13)
4 Examples
Here we briefly state two classical physics expressions that obey all the symmetries, and show how to formulate them in terms of invariant scalars.
Total mechanical energy: In Newtonian gravity, the total mechanical energy T of n particles with scalar masses my vector positions r,, and vector velocities et is a scala function:
T Σ my |
Σ R
Gm;
(14)
In-ryl
where G is Newton's ''s constant (a fundamental constant, and hence scalar). Since |a| = (a (aTa)' /2, this expression (14) is manifestly constructed from functions only of scalars m, and scalar products of vectors. It is also worthy of note that the positions T; only appear in differences of position.
Electromagnetic force law: The total electromagnetic force F acting on test particle of charge, 3-vector position, and 3-vector velocity (q, r, u) given set of other charges (g;, (e, Us) is a vector
function 41
F
n i== kqq |
Ei= k
vx (x(exx(r-re)) 1rrr-113
(15)
electrostatic force
magnetic force
where & an electromagnetic constant, cs the speed of light, and x b represents the cross product that produces a pseudo-vects vector perpendicular to vectors a and b according to the right-hand rule. This doesn't 't obviously obey our equivariance requirements, because cross products deliver parity- violating pseudo-vectors: these can't be written in -33)-equivariant form. However, cross of cross of vectors is vector, so this expression in fact -quivvaav,,,,, as are all forces (because forces must be O(3)-cquivariant vectors in order for the theory to be sell-consistent).
We can expand the vector triple product using the identity (b x e)
(a (aTc) (a b)
(r
ri)
(v (r T:)) u. (e (v) (r-r;)
F
k kgq
kqq
r:
c r r
i=1
(1- 1 T Di
(1- 2e
(r
(u (r r:))
kgg
kggs
(16)
'The energy is not scalar in special relativity or general relativity, but it is scalar in Newtonian physics.
8

---

where the quantity u oy is the scalar product of the velocities. All of the quantities are now straight- forwardly functions of invariant scalar products times the input vectors.
How many scalars are needed?
Our analysis in Section shows that the invariant and equivariant functions of interest (under actions in Table 2 from groups in Table D with input vectors U1,.., Un, can be expressed in terms of the This greatly simplifies the parameterization of such functions, but it signif- icantly increases the number of features if n > d. In Appendix G we remark that the scalars can be uniquely determined by a small subset of size approximately (d (d+1) n. This is related to the rigidity theory of Gram matrices (77 that answers when there exists unique set of vectors that realize partial set of distances, and it is closely related to the low rank matrix completion problem [83]. Furthermore, there is vast literature studying high probability robust reconstruction of all scalars from a random subset via convex optimization techniques [1]. Recently developed optimization techniques on Gram matrices could provide efficient algorithms to learn invariant and equivariant functions 42
scalars (ee "=)=
6 Limitations and caveats
Although the principal results presented here work for many groups, and work naturally at all spatial dimensions d (unlike methods based on irreps, for example). they do not solve all problems for all use cases of equivariant machine learning. For one, there are myriad groups-and especially discrete groups that apply to physical and chemical systems where invariant and equivariant functions do not have such nice characterization. Such is the case for the GNNs discussed in Appendix A
One example of a situation in which our formulation might not be practical is provided by multipole expansions (for example, those used in the fast multipole method (5] and n--ody networks ) In the fast multipole method, a hierarchical spatial graph is constructed, and high-order tensors used aggregate information from lower-level nodes into higher-level nodes. This aggregation is concise and linear when it is performed using high-order tensors; this aggregation is hard (or maybe impossible) when only scalars can be transmitted, without the use of the irreps of the relevant symmetry groups This is a research direction we are currently exploring.
Another example that suggests that universal functions might be cumbersome is the current forms in which classical theories-suue as electromagnetism and general relativity are written. For example, in Section E we showed that the electromagnetic force law can be written in the form of functions of scalars and scalar products times vectors, but that is not how the theory is tradi- tionally written. It is traditionally written in terms of the magnetic field (a pseudo-vector) or the electromagnetic tensor (an order-2 tensor). As another example, general relativity is traditionally written in terms of contractions of an order-4 curvature tensor. That is, although the theories can in principle be written in the forms we suggest, they will in general be much more concise or simple or clear in forms that make use of higher-order or non-equivariant forms.
Finally, although our results apply to many physically relevant groups, they do not encode all of the symmetries of classical physics. For example, one critical symmetry is the dimensional or units sym- metry: You cannot add or subtract terms that have different units (positions and forces, for example). This symmetry or consideration has implications for the construction of valid polynomials. It also implies that only dimensionless (unitless) scalar quantities can be the arguments of large classes of nonlinear functions, including exponentials sigmoids. These additional symmetries must be enforced at present with some additional considerations of network architecture or constraints.
We also note that in this work we have characterized global symmetries, which act each point in the same way. Our characterization does not obviously generalize to gauge symmetries, which local symmetries that apply changes independently to spatially separated points (see e.g. [II). That said, we believe our model could encompass general gauge symmetries if we replace the lo- cal metric by any position-depennent metric A,. Recent work has shown that equivariance under gauge symmetries is possible in the realm of convolutional neural networks by defining coordinate- independent kernels [92] In our case, we would have to propagate spatially separated vectors the same location in order to pass them to a locally invariant function, requiring the operation of parallel

---

MLP
101
104
MLP+Aug
EMLP (NLP-5015)
109
Scalors (I)
Test MSE
10-1
1
10-2
5013)
10-3
10
EMLP-
Scalars
10a
10a
10a Training Set Size
104
Training Set Size
Figure 1: Test error a function of training size for (Left) the O(5)-invariant task, and (Right) the O(3)- equivariant task. Scalars (I) denotes the MLP model using the scalars method for 0(5)-invariance construction, and Scalars (E+P) denotes MLP model using scalars method -()-evuvveee and permutation invariance construction. MLP denotes standard multilayer perceptron, and MLP+Aug denotes an MLP that been trained with data augmentation
given symmetry group. EMLP-G denotes EMLP models from A with different relevant symmetry groups G. For both tasks, the scalar method outperforms all other methods. The shaded regions depict 95% confidence intervals taken over runs.
transport. This operation would then also have obey our invariance characterization; believe this is possible and we will detail it in future work.
7 Numerical experiments
We demonstrate our approach using scalar- based multi perceptrons (MLP) two toy learning
tasks from [28]: an O(S)-invriiant task and an equivvaraa task. Further numerical experiments with these methods applied to dynamical systems appear in (96). The code is available GitHut
and it reuses much of the functionality provided by EMLP (28]
0(S)-invarian task: Given observations of the form (zf a5.
=) where #f. TR and
an O(5)-invariant scalar function, we aim to learn f. In this case f is the example from 28
|
x x2
f(11, ( II)
11, E
(17)
2
|
|
We model f using Lemma ! namely f(z1, = g I1,
x1
22 x2) where g (R)
+ R
could be any function. We learn g by implementing as an MLP.
-uuiviquivr task: In this task, the data are n = 5 point masses and positions (m, x)) and the goal is to predict the matrix I = Σi m2 xz - -x,²] ). To this end we aim to learn
h: 3335 3x3
(18)
(m; F4 ) + I.
The function is -33)-qquivariant in positions, and Ss-invariant, namely, for all Q 0(3) and E S,, we have h( (1)) Q0 ) Qh((m;, x.) By Proposition 10 we know there exists functions for fr. and f2 such that
5 Σ
h((m,, x))_.))
folæT x4 mi, {oy 21, my, milu,es) x,pe
5
Σ (>1=
f1 xz, m., mj, 1 æı, my, mi mihaiss)) T.a + fello, æj my m, m,)
Here the set notation means that the function f; is permutation-invariat with respect the inputs in the set. The function h is -uuiiqquia by construction (see Section 3). We model fo. fi and fa with MLPs on deep sets 100..
10

---

Acknowledgements: It is pleasure to thank Tim Carson (Google), Miles Cranmer (Princeton), Johannes Klicpera (TUM), Risi Kondor (Chicago), Lachlan Lancaster (Princeton), Yuri Tschinkel (NYU), Fedor Bogomolov (NYU), Gregor Kemper (TUM), and Rachel Ward (UT Austin) for valu- able comments and discussions. We especially thank Gerald Schwartz. (Brandeis) for indicating the line of the polynomial argument to and Mare Finzi (NYU) for their help with the EMLP code- base. SV was partially supported by NSF DMS 2044349, the NSF-Simons Research Collaboration on the Mathematical and Scientific Foundations of Deep Learning (MoDL) (NSF DMS 203 1985), and the TRIPODS Institute for the Foundations of Graph and Deep Learning at Johns Hopkins University. KSF was supported by the Future Investigators in NASA Earth and Space Science and Technology (FINESST) award number SONSSC20K1545.
References
[1] A. Alex, M. Kalus, A. Huckleberry, and von Delft. A numerical algorithm for the explicit calculation of SU(N) and SL(N, C) Clebsch-Gordan coefficients. Journal of Mathematical Physicx, 52(2):223507, 2011.
[2] B. Anderson, T. S. Hy, and R. Kondor. Cormorant: Covariant molecular neural networks. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Bil Alché-Buc, E. Fox, and R. Garnett, editors, Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[3] A. K. Aniyan and K. Thorat. Classifying radio galaxies with the convolutional neural network. The Astrophysical Journal Supplement Series, 230(2):20, Jun 2017.
[4] W. Azizian and M. Lelarge. Characterizing the expressive power of invariant and equivariant graph neural networks. arXiv:2006. 15646, 2020.
[5] R. Beatson and L. Greengard. A short course on fast multipole methods. Wavelets, Multilevel Methods and Elliptic PDEs, 1:1-37, 1997.
[6] Y. Bengio, A. Lodi, and A. Prouvost. Machine learning for combinatorial optimization: a methodological tour d d'horizon. European Journal of Operational Research, 2020.
[7] A. Bietti, L. Venturi, and Bruna. On the sample complexity of learning with geometric stability. arXiv preprint arXiv:2106.0711 07148, 2021.
[8] A. Bogatskiy, B. Anderson, J. Offermann, M. Roussi, D. Miller, and Kondor. Lorentz group equivariant neural network for particle physics. In International Conference on Machine Learning, pages 992-1002. PMLR, 2020.
[9] G. Bouritsas, F. Frasca, S. Zafeiriou, and M. M. Bronstein. Improving graph neural network expressivity via subgraph isomorphism counting. arXiv:2006.0922 09252, 2020.
[10] D. Boyda, G. Kanwar, S. Racanière, D. J. Rezende, M. S. Albergo, K. Cranmer, D. C. Hack- ett, and P. E. Shanahan. Sampling using SU(N) gauge equivariant flows. Phys. Rev. D, 103(7):074504, 2021.
[1]] M. M. Bronstein, J. Bruna, T. Cohen, and P. Velivckovi'c. Geometric deep learning: Grids, groups, graphs, geodesics, and gauges. ArXiv, abs/2104.13475, 2021.
[12] E. J. Candes and Recht. Exact low-rank matrix completion via convex optimization. In 2008 46th Annual Allerton Conference on Communication, Control, and Computing, pages 806-812, IEEE, 2008
[13] Q Cappart, D. Chételat, E. Khalil, A. Lodi, C. Morris, and P. Veličković, Combinatorial optimization and reasoning with graph neural networks. arXiv:2102.09544 2021.
[14] 7. Chen, I.. Chen, S. Villar, and J. Bruna. Can graph neural networks count substructures? In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Neural Information Processing Systems, volume 33, pages 10383-10395. Curran Associates, Inc., 2020.
[15] Z. Chen, L. Li, and J. Bruna. Supervised community detection with line graph neural net- works. International Conference on Learning Representations, 2019.

---

[16] Z. Chen, S. Villar, L. Chen, and J. Bruna. On the equivalence between graph isomorphism testing and function approximation with gnns. In Neural Information Processing Systems, pages I 15868-15876, 2019.
[17] K. Choo, G. Carleo, N. Regnault, and T. Neupert, Symmetries and many-body excitations with neural-netwood quantum states. Physical Review Letters, 121:167204, Oct 2018.
[18] T. Cohen, M. Weiler, B. Kicanaoglu, and M. Welling. Gauge equivariant convolutional net- works and the icosahedral enn. In International Conference on Machine Learning, pages 1321-1330. PMLR, 2019.
[19] M. D. Cranmer, S. Greydanus. S. Hoyer, P. W. Battaglia, D. Spergel, and S. Ho. Lagrangian neural networks, arXiv:2003.04633 04630, 2020.
[20] M. Defferrard, X. Bresson, and Vandergheynst. Convolutional neural networks on graphs with fast localized spectral filtering. In neural information processing systems, pages 3844- 3852, 2016.
[21] S. Dieleman, K. W. Willett, and J. Dambre. Rotation-invariat convolutional neural net- works for galaxy morphology prediction. Monthly Notices of the Royal Astronomical Society, 450(2):441-1459, Apr 2015.
[22] H. Domínguez Sánchez, M. Huertas- -omppay, M. Bernardi, D. Tuccillo, and J. L. Fischer. Improving galaxy morphologies for sdss with deep learning. Monthly Notices of the Royal Astronomical Society, 47663):3661-3677 Feb 2018
[23] D. K. Duvenaud, D. Maclaurin, J. Iparraguirre, R. Bombarell, T. Hirzel, A. Aspuru- Aspuru-Guzik, and R. Adams. Convolutional networks on graphs for learning molecular fingerprints. In Neural Information Processing systems, pages 2224-2232, 2015.
[24] N. Dym and H. Maron. On the universality of rotation equivariant point cloud networks. arXiv:2010.02449, 2020.
[25] A. Einstein. Die Grundlage der allgemeinen Relativitätstheorie. Annalen der Physik, 354(7):769 822, Jan. 1916.
[26] B. Elesedy and S. Zaidi. Provably striet generalisation benefit for equivariant models. arXiv preprint arXiv:2102.1033: 2021.
[27] M. J. Fenton, A. Shmakov, T.-W. Ho. S.-C. Hsu, D. Whiteson, and P. Baldi. Permu- tationless Many-Jet Event Reconstruction with Symmetry Preserving Attention Networks. arXiv:2010.09206, 2020.
[28] M. Finzi, M. Welling, and A. G. Wilson. A practical method for constructing equivariant multilayer perceptrons for arbitrary matrix groups arXiv:2104.09459 2021.
[29] F. Fuchs, D. Worrall, V. Fischer, and M. Welling. Se (3)-transformers: 3d tronssaanllaion equivariant attention networks. Neural Information Processing Systems, 33, 2020.
[30] J. Gilmer, S. S. Schoenholz, P. F. Riley, O. Vinyals, and G. E. Dahl. Neural message passing for quantum chemistry. In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pages 1263-1272. JMLR. org, 2017.
[31] R. González, R. Muñoz, and C. Hernández. Galaxy detection and identification using deep learning and data augmentation. Astronomy and Computing, 25:103-109, 2018.
[32] R. Goodman and N. R. Wallach. Symmetry, representations, and invariants, volume 255. Springer, 2009.
[33] M. Gori, G. Monfardini, and F. Scarselli. A new model for learning in graph domains. In Pro- ceedings. 2005 IEEE International Joint Conference on Neural Networks, 2005., volume 2,
pages 729-734. IEBE, 2005.
12

---

[34] S. Greydanus, M. Dzamba, and J. Yosinski. Hamiltonian neural networks. In H. Wallach, Larochelle, A. Beygelzimer, F. d' Alché-BBuc E. Fox, and Garnett, editors, Neural Infor- mation Processing Systems, volume 32. Curran Associates, Inc., 2019.
[35] B. Gripaios, W. Haddadin, and C. G. Lester. Lorentz-and permutation-invariee of particles. Journal of Physics A: Mathemutical and Theoretical, 15(15):155201, 2021.
[36] Y. Guo, H. Wang, Q. Hu, H. Liu, L. Liu, and M. Bennamoun. Deep learning for 3d point clouds: A survey. IEEE Transactions on Pattern Analysis and machine intelligence, 2020.
[37] W. Haddadin. Invariant polynomials and machine learning. arXiv preprint arXIv:2104.12333,
2021.
[38] K. Huang. Traffic agent movement prediction using resnet- based model. In 2021 6th Interna- tional Conference on Intelligent Computing and Signal Processing (ICSP), pages 136-139. IEEE, 2021.
[39] M. Huertas- Huertas-Company, R. Gravet, G. Cabrera- Vives, P. G. Pérez-Gonzélez, S. Kartaltepe, G. Barro, M. Bernardi, S. Mei, F. Shankar, P. Dimauro, and et al. A catalog of visual-like mor- phologies in the 5 candels fields using deep learning. The Astrophysical Journal Supplement Seriex, 221(1):8, Oct 2015.
[40] A. Ibort, A. López Yela, and Moro. A new algorithm for computing branching rules and Clebsch-Gordan coefficients of unitary representations of compact groups. Journal of Math- ematical Physics, 58(10): 2017.
[41]
Jackson. Classical Electrodynamics; 2nd ed. Wiley, New York, NY, 1975.
[42] A. Jalali, M. Fazel, and Xiao. Variational gram functions: Convex analysis and optimiza- tion. SIAM Journal Optimization, 27(4):2634- 2017.
[43] C. K. Joshi, T. Laurent, and X. Bresson. An efficient graph convolutional network technique for the travelling salesman problem. arXiv: 190.:::906.01227, 2019.
[44] N. Karalias and A. Loukas. Erdos goes neural: an unsupervised learning framework for combinatorial optimization graphs. 1XXiv:2006.104433 2020.
[45] K. Kashinath, M. Mustafa, A. Albert, J. Wu, C. Jiang, S. Esmacilzadeh, K. Azizzadenesheli, R. Wang. A. Chattopadhyay, A. Singh, et al. Physics-informed machine learning: case stud- ies for weather and climate modelling. Philosophical Transactions of the Royal Society A, 379(2194):20200033, 2021.
[46] N. Keriven and G. Peyré. Universal invariant and equivariant graph neural networks. In Neural Information Processing Systems, pages 7092-7101, 2019.
[47] T. N. Kipf and M. Welling. Semi-supervised classification with graph convolutional networks. 2016.
arXiv:1609.028
[48] J. Klicpera, F. Becker, and S. Günnemann. Gemnet: Universal directional graph neural net- works for molecules. arXiv preprint arXiv:2106.088 200.000000.,903, 08903, 2021.
[49] R. Kondor. N-body networks: a covariant hierarchical neural network architecture for learning atomic potentials. arXiv:1803.0 015588 2018
[50] R. Kondor, H. T. Son, H. Pan, B. Anderson, and S. Trivedi. Covariant compositional networks for learning graphs. arXiv:1801.0214 2018.
[51] Y. LeCun, Y. Bengio, and G. Hinton. Deep learning. Nature, 521(7533):336 444, 2015.
[52] LeCun, B. Boser, S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and D. Jackel. Backpropagation applied to handwritten zip code recognition. Neural Computation, 1(4):541-55 1989.
[53] J. Lee, I. Park, I. J. Watson, and Yang. Permuutrnnto Jet- let-Parton Assignment using Self-Attentin Network. arXiv:2012.03342, 2020.
13

---

[54] M. Leshno, V. Y. Lin, A. Pinkus, and S. Schocken. Multilayer feedforward networks with nonpolynomial activation function can approximate any function. Neural Networks, 6(6):8118861) 1993.
[55] Z. Li, S. Yang, G. Song, and L. Cai. Hamnet: Conformation-guuieed molecular representation with hamiltonian neural networks, 2021.
[56] J. Ling, R. Jones, and J. Templeton. Machine learning strategies for systems with invariance properties. Journal of Computational Physics, 318, 05 2016.
[57] J. Ling, A. Kurzawski, and Templeton. Reynolds averaged turbulence modelling using deep neural networks with embedded invariance. Journal of Fluid Mechanics, 807:155-166, 2016.
[58] J. Lintott, K. Schawinski, A. Slosar, K. Land, S. Bamford, D. Thomas, M. Raddick, R. C. Nichol, A. Szalay, D. Andreescu, P. Murray, and J. Vandenberg. Galaxy Zoo: morphologies derived from visual inspection of galaxies from the Sloan Digital Sky Survey*. Monthly Notices of the Royal Astronomical Sociery, 389(3): 119-111991119 2008.
[59] A. Loukas. What graph neural networks cannot learn: depth vs width. In International Conference Learning Representationss 2019.
[60] D. Luo, G. Carleo, B. K. Clark, and Stokes. Gauge equivariant neural networks for quantum lattice gauge theories, 2020.
[61] D. Luo, Z. Chen, K. Hu, Z. Zhao, V. M. Hur, and B. K. Clark. Gauge Invariant Autoregressive Neural Networks for Quantum Lattice Models. arXiv:2101.07243 2021.
[62] M. Lutter, C. Ritter, and J. Peters. Deep lagrangian networks: Using physics as model prior for deep learning. In 7th International Conference on Learning Representations (ICLR). ICLR, May 2019.
[63] H. Maron, H. Ben-Hamu, N. Shamir, and Y. Lipman. Invariant and equivariant graph net- works. In International Conference Learning Representations, 2018.
[64] H. Maron, E. Fetaya, N. Segol, and Y. Lipman. On the universality of invariant networks. In International Conference on Machine Learning, pages 4363-4371, 2019.
[65] H. Maron, O. Litany, G. Chechik, and E. Fetaya, On learning sets of symmetric elements. In International Conference on Machine Learning, pages 6734-6744 6744. PMLR, 2020.
[66] S. Mei, T. Misiakiewiez, and A. Montanari. Learning with invariances in random features and kernel models. arXiv preprint arXiv:2102.132199 2021.
[67] C. Morris, M. Ritzert, M. Fey, W. L. Hamilton, J. E. Lenssen, G. Rattan, and M. Grohe. Weisfeiler and leman go neural: Higher-order graph neural networks. Association for the Advancement Artificial Intelligence, 2019.
[68] R. Murphy, B. Srinivasan, V. Rao, and B. Ribeiro. Relational pooling for graph representa- tions. In International Conference on Machine Learning, pages 4663-4673, 2019.
[69] E. Noether. Invariant variation problems. Transport Theory and Statistical Physics, 1(3):186- 207, 1971.
[70] A. Nowak, S. Villar, A. Bandeira, and J. Bruna. A note on learning algorithms for quadratic assignment with graph neural networks.
2017.
[71] D. L. Pimentel-Alareón, N. Boston, and R. D. Nowak. A characterization of deterministic sampling patterns for low-rank matrix completion. IEEE Journal of Selected Topics in Signal Processing, 10(4):623-633 666, 2016.
[72] V. L.. Popov and F. B. Vinberg. Invariant theory. In Algebraic geometry IV, pages 123-278. Springer, 1994.
14

---

[73] C. R. Su. K. Mo, and L. J. Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 652-660, 2017.
[74] R. Qi. L. Yi, H. Su, and L. J. Guibas. Pointnet++ deep hierarchical feature learning on point sets a metric space. In Neural Information Processing Systems, pages 5105-5114, 2017.
[75] S. Ravanbakhsh, J. Schneider, and B. Poczos. Equivariance through parameter-sharing. Pro- ceedings of the 34th International Conference on Machine Learning, 2017.
[76] M. M. G. Ricci and T. Levi-Civita. Méthodes de calcul différentiel absolu et leurs applica- tions. Mathematische Annalen, 54(1):125-201, 1900.
[77] B. Roth. Rigid and flexible frameworks. The American Mathematical Monthly, 88(1):6-21, 1981.
[78] C. Roth and A. H. MacDonald. Group Convolutional Neural Networks Improve Quantum State Accuracy. wiXiv2204.00088, 2021.
[79] A. Sanchez-Gonzalez, V. Bapst, K. Cranmer, and Battaglia. Hamiltonian Graph Networks with ODE Integrators. arXiv: 190.99299,790,, 2019.
[80] V. G. Satorras, E. Hoogeboom, and M. Welling. E(n) equivariant graph neural networks. arXiv:2102.09844, 2021.
[81] F. Scarselli, M. Gori, A. C. Tsoi, M. Hagenbuchner, and G. Monfardini. The graph neural network model. IEEE Transactions on Neural Networks, 20(1):61-80, 2008.
[82] T. Schütt, O. T. Unke, and M. Gastegger. Equivariant message passing for the prediction of tensorial properties and molecular spectra, 2021.
[83] A. Singer and M. Cucuringu. Uniqueness of low-rank matrix completion by rigidity theory. SIAM Journal on Matrix Analysis and Applications, 1221-166,, 2010.
[84] B. Tahmasebi and S. Jegelka. Counting substructures with higher-order graph neural net- works: Possibility and impossibility results. urXiv:2012.03177 2020.
[85] N. Thomas, T. Smidt, S. Kearnes, L. Yang, L. Li, K. Kohlhoff, and P. Riley. Tensor field networks: Rotation-and translation-equivarianl neural networks for 3d point clouds. arXivv/802.088: 2018
[86] L. N. Trefethen and D. Bau III. Numerical linear algebra, volume 50. Siam, 1997.
[87] P. Veličković, G. Cucurull, A. Casanova, A. Romero, Liô, and Bengio. Graph attention networks. In International Conference on Learning Representations, 2018.
[88] T. Vieijra, C. Casert, J. Nys, W. De Neve, J. Haegeman, J. Ryckebusch, and F. Verstraete. Restricted boltzmann machines for quantum states with non-abelian or anyonic symmetries. Physical Review Letters, 124()), Mar 2020.
[89] R. Wang, K. Kashinath, M. Mustafa, A. Albert, and R. Yu. Towards physics-informed deep learning for turbulent flow prediction. In Proceedings the the 26th ACM SIGKDD International Conference Knowledge Discovery & Data Mining, pages 1457-1466, 2020.
[90] R. Wang, R. Walters, and R. Yu. Incorporating symmetry into deep dynamics models for improved generalization. In International Conference on Learning Representations, 2021.
[91] Z. Wang, C. Wang, S. Zhao, S. Du, Y. Xu, B.-L. Gu, and W. Duan. Symmetry-adapted graph neural networks for constructing molecular dynamics force fields, 2021.
[92] M. Weiler, Forré, E. Verlinde, and M. Welling. Coordinate independent convolutional networks isometry and gauge equivariant convolutions on riemannian manifolds, 2021.
[93] H. Weyl. The Classical Groups. Princeton University Press, 1946.
15

---

[94] K. Xu, W. Hu, J. Leskovec, and S. Jegelka. How powerful are graph neural networks? In International Conference on Learning Representations, 2018.
[95] W. Yao, A. S. Bandeira, and Villar. Experimental performance of graph neural networks on random instances of max-cut. In Wavelets and Sparsity XVIII, volume 11138, page 1 1380S. International Society for Optics and Photonics, 2019.
[96] W. Yao, K. Storey-Fisher, D. W. Hogg, and S. Villar. simple equivariant machine learning method for dynamics based on scalars, 2021.
[97] D. Yarotsky, Universal approximations of invariant maps by neural networks. arXiv: 10006 2018
[98] R. Yu, P. Perdikaris, and A. Karpatne. Physics-guides ai for large-scale spatiotemporal data. Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Min- ing, KDD "21, page 4088 4089, New York, NY, USA, 2021. Association for Computing Machinery.
[99] M. Zaheer, S. Kottur, S. Ravanbakhsh, B. Póczos, R. Salakhutdinov, and A. J. Smola. Deep sets. In Neural Information Processing Systems, pages 3394-3404, 2017.
[100] M. Zaheer, S. Kottur, S. Ravanbakhsh, B. Poezos, R. R. Salakhutdinov, and A. Smola. Deep sets. In Advances in Neural Information Processing Systems, volume 30, 2017.
[101] Z. Zhang, B.-S. Hua, D. W. Rosen, and S.-K. Yeung. Rotation invariant convolutions for 3d point clouds deep learning. 2019 International Conference on 3D Vision (3DV), pages 204-213. IEEE, 2019.
[102] Y. Zhao, T. Birdal, J. E. Lenssen, E. Menegatti, L. Guibas, and F. Tombari. Quaternion equivariant capsule networks for 3d point clouds. In European Conference on Computer Vision, pages 1-19. Springer, 2020.
[103] Y. D. Zhong, B. Dey, and A. Chakraborty. Symplectic ode-net: Learning hamiltonian dynam- ics with control. In 7th International Conference on Learning Representations (ICLR). ICLR, May 2020.
16


Reviews:
[{"review_id": "ZhmIT1G44_o", "reviewer": "Reviewer_K6Cj", "summary": "The paper introduces characterizations of invariant and equivariant functions with respect to various symmetries. Starting from basic invariance with respect to rotation, translation, and permutation, the authors build up their toolkit to parametrize equivariant functions with respect to common symmetry groups in physics, including Euclidean, Lorentz, and Poincaré groups. The authors finally provide an example of applying their ideas to build a message passing neural network for learning the symmetries in the n-body problem.", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Quality**: I checked most of the proofs and to the best of my knowledge, they seem to be correct.\n\n**Clarity**: The paper is generally well-written in terms of motivation and related work. The section on equivariant maps can be made easier to read if the Definition, Lemma, and Proposition titles are boldfaced, to distinguish these from the main text. Many of the sum symbols in the Appendix are missing upper and lower indices, and these should be added.\n\n**Originality**: The authors collect the results on the characterization of equivariant functions in a different perspective from most previous papers, which focus exclusively on a few specific physics problems and their corresponding functions. I am not entirely familiar with the literature to comment on the originality of the lemmas, though at least some results in the collection seem to be well-known or easy extensions (e.g. that of rotation and translation), while others seem to be original. \n\n**Significance**: On the good side, the paper provides a good collection of results for the characterization of invariant and equivariant functions with respect to various symmetries that may be useful for future research in this area. The message passing neural networks framework also gives a good way of incorporating different types of symmetries in one network. \n\nMy main concerns/suggestions for improvement are as follows:\n\n(1) The section on estimating the number of scalars only gives a high-level discussion, which is not very satisfying. This can be made better if the authors can extend their discussion into several theorems, and give an explicit comparison to results for previous networks (which are mentioned in Appendix A, though also given in very high level without explicit theorems).\n\n(2) While the framework makes sense to me in the theoretical sense, I am rather unconvinced on the practicality of the method. The paper would be much more solid if the authors can provide an empirical comparison of their proposed model with the current state-of-the-art models. \n\n**Additional Comments/Questions**: Is there proof for Proposition 9? I do not seem to see the proof either in the paper or the appendix. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper contains a limitation section that addressed many questions that I have. My remaining concerns are listed in the \"Main Review\" section.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "10"}, "extra_text": {}}, {"review_id": "TiRpHBmJSzo", "reviewer": "Reviewer_9FH3", "summary": "The authors theoretically discuss the universal approximating function space for various symmetries that frequently hold in classical physics. They find that all invariant scalar functions and equivariant vector functions can be expressed by a collection of scalar products and contractions. The authors’ findings can provide a simple characterization of such invariant and equivariant functions. It would be particularly useful when modeling and learning physical phenomena that naturally manifest such symmetries.", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality and quality: The authors clearly provide two important facts on representing invariant and equivariant functions with “scalars”. Then, they apply these facts for truly revealing some important symmetries, e.g., orthogonal, rotational, translational, Euclidean, Lorentzian, Poincare, and permutations. To me, these findings are interesting and technically sound. \n\nClarity: This paper is generally well-written. The motivation and main contribution of the paper are clearly presented. Although I am a beginner at invariant theory, I was able to follow the paper to a certain degree.\n\nSignificance: Enforcing symmetries is highly desirable for learning the laws of physics. This paper contributes a solid theoretical background for such a job. The authors also discuss the practical usefulness of their findings in Section 5 (in terms of scalability), Section 6 (in terms of limitations), and Section 7 (in terms of practical model building considerations), which makes their paper be highly appealing for practitioners like me.\n\nOverall, I think this paper is a good contribution to both machine learning and physics communities. Therefore, I would like to vote to accept this paper. However, since I am not an expert in this field, I cannot exhaustively evaluate this contribution, especially in terms of originality and quality. Thus, my rating may be updated after discussing with the authors or other expert reviewers. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors discuss the limitation of their work in Section 6 of the paper.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "OczDz1M_He", "reviewer": "Reviewer_D3qF", "summary": "The paper presents a mathematical idea for how to define a universal class of invariant and equivariant functions. These functions act on a collection of d-dimensional points, and output a single number or a vector. The group acts diagonally, applying the same transformation to each point. Using results from classical invariant theory, it is shown that such in/equivariant functions can always be written as a function of the collection of inner products between the d-dimensional vectors (or some subset thereof). This general idea is worked out for the groups that are most important in physics problems. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I found this paper very interesting to read. It contains a thorough mathematical study of in/equivariant functions of collections of points, and suggests a research direction that is different from the current mainstream approach in geometric/equivariant deep learning. The paper is well written and should be accessible to most machine learning researchers with a bit of effort.\n\nAlthough clearly a lot of work has gone into this work and the paper feels polished enough, the lack of implementation and experiments makes me question whether the paper is ready for prime time. In principle I am happy to accept purely theoretical papers, but in this case the main contribution is an *idea* for how to build models for learning problems in physics, that may or may not work well in practice. Although it is nice to know that this class of models/functions is universal in principle, the more important question is how well it works in practice. We cannot know this without experiments, and there are some reasons to think that this approach may not work that well (see below).\n\nAt its core, this paper suggests going back to the “feature engineering followed by learning” approach that was popular before deep learning where “end-to-end representation learning” is used. The features in this case are invariant inner products, which are shown to capture all the relevant information. Although feature engineering might work for the type of problems of interest in this work, we can’t know without experiments, and the overall trend of the field suggests that the DL approach is often superior.\n\nOne potential weakness of the proposed approach is that although it leverages the geometric prior of symmetry, it ignores scale separation / locality (See the book “Geometric Deep Learning: Grids, Groups, Graphs, Geodesics & Gauges” which proposes these as the two key geometric priors of geometric DL). That is, all points interact with all other points immediately at the input layer / feature computation stage. For many problems it may make more sense to have nearby points interact first, apply coarsening / pooling, and then gradually allow longer range interactions, as happens for instance in a convolutional network. \n\nThe paper suggests that one could make the whole model into a polynomial. Although many classical physics situations can be captured in this way,polynomials can be unstable to deformations that are not part of the group, and it is not clear if such models will share the benign optimization landscapes that we find with neural networks built as compositions of learnable linear layers and pointwise (non-polynomial and often non-analytic) nonlinearities.\n\nOf course these are just speculations and I would encourage the authors to test their idea on real-world problems to settle this question.\n\nI would note that the idea presented in this paper can be generalized via the use of computational invariant theory. The paper assumes a particular kind of action of the group (v1, …, vn) -> (g v1, …, g vn). If we stack the vi into a long vector, we see that we have a representation of the group that is block diagonal with identical blocks. Instead one could allow for an arbitrary representation. In case the group is algebraic (specified by a polynomial constraint such as Q^T Q = I) and the representation is algebraic, there exist algorithms that will automatically generate, via the use of Grobner bases, a (minimal) set of generators for the ring of polynomial invariants.\n\nA final quibble, which does not affect my overall judgement of the paper, pertains to the use of the word “gauge”. Gauge symmetry should not be confused with “coordinate freedom” as done in the abstract, or with global symmetries such as those in table 1 & 2. Properly understood, a gauge transformation is an automorphism of a principal bundle that fixes the base space (see e.g. “Mathematical Gauge Theory” by Hamilton). The general setting is one where we have a base space (e.g. space time), with at each point a “fiber” (a vector space, whose dimensions may or may not be related to directions in the base space). A gauge transformation applies a change of frame to each fiber independently, and (for continuous base spaces) the group of gauge transformations is thus infinite dimensional, unlike the groups considered here. The groups considered in this paper act on the base space R^d, and thus do not fix the base space as a gauge transformation would.\n\nIn conclusion, although this is a very substantial and interesting piece of work, it is still in the idea stage. Perhaps one can argue that the ideas are interesting enough to warrant publication as is, but personally I think that it would be better to first validate the ideas before publishing at a conference.\n\nTypos\np3 : “deep nets” -> “deep sets”\n\n\nPost rebuttal edit:\nSince the authors have now provided an implementation that appears to work well, I have increased my score.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "Gt_RnbxNN86", "reviewer": "Reviewer_WMkN", "summary": "The paper presents an approach to parameterization of functions that are invariant or/(and) equivariant under the action symmetries commonly present in physical systems. The construction of such parametrization is based on known mathematical results relating the dependence structure of invariant/equivariant maps with scalar and vector outputs.\n\nAuthors explicitly construct parameterizations for cases of scalar functions that are: (1) invariant and equivariant under O(d) and SO(d) groups; translation invariant; Lorentz symmetry; as well as several combinations. Relevance to classical laws of nature is provided via examples of physical laws in classical mechanics and electromagnetism.\n\nAuthors discuss limitations and caveats of the presented approach and note that the original construction can be improved in terms of the number of input features by relying on the result from the low-rank matrix completion problem.", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper discusses a relevant topic of efficient encoding of symmetry properties into neural networks by construction. The paper is written clearly and I especially would like to highlight a detailed section on limitations of the proposed method.\n\nWhile the presented approach is perfectly valid, I am not convinced that it is novel and significant enough to be of interest to a wider community. The idea of building-in invariances and equivariances by construction has been present in research papers on using ML for natural sciences for some time (e.g. [1]). While there is also interest in exploiting the symmetry properties in many other areas of machine learning besides science oriented applications, the presented approach is rather challenging to generalize to arbitrary/apriori unknown symmetries.\n\n[1] Ling, J., Kurzawski, A., & Templeton, J. (2016). Reynolds averaged turbulence modelling using deep neural networks with embedded invariance. Journal of Fluid Mechanics, 807, 155-166. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "This work does not have negative societal impact. The limitations are addressed adequately.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "-49lhd1MD4O", "reviewer": "Reviewer_Rogp", "summary": "This paper makes an interesting observation using standard ideas from classical invariant theory. Specifically, that the design of networks equivariant to various symmetry groups in classical physics can be done using a collection of invariant scalars. This suggests an attractive alternative approach for the design of equivariant networks. This approach circumvents the need for careful usage of the irreps, solving particular Clebsch-Gordan decompositions (which might not also be available) and so on; something inherent in a lot of current work in the area. The characterization of equivariant functions using such scalars are provided for various groups relevant to classical physics i.e. the orthogonal, rotation, translation, Euclidean, Lorentz, Poincare and permutation groups. Two examples relevant to physics are provided along with some observations on model building considerations. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Much recent effort has gone into the careful design of group/gauge equivariant neural networks that incorporate various task specific symmetries. A lot of activity has also been seen in the use of such networks in various applications in physics. Some of these approaches require irreducible representations of the group under consideration and also need to solve the relevant Clebsch-Gordan problem, which might not always be possible. The main contribution of the paper is the observation that for an array of groups relevant to classical physics (orthogonal, rotation, translation, Euclidean, Lorentz, Poincare and permutation), such approaches are not necessary. It provides a tractable characterization for the space of equivariant functions (to these groups) by showing that they can be constructed only from a subset of invariant scalars (scalar products and contractions of the input vector or tensor). \n\nThe key to the above observation are some classical and very well known results from invariant theory (going back to, and stated in their earliest forms, by Weyl). For n input vectors v_1,...v_n in R^d, Lemma 1 (which is due to Weyl) shows that that O(d) invariant functions are functions of the invariant scalars v_i ^T v_j. Lemma 2 gives an analogous result for SO(d) which uses d x d determinants of the inputs when arranged in a d by n matrix V. Proposition 4 extends the result in Lemma 1 to equivariant functions, while proposition 5 does so for Lemma 2. Similar results are shown for E(d), Lorentz, Poincare and permutation groups. Two illustrative examples are provided (total mechanical energy and electromagnetic force law), showing how to formulate them in terms of such invariant scalars -- which can then used in a machine learning system. Section 5 addresses the question on how many scalars might be necessary, leveraging results from literature adjacent to matrix completion and compressed sensing. This section shows that it is not necessary to use all the scalars obtained from pairwise operations, but only a relevant subset of them. Finally some model building considerations are discussed. \n\n\nCentral to the paper is essentially the observation that the design of equivariant networks in the context of physics (and more broadly) can be greatly simplified using the above ideas. Another relevant contribution is its scope -- it focuses on groups important in classical physics. The paper is also extremely well written, with most of the ideas fleshed out in admirable detail in the appendix. However, despite these, I feel conflicted about recommending acceptance. The results in the paper are essentially well known or minor extensions of classical results in invariant theory -- which by itself is not a problem. As I have mentioned above, the chief import of the paper is the observation that these can be relevant to constructing equivariant NNs without resorting to complex acrobatics with regard to the irreps etc. The use of such invariant scalars in the context of physics and machine learning does appear from time to time (although granted not for these set of groups and the broader scope that the paper covers). For instance see: 1. Lorentz- and permutation-invariants of particles, Ben Gripaios, Ward Haddadin, and Christopher G Lester, Journal of Physics A: Mathematical and Theoretical, Volume 54, Number 15; 2. Invariant polynomials and machine learning, Ward Haddadin, arXiv:2104.12733v1. Papers such as \"Moment Tensor Potentials\" by Shapeev are also relevant. \n\nSome of these considerations make numerical experiments comparing the framework, on at least one standard task, quite pressing and essential (even if it has enough material in it for others to build systems using these ideas and actually leading to publications themselves). Some experimental validation is essential because it is not at all clear whether using scalars in the way described will lead to practical networks. The framework described in the paper will also need reformulation of classical theories at times. It is also not clear whether such reformulations will lead to numerically stable networks. I am worried that the proverbial devil will be in the (implementation) details, and that's where a lot of work will be required. Because of these considerations, despite liking the paper, I am currently leaning towards a score of 5. If during the rebuttal, the authors can convince me otherwise, I would be more than happy to raise my score. \n\nMinor comments:\n\n- \"convolutional symmetry\" is a phrase that appears in the paper in a few places. Is this standard terminology? Unless I am missing something, this doesn't sound quite right. \n- line 100: typo -- looks like it should say deep sets (instead of deep nets).\n- lines 119 - 122: This might be a matter of opinion. But I don't quite see how the universal approximation theorem using non-polynomial activations ([46] in the paper) inspired some of the recent equivariant models. In the earlier equivariant NN papers (before universality results for equivariant networks started to appear), these results are either not cited, or had perfunctory citations.\n- equation 17 seems to have a typo. More specifically, it overcounts the energy. The second summation should go over j > i", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations are discussed in detail and with candor. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Equivariant neural networks are very important for applications of ML to physics problems. This paper explores the idea of building equivariant neural networks from invariant scalar functions which are simpler to build. This principle is well-known and commonly used in physics but has not been extensively explored in the ML literature for equivariant model building. 

The paper was originally presented as a general mechanism for building Gauge equivariance --- however all the presented theory only considers global (coordinate) symmetries. A few extra ingredients are needed to build Gauge equivariance on top of the presented theory such as the parallel transport of tensors at different locations to build local invariants. The authors promised the paper was modified to its correct scope.

Concerns have been raised regarding the lack of experiments in the original submission, but the authors added an implementation of these ideas and experiments to an anonymised git repository. The experimental results are a bit preliminary but look promising.
Decision: Accept (Poster)