Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The paper characterizes scalar and vector functions that satisfy the symmetries of classical physics. Concretely, functions that are equivariant and invariant with respect to a list of (relatively) standard groups (such as $SO(d)$, $E(d)$, etc.) are characterized; the authors additionally show how to express a couple of physically meaningful equivariant functions in their framework. 
main_review: Strengths

1. Overall, the paper is clear and well-written. The quality of the presentation is high; lemmas and propositions are well-formatted, derivations are well-justified. 

2. The paper presents a unified characterization of a fairly general class of equivariant functions, where the symmetries are specified via a canonical set of group actions oft-seen in physics.

Weaknesses

1. Although the paper characterizes a fairly general class of equivariant functions relevant in physics, arguably the implications of this characterization are not particularly significant. Namely, physicists are already keenly aware of the symmetries inherent to their problems and do a good job of incorporating these symmetries in their models [a,b,c]. Whether it be Lorentz group equivariance [a], or equivariance to arbitrary Lie groups [c], models are built to respect symmetries. I was interested in seeing more about what the authors had to say regarding the implications of their characterization to model-building, but this was only briefly mentioned (Section 7); no comparison was made to prior models and the paper did not point out any potential inefficiencies that could be remedied via their characterization. By itself, the characterization is not particularly interesting. As exhibited by the paper, the proofs are straightforward, methodologically not much original or interesting computation was presented.

2. The paper fails to provide a sufficient comparison to prior work done in modeling physical symmetries. Section 7 gives a brief description in which particle position is predicted by taking into account various physical symmetries; how does this compare to work that already models these symmetries and does a similar prediction, say [c]? It seems the paper stops short of actually going through with building a model, or pointing out how prior work is inefficient/can be improved with the given characterization, or providing some convincing consequential implication of the work. This makes the paper seem unfinished, or perhaps preliminary.

3. Narratively, I take umbrage with how the paper was presented. I believe the paper spends too many pages on characterizing symmetric functions, and too few pages on model-relevant considerations. How one builds the model in physics or in machine learning is frequently as, if not more, important as characterization of symmetries abstractly. See for example the discussion of model symmetries in [b], which describes how one builds SU(N) gauge equivariant flows (note the extensive interplay of symmetry and model considerations). Discussing the implications of the characterization in such a model-specific setting would help in demonstrating usefulness.

Verdict

Although the paper is well-written and presents a unified characterization of a fairly general class of equivariant functions relevant in physics, it is preliminary, in that it barely begins to address the implications of this characterization to modeling in physics. By itself, the theory is not particularly interesting and the proofs given are short and straightforward. As such, I recommend a rejection.

Minor Corrections

The paper is, for the most part, well-written, and the ideas are clearly communicated. Some minor corrections are given below:

L135: "Implementating" -> "Implementing"

References

[a] Bogatskiy, A., Anderson, B.M., Offermann, J., Roussi, M., Miller, D.W., & Kondor, R. (2020). Lorentz Group Equivariant Neural Network for Particle Physics. ArXiv, abs/2006.04780.

[b] Boyda, D., Kanwar, G., Racanière, S., Rezende, D.J., Albergo, M.S., Cranmer, K., Hackett, D., & Shanahan, P. (2020). Sampling using SU(N) gauge equivariant flows. ArXiv, abs/2008.05456.

[c] Finzi, M., Stanton, S., Izmailov, P., & Wilson, A.G. (2020). Generalizing Convolutional Neural Networks for Equivariance to Lie Groups on Arbitrary Continuous Data. ICML.


### Post-rebuttal Update

I have raised my score from a 4 to a 5.
limitations_and_societal_impact: The authors have adequately addressed the limitations of the work. There is even a separate section, Section 6, that addresses limitations directly. The fact that the work does not characterize functions symmetric with respect to action by (most) discrete groups is stated explicitly, as well as the fact that dimensional (unit) symmetry is not addressed.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper proves several simple characterizations for invariant and equivariant function for groups that are of interest to physics applications such as the orthogonal group/Lorentz group together with translations and permutations.  

main_review: Positive points:
-- the paper addresses a timely and interesting problem: encoding invariance and equivariance in deep learning models. While incorporating permutation symmetries in deep models is relatively well understood, our understanding of continuous symmetries like the ones addressed in this submission is still being developed. 
-- The paper is very well written and easy to follow. Great job!
-- The characterizations discussed are simple and elegant and could serve as inspiration for future models

Negative points: 
-- It is not obvious which of the theorems/propositions were known before and which were not. It is of course well known that the first fundamental theorem holds. What about its generalization to equivariant functions? And the permutation equivariance part?
-- Although the characterization in the paper seems useful - it is not entirely clear how to leverage them in a deep learning framework, and I believe this can be an even more important contribution. There are two sections that try to address this issue: section 7 and the application of the message passing algorithm in the appendix. I would give a much larger room to this discussion and even implement the suggested method. To make room, Section 6 can be shortened. There is also no discussion of the relationship between the proposed algorithm and recent O(3) graph models.  

Originality: The paper leverages prior work from invariant theory and although I know some of the propositions are new, it is not clear to me which ones. This can be fixed in a revision.

Quality: A high-quality paper addressing an interesting problem.

Clarity: Very clearly written.

Significance: This paper can pave the way for future research in the field. Implementing the suggested model would strengthen it.

Summary: I like the paper, but the authors should address the above-mentioned issues, especially with regards to the new model. I'm open to raising my score as a result of the rebuttal.

limitations_and_societal_impact: The limitations are discussed.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: In this paper, the authors study the characterisation of group invariant and equivariant functions in terms of a collection of group invariant scalars, such as the dot product between two vectors for O(n) invariance. These contrast with the mainstream approach in geometric deep learning of solving constraints on the parameter space that make the network architecture invariant/equivariant. The paper is purely theoretical and contain no experiments.
main_review: Originality:
The characterisation of invariant functions with respect to the orthogonal groups in terms of appropriate inner-products is well-known, however as far as I am aware, the characterisations of equivariant functions given here are new (the case of O(n) is probably known given its simplicity). While it is possible that similar results exist somewhere in the literature, presenting it in this fashion may be beneficial to machine learning researchers looking to construct simple parameterisations of invariant/equivariant functions without relying on the heavy machinery of group representation theory, which seems to be a common approach in the literature. This also means that it would make it more appealing if it were more approachable to a wider audience.

Quality:
The proofs in the main text and appendix are rigorous and mathematically sound (although there are some presentation issues, see below). Experiments on simple datasets would have been nice to see however, which would have made a stronger case for the algorithm and enabled a discussion on the practicality/computational cost (try e.g. on the kaggle galaxy dataset that is mentioned in §2 and other simple physical models such as the pendulum). **Update:** The authors have addressed this point adequately and now have convincing experiments on synthetic and real data.

Clarity:
The writing in the main body is fairly easy to follow and the authors have done a good job at conveying the main ideas, however there are several misprints and some unclear points, which I list in the following.
- The use of the term gauge equivariance in the title may be misleading. The paper only deals with global symmetries, whereas gauge symmetry in general refers to local symmetries. **Update:** The authors have agreed to remove the word "gauge" from the title to avoid confusion. 
- The authors put an emphasis on constructing equivariant polynomial functions at several points in the paper. I did not understand the purpose of this if the framework already contains a universal approximation of general equivariant nonlinear functions.
- Conclusion is missing.
- Line 29. There should be a right bracket ")" after "...linear momentum"
- Lemma 1. If this is a classic theory, is it necessary to include a proof?
- Line 311-313. The line "In particular invariant functions ... where $M$ is either $V^TV$ or $V^T \Lambda V$" seems to be incomplete.
- Line 726-727. The line "Isomorphically, $G_W$ is $SO(m)$ and its action on $W$ is $SO(m)$’s canonical action on $R^m$" in the appendix is difficult to understand and needs improvement.
- Line 737-738. For clarity, I would suggest changing "Thus it and $v_1,...,v_n$ span $R^d$. So in all cases, i.e., for any tuple $(v_1,...,v_n)$, there exists a solution in the $a_t$ and $a_S$ to..." to "Thus, $v_1,...,v_n$ and its generalised cross product span $R^d$. So in all cases, i.e., for any tuple $(v_1,...,v_n)$, there exists $a_t$ and $a_S$ such that ... holds."
- Line 316. I suggest changing "We note that the results from [59] don’t assume the matrix M is positive semi-definite" to "We note that the results from [59] do not assume that the matrix M is positive semi-definite"

Significance:
The work may be useful for incorporating physical inductive bias in neural networks without relying on tools from group representation theory. However the lack of experimental verification and seemingly little use case may make it less appealing to the general audience. **Update:** I have changed my view regarding the experiments in light of the rebuttal.


limitations_and_societal_impact: The authors have explicitly acknowledged some limitations of this approach in the paper, including:
- It only deals with the orthogonal family of transformations and the corresponding construction of invariant scalars for other types of groups is not clear cut.
- Expressions are cumbersome when inputs are general tensors, which arises often in physics.

In addition to these, I would also point out the following:
- Inability to deal with scalar fields and more general sections of vector bundles, which are natural objects in computer vision and in physical field theories (the technique in this work seems to only deal with point-mass-type systems). For example, group convolutions deal with group equivariance of scalar fields while steerable CNNs and gauge equivariant CNNs deal with equivariance on sections of general vector bundles. The approach in this paper do not include these settings.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper proposes explicit constructions of invariant and equivariant functions for a range of symmetry groups and specific group actions that are relevant in classical physics. The symmetry groups under consideration, listed in Table 1, are the orthogonal groups $O(d)$, rotations $SO(d)$, translations $\mathbb{R}^d$, Euclidean group $E(d)$, Lorentz group $O(1,d)$, Poincare group $IO(1,d)$ and permutation group $S_n$ (Table 1). Except for the permutation group, these groups act via their standard action on vectors in $\mathbb{R}^d$ and leave scalars invariant (Table 2). The permutation group is assumed to act by permuting a set of vectors in $\mathbb{R}^d$. The main contribution of the paper are theorems that characterize the spaces of equivariant vector-valued functions $f: (\mathbb{R}^d)^n \to \mathbb{R}$ and invariant scalar-valued functions $h: (\mathbb{R}^d)^n \to \mathbb{R}^d$. It is proven that these functions can generally be written in terms of functions of all $n^2$ pairwise inner products and subdeterminants of the $n$ input vectors. This result suggests an universal way of parameterizing equivariant/invariant neural networks. An important difference to prior work is that no constraints on the neural connectivity are required but just unconstrained functions of the inner products and subdeterminants.

Two classical examples from physics exemplify the results of the theorems.
A potential issue of the proposed parametrizations is that the number $n^2$ of pairwise inner products is quadratic in the number $n$ of input vectors. The authors briefly discuss some results which suggest that the number of inner products that is required in practice is almost always of size approximately $(d+1)n$, which is of the same order as the original input ($dn$ neurons).

No empirical analysis of the parametrization and comparison with alternative methods is provided.
main_review: Originality:

The methods leverage classical results from invariant theory, and are therefore firmly grounded in the mathematical literature. In contrast to most related	work, the method does not requite the user to implement irreducible representations or constraints on the network architecture. This is very useful as it greatly simplifies implementations. The ideas of the paper seem to have been explored before, for instance in Gripaios et al (2021). The related work section gives a comprehensive overview of the field.

Quality:

The submission is mathematically sound and all claims are formally proven. The authors are honest about the about strengths and weaknesses of the work, which are in particular discussed in the limitations section 6. As a purely theoretical paper, the submission would be complete. However, the proposed method is not empirically evaluated and benchmarked against alternative approaches. This makes the paper as a whole incomplete, which justifies my final score.

Clarity:

The paper is well organized and the presentation of the theoretical results in section 3 is very clear. Section 4 provides some examples of these results in classical physics. The results in section 5 seem a bit suggestive and are not formulated in terms of a theorem.

The title of the paper refers to "gauge-equivariant machine learning", however, the relation to local gauge symmetries is unclear. The covered symmetry groups are rather global symmetries of Euclidean spaces or Minkowski spacetime, acting on all vectors in the same way. The authors should either adapt the title or clarify in how far the results relate to gauge symmetries.

The practical relevance of the method remains due to the missing empirical evaluation entirely unclear.

Significance:

The significance of the results can not be assessed without an implementation and comparison with alternative approaches. Such an empirical study is in my opinion indispensable as experience shows that theoretical results do often not directly translate into practice but that further tweaks are required. An implementation could furthermore support the claim from section 5 that not all $n^2$ inner products are required in practice, but only $(d+1)n$.

Overall score:

The results of the paper look very promising and are of potentially high relevance to the community. However, without an empirical evaluation, at least on some toy problems, the submission remains incomplete. I would like to encourage the authors to add an experimental section to the submission. As it is currently still missing, I need to clearly reject the paper in its current state.

limitations_and_societal_impact: Limitations of the methods are honestly discussed in section 6.
review_title: null

Meta-review: The paper originally got 1 "Clear reject", 2 "Ok but not good enough - rejection"s, and 1 "Marginally above the acceptance threshold", all with relatively high confidences. The author rebuttals were successful and all the four reviewers raised their scores, resulting 1 "Marginally below the acceptance threshold", 1 "Marginally above the acceptance threshold", and 2 "Good paper, accept"s. The AC found that the theory is valuable to the community, although it is a direct application of existing invariant/equivariant theories in mathematics. Thus the AC recommended acceptance. Nonetheless, the authors should remove "gauge" in the paper title and revise according to the reviewers' (especially Reviewer StjN's) concerns.
Decision: Accept (Poster)