Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: In the manuscript, the authors propose a method to estimate continuous symmetry of data through the estimation of Lie rings, rather than the Lie groups themselves.
To the best of my knowledge, there has never been a method for nonparametric estimation of Lie rings with linear representations, and I believe that the method is very novel and significant in the study of equivariant neural networks.
main_review: As mentioned above, I believe that the method which is described in this manuscript has both significant novelty and importance that should be urgently shared with researchers in the same field.
Although I acknowledge the novelty of the proposed method, the current manuscript is unclear what is the purpose of the research and the novelty of the whole research resulting from the development of the method. I think that the paper should be accepted after this ambiguity has been fixed.
In the following, I pointed out the unclear points according to the novelty or research purpose pointed out by the authors.


1. We propose the Lie algebra convolutional network (L-conv), a building block for constructing group equivariant architectures.
>The authors state that the estimation error of the Lie group by the proposed method is $O(η^{p+1})$.
In the estimation of Lie rings, only information in the neighborhood of a point is used.
Therefore, the estimation error of the Lie group by the proposed method may be much larger than that of the direct estimation of the Lie group due to noise in the data, training error of DNN[1], or the large curvature of the manifold formed by the Lie group.
In this case, the accuracy of the task or the symmetry estimation will be lower than the previous studies that introduce the constraint of the Lie group or estimate the Lie group directly.
Can you provide any discussion or numerical experiments that show that this does not occur?


2. We show that multi-layer L-conv can approximate group convolutional layers, including CNN, and find graph convolutional networks to be a special case of L-conv.
>This is a very interesting discussion, but the meaning of this discussion in this research is unclear.
What advantage does this property give to L-conv as a network model of deep neural networks?
If the advantage includes higher performance compared to CNNs and GCNs, then the numerical results in the supplemental material D should be mentioned here as well.


3. The Lie algebra generators in our L-conv can be learned to automatically discover symmetries.
>As far as I know, there are methods that also learn Lie groups [2] or estimate Lie-ring from trained DNNs [1]. 
What is the advantage of the proposed method?


4. We find that MSE loss function for L-conv generalizes important Lagrangians in physics.
> I could not understand why "The MSE is given by $I$".
Are you making the assumption that the error is given in quadratic form?
If so, can $I$ represent any quadratic-form likelihood function with symmetries?


5. We find generalization error and equivariance in the loss landscape can be expressed as Euler-Lagrange equations and Noether currents, transferring ideas from physics to ML.
> About the discussion about this novelty (from Line 283 to 296), it seems to be obvious that if the Lagrangian satisfies the principle of least action (Robustness to the perturbation $\delta \phi$), then it satisfies Hamilton's principle (variational principle).
Therefore, it could not seem to be an element in itself to consist the novelty of the manuscript.
About the discussion about this novelty (from Line 297 to 306), the discussion could not seem to be an element in itself to consist the novelty of the manuscript either since the discussion only contains a description of the basic knowledge of analytical mechanics. 
In order to increase the importance of the discussion, I would like to see a more detailed description of how the obtained knowledge can be used in concrete ways.
Specifically, I did not feel the following statements were concrete enough.
(Line 304) "It would be interesting to see if these conserved currents can be used in practice as an alternative way for identifying or discovering symmetries. "

6. There are the other major comments about "7 Discussion" as described below:
Authors have pointed out the following as useful aspects of the findings of this research.
(Line 325)"But perhaps the most interesting discovery here is the connection made with symmetry encoding models in physics. This is significant for two reasons. "
However, I could not determine if the two points presented have significance for the following reasons, respectively.

6-1. About (Line 326) "One is that methods for dealing with symmetry used in physics acquire important meanings in ML, as our discussion on the loss function, generalization and conservation laws show. "
>It is true that paragraphs such as Line 307 discuss the correspondence between physics and machine learning, but it does not specifically describe the prospects of how they can be connected to the development of machine learning algorithms.
Therefore, I can not understand there is any significance in this finding.


6-2. About "The second reason is for scientific machine learning. Physicists usually use the simplest forms for the loss function, similar to the MSE loss above. Many areas of physics suffer from this because these ansatze diverge in many cases or fit the data poorly. Hence, more complex L-conv loss functions can potentially serve as more advanced ansatze for solving physics problems."
>I would like you to describe what kind of physical problems you can solve here as well.
Personally, since the estimation of the Lie ring is important in the conservation law estimation of dynamical systems, I am interested in whether the method can be applied to such problems.
Also, if the authors use quantum field theory as the target of the discussion, I would like you to discuss whether discrete symmetry can also be treated in the framework of the proposed method.
That is because quantum field theory has many discrete symmetries.

>6-1 and 6-2 are related to the concerns in point 5.


There are some minor comments as described below:
>1. I understand that the Lie rings estimated by the method are restricted to those realized by linear transformations.
This restriction is looser than that of the representation of Lie groups.
Is there any advantage to this relaxation?

>2. Figure 4 is not cited in the text.

>3. (Line 197) Equation 58 is not found in the text.
Also, Equation 58 in Supplemental material does not seem to be the relevant equation.

>4. (Line 262) "assme" -> "assume"  typo?

>5. (Line 286) "here s" -> "here is"  typo?

>6. (Line 86) $\epsilon ^i$ seems to be undefined.


References:

[1] Mototake, Yoh-ichi. "Interpretable conservation law estimation by deriving the symmetries of dynamics from trained deep neural networks." Physical Review E 103.3 (2021): 033303.

[2] Zhou, Allan, Tom Knowles, and Chelsea Finn. "Meta-learning Symmetries by Reparameterization." International Conference on Learning Representations. 2020.
limitations_and_societal_impact: I described one of my concerns about the limitation of the method as major comment 1 on the Main review.
In addition to this, I point out that there is no discussion on the limitation of the method by restricting the Lie ring to realization by linear transformations. If possible, please discuss what kind of symmetries cannot be handled by the proposed method.


I do not think there is a negative societal impact.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: They propose a model for approximating invariant functions for Lie groups, the Lie algebra convolutional network (L-conv). While convolutional networks for ordinary groups require some kind of discretization, their model does not require it. Also, Lie algebra generators in our L-conv can be learned to automatically discover symmetries.
main_review: This was a difficult paper to judge. First of all, I can assure that the mathematical instrumentation of their method is solid.
LieConv. and other group convolutions have the disadvantage that the integral over the Haar measure needs to be approximated by a discrete approximation when implemented, which prevents it from being an equivariant function in the strict sense. In their method, the convolution for the group is approximated using an equivariant function, and the model gives a strictly equivariant function.If you look at Theorem 1, you will also see that their model is an universal approximator for functions written in group convolution. If it is an universal approximator for equivariant functions, it should be clearly stated.
I considered the experimental part of this paper to be weak. Although it has the strength of giving an exact invariant function, it is unclear in the current paper whether the previous methods or theirs are stronger for the actual invariant task. This is because their model also has a part that approximates the convolution, and it is not clear at this stage whether the approximation accuracy or the approximation accuracy by discretization is better.
It would be good to have some comparison experiments with Finzi's model, for example, to clarify this part.
There are no major problems with the structure of the paper.
Here are some typos I found.
105 expand
197 L-conv equation 58
limitations_and_societal_impact: No.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper proposes a method to build equivariant neural networks by defining the layers in terms of the generators of the Lie group to which the NN is to be equivariant to. At the core lies the idea that any function (or conv kernel) with local support can be expanded in the Lie algebra basis via a Taylor approximation-like approach. Then, by observing that any function can be approximated with a basis of (shifted) local functions, it can be concluded that the presented approach in theory can approximate any group convolution. In addition, the authors the investigate the option to let go of the user-imposed constraint of being equivariant to a specific group, and instead let the neural network learn the generators (of the unknown group) in addition to the learnable weights that parametrize the layers. This is an open problem and the authors make promising steps in this direction.

Finally, the authors draw connections to physics, but I must admit, this part was too much for me and I did not get it, but possibly could have by investing more time in the paper. 
main_review: [overall impression]
Overall the paper presents very interesting ideas and has high theoretical value. Especially the Lie algebra approach in which convolution kernels are treated as a superposition of localized functions is appealing and leads to a new class of implementations of group convolutions. Additionally it allows to further explore the option to learn the symmetries that are relevant for a task. The unique approach gives new insights in the field of equivariant deep learning and that's where the value of the paper is. Furthermore, the paper is packed with important relevant insights, but at the same time I consider this it weakness as it makes it hard to distill the important parts. I do believe the paper can still significantly improve from more intuitive explanations; after rereading the paper and writing the review I am starting to see the brilliance of it but it does require quite some effort. So despite it being a paper that is hard to comprehend (could be just me), the vastness of sound theoretical insights makes the paper in my opinion worthwhile a publication at NeurIPS.

[readability/too much content]
As mentioned, my main concern is the density of the paper. In my opinion the paper is quite technical and there is simply too much information in it. It would have been nice if it were more focussed. As a reader I often struggled to grasp the general ideas as the emphasis is mostly on the technical details and I spend a lot of time on getting intuition or interpretation right. Providing layman's explanation/interpretation of the equations could improve readability of the paper (even for people already familiar to Lie group equivariant methods). More intuition probably comes at the cost of reducing the amount of information in the main paper. In my opinion, section 6 was a bit detached from the rest of the paper and was probably not necessary. I do appreciate such connections but it felt a bit too much (it could form the basis for a separate paper).

My detailed comments are as follows.

[discretization issues?]
Abstract, intro and Line 323: "... to encode many irreps or discretizing the group":
A key feature of the method seems to be that it doesn't require group discretization as opposed to other methods, but I am not sure if this is really the case. The definition of the L-conv layer has an obvious dependency of g on the left-hand side of the equation. Doesn't this mean one has to evaluate it on a grid on the group G? I think here I misunderstood something as it seems that in Equation 16 the dependency on g is lost. If I am indeed mistaken here, please use my ignorance as queue that more intuition about the equations need to be provided in the paper. 

Regardless, the data is always discretized (see section 4 tensor notation) and so will the generators/left-invariant vector fields (or at least the regular representations/action of g on the vector representations of the data). So I don't understand the argument of not using a discretization as the proposed method still seems to be prone to discretization artifacts. If possible, try to address my concern regarding the discretization issue.

[Lie conv methods are involved?]
Line 32. "however, their approach is quite involved, ..." and line 39 "and this task is quite tedious". I disagree here, I think such Lie group based methods are in fact one of the easiest ways to build G-CNNs (especially compared to steerable G-CNNs) as it allows for a generic and modular code base in which only a few operations (group product, inverse and log) need to be defined after which implementations follow standard procedures. See e.g. the PoinConv method in Finzi et al. (already cited) or on a dense grid as in Bekkers et al. (missing cite, see ref below).

*ref*
Erik J Bekkers. B-spline cnns on lie groups. In International Conference on Learning Representations, 2019.

[group quotients?]
With respect to the lifting, i.e., replacing points x=g x0 with g. I think it is important to remark that g may not be unique (e.g. when x comes from a homogeneous space that is a quotient of the group, e.g., when x \in R^2 and G=SE(2)). It currently reads as if there is only one g for each x. If this is not the case, is this a problem?

[Equation 6]
Equation 6 is important but it is hard to follow what is precisely happening. Some intuition here may be helpful.
If I understand correctly the function f(g) is locally approximated with a first order Taylor expansion? 
And because the kernel is localized the evaluation boils down to multiplying this approximation locally with W^0 as this like the average value of that the kernel is assumed to have and can locally be assumed to be constant?
Then d/dg is a bit abstract, this is "just" the differential of f, right? Which is a covector and multiplying it with g L_i filters out the i-th component?

A thing that confused me is that now all of a sudden the integration measure moves up-front in the integral whereas in eq 3 and 4 it is at the end. Is there a particular reason for this?

[Figure 2 is very insightful! ]

[Related work]
The approach taken in this paper turns out to be very related to two particular works. Firstly, also in Bekkers' work on B-Spline CNNs on Lie groups, convolution kernels are expanded in a basis of highly localized functions, namely B-splines. See e.g. Figure 3 of that paper. The result of group convolution with such kernels can then be obtained by expnding the kernels and to the regular g-conv, or by shifting the underlying function to the original and take the inner products with the basis functions, which is similar to the work presented in this paper.

Secondly, the work by Smets et al. on PDE based G-CNNs seems very related. In it, layers are defined by PDEs expressed in left-invariant vector fields, similar to this work. In particular, the convection term of their PDEs is computed by 1, constructing left-invariant vector fields from a given Lie algebra basis, 2 use them as differential operators (like in this paper equation 10) and 3, multiply them with corresponding weights (like in this paper). The viewpoint is however rather different (PDE vs local approximation viewpoint). 

*ref*
Smets, B., Portegies, J., Bekkers, E., & Duits, R. (2020). PDE-based group equivariant convolutional neural networks. arXiv preprint arXiv:2001.09046.

[deep L-conv networks?]
It is a bit unclear how the experiment with learning the symmetries could generalize to deeper architectures. The current experiments only use one group equivariant layer (as 3 recurring L-convs). Are such layers also suitable for constructing deep architectures?
If so, would this mean that the generators need to be the same throughout all layers?

[practical details of learning the generators]
Line 199: "The L_i can be either inserted... or learned to discover symmetries" In the latter case, how is L_i parametrized? This is generally a very large and dense matrix, or am I mistaken? Is there any constraint put on the L_i matrices that are to be learned?



limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The paper introduces an equivariant L-Conv layer, a building block that can be used to construct group equivariant layers. These L-conv layers are used to discover symmetries in data, and various connections between L-Conv and concepts in physics are made.

main_review: The paper lacks focus and it is difficult to see what the main contribution of the paper is. Each contribution in the list of contributions in the Intro is presented poorly, which makes it difficult to assess the strenghts and weaknesses of L-Conv and its use for discovering symmetries from data.

- Majority of paper is on L-Conv, a new group equivariant layer, suggesting that this new methodology is the main focus and this is a methodology paper. If this were the case, there should be more details in the main paper about the practical benefits of L-Conv over baseline group equivariant layers such as G-conv, shown by experimental results. However all experimental results (except from a brief, insufficiently described result in Figure 3) are placed in the appendix, making the practicality of L-Conv difficult to assess. Furthermore, the results on images in Appendix D comparing to group equivariant baselines such as LieConv are disappointing - all models show test accuracy < 0.5, and LieConv seems to outperform L-Conv despite having much fewer parameters, except the rotated&scrambled image case which is a rather contrived scenario.

- The title of the paper, on the other hand, suggests that symmetry discovery is a main component of the paper. However there is very little description of this in the main paper (a single paragraph in Section 3.1), and due to the brevity it is difficult to understand how symmetry discovery is happening and how the reader is supposed to interpret the learned L matrix in Figure 3. The task in Figure 3 seems to be a task trying to predict rotation angles between two rotated versions of the same input, which I'd argue is not really an instance of 'symmetry discovery'. Symmetry discovery usually refers to the case when given a G-equivariant/invariant task with unknown G, one has to narrow down the set of possible groups G. However for the task in Figure 3, any model that solves the task has to be rotation equivariant, so there is no discovery of symmetry involved here. It would be more relevant/interesting to see what L_i is learned by L-Conv when used on a more standard invariant/equivariant task such as image classification or object localisation, to see if L-Conv can learn translational/rotational symmetry. Furthermore when L_i is learned, the number of lie algebra generators (the number of L_i learned) determines what groups can/cannot be learned, which seems to be a limitation of the method for symmetry discovery.

- Re connections between MSE loss function/generalisation error for L-Conv and concepts in Physics, it's not clear what are the insights gained from these connections. The paper merely points out these connections, but does not expand on what can be learned from such connections.

- The main paper reads to be a summary of the appendix, with considerable overlap between the main paper and the appendix (e.g. Sections 3&6 <-> Appendix A, Section 4 <-> Appendix B.1). I find myself having to read the appendix to make sense of the main paper - this is another example showing that the paper is disorganised, since the roles of the main paper and the supplementary material are reversed.

Introduction of L-Conv layer in Section 3 is confusing.
 
- Equation (6) is hard to follow. The equality in the first line seems to be from Taylor expansion (it'd be good to mention this explicitly for clarity), but the jump from line 1 to 2 in Equation (6) seems erroneous. Shouldn't the second line be W^0 f(g) + c \bar{epsilon}^i g L_i df/dg instead? Hence the last line of equation (6), where you seem to have gone from the Taylor expansion back to the analytic expression, seems unjustified.

- In Section 3, it's not clear at all how L-conv is implemented in practice (e.g. what delta_eta is and how W^0, \bar{epsilon}_i are computed), and information on this only appears at the end of page6, towards the end of the paper, at which point I found myself having to go back to Section 3 to make sense of it. This is an instance showing that the paper is poorly organised and presented.

- Section 3.1 on approximating G-Conv using L-conv doesn't seem to tell us anything about how L-Conv can be made to approximate G-Conv in practice. There is no information about how the local kernels kappa_k(g) should be chosen given a G-Conv kernel \kappa(g), and there is no detail about the error introduced by taking a finite number of L-Convs to approximate a G-Conv. The statement "L-Convs can approximate G-Conv" doesn't mean much if there is no analysis on the approximation error.

- tensor notation further makes things all the more confusing, and seems unnecessary since all "tensors" are just matrices or vectors.

limitations_and_societal_impact: The authors do not mention any limitations of their work. There are clear limitations are in the lack of clarity and organisation in the presentation of the paper, which is making it difficult to assess the limitations of the proposed methods and ideas in the paper.
review_title: null

Meta-review: In this work, the authors build group equivariant neural network layers. These are built from convolutions with sufficiently local functions that can be lifted from the Lie group to the Lie algebra. This approach is sufficiently general to encompass previous equivariant architectures. This approach also comes with a universal approximation property, which the authors need to highlight.

== Why accept this paper?
The paper offers a clearly creative and novel contribution, with a number of high value theoretical hindsights. The community will greatly benefit from having access to these ideas, and I look forward to seeing what and how the present authors, or other researchers, will build on top of those ideas.

== Why not a higher rating?
As mentioned by all reviewers, this paper is particularly hard to read. Part of the reason is that it deals with a difficult problem, but we can only praise the authors for this, not blame them. But another reason for the difficulty to read the paper is its very high density with a lack of focus and clarity. Following the discussion with reviewers, the authors received a number of suggestions to improve their paper, and they should implement as many of those suggestions as possible.
Decision: Accept (Poster)