Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: This paper describes an approach to making deep learning robust to arbitrary symmetries. To this end, the authors introduce a group equivariance architecture of neural networks which ensures that the function learned by the network has the equivariance property with respect to the symmetry group action (roughly speaking, it ensures that the symmetry affects the input and output of the function in the same way, or that the application of the symmetry does not influence the how the function works).

The building block of the group equivalence architecture is the L-conv layer, i.e. a layer which models a Lie algebra. The L-conv layer works somewhat like the convolutional layer, but instead of using the kernel mechanism, it multiplies the previous layer by L_i which is the basis spanning the Lie algebra around the corresponding Lie group’s identity element. The authors relate to the condition (provided in existing literature) for a feedforward network layer to be equivariant under a group action of G, and show that the L-conv layer satisfies this condition (Propositions 1 and 2).

Subsequently, it is shown that fully connected layers, convolutional layers as well as graph convolutional networks can be expressed with L-conv layers. Further, the authors compare their approach to a very similar one submitted concurrently to ICLR. They also consider the special case of linear regression and show that the symmetry Lie group can be described analytically. Finally, experiments comparing the L-conv based architecture to CNNs and fully connected layers on standard deep learning benchmark data subject to rotation and random permutations are presented.


*****Strengths:*****

Modelling arbitrary symmetries and equivariance / invariance relations with continuous groups is an important topic that has sparked much interest recently in various application areas from computer vision to computational chemistry. The work presented in this paper is certainly relevant.

I found the paper well organised and easy to follow.

While the approach presented here is similar to a couple of papers reviewed in related work (especially Zhou et al.), I think that the description of the symmetry relations in the language of Lie groups and Lie algebra bases is novel and well presented here.


*****Weaknesses:*****

The main weakness of this paper, in my view, is that no description of L_i learning is provided. To some extent, I am not sure if I understand the process correctly after reading the paper. While the theory appears to guarantee that correctly trained L_i are able to produce L-conv layers acting as fully connected layers, convolutional layers or graph convolution networks while ensuring the equivariance property, no analysis is given as to how feasible it is to find the correct basis L_i. In my understanding, the entire contribution of the paper hinges on the ability to find L_i and thus it should be given more space in the paper. The only comment on this is given in the experiment section, where the authors mention an additional autoencoder being employed. Meanwhile, from section 2 and equation 6, one could guess that L_i could be learned directly by sgd. If the latter is not the case, I think a discussion as to why not would make the paper clearer. If there are other theoretical results on the structure / properties of Lie algebra bases that the authors make use of, I think they should be provided too.

In the experiment section, the only experiment concerns rotation as the modelled symmetry. As the goal of the paper is modelling arbitrary symmetries, i.e. all functions with the equivariance property wrt some Lie group, would it not be more convincing to devise an arbitrary set of transformations (satisfying equation 1) without a simple analytical form (as in the case of rotation) and show that one can guarantee robustness against this arbitrary set? The current experiment could be beaten by any approach parametrising rotational symmetry.


*****Questions / feedback:*****

See weaknesses section.


*****Typos:*****

Theorem 1 (the third sentence) should be rephrased -> for this problem to be equivariant?

Equation 15: so -> SO

Section 5: fing -> find


*****Post Rebuttal*****

I would like to thank the authors for their comments. They clarified most of my doubts regarding the paper, most importantly about learning the basis $L_i$. While I agree with reviewer 3 that there is room for improvement when it comes to experimental evaluation, I appreciate the changes made during the rebuttal period and keep my original score.
review_title: An interesting description of group equivariance with Lie algebras.

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: Summary:

The paper meticulously builds a theoretical framework for Lie-algebra convolutional layers and then goes on to show how CNNs, GCNs and FC layers are a special case of L-convs. The paper also demonstrates how the underlying generators can be learnt from data and provide convincing supporting experimental results. The proposed L-conv layers also use much fewer parameters compared to previous works. 

Key strengths:

The theoretical framework developed in this paper, starting from Lie groups and equivariance and invariance definitions is very elegant and convincing. I checked the maths at each step and am convinced that it is correct, to the best of my knowledge. I did need to refer to Hall 2015 though. Intuitively as well as mathematically, it makes sense to me. The comparison to MSR (Zhou 2020) seems fair to me. 
The experiments, though limited, in the main paper, are quite convincing. The experiments are cleverly constructed and provide enough justification to support the utility of L-conv layers in comparison to CNN and FC layers. 

Questions:

For Figure 2: For CIFAR100 and FashionMNIST, CNN seems to do better on "rotated+scrambled" compared to "rotated". What is the reason behind that? This is not seen in any other method or dataset.  

Suggestions for improvements:

1. The paper inherently assumes familiarity with Lie groups/Lie algebra or even exp/log of matrices which I am familiar with, but not all readers will be. Therefore, instead of citing Hall 2015, it would be good to cite Sections within the textbook. This will aid uptake of an important mathematical sub-field. 
2. There is no mention of accompanying code in the manuscript. Would the authors consider making it available upon acceptance? It would help further research in this area. 
3. The section on linear regression (3.1) seems to occur again in an expanded form in the supplementary material (Sec C). The derivation in the supplementary material was a little bit clearer. 
4. It would be worthwhile checking the paper for typos. A few that I noted: larest, wight, "a too many"
5. Figure 1 is not easy to interpret. A substantial caption would be beneficial. It's also not referenced in the text. 

Overall comments:
The main contribution of this paper is the development of the theoretical framework for Lie-algebra convolutions. The paper does so very convincingly and I regard this as an important contribution to the area of deep learning. This may open the door to the field using the correct inductive biases for many problems in vision, speech and physics. I enjoyed reading this paper, including the supplementary material that makes a start on imposing orthogonality during regularization for complex-valued neural networks. 
review_title: Learning symmetries through Lie-algebra convolutions

final_score: rating: 6: Marginally above acceptance threshold
review_content: ### Summary
The paper presents a new approach for building group-equivariant neural networks. The authors propose *L-conv*, a layer which is equivariant to transformations from a group $G$ in the neighborhood of identity. They show that a network with a sufficient number of such layers is $G$-equivariant as a whole. Additionally, the authors propose to learn the structure of the group directly from the training data instead of incorporating it in the network *a priori*.

### Strengths:
1. The paper is interesting and is easy to follow. The problem the authors are solving is clearly stated from the very beginning and is described in the title. 
2. Proposition 1 is insightful. Equations 4 and 5 contain a novel parametrization of convolutional kernels. It also allows considering standard convolution from a different perspective.
3. Proposition 2 reveals a useful connection between the  Lie algebra-Lie group correspondence and a network of *L-conv* layers.

### Weaknesses:
1. **Experiments**.  The conducted experiments do not demonstrate the advantage of the proposed models over other models that use the power of data symmetry. The advantage of the proposed models over conventional CNN is not properly demonstrated.
    * The decision of using very shallow networks is not clear. An experiment with a 1-layer CNN is not directly generalizable to deeper networks. 
    * Paragraph **Learning $L_i$ during Training** does not contain enough information for a clear understanding of the chosen models. A diagram of the chosen models or a table could make it more clear.
    * Figure 2, MNIST. The *L-conv* model contains 2 times more parameters than the standard CNN. It demonstrates $\approx 95$ \% while the standard CNN demonstrates $\approx 93$ \%. It is not clear whether the improvement is caused by the increased number of trainable parameters or by the proposed layer. The same argument is applicable to the CIFAR10/100 and the FashionMNIST experiments.
    * There are no demonstrations of the learned generators $L$. A demonstration of it will make it possible to understand whether the learned structure matches the known structure of the data. For example the rotation operator for Rotation MNIST. 
    * It is worth understanding whether a network of *L-conv* layers converges to a group equivariant network or proposition 2 is purely theoretical. 
2. Paragraph **Notation** and the proposed notation itself do not seem useful. 
    * The Einstein summation rule is used only in Equation 4 which can be easily rewritten with $\sum$. The rewritten version will be more readable. 
    * Equation 2 is not used in the paper at all.  
    * Some letters are used multiple times while meaning different things. For example $f$ is a structure constraint in Equation 2, it is a neural network in paragraph 1 of Section 2, and it is a point-wise activation function later in the same Section.

### Decision
To sum up, the paper proposes an interesting method for building group equivariant neural networks which allows for estimating symmetries from the data instead of incorporating it in the network *a priori*. However, the experimental results are not sufficient for proving the advantage of the proposed approach. The theoretical contribution of the paper is not sufficient for considering it as pure-theoretical. The paper seems raw for acceptance at the current stage.


# Revision
After a fruitful conversation with the authors I am changing my decision.
The conducted experiment presented in Figure 4 confirms an advantage of L-conv.
The paper has a room for improvement. However, it contains valuable discoveries and can be clearly placed in the field of symmetry-aware models. The proposed approach is well-described and is properly evaluated on shallow networks. While the generalization of the results to deeper models is not possible (and the reasons are described by the authors correcly), the approach is interesting by itself and is important to the field.


review_title: An interesting approach 

Meta-review: The decision for this paper is quite difficult: the methodological ideas are interesting, such as learning the generators directly, but the experimental results are relatively weak. Perhaps learning the generators is too unconstrained. Moreover, the proposed 'L-conv' builds heavily on prior work such as the 'LieConv' model of Finzi et. al, which is not made very transparent in the current narrative. And LieConv does provide better performance than L-Conv --- this should also be made clear in the text. This was a very difficult call, and after extensive discussion, the decision is intended to be in the long term best interests of the paper. The ideas are interesting and warmly appreciated, the reviewers appreciated aspects of the response, and the project is sufficiently promising that it was felt that their impact would be much greater if the experimental execution were strengthened, such that the project largely terminating at this point would do it a disservice in the long run. There was a general feeling that this is still a 'work in a progress' and was somewhat rush-written. The authors are strongly encouraged to continue pursuing this work, strengthening the experiments and narrative,  as above.
Decision: Reject