Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: The paper adds an interesting new perspective to equivariant neural nets. However, the actual construction looks equivalent to steerable neural nets to me (see the papers by Cohen and Welling). The generalization of steerable nets has been published under the name "gauge equivariant neural nets", it would be very interesting to chart out the exact connections between these concepts. 

The authors mention that Z^{\times G} is not the only possible lifting space. I believe that the general case would be Z^V where V is a representation of G. 

Many of the earlier papers on equivariant nets were written in the language of representation theory. It is interesting that similar nets can be constructed by purely group theoretic methods, but I really think that ultimately they are same thing. Consequently, I would expect all the experimental results to be identical.

What would make this paper really valuable for didactic purposes is if these connections were carefully mapped out and presented with intuitive diagrams and examples.
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: Motivated by group action theory, this paper proposes a method to obtain ‘equivariant’ neural nets given trained one, where ‘equivariant’ refers to a network that gives identical output if certain symmetry of the dataset is performed on the input (for example, if we rotate a sample the predicted class should not change). 

After reading the paper, I don’t understand the experiments section. In particular, it is not clear to me how the proposed method differs from regular data augmentation, as to my understanding, the input to conv1 is copied 4 times and performed rotation for 0, 90, 180 and 270 degrees and the 4 times increased number of parameters (in-depth) of conv1 are shared. Furthermore, the same rotations are performed to the input to the second layer-conv 2: as the augmentation is cyclic I don’t understand why the authors perform this operation second time. Could the authors elaborate on this? After reading this section, I don’t understand the proposed fine-tuning procedure (pre-train, finetune and test): (1) what is the accuracy of the pre-trained network that was started from? (2) how is the initial network fine-tuned and modified? (as the authors mention that during training the samples are not rotated). Also, I am confused with the first sentence on page 8: ‘the complexity of the constructed network does not grow in terms of the number of parameters’. It would be useful if the results in Fig. 4 are more clearly illustrated.

Is the order or increased computation 4x4x4? It would be useful to compare the method (computation & performance) with a baseline where the dataset is enlarged with data augmentation. The authors mention in the introduction that this increases training overhead, whereas the proposed practical method increases the computation at inference as well as the memory footprint of the model and the forward pass. It would be useful if the authors compared empirically with baselines with (1) data augmentation (2) network with an increased number of parameters (same as the proposed one).

In summary, the idea of using group action theory seems interesting. However after reading the paper, it is not clear to me how the idea is carried out, and although the authors provide theoretical justification, it is not clear how this connects with the practical proposed method and whether it outperforms standard data augmentation (see above). Moreover, I find the writing of the paper quite unclear (see discussion above and examples below).

- If digits 6 or 9 are rotated the label changes, how does the proposed method handle this?
- page 8, conclusion: The authors claim that the proposed approach yields a ‘significant reduction in the design and training complexity’. I don’t understand relative to what this comparison refers to, as the regular data augmentation approach is more straightforward in my opinion. Also, given that this is pointed as an important contribution, in my opinion, an empirical comparison must be done with such baselines (see above).
- Page 1: it is mentioned that ‘the number of parameters can be the same as the original network’ but the experiments do not include such architecture. After reading the paper I don’t understand how such a network can be implemented and whether it works.

— Minor —
- Page 1 & 1par-Pg2: I don’t understand what the authors mean by ‘uniformly *across layers* of NN? 
- Page 2: In these existing works, … I don’t understand this sentence
- Page 2: our .. method use -> uses
- Page 2: map over the orbits. I don’t understand this
- Page 2: the first truly equivariant NN. After reading Sec 5 I don’t understand this point.
- Sec. 4: how to equivarifying -> equivarify
- Page 4: ‘pick a generator’, would recommend elaborating this term or only mentioning g as an element of G for clarity for readers unfamiliar with group theory
- What is the testing accuracy if rotated for different angles than trained (e.g. 45 degrees)?
review_title: Official Blind Review #4

final_score: rating: 3: Weak Reject
review_content: In this paper, the authors propose a method for making a neural network equivariant. Their method also can be applied to make each layer equivariant too. 

Strengths:
-- The paper is very well written and easy to follow with clear notation. 

-- The derivations seem to be correct.


Weaknesses:
-- The experiment is nice but very limited and does not demonstrate the benefits of having an equivariant network. For example, the authors do not report the accuracy of recovering the original (0) rotation.

-- The novelty of the work is questionable. While the development is different, the final example for equivarification of a neural network is very similar to the existing works by Cohen and Welling.

-- There are other works on equivarification that are missed by this paper. For example, consider the following paper:
Lenssen, J. E., Fey, M., & Libuschewski, P. (2018). Group equivariant capsule networks. In NeurIPS.

-- The layer-wise equivariant method does have extra computational overheads.

-- The fact that we have to specify the groups that we want to make the network equivariant with respect to is a limitation. The promise of capsule networks, in contrast, is to "ideally" learn the pose (variation) vectors in a data-driven way.
Sabour, S., Frosst, N., & Hinton, G. E. (2017). Dynamic routing between capsules. In NeurIPS.

-- The following statements need more explanation:
  * "However, these may require extra training overhead as the augmented data increase."
review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: In this work, the authors employ concepts from group theory to turn an arbitrary feed forward neural network into an equivariant one, i.e. a network whose output transforms in a way that is consistent with the transformation of the input. To this end, the authors first introduce the basic concepts of group theory required to follow their work and provide a comprehensive definition of equivariance. They then explain how to equivarify (w.r.t. a finite group G) a given neural network, and present experimental results on rotated MNIST digits to support their approach.

I find that the proposed approach is very elegant and addresses a highly important question, namely how to devise or modify an architecture such that it preserves symmetries. In my opinion, the current paper makes an interesting theoretical contribution towards achieving this goal, but it also has several shortcomings that are detailed below. Based on these shortcomings, I recommend rejecting the paper but I would be willing to increase the score if these points were addressed in sufficient detail.

Major comments:

1) Scaling
The authors mention in the abstract that ‘although the network size scales up, the constructed equivariant neural network does not increase the complexity of the network compared with the original one in terms of the number of the parameters.’ Based on Eq. (3.0.2) and Fig. 3, it is my understanding that n evaluations of each data point (input to a layer) are required for a cyclic group of order n. If the outputs of a layer are then concatenated, the input dimension of the subsequent layer grows by a factor of n. I would therefore argue that out-the-box application of the proposed approach does increase the number of variables dramatically and that the abstract is misleading in this respect. The authors briefly comment on this point with one sentence in the third paragraph of the introduction. However, I would appreciate if this point was addressed in more detail, for example in a dedicated paragraph after the theory is introduced. Please also address the question of whether variable sharing is essential from an equivariance point of view, or whether it’s simply a necessity to prevent an explosion of the number of parameters. Furthermore, convolutions encode translational symmetry which may be beneficial for the current application but may not be desirable for other datasets. A few comments for clarification would be very helpful. Finally, the equivarified network seems to increase the required number of computations significantly compared to the original one, which I find worth a comment .

2) Experiment
The authors only consider a convolutional architecture and say that ‘in order to illustrate flexibility we choose not to simply equivarify the original network’. However, to me one of the main advantages of the paper seems to be that you can take this approach to equivarify any FFN. It would therefore be interesting to see this approach be applied to different networks starting with a simpler one, e.g. a 2-layer MLP. The authors could then compare the original network to the equivarified one with and without variable sharing. That would not only help the reader understand the approach better but also be much more in line with the main motivation of the paper. Then adding a second experiment, e.g. a convolutional architecture, to demonstrate flexibility would be very interesting. With regard to Fig. 4, I think there may be better ways of summarising the results than dumping 160 numbers of which only 4 seem to be of interest. The message seems to be that the network yields identical probabilities irrespective of the degree of rotation. What I find surprising is that all numbers are actually identical (shifted by 10). Is this by construction?

3) Limitations
As indicated in the second paragraph of Sec. 4, this approach is limited to finite groups and the authors only consider image rotations w.r.t. the cyclic group of degree 4. Although I appreciate that this is meant to serve as a toy problem to illustrate that the approach works, I do not think that rotations by a constant angle are very interesting. What would be really interesting is equivariance w.r.t continuous rotations (Lie Groups), e.g. the SO(2) in this particular case. I doubt that an extension to the SO(2) is straightforward within the current theoretical framework. However, even if that is the case, I would appreciate if the authors could comment on this in a paragraph.

Minor comments:

i) There are many typos and grammar mistakes in the paper:
‘any feedforward networks’ -> ‘any feedforward network’.
‘enables to design’ -> ‘enables us to design’
‘our proposed equivarification method use’ -> ‘our proposed equivarification method uses’
‘traditional sense multiplication’ -> ‘traditional sense of multiplication’
‘a group G acts’ -> ‘a group G that acts’
‘neural network that defined on the’ -> ‘neural network that is defined on the’
‘which achieves promising performance’ -> ‘which achieve promising performance’
‘supplymentary material’ -> ‘supplementary material’ 
Etc.

ii) I think there may be a mistake in the 3rd point of Definition 3.0.3: For consistency with the previous definitions and with Fig. 1, shouldn’t F map from X to Z and \hat F from X to \hat Z? 

iii) Last paragraph of Sec 3: ‘then after the identification \hat F becomes a map from Z to...'.  Should it be ‘a map from X to ..’?

iv) In Definition 3.0.3 you define the tuple (\hat Z, T, p) to be a G-equivarification, but in the paragraph below you call the G-product itself a G-equivarification (without including T and p). 

v) Footnote 2: You could correct for that and present the theory shifting by g instead of g^-1 to make it easier for the reader to follow. Or, at least, give a reference to the footnote earlier on in Example 4.0.1 to avoid confusion.

vi) Unless there is a special reason for this, I would suggest changing the enumeration of definitions, lemmas, examples and equations, i.e. (3.0.1) -> (3.1), etc...


*********************************************************
I increased my rating based on the authors addressing many of my comments. 
*********************************************************

review_title: Official Blind Review #3

Meta-review: This paper proposes a way to construct group equivariant neural networks from pre-trained non-equivariant networks. The equivarification is done with respect to known finite groups, and  can be done globally or layer-wise. The authors discuss their approach in the context of the image data domain. The paper is theoretically sound and proposes a novel perspective on equivarification, however, the reviewers agree that the experimental section should be strengthened and connections with other approaches (e.g. the work by Cohen and Welling) should be made clearer. The reviewers also had concerns about the computational cost of the equivarification method proposed in this paper. While the authors’ revision addressed some of the reviewers’ concerns, it was not enough to accept the paper this time round. Hence, unfortunately I recommend a rejection.
Decision: Reject