Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper describes an approach to applying attention in equivariant image classification CNNs so that the same transformation (rotation+mirroring) is selected for each kernel. For example, if the image is of an upright face, the upright eyes will be selected along with the upright nose, as opposed to allowing the rotation of each to be independent. Applying this approach to several different models on rotated MNIST and CIFAR-10 lead to smaller test errors in all cases.

Overall, this is a good idea that appears to be well implemented and well evaluated.  It includes an extensive and detailed bibliography of relevant work. The approach seems to be widely applicable. It could be applied to any deep learning-based image classification system. It can be applied to additional transformations beyond rotation and mirroring.

The one shortcoming of the paper is that it takes a simple idea and makes it somewhat difficult to follow through cumbersome notation and over-mathmaticization. The ideas presented would be much clearer as an algorithm or more code-like representation as opposed to as equations. Even verbal descriptions could suffice. The paper is also relatively long, going onto the 10th page. In order to save space, some of the mathematical exposition can be condensed.

In addition, as another issue with clarity, the algorithm has one main additional hyperparameter, r_max, but the description of the experiments does not appear to mention the value of this hyperparameter. It also states that the rotated MNIST dataset is rotated on the entire circle, but not how many fractions of the circle are allowed, which is equivalent to r_max.
review_title: Official Blind Review #1

final_score: rating: 8: Accept
review_content: [Update after rebuttal period]

While I still find the paper somewhat hard to parse, the revision and responses have addressed most of my concerns. I think this paper should be accepted, because it presents a novel and non-trivial concept (rotation-equivariant self attention).


[Original review]

The authors propose a self-attention mechanism for rotation-equivariant neural nets. They show that introduction of this attention mechanisms improves classification performance over regular rotation-equivariant nets on a fully rotational dataset (rotated MNIST) and a regular non-rotational dataset (CIFAR-10).

Strengths:
+ States a clear hypothesis that is well motivated by Figs. 1 & 2
+ Appears to accomplish what it claims as contributions
+ Demonstrates a rotation-equivariant attention mechanism
+ Shows that its introduction improves performance on some tasks

Weaknesses:
- Unclear how the proposed attention mechanism accomplishes the goal outlined in Fig. 2d
- Performance of the authors' evaluations of the baselines is lower than reported in the original papers, casting some doubt on the performance evaluation
- The notation is somewhat confusing and cumbersome, making it hard to understand what exactly the authors are doing
- No visualisation or insights into the attention mechanism are provided

There are three main issues detailed below that I'd like to see addressed in the authors' response and/or a revised version of the paper. If the authors can address these concerns, I am willing to increase my score.

1. The motivation for the attention mechanism (as discussed in the introduction and illustrated in Fig. 2) seems to be to find patterns of features which commonly get activated together (or often co-occur in the training set). However, according to Eq. (9), attention is applied separately to orientations of the same feature ($A_i$ is indexed by i, the channel dimension), and not across different features. Since the attention is applied at each spatial location separately, such mechanism only allows to detect patterns of relative orientations of the same feature appearing at the same spatial location. The motivation and utility of such formulation is unclear, as it appears to be unable to solve the toy problem laid out in Fig. 2. Please clarify how the proposed mechanism would solve the toy example in Fig. 2.

2. The only real argument that the proposed mechanism is useful are the numbers in Table 1. However, the experimental results for CIFAR-10 are hard to compare to the baselines because of differences in reported and reproduced results. I would appreciate a clarification about the code used (was it published by the authors of other papers?) and discussion of why the relative improvement achieved by the proposed method is not an artefact of implementation or optimisation issues. 

3. The exposition and notation in section 3.1 is very hard to follow and requires substantial improvement. For instance, the sections "Attention and self attention" and "Compact local self attention" seem to abstract from the specific case and use x and y, but it is unclear to me what x and y map to specifically. Maybe also provide a visualization of how exactly attention is applied.


Minor comments/questions:

- If the attention is applied over the orientations of the same feature, why does it improve the performance on Rotated MNIST (which is rotation invariant)?

- I assume the attention matrix $A_i$ is different for each layer, because the features in different layers are different and require different attention mechanisms. However, unlike F and K, A is not indexed by layer l.

- It would be good to provide the standard deviation for the reported results on CIFAR-10 to see if the improvement is significant.
review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: [Post-rebuttal update]

Having read the rebuttals and seen the new draft, the authors have answered a lot of my concerns. I am still unsatisfied about the experimental contribution, but I guess producing a paper full of theory and good experiments is a tall ask. Having also read through the concerns of the other reviews and the rebuttal to them, I have decided to upgrade my review to a 6.

*Paper summary*

The paper combines attention with group equivariance, specifically looking at the p4m group of rotations, translations, and flips. The basic premise is to use a group equivariant CNN of, say, Cohen and Welling (2016), and use self-attention on top. The authors derive a form of self-attention that does not destroy the equivariance property.


*Paper decision*

I have decided that the paper be given a weak reject. The method seems sound and I think this in itself is a great achievement., But the experiments lack focus. Just showing that you get better accuracy results does not actually test why attention helps in an equivariant setting. That said, I feel the lack of clarity in the writing is actually the main drawback. The maths is poorly explained and the technical jargon is quite confusing. I think this can be improved in a camera-ready version or in submission to a later conference, should overall acceptance not be met.


*Supporting arguments*

I enjoyed the motivation and discussion on equivariance from a neuroscientific perspective. This is something I have not seen much of in the recent literature (which is more mathematical in nature) and serves as a refreshing take on the matter. There was a good review of the neuroscientific literature and I felt that the conclusions, which were draw (of approximate equivariance, and learned canonical transformations) were well motivated by these paper.

The paper is well structured. That said, I found the clarity of the technical language at times quite difficult to follow because terms were not defined. By way of example, I still have trouble understanding terms like “co-occurence” or “dynamically learn”. In the co-occurence envelope hypothesis, for instance, what does it mean for a learned feature representation to be “optimal in the set of transformations that co-occur”. Against what metric exactly would a representation be optimal? This is not defined.

That said, I feel that the content and conclusions of the paper are technically sound, having followed the maths, because the text was too confusing.


*Questions/notes for the authors*

- I would like to know whether the co-occurence envelope hypothesis is the authors’ own contribution. This was not apparent to me from the text.
- I’m not sure what exactly the co-occurence envelope is. It does not seem to be defined very precisely. What is it in layman’s terms?
- I found the section “Identifying the co-occurence envelope” very confusing. I’m not sure what the authors are trying to explain here. Is it that a good feature representation of a face would use the *relevant* offsets/rotations/etc. of visual features from different parts of the face, independent of global rotation?
- Is Figure 1 supposed to be blurry?
- At the end of paragraph 1 you have written: sdfgsdfg asdfasdf. Please delete this.
- I believe equation 4 is a roto-translational convolution since it is equivariant to rotation *and translation*. Furthermore, it is not exactly equivariant due to the fact that you are defining input on a 2D square grid, but that is a minor detail in the context of this work.
- Now that we have automatic differentiation, is the section on how to work out the gradients in Equations 5-7 really necessary?
- In equation 8 (second equality), you have said f_R^l(F^l) = A(f_R^l(F^l)). How can this be true if A is not the identity? Giving the benefit of the doubt, this could just be a typo.
- Please define \odot (I think it’s element-wise multiplication).
- Are you using row-vector convention? That would resolve some of my confusion with the maths.
- You define the matrix A as in the space [0,1]^{n x m}. While sort of true, it is more precise to note that each column is actually restricted to a simplex, so A lives in a subspace of [0,1]^{n x m}.
- I think it would have been easier just say that you are using a roto-translation or p4m equivariant CNN with attention after each convolution. Then you could derive the constraint on the attention matrices to maintain equivariance. It would be easier to follow and make easy connections with existing literature on the topic.
review_title: Official Blind Review #3

Meta-review: The paper proposes an attention mechanism for equivariant neural networks towards the goal of attending to co-occurring features. It instantiates the approach with rotation and reflection transformations, and reports results on rotated MNIST and CIFAR-10. All reviewers have found the idea of using self-attention on top of equivariant feature maps technically novel and sound. There were some concerns about readability which the authors should try to address in the final version. 
Decision: Accept (Poster)