Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: Summary:
This paper proposes global self-attention networks for image recognition. The proposed GSA module is consists of two parallel layers,  a content attention layer, and a positional attention layer. Experimental evaluations are conducted on ImageNet and Cifar100, and the results are promising.

Clarity:
1. I think this paper is moderate. The content attention layer can be treated as calculating the correlations between each slice of the feature maps, then reweighting/aggregating information through the channel dimension. It is not spatial attention and maybe we cannot say it is global attention.
2. The complexity of the proposed content attention layer is O(Nd_kd_out), and classical global attention like non-local is O(NNd_k). The author should not ignore the channel values as they are large in the top layers.
3. From table 4, the position attention layer is much more important than the content attention layer (77.4 vs 70.8), the position attention is local attention (not global) and also the content is not global spatial attention (for capturing long-range context). Thus I am not convinced by the claim of the paper. Besides, from previous papers (Hu et al. (2019), Ramachandran et al. (2019), Zhao et al. (2020)), we know that local self-attention is already enough, and global attention is not really needed.
review_title: Not so convinced by the claims in the paper

final_score: rating: 5: Marginally below acceptance threshold
review_content: # Post-rebuttal update

I would like to thank the authors for the detailed feedback. I am now convinced about the statistical significance of the results. Regarding the additional study, while it is true that the combination of the changes, in addition to the softmax, was what made the results improve, the change is quite minor. Also, the biggest change comes from the fact that softmax is removed. The reviewer also finds the explanation of why this happens to be handwavy.

Given the concurrent work [Wang et al. 2020] and the incremental nature of the innovation, the reviewer is not sure of the benefits of the current paper to be published at ICLR.

# Summary

The paper proposes a Global Self Attention (GSA) layer, that encapsulates both global and local self-attention, by computing them in parallel and merging the two attentions together. The method is tested on ImageNet and CIFAR-100, demonstrating state-of-the-art results.

While the paper delivers state-of-the-art performance, the novelty of the paper is weak. The enhancements in this paper are also minor, making the paper borderline. The paper currently makes strong claims about the benefit of the proposed method, which are not all obvious. Thus, my preliminary rating for the paper is borderline reject.

# Strengths

The main strength of the paper is its performance and the extensive comparison against the state-of-the-art. The method outperforms all compared methods, albeit by a small margin. CIFAR-10 results are also interesting, but there is no comparison against other self-attention.

The paper is easy to follow, with detailed coverage of related works. 

The ablation study demonstrates which components are important.

# Weaknesses

## Weak novelty, sigmoid, & regarding Wang et al. (2020)

Components that constitute GSA are already shown to be highly effective. The main novelty of the paper then seems to be that the softmax is not used at certain locations and that the two different types of attention are combined together. Thus, the novelty of the paper is limited, especially considering the fact that the gain in performance is marginal. Furthermore, while the focus is on the combination of the two types of attention, which does seem to help in the ablation study, I wonder whether the removal of the softmax had a larger effect. This makes the argument of the paper weak. This leads to my second concern about the paper.

The removal of the sigmoid seems to have had a large effect. At the end of section 3.1.1, it is mentioned that removing softmax increases performance by 0.9% absolute. Considering that the amount of gain from global attention is around 1.1% absolute, this means that without the softmax, there is almost no gain from introducing global attention. Furthermore, I wonder if Wang et al. (2020), without the softmax as in this paper would provide enhanced results as in this paper. 

Thus, it is my current understanding that the method, with softmax and without content attention is similar, if not identical, to Wang et al. (2020). An additional ablation study where softmax is removed from Wang et al. (2020)'s method is necessary to clarify where the performance gain is really coming from.

Also, I am not sure why the paper has chosen the Conv-stem from Wang et al. (2020) and not the full version. Could the authors clarify this?

## Not significantly better

The tone of the paper, regarding the experiments, also needs to be weakened. The method, compared to prior work, improves existing work only slightly. For example, in Table 3, the gap in top-1 performance 0.3% in the case of Resnet-50, 0.9% in ResNet-101, and 0.1% compared to the full version of Wang et al. (2020). I also wonder if this satisfies statistical significance, given that each training session will give slightly varying results. 

review_title: Review

final_score: rating: 4: Ok but not good enough - rejection
review_content: 


**Update after rebuttal** : I want to thank the authors for their rebuttal. However, after reading all responses and all new results presented, I still think that most of the weaknesses of this paper are still present. Two notes that I urge authors to take into account:

* Even the updated Figs 2 and 3 are highly misleading: sure, the gains over a baseline ResNet network are big, but the comparisson should be vs other backbones that combine convolutions with attentions. 
* The authors should clearly say that they do use a convolution for conv1, and their modules start on top of that first spatial convolution.

----------------------------------------

Summary:
The authors present a composite attention module that complement the global attention with axial attentions over rows and columns. They replace 3x3 convolutions with this module on ResNets and show they can outperform convoultion and convolution+ attention based models on ImageNet.

Strong points: The authors present a method that replaces almost all convolutions of a CNN with self-attention variants and gives high results for ImageNet.

Weak points:
Many parameters of the method are chosen arbitrary and are not really ablated; the same stands for alterations in eg the "content attention" layer which is a variant of self-attention, but not exactly the same, since the softmax is applied only on the key values. For a method that argues that one should replace convolutions with attentions, a lot more ablations on hyperparameter stability and effect would be needed, not just a "winning combination", as well as experiment on more than one dataset (CIFAR experiments are for transfer learning).

Specifically:

A) In general, a lot of ablations are missing. The effect of the number of heads, the dimension parameter $d_k$ and the very important region parameter $L$ are not ablated. In fact, parameter $L$ is never explicitly set in the main text - the end of the appendix states "the default version of GSA sets L = max{h, w}" - this should be clearly set and ablated in the main text.  The term "global axial attention" is mentioned in the text, but the text does not say that global axial attention is the "default case". 

B) Sure, not having a softmax as in the common decomposed self-attention allows for more flexibility, but the gains from this change are not really isolated nor ablated; In the text the authors say that there is a decrease by 0.9% on ImageNet without specifying any details (which model? compared to what?). This seems like a tweak that affect performance significantly, that is not ablated.
Qb1) What is the performance of the proposed module when keeping everything the same but just substituting the proposed "content attention" with the common way of using softmax, eg what is used in (Chen et al 2018) or (Shen et al 2018)? If there is a large drop, how would you explain this? 

C) The section that presents the GSA network (3.2) is very short, incomplete and  highly unclear:  
Qc1)  the authors say they "replace all 3 × 3 convolution layers in ResNet-50" , but the first conv1 in ResNet is not 3x3: Do you replace *all* convolutions? even the very first one, or is there a 7x7 convolution in your GSA network? If so, this should be clearly stated and not implied.
Qc2) How many layers of self-attention do you have in each block, is it exactly as many as convolutions?  how do you end up with 18M Parameters and these FLOPS?

D) The paper presents results *only* on the ImageNet dataset (There are some results on CIFAR in the appendix, but models there are only fine-tuned from ImageNet pretrained, and not from scratch); This is a dataset that has no true test set and all methods are overfitting the validation set; this is a dataset where different hardware setups give different baseline numbers. I understand that this is common practice, but, as most other papers do, I would also expect results at at least one more dataset or task to be fully convinced that this approach has something novel to offer. What is more, the curves in Figures 2,3 are against the vanilla ResNet architecture and seem a bit outdated;  a number of more recent papers have taken convolutions to much higher performance with minimal overheads (eg SE-Nets) that would not really show such big differences. 
Qd1) Baseline performance (on the same setup) is highly important and not really presented in Table 3: what would a baseline ResNet-50 achieve at the authors's hardware, and for the same training setup (90 epochs using stochastic gradient descent with momentum of 0.9, cosine learning rate schedule with base learning rate of 0.1, weight decay of 10−4, and mini-batch size of 2048 as well as with the same label smoothing)? This would enable a more fair comparison between Convolutions and a network of (almost) only self-attentions. 
Qd2) Are Resnet numbers in Figures 2,3 reproduced or copied from the 2016 resnet paper?

E) In a missing related work [Chen et al "Graph-based global reasoning networks." CVPR 2019] use global attention + a graph convolution in a module they call GloRe; the graph convolution there seems to be something similar to axial attention, but over nodes on a graph and not positions. Similar to this paper that restricts softmax only on the keys , the softmax in GloRe  is also removed for similar reasons. This is a paper that would be nice to be discussed and compared to - a comparison is also missing from Table 3; that paper reports that a Resnet-50 with 3 Glore units reaches 78.4 top-1 accuracy with 5.2 GFLOPS (and 30M params), vs 78.5 for the approach proposed here with 7.2 GFLOPS.  

Notes and questions:
* The name of the module focuses on "global" but this is a characteristic of many other attentions cited (Wang et al. (2018); Yue et al. (2018); Chen et al. (2018); Shen et al. (2018); Huang et al. (2019)) and doesn't seem fitting to be the main thing titled here. The approach adds positional/axia attention side by side to global; this could/should be reflected in the title
* It would be interesting to see if distillation could help this model as much as it does convolutional resnets.
* Figures 2,3 would be clearer if markers were indicated for the 3 datapoints of each line. 

review_title: A attention-based model that rivals convolutional models on ImageNet

final_score: rating: 5: Marginally below acceptance threshold
review_content: There have been multiple attempts to use self-attention in computer vision backbones for image classification and object detection. Most of these approaches either tried to combine convolution with global self-attention, or replace it completely with local self-attention operation. The proposed approach naturally combines the two, by employing query-key-value switching trick, with axial positional attention.

The approach is very similar to Axial Deeplab (Wang et al. 2020, ECCV 2020 Spotlight), with the main difference that Axial Deeplab uses axial self-attention in the content part, instead of global self-attention, for computational efficiency reasons. The authors claim superiority of their approach, and provide an experimental comparison, showing that there is almost no benefit from axial content attention, which is suspicious - could be an implementation issue?

Overall, the benefit of content attention is not clear from the experiments, as it has more parameters, it is not surprising that it yields higher accuracy and slower runtime. An experiment with the same number of parameters with/without content part is missing.

Regarding softmax on queries, I think the motivation behind was activation normalization. I am curious if it poses additional training difficulties and if the authors tried other activations similar to Katharopoulos et al. (Transformers are RNNs), e.g. ReLU and scalar normalization.

CIFAR-100 experimental setup is uncommon and very surprising, as the authors use ImageNet pretrained models and finetune on CIFAR-100 with large input images, in contrast to training from scratch on CIFAR-100. I suggest that these experiments are updated, otherwise it is extremely difficult to reproduce the paper with a limited computational budget.

Regarding the reproducibility, releasing the code for such methods must be a requirement, as reproducing similar approaches like stand-alone self attention is notoriously difficult, and I don’t think the community managed to do it.

Axial Deeplab paper has been online for 6 months before ICLR submission deadline, the novelty over it is very limited, but I would leave for ACs to decide if it is concurrent or prior work. Disregarding this, the paper has valuable findings but needs more rigorous experimental evalution, as I suggested in my review.
review_title: Good but needs more thorough experimental evaluation

Meta-review: Paper was reviewed by four expert reviewers. Unfortunately all reviewers, uniformly felt that paper fell marginally bellow bar and argue for rejection. A number of concerns have been identified by the reviewers in the review phase.  Those included: (1) lack of novelty [Reviewer3, Reviewer4], (2) lack of various ablations [Reviewer 2], (3)  issues with experimental setup [Reviewer4], and (4)  lack of significant improvements in performance [Reviewer 1, Reviewer 3]. While authors addressed some of the concerns with provided experiments and ablations during the rebuttal, Reviewers remained unconvinced on the main concerns of novelty and significance. As such, the reviewers are unanimous in their assessment and AC does not see a reason to overturn this consensus. 
Decision: Reject