Global Self-Attention Networks for Image Recognition
Authors: Zhuoran Shen
Irwan Bello
Raviteja Vemulapalli
Xuhui Jia
Ching-Hui Chen
Venue:  (2021)
Keywords: self-attention
neural network architecture
image classification
semantic segmentation

Abstract: Recently, a series of works in computer vision have shown promising results on various image and video understanding tasks using self-attention. However, due to the quadratic computational and memory complexities of self-attention, these works either apply attention only to low-resolution feature maps in later stages of a deep network or restrict the receptive field of attention in each layer to a small local region. To overcome these limitations, this work introduces a new global self-attention module, referred to as the GSA module, which is efficient enough to serve as the backbone component of a deep network. This module consists of two parallel layers: a content attention layer that attends to pixels based only on their content and a positional attention layer that attends to pixels based on their spatial locations. The output of this module is the sum of the outputs of the two layers. Based on the proposed GSA module, we introduce new standalone global attention-based deep networks that use GSA modules instead of convolutions to model pixel interactions. Due to the global extent of the proposed GSA module, a GSA network has the ability to model long-range pixel interactions throughout the network. Our experimental results show that GSA networks outperform the corresponding convolution-based networks significantly on the CIFAR-100 and ImageNet datasets while using less number of parameters and computations. The proposed GSA networks also outperform various existing attention-based networks on the ImageNet dataset.

Under review as a conference paper at ICLR 2021
GLOBAL SELF-ATTENTION NETWORKS FOR IMAGE RECOGNITION
Anonymous authors Paper under double-blind review
ABSTRACT
Recently, a series of works in computer vision have shown promising results on various image and video understanding tasks using selff-atennton. However, due to the quadratic computational and memory complexities of self-attention, these works either apply attention only to low-resolution feature maps in later stages of deep network or restrict the receptive field of attention in each layer to small local region. To overcome these limitations, this work introduces a new global self-attention module, referred to the GSA module, which is efficient enough to serve as the backbone component parallel layers: a content attention layer that attends to pixels based only on their content and a positional attention layer that attends to pixels based on their spatial locations. The output of this module is the sum of the outputs of the two layers, Based on the proposed GSA module. we introduce new standalone global attention-based deep networks that use GSA modules instead of convolutions model pixel interactions. Due to the global extent of the proposed GSA module, a GSA network has the ability model rong-range pixel interactions throughout the network. Our experimental results show that GSA networks outperform the corresponding convolution-base
deep network. This module consists
networks significantly on the CIFAR-1O0 -10 and ImageNet datasets while using less parameters and computations. The proposed GSA networks also outperform various existing attention iraset networks on the ImageNet dataset.
INTRODUCTION
Self-attention is a mechanism in neural networks that focuses modeling long-range dependencies. Its advantage in terms of establishing global dependencies over other mechanisms, e.g., convolution and recurrence, has made it prevalent in modern deep learning. In computer vision, several recent works have augmented Convolutional Neural Networks (CNNs) with global self-attention modules and showed promising results for various image and video understanding tasks (Bello et al. 2019 Chen et al. 2018; Huang et al. 2019 Shen et al.) 2018 Wang et al. [0188] Yue et al. 2018). For brevity, in the rest of the paper, we refer to self-attention simply attention.
The main challenge in using the global attention mechanism for computer vision tasks is the large spatial dimensions of the input. An input image in a computer vision task typically contains tens of thousands of pixels, and the quadratic computational and memory complexities of the attention mechanism make global attention prohibitively expensive for such large inputs. Because of this, earlier works such as Bello et al. (2019): Wang et al. (2018) restricted the use of global attention mechanism low-resolution feature maps later stages of a deep network. Alternatively, other recent works such as Hu et al. (2019); Ramachandran et al. (2119; Zhao et al.] (2020) restricted the receptive field of the attention operation small local regions. While both these strategies are effective capping the resource consumption of attention modules, they deprive the network of the ability to model long-range pixel interactions in its early and middle stages, preventing the attention mechanism from reaching its full potential.
Different from the above works, Chen et al. (2018) Huang et al. (2019); Shen et al. (2018) Yue et al. 2018)) made the global attention mechanism efficient by either removing the softmax normalization on the product of queries and keys and changing the order of matrix multiplications involved in the attention computation (Chen et al. 2018 Shen et al. 2018 Yue et al., 2018) decomposing

---

Under review as a conference paper at ICLR 2021
one global attention layer into sequence of multiple axial attention layers (Huang et al. 2019) However, all these works use content-only attention which does not take the spatial arrangement of pixels into account. Since images spatially-structureed inputs, attention mechanism that ignores spatial information is not best-suited for image understanding tasks on its own. Hence, these works incorporate attention modules auxiliary modules into standard CNNs.
To address the above issues, we introduce a new global self-attention module, referred to as the GSA module, that performs attention taking both the content and spatial positions of the pixels into account. This module consists of two parallel layers: a content attention layer and a positional attention layer, whose outputs are summed at the end. The content attention layer attends to all the pixels at once based only on their content. It uses an efficient global attention mechanism similar to Chen et al. (018)) Shen et al. (2018) whose computational and memory complexities are linear in the number of pixels. The positional attention layer computes the attention map for each pixel based on its own content and its relative spatial positions with respect to other pixels. Following the axial formulation (Ho al. 2019 Huang et al. 2019), the positional attention layer is implemented as column-only attention layer followed by a row-only attention layer. The computational and memory complexities of this axial positional attention layer are O(NVN) in the number of pixels.
The proposed GSA module is efficient enough to act as the backbone component of deep network. Based on this module, we introduce new standalone global attention based deep networks, referred to as global self-attention networks. A GSA network uses GSA modules instead of convolutions to model pixel interactions. By virtue of the global extent of the GSA module, a GSA network has the ability to model long-range pixel interactions throughout the network. Recently, |Wang et al. (2020) also introduced standalone global attention-based deep networks that use axial attention mechanism for both content and positional attentions. Different from Wang et al. (2020), the proposed GSA module uses a non-axial global content attention mechanism that attends to the entire image at once rather than just row or column. Our experimental results show that GSA-ResNet, a GSA network that adopts ResNet (He et al., 2016) structure, outperforms the original convolution-basee ResNet and various recent global or local attention- ResNets on the widely-used ImageNet dataset.
MAJOR CONTRIBUTIONS
GSA module: We introduce a new global attention module that is efficient enough to act as the backbone component of deep network. Different from Wang et al. (2018); Yue et al.) (2018); Chen et al. (2018); Shen et al. (2018); Huang et al. (201)), the proposed module attends pixels based
both content and spatial positions. Different from Zhao et al. (2020); Hu et al. (2019): Ramachandran et al. (2019), the proposed module attends to the entire input rather than a small local neighborhood. Different from Wang al. 22020), the proposed GSA module uses a non-axial global content attention mechanism that attends to or column.
the entire image at once rather than just
. GSA network: We introduce new standalone global attention-based networks that use GSA modules instead of spatial convolutions to model pixel interactions. This is one of the first works (Wang et al. (2020) being the only other work) explore standalone global attention-based networks for image understanding tasks. Existing global attention-baser works insert their attention modules into CNNs as auxiliary blocks at later stages of the network, and existing standalone attention- based networks use local attention modules.
Experiments: We show that the proposed GSA networks outperform the corresponding CNNs significantly on the CIFAR-100 and ImageNet datasets while using less parame- ters and computations. We also show that the GSA networks outperform various existing iitention-basee networks including the latest standalone global attention-based network of Wang et al. (2020) on the ImageNet dataset.
2 RELATED WORKS
2.1 AUXILIARY VISUAL ATTENTION
Wang et al. (2018) proposed the non-local block, which is the first adaptation of the dot-product attention mechanism for long-range dependency modeling in computer vision. They empirically verified its effectiveness video classification and object detection. Follow-up works extended it to
2

---

Under review as a conference paper at ICLR 2021
different tasks such as generative adversarial image modeling (Zhang et al. [2099] Brock et al. 2019), video person re- re-identification (Liao et al.) 2018). image de-raining (Li et al. 2018) etc. Several recent works focused mitigating the high computational cost of Wang al. (2018). Chen et al. (2088); [hhnn et al. (2018) utilized the associative property of matrix multiplictiion to reduce the complexity from quadratic linear. |Huang al. (2019) proposed to decompose global attention into row attention and column attention to save resources.
Recently, a series of works (Sun et al. 2019 Carion al. 2020) have used Transformers (Vaswani et al. 2017) for various computer vision applications. These works first use a deep CNN to extract semantic features, and then use Transformer to model interactions among the high-level semantic features. For example, Carion et al. (2020) used a Transformer to model object-level interactions for object detection, and [unn et al. (2019) used Transformer to model inter-frame dependencies for video representation learning.
All these methods use attention modules as auxiliary modules to enhance long-range dependency modeling of CNN, and relegate most of the feature extraction work to the convolution operation. In contrast, GSA network uses attention as the primitive operation instead of spatial convolution.
2.2 BACKBONE VISUAL ATTENTION
Bello al. 2019) were the first to test attention as a primitive operation for computer vision tasks. However, they used the costly non-local block (Wang et al., (0188) which prevented them from fully replacing convolutional layers. |Ramachandran et al. (2019). y, Hu et al. (2019) and /Zao et al. (220)) solved this problem by limiting the receptive field of attention contrast to these works, the proposed GSA network uses global attention throughout the network and is still efficient. Recently, Wang et al. (2020) used axial decomposition make global attention efficient. Different from them, the proposed GSA network uses a non-axial global content attention mechanism which is better than axial mechanism as later shown in the experiments.
local neighborhood. In
GLOBAL SELF- ALTEENTIOO NETWORK
3.1 GLOBAL SELF- MODULE
Let F E RWHx and F E WWH respectively, denote the (spatially) flattened input and output feature maps of the proposed GSA module. Here, W, H represent the spatial dimensions, and din; dout represent the channel dimensions. Each pixel in the output feature map is generated by aggregating information from every pixel the input feature map based on their content and
spatial positions. Let K [ky] Wxxd, Q [qy] E IR Wxx,, and V = [Dty] E g X respectively denote the matrices of keys, queries, and values generated using three 1x convolutions on the input feature map F' Here, de denotes the number of channels used for keys and queries. Each row in these matrices corresponds to one input pixel. The proposed GSA module (see Fig. consists of two parallel layers: a content attention layer and positional attention layer.
3.1.1 CONTENT ATTENTION LAYER
This layer uses the keys, queries, and values to generate new features F = [f E RW Xxdu using the following content-based global attention operation:
F = (p V).
(1)
where KJ denotes the matrix transpose of K, and p denotes the operation of applying softmax normalization for each row separately. This attention operation can interpreted as first aggre- gating the pixel features in V into d2 global context vectors using the weights in p [K and then redistributing the global context vectors back to individual pixels using the weights in Q. The computational and memory complexities of this operation are O(N) in the number of pixels.
This attention operation similar to the attention operation used in Chen et al. 2018 ; Shen et al. (2018) except that it does not softmax normalization queries. Normalizing the queries strains the output features to be convex combinations of the global context vectors. As these con- straints could restrict the expressive power of the attention mechanism, remove the softmax
3

---

Under review as a conference paper at ICLR 2021
1x1 Conv
K: WH x 4,
Content attention
Ft: WH
1x1 Conv
Q: Wh'x
1x1 Conv
V: WH
Column-only PAlayer
Row-only PA layer
F WHx d
L d
R' R
Positional attention
Figure 1: Proposed GSA module: The keys, queries, and values (generated using 1 x convolutions) are processed by the content attention and positional attention layers in parallel. The positional attention layer is split into volumn-only and row-only positional attention layers, which use learned relative position embeddings Rs and R' keys. Finally, the outputs of the content and positional attention layers are summed to generate the output of the GSA module. Here, BN denotes batch normalization (loffe & Szegedy 2015), and PA stands for positional attention.
normalization on queries. This allows the output features to span the entire subspace of the da global context vectors. When we experimented with softmax normalization on the queries, the top-1 accuracy on the ImageNet validation dataset decreased significantly (1%).
3.1.2 POSITIONAL. ATTENTION LAYER
The content attention layer does not take the spatial positions of pixels into account, and hence, is equivariant to pixel shuffling. So, on own, it is not best-suited for tasks that deal with spatially- structured data such as images. Inspired by [ello et al. (2019); RRamachandran et al. (2019); [Shw jtt al. 2018), we address this issue by using a positional attention layer that computes the attention map for pixel based on its own content and its relative spatial positions with respect to its neighbors. For each pixel, our positional attention layer attends to its L X spatial neighbors. Inspired by implement this attention layer column-only attention layer followed by a row-only attention layer. In a column-only attention layer, an output pixel only attends to the input pixels along its column, and in a row-only attention layer, an output pixel only attends to the input pixels along its row. Note that column-only attention layer followed by row-only attention layer effectively results in information propagation over the entire L x L neighborhood.
the axial formulation (Ho et al. 2019 Huang et al.) 2019),
Let A {- 1ي1 O, ر be set of L offsets, and R [r IR Lxd denote the matrix of L learnable relative position embeddings corresponding to L spatial offsets 5 E A along a column. Let V = [ E RR I.duut be the matrix consisting of the values the L column neighbors of pixel (a, b). Let fa denote the output of the column-only positional attention layer at pixel (a, b). Then, our column-only positional attention mechanism, which uses the relative position embeddings R as keys, can be described using
= - Ve.
(2)
where qab is the query at pixel (a, b). Since each pixel only attends to L column neighbors. the computational and memory complexities of this column-on positional attention layer are O(NL), where N is the number of pixels. Similarly, row-only positional attention layer with ONNL) com- putational and memory complexities can be defined using T. learnable relative position embeddings R' [r]] E Lxxd, corresponding to the L row neighbors. In the case of global axial attention, the neighborhood spans the entire column or row resulting in O computational and memory complexities.
The final output feature map of the GSA module is the sum of the outputs of the content and posi- tional attention layers.
3.2
GSA NETWORKS
A GSA network is a deep network that uses GSA modules instead of spatial convolutions to model pixel interactions. Table [ shows how GSA network differs from various recent attention-based
4

---

Under review as a conference paper at ICLR 2021
Table 1: Properties of recent attention-based networks
Attention module
Method
Global
Content
Positional
Attention + CNN combination
Wang et
20118
Chen ct al. 2018
Few attention modules are inserted in between residual blocks
Yue ct al. 2018
Shen et al.
2018
Huang ct al. 2019
Carion ct
20200
Attention modules are added at the end
Sun
2019
Bello al.
(001))
Convolution layers are augmented with attention modules in parallel
Hu et al.
2019)
Ramachandran
2019
Convolution layers replaced attention modules
Zhao el al. (2020)
Wang et al. (2020
This work
networks. All existing works except Wang al. (2020) either insert their attention modules into
CNNs as auxiliary blocks (Bello et al. [2019]Chhen et al. 2018 Huang et al. 2019 Shen et al. 2018 SSn Sun al. 2019) at later stages of the network or constrain their attention mechanism to small local regions (Hu et al.) 2019 Ramachandran et al.)
Wang et al. 2018 Yue et | 2018 Carion et al. 2020
2019 Zhao et al. 2020). contrast, GSA network replaces spatial convolution layers deep network with global attention module and has the ability to model long-range pixel interactions throughout the network. While Wang et al. (2020) also introduces a global attention module as an alternative for spatial convolution, their module uses axial attention mechanism for both content and positional attention. In contrast, the proposed GSA module uses a non-axial global content attention mechanism that attends to the entire image once rather than just row column.
3.3 JUSTIFICATIONS
The proposed GSA module uses direct global attention operation for content attention and an axial attention mechanism for positional attention.
Why not axial content attention? Axial attention is a mechanism that approximates direct global attention with column-only attention followed by row-only attention. In the proposed global content attention layer, two pixels (i, and (p. q) interact directly based only on their content. In contrast, in fillmm--olloweddby--ooo axial content attention layer, pixels (i.j) and (p.g) would interact through pixel (p,j), and hence, their interaction would be undesirably controlled by the content at (p, Therefore, the proposed direct global attention is better than axial mechanism for content attention. This is also verified by the experimental results in Table
2 which show that the proposed GSA module that uses direct global content attention significantly better than axial attention.
Why not direct global positional attention? It is important attend to pixels based on relative positions (instead of absolute positions) to maintain translation equivariance. In the case of content attention, each pixel has unique key, and hence, we can multiply keys and values first to make the attention mechanism efficient. This is not possible in the case of positional attention since the key at a pixel varies based on its relative position with respect to the query pixel. Hence, we use axial mechanism to make positional attention efficient. While axial attention is not good for content attention (as explained above), it is suitable for positional attention. The relative position between pixels (i. and (p, q) is strongly correlated the relative positions between pixels (i. and (p.j), and between pixels (p. q) and (p. j). So, routing position-ber based interaction between (i.j) and (p. g) through (p. j) works fine.
5

---

Under review as a conference paper at ICLR 2021
CNN
CNN
79
70
scccurcc
Top accuraccy
Tup eccurety
78
138
CNN C5A-Nat
70
76-
15
20
25
30
35
40
6
10
12
14
20
30
40
50
Parameters (K)
FLOPs
(G)
Latency (ms)
Figure 2: Comparison between ResNet-(38, 50, 101) structure-based CNNs and GSA networks.
GSA networks clearly outperform CNNs while using less parameters, computations, and runtime.
4 EXPERIMENTS
Model Unless specified otherwise, we use GSA-ResNet-50, a network obtained by replacing all 3 x 3 convolution layers in ResNet-50 (He al. 2016) with the proposed GSA module. We use an input size of 224 x 224, and for reducing the spatial dimensions, we use x 2 average pooling layers (with stride immediately after the first GSA module in the second, third and fourth residual groups. The number of channels for K, Q. V in each GSA module are set to be the same as the corresponding input features. We use multi-head attention mechanism (Ramachandran al. 2019) Vaswani et al. (217)) with 8 heads in each GSA module. The relative position embeddings are shared across all heads within a module, but not across modules. All 1 convolutions and GSA modules are followed by batch normalization (loffe & Szegedy 2015).
Training and evaluation All models are trained and evaluated on the training and validation sets of the ImageNet dataset (Russakovsky et al., 2015). respectively. They are trained from scratch for 90 epochs using stochastic gradient descent with momentum of 0.9, cosine learning rate schedule with base learning rate of 0.1, weight decay of 10 10-4, and mini-batch size of 2048. use standard data augmentations such as random cropping and horizontal flipping. Following recent attention- based works (Ramachandran et al. 2019 Zhao et al., 22200] Wang et al. 2020), we also use label smoothing regularization with coefficient 0.1. For evaluation, we use single 224 x 224 center crop. While computing FLOPs, multiplications and additions are counted separately. For reporting runtime, measure inference time for single image on TPUv3 accelerator.
4.1 COMPARISON WITH CONVOLUTION OPERATION
Figure compares ResNet- (38,50018 structure-based CNNs and GSA networks. The GSA net- works outperform CNNs significantly while using less parameters, computations and runtime. These results clearly shows the superiority of the proposed global attention module over the widely-used convolution operation. With increasing popularity of attention-based models, we hope that hardware accelerators will be further optimized for attention dieention-based operations and GSA networks will become much more faster than CNNs in the near future.
4.2 COMPARISON WITH AXIAL ATTENTION
The GSA module uses global content attention mechanism that attends to the entire image at once. To validate the superiority this attention mechanism over axial attention, Table 2 we compare the proposed GSA module with a global attention module that attends based on both content and positions similar to Ramachandran et al. (2019) but in an axial fashion. The GSA module clearly outperforms the axial alternative. Also, the performance of our axial positional attention alone is comparable to the axial attention that uses both content and positions suggesting that axial mecha- nism not able to take advantage of content-only interactions (see Section 3.3 for justification).

---

Under review as a conference paper at ICLR 2021
Table 2: Comparison of the proposed GSA module with axial attention
Attention module
Top-1 acc.
Top-5 acc.
Parameters
FLOPs
Runtime
Proposed GSA module
78.5
93.9
18.1 M
7.2
31.7 ms
Axial content and positional attention
77.5
93.6
18.1 M
7.3
32.5 ms
Only axial positional attention
77.4
93.5
16.8
6,4 28.2 ms
Table 3: Comparison of GSA networks (3 x 3 convolutions replaced with GSA modules) with recent attention-base based approaches. Note that Wang er al. (2020) is the conv-stem version since GSt-Net uses a conv stem. M-ResNet-50 is modified ResNer-50 which halves the number of input and output channels of all residual blocks and scales the number of filters in every layer by 1.125
Structure
Method
Top-1 acc.
Top-5 acc.
Params
FLOPs
Runtime
ResNet-50
Chen et al
2018
77.0
93.5
Shen el a
2018
77.3
93.6
26.2 M
9.9 G
Hu et al.
2019)
77.3
93.6
23.3 M
8.6
Ramachandran et al.
2019
77.6
18.0 M
7.0
131.9 ms
Yue et al. (2018
77.7
93.6
Bello al
(2019
77.7
93.8
25.8 M
8.3
34.9 ms
Zhao al.
2020
78.2
93.9
20.5 M
6.6
This work
78.5
93,9
18.1 M
7.2G
31.7 ms
ResNet-101
et al.
2019)
78.5
94.3
42.0 M
16.0G
Bello (0019
78.7
94.4
45.4 M
16.1 G
61.9
This work
79.6
94.5
30.4 M
12.2
57.2 ms
M-ResNet-50
Wang et al.
2020
77.5
12.4 M
5.6
41.4 ms
This work
78.2
M
6.0
31.1 ms
4.3 COMPARISON WITH EXISTING AITENTION-BASEL APPROACHES
Table 3 compares GSA networks with recent attention- based networks. The GSA networks achieve better performance than cxisting global and local attention- networks while using similar or less number of parameters and FLOPs, except when compared Zhao al. (2020); Wang et al. 2020) which use slightly fewer FLOPs. Compared to local attention based works (Hu et al. 2019 Ramachandran al., 2019 Zhao et al. 2220), the proposed GSA network takes advantage of global attention throughout the network and produces better results. Compared to /Shen et al.) (2018); y Yue al. (2018); [Bello al. ;); Chen al. (2018) which insert few attention modules auxil- iary blocks into a CNN, the proposed GSA network uses global attention through out the network. Compared to Wang al. (2020). the proposed GSA network uses a non-axial global content atten- tion which is better than axial mechanism. To report runtime for other methods, we measure single image inference time on a TPUv3 accelerator using the code provided by the corresponding authors.
4.4 ABLATION STUDIES
4.4.1 IMPORTANCE OF INDIVIDUAL COMPONENTS
As described Section 3 GSA module consists of three components: a content attention layer, a column-only positional attention layer, and row-only positional attention layer. Table shows the results for different variants of the proposed GSA module obtained by removing one or more of its components. As expected, the module with all three components performs the best and the content-only attention performs poorly (7.7% drop in the top-1 accuracy) since it treats the entire image bag of pixels. This clearly shows the need for positional attention that is missing in many existing global attention aitention-based works (Chen et al. 2018: Wang et al.) 2018) Yue et al.) 2018). Interestingly, for positional attention, column-only attention performs better than row-only attention

---

Under review as a conference paper at ICLR 2021
Table 4: Comparison of different variants of the proposed GSA module
Table 5: Effect of replacing convolutions with GSA modules (indicated using v) at different stages of ResNet-50
Attention component
Accuracy
Content
Col.
Row
Top-1
Top-5
Residual group
Accuracy
78.5
93.9
1
2
4
Top-1
Top-5
Runtime
V
V
77.4
93.5
4
V
v
78.5
93.9
31.7 ms
V
76.2
92.7
V
/
78.7
94.1
28.5 ms
V
75,4
92.2
V
v
78.5
94.1
24.5 ms
v 77.7
93.7
23.2 ms
V
72.6
90.8
70.2
89.4
ResNet-50 CNN
76.9
93.5
22.6 ms
70.8
89.5
ResNet-101 CNN
78.7
94.4
39.3 ms
(row3 row4 and row5 vs row6) suggesting that modeling pixel interactions along the vertical dimension is more important than the horizontal dimension for categories in the ImageNet dataset.
4.4.2 WHERE IS GLOBAL ATTENTION MOST HELPFUL?
replaces spatial convolution with the proposed global attention module in all residual groups of ResNet-50. Table how the performance varies when global attention replaces spatial convolution only in certain residual groups. Starting from the last residual group, as we move towards the earlier stages of the network, replacing convolution with attention improves the performance consistently until the second residual group. Replacing convolutions in the first residual group results in a slight drop in the performance. These results show that the global attention mechanism is helpful throughout the network except in the first few layers. This is expected behavior since the first few layers of a deep network typically focus on learning low-level features. It is worth noting that by replacing convolutions with the proposed GSA modules in the second, third and fourth residual blocks of ResNet-50, we are able to achieve same top-1 accuracy as convolution- based ResNet-100 101 while being significantly faster.
Our default
4.5 RESULTS ON CIFAR- 100 (KRIZHEVSKY & HINTON 2009
Similar to the ImageNet dataset, the proposed GSA networks outperform the corresponding CNNs significantly on the CIFAR- 100 dataset while using less parameters, computations, and runtime. Improvements in the top-1 accuracy with ResNet- (38, 50, 101} structures are 2.5%, 2.7% and 1.6%, respectively. Please refer to Fig. 3 and Table the Appendix for further details.
5 CONCLUSIONS
In this work, we introduced a new global self-attention module that takes both the content and spatial locations of the pixels into account. This module consists of parallel content and positional attention branches, whose outputs are summed at the end. While the content branch attends to all the pixels jointly using an efficient global attention mechanism. the positional attention branch follows axial formulation and performs column-only attention followed by row-only attention. Overall, the proposed GSA module is efficient enough be the backbone component of a deep network. Based on the proposed GSA module, we introduced GSA networks that use GSA modules instead of spatial convolutions. Due to the global extent the proposed GSA module, these networks have the ability to model long-range pixel interactions throughout the network. We conducted experiments on the CIFAR- 100 and ImageNet datasets, and showed that GSA networks clearly outperform their convolution- based counterparts while using less parameters and computations. We also showed that GSA networks outperform various recent local and global sieetiion-based networks. In the near future, plan to extend this work other computer vision tasks.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangging Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dan Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vin- cent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Watten- berg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. TensorFlow: Large-scale machine learning on heterogeneous systems, 2015. tensorflow.org.
org/ Software available from
Irwan Bello, Barret Zoph, Ashish Vaswani, Jonathon Shlens, and Quoe V Le. Attention augmented convolutional networks. In ICCV, 2019.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In ICLR, 2019.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. wrXiv:2005.12872 2020.
End-to-end object detection with transformers. arXiv preprint
Yunpeng Chen, Yannis Kalantidis, Jianshu Li, Shuicheng Yan, and Jiashi Feng. A Double attention networks. In NeurlPS, 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, 2016.
Jonathan Ho, Nal Kalchbrenner, Dirk Weissenborn, and Tim Salimans. Axial attention in multidi- mensional transformers. arXiv preprint 1rXiv: 2019.
Han Hu, Zheng Zhang, Zhenda Xie, and Stephen Lin. Local relation networks for image recognition. In ICCV, 2019.
Zilong Huang. Xinggang Wang, Lichao Huang, Chang Huang, Yunchao Wei, and Wenyu Liu. CC- Net: Criss-cross attention for semantic segmentation. In 2019.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In ICML, 2015.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Tech- nical Report, 2009.
Guanbin Li, Xiang He, Wei Zhang, Huiyou Chang, Le Dong, and Liang Lin. Non-locally enhanced encoder-decoder network for single image de-raining. In ACMMM, 2018.
Xingyu Liao, Lingxiao He, and Zhouwang Yang. Video-based person re- re-identification via 3d con- volutional networks and non-local attention. arXiv preprint arXiv:1807.05077 2018.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury. Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy. Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems 32, pp. 8024 8035. Curran Associates, Inc., 2019. URL Utt.stteee/ppppe.............//p pdf
librar.r...... library..
Prajit Ramachandran, Niki Parmar, Ashish Vaswani, Irwan Bello, Anselm Levskaya, and Jonathon Shlens. Stand-alone self-attention in vision models. In NewrIPS, 2019.

---

Under review as a conference paper at ICLR 2021
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang. Andrej Karpathy, Aditya Khosla, Michael S. Bernstein, Alexander C. Berg, and Fei-Fei Li. ImageNet large scale visual recognition challenge. International Journal of Computer Vision, 115(3):211-255 2015.
Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position representa- tions. In NAACL-HLI, 2018.
Zhuoran Shen, Mingyuan Zhang, Shuai Yi, Junjie Yan, and Haiyu Zhao. Efficient attention: Self- attention with linear complexities. arXiv preprint arXiv:/8/2.0f1 2018.
Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid. Videobert: A joint model for video and language representation learning. In ICCV, 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. NeurIPS, 2017.
H. Wang, Zhu, Green, Adam, A. Yuille, and L.-C. Chen. Axial-DeepLab: Stand-alone axial-attention for panoptic segmentation. arXiv preprint arXiv:2003.0785; 2020.
Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In CVPR, 2018.
Kaiyu Yue, Ming Sun, Yuchen Yuan, Feng Zhou, Errui Ding, and Fuxin Xu. Compact generalized non-local network. NeurlPS, 2018.
Han Zhang, lan J. Goodfellow, Dimitris N. Metaxas, and Augustus Odena. Self-attention generative adversarial networks. In ICML, 2019.
Hengshuang Zhao, Jiaya Jia, and Vladlen Koltun. Exploring self-attention for image recognition. In CVPR, 2020.
A CIFAR- 100 EXPERIMENTS
All the models are trained and evaluated on the training and test splits of CIFAR-100, respectively. They trained for 10K steps starting from ImageNet pretrained weights using stochastic gradient descent with momentum of 0.9, weight decay of 10-2, and mini-batch size of 128. We use an initial learning rate of 5 10 and reduce it by factor of 10 after every 3K steps. For both training and evaluation, we use 224 X 224 input images.
Fig. 3 compares ResNet- (3,Nct-(38.00.. (3,5001000) 50,101) structure-based CNNs and GSA networks on the CIFAR-100 dataset. Similar to ImageNet results, GSA networks outperform CNNs significantly on the CIFAR- 100 dataset while using less parameters, computations, and runtime. reports all the numbers corresponding to the plots in Fig. 2 and Fig.
CNN
84.0
84 0-
84.0
eccurecs
83.5
Top-4 accunarg
63.5
Top-1 accroey
83.5
83.0
63.0
83.0
82.5
62.5
82.0
82.0
82.0
81.5
81.5
CNN
81.5
CNN
81.0
CSA-Net
15
20
25
30
35
45
13
14
16
30
40
So
Paramiteess (M)
FLOPs (G)
Lacency (ms)
Figure 3: Comparison between ResNet-(38, 50, 101) structure-based CNNs and GSA networks. GSA networks clearly outperform CNNs while using less parameters, computations, and runtime.
10

---

Under review as a conference paper at ICLR 2021
Table 6: Comparison between CNNs and GSA networks
ImageNet
1IFAR-100
Structure
Operation
Top-1 acc.
Top-5 acc.
Top-1 acc.
Params
FLOPs
Runtime
ResNet-38
Convolution
75.9
92.9
80.8
19.6 M
6.4
17.2 ms
GSA
(+1.9) 77.8
93.6
(+2.5) 83.3
14.2 M
5.9
24.7 ms
ResNet-50
Convolution
76.9
93.5
81.2
25.6
8.2
22.6 ms
GSA
(+1.6) 78.5
93.9
(+2.7) 83.9
18.1
7.2G
31.7 ms
ResNet-101
Convolution
78.7
94.4
82.8
44,5 M
15.6 G
39.3 ms
GSA
(+0.9) 79.6
94.5
(+1.6) 84.4
30.4 M
12.2 G
57.2 ms
B MATHEMATICAL IMPLEMENTATION DETAILS
This section presents mathematical implementation details of the Global Self-Attention (GSA) mod- ule to supplement the high-level description in Section 3 of the paper.
For conciseness and better resemblance of the actual implementation, this section the Einstein notation"] Note that both TensorFlow Abadi et al. (2015) and PyTorch Paszke et al. (2019) pro- vide direct support for the Einstein notation, through t w.f.einssen einsum () and torch e e einsum (), respectively. Therefore, there are direct TensorFlow/PyToree transcriptions for all equations in this section.
Assume the input X is rank-3 tensor of shape h x w x d, for h the height, w the width, and d the number of channels.
KQV layer The first step is to compute the keys K, queries Q, and values V from X using separate 1 (i.e. point-wise) convolution layers. Then, the module splits K, Q, V each into equal-size slices along the channel dimension for the attention heads. An efficient implementation fuses the two steps into
Kaynk W((K) drkk Xayd
Quynk W (Q)
dnk
Vayno W (V) due Xayds
where ((K), (()), W(V) are the corresponding weights, x, y are the spatial dimensions, n the head dimension, and d, k, e the channels dimensions for the input, the keys and queries, and the values, respectively.
Content attention As Section 3 of the paper describes, within each head the module matrix multiplication to implement content attention. The actual implementation parallelizes the process across all heads by computing
K = o(K),
Onko Kaynk V Vaynar yC Ca
(4)
where a represents softmax along the spatial dimensions (æ,, ().
Positional attention a batch normalization layer, and row-only attention sub-layer. Since the column-only and row- only sub-layers are symmetric, this section only presents the implementation for the column-only sub-layer.
The positional attention layer consists of column-only attention sub-layer,
'The Einstein notation is compact convention for linear algebra operations Albert Einstein devel- oped. htttpp://en..wikiiddaa... jar jet.erenee//aaii-ggutttttoo
widd http://w.ww../////////oo//.o///e http:///wwww.iiisiii..i.... instiinnennn iestteinnnottttii nonatiio provides a reference. https:
gives an intuitive tutorial.

---

Under review as a conference paper at ICLR 2021
The layer maintains relative position embedding matrix R E R(2h-1)* for h the image height and & the number of channels. Each of the 2h rows corresponds to a possible vertical relative shift, from - (h 1) to h - 1. The first step is to re-index this matrix from using relative shifts to absolute shifts. To achieve this goal, the module creates a re-indexing tensor where
I.,.r = if x =r Ta,irr = O, otherwise,
1 L,
(5)
where L is the maximum relative shift to attend to. The default version of GSA sets L = max (h, w so that the positional attention is global.
Then, the module computes the position embedding tensor whose indices are the absolute shifts
Park Liir Ryk-
(6)
Now, the output of the column-only attention sub-layer is
Sayin Quynk Pzike
yH syne
Suyen Vayna-
After obtaining YH, the module applies batch normalization to it and uses it the input to the row-only sub-layer to generate y the final output of the positional attention layer.
Final fusion After computing the outputs of the content and positional attention layers, the final output simply
Y == +Y
(8)
Comparison competing approaches The implementation of the GSA module only consists of 8 Einstein-notation equations and other equations, each of which corresponds to one line of code in FeosorFlow or PyTorch. The implementation substantially simpler in comparison competing approaches Ramachandran et al. (2019); Zhao et al. (2020) using local attention which requires custom kernels.
12


Reviews:
[{"review_id": "kWTqXEnEun8", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis paper proposes global self-attention networks for image recognition. The proposed GSA module is consists of two parallel layers,  a content attention layer, and a positional attention layer. Experimental evaluations are conducted on ImageNet and Cifar100, and the results are promising.\n\nClarity:\n1. I think this paper is moderate. The content attention layer can be treated as calculating the correlations between each slice of the feature maps, then reweighting/aggregating information through the channel dimension. It is not spatial attention and maybe we cannot say it is global attention.\n2. The complexity of the proposed content attention layer is O(Nd_kd_out), and classical global attention like non-local is O(NNd_k). The author should not ignore the channel values as they are large in the top layers.\n3. From table 4, the position attention layer is much more important than the content attention layer (77.4 vs 70.8), the position attention is local attention (not global) and also the content is not global spatial attention (for capturing long-range context). Thus I am not convinced by the claim of the paper. Besides, from previous papers (Hu et al. (2019), Ramachandran et al. (2019), Zhao et al. (2020)), we know that local self-attention is already enough, and global attention is not really needed.", "extra_scores": {}, "extra_text": {}}, {"review_id": "h0oKsquPJsR", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\n\n\n**Update after rebuttal** : I want to thank the authors for their rebuttal. However, after reading all responses and all new results presented, I still think that most of the weaknesses of this paper are still present. Two notes that I urge authors to take into account:\n\n* Even the updated Figs 2 and 3 are highly misleading: sure, the gains over a baseline ResNet network are big, but the comparisson should be vs other backbones that combine convolutions with attentions. \n* The authors should clearly say that they do use a convolution for conv1, and their modules start on top of that first spatial convolution.\n\n----------------------------------------\n\nSummary:\nThe authors present a composite attention module that complement the global attention with axial attentions over rows and columns. They replace 3x3 convolutions with this module on ResNets and show they can outperform convoultion and convolution+ attention based models on ImageNet.\n\nStrong points: The authors present a method that replaces almost all convolutions of a CNN with self-attention variants and gives high results for ImageNet.\n\nWeak points:\nMany parameters of the method are chosen arbitrary and are not really ablated; the same stands for alterations in eg the \"content attention\" layer which is a variant of self-attention, but not exactly the same, since the softmax is applied only on the key values. For a method that argues that one should replace convolutions with attentions, a lot more ablations on hyperparameter stability and effect would be needed, not just a \"winning combination\", as well as experiment on more than one dataset (CIFAR experiments are for transfer learning).\n\nSpecifically:\n\nA) In general, a lot of ablations are missing. The effect of the number of heads, the dimension parameter $d_k$ and the very important region parameter $L$ are not ablated. In fact, parameter $L$ is never explicitly set in the main text - the end of the appendix states \"the default version of GSA sets L = max{h, w}\" - this should be clearly set and ablated in the main text.  The term \"global axial attention\" is mentioned in the text, but the text does not say that global axial attention is the \"default case\". \n\nB) Sure, not having a softmax as in the common decomposed self-attention allows for more flexibility, but the gains from this change are not really isolated nor ablated; In the text the authors say that there is a decrease by 0.9% on ImageNet without specifying any details (which model? compared to what?). This seems like a tweak that affect performance significantly, that is not ablated.\nQb1) What is the performance of the proposed module when keeping everything the same but just substituting the proposed \"content attention\" with the common way of using softmax, eg what is used in (Chen et al 2018) or (Shen et al 2018)? If there is a large drop, how would you explain this? \n\nC) The section that presents the GSA network (3.2) is very short, incomplete and  highly unclear:  \nQc1)  the authors say they \"replace all 3 × 3 convolution layers in ResNet-50\" , but the first conv1 in ResNet is not 3x3: Do you replace *all* convolutions? even the very first one, or is there a 7x7 convolution in your GSA network? If so, this should be clearly stated and not implied.\nQc2) How many layers of self-attention do you have in each block, is it exactly as many as convolutions?  how do you end up with 18M Parameters and these FLOPS?\n\nD) The paper presents results *only* on the ImageNet dataset (There are some results on CIFAR in the appendix, but models there are only fine-tuned from ImageNet pretrained, and not from scratch); This is a dataset that has no true test set and all methods are overfitting the validation set; this is a dataset where different hardware setups give different baseline numbers. I understand that this is common practice, but, as most other papers do, I would also expect results at at least one more dataset or task to be fully convinced that this approach has something novel to offer. What is more, the curves in Figures 2,3 are against the vanilla ResNet architecture and seem a bit outdated;  a number of more recent papers have taken convolutions to much higher performance with minimal overheads (eg SE-Nets) that would not really show such big differences. \nQd1) Baseline performance (on the same setup) is highly important and not really presented in Table 3: what would a baseline ResNet-50 achieve at the authors's hardware, and for the same training setup (90 epochs using stochastic gradient descent with momentum of 0.9, cosine learning rate schedule with base learning rate of 0.1, weight decay of 10−4, and mini-batch size of 2048 as well as with the same label smoothing)? This would enable a more fair comparison between Convolutions and a network of (almost) only self-attentions. \nQd2) Are Resnet numbers in Figures 2,3 reproduced or copied from the 2016 resnet paper?\n\nE) In a missing related work [Chen et al \"Graph-based global reasoning networks.\" CVPR 2019] use global attention + a graph convolution in a module they call GloRe; the graph convolution there seems to be something similar to axial attention, but over nodes on a graph and not positions. Similar to this paper that restricts softmax only on the keys , the softmax in GloRe  is also removed for similar reasons. This is a paper that would be nice to be discussed and compared to - a comparison is also missing from Table 3; that paper reports that a Resnet-50 with 3 Glore units reaches 78.4 top-1 accuracy with 5.2 GFLOPS (and 30M params), vs 78.5 for the approach proposed here with 7.2 GFLOPS.  \n\nNotes and questions:\n* The name of the module focuses on \"global\" but this is a characteristic of many other attentions cited (Wang et al. (2018); Yue et al. (2018); Chen et al. (2018); Shen et al. (2018); Huang et al. (2019)) and doesn't seem fitting to be the main thing titled here. The approach adds positional/axia attention side by side to global; this could/should be reflected in the title\n* It would be interesting to see if distillation could help this model as much as it does convolutional resnets.\n* Figures 2,3 would be clearer if markers were indicated for the 3 datapoints of each line. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "W8IKyehQF13", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "There have been multiple attempts to use self-attention in computer vision backbones for image classification and object detection. Most of these approaches either tried to combine convolution with global self-attention, or replace it completely with local self-attention operation. The proposed approach naturally combines the two, by employing query-key-value switching trick, with axial positional attention.\n\nThe approach is very similar to Axial Deeplab (Wang et al. 2020, ECCV 2020 Spotlight), with the main difference that Axial Deeplab uses axial self-attention in the content part, instead of global self-attention, for computational efficiency reasons. The authors claim superiority of their approach, and provide an experimental comparison, showing that there is almost no benefit from axial content attention, which is suspicious - could be an implementation issue?\n\nOverall, the benefit of content attention is not clear from the experiments, as it has more parameters, it is not surprising that it yields higher accuracy and slower runtime. An experiment with the same number of parameters with/without content part is missing.\n\nRegarding softmax on queries, I think the motivation behind was activation normalization. I am curious if it poses additional training difficulties and if the authors tried other activations similar to Katharopoulos et al. (Transformers are RNNs), e.g. ReLU and scalar normalization.\n\nCIFAR-100 experimental setup is uncommon and very surprising, as the authors use ImageNet pretrained models and finetune on CIFAR-100 with large input images, in contrast to training from scratch on CIFAR-100. I suggest that these experiments are updated, otherwise it is extremely difficult to reproduce the paper with a limited computational budget.\n\nRegarding the reproducibility, releasing the code for such methods must be a requirement, as reproducing similar approaches like stand-alone self attention is notoriously difficult, and I don’t think the community managed to do it.\n\nAxial Deeplab paper has been online for 6 months before ICLR submission deadline, the novelty over it is very limited, but I would leave for ACs to decide if it is concurrent or prior work. Disregarding this, the paper has valuable findings but needs more rigorous experimental evalution, as I suggested in my review.", "extra_scores": {}, "extra_text": {}}, {"review_id": "Cp0FV7T2tGa", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "# Post-rebuttal update\n\nI would like to thank the authors for the detailed feedback. I am now convinced about the statistical significance of the results. Regarding the additional study, while it is true that the combination of the changes, in addition to the softmax, was what made the results improve, the change is quite minor. Also, the biggest change comes from the fact that softmax is removed. The reviewer also finds the explanation of why this happens to be handwavy.\n\nGiven the concurrent work [Wang et al. 2020] and the incremental nature of the innovation, the reviewer is not sure of the benefits of the current paper to be published at ICLR.\n\n# Summary\n\nThe paper proposes a Global Self Attention (GSA) layer, that encapsulates both global and local self-attention, by computing them in parallel and merging the two attentions together. The method is tested on ImageNet and CIFAR-100, demonstrating state-of-the-art results.\n\nWhile the paper delivers state-of-the-art performance, the novelty of the paper is weak. The enhancements in this paper are also minor, making the paper borderline. The paper currently makes strong claims about the benefit of the proposed method, which are not all obvious. Thus, my preliminary rating for the paper is borderline reject.\n\n# Strengths\n\nThe main strength of the paper is its performance and the extensive comparison against the state-of-the-art. The method outperforms all compared methods, albeit by a small margin. CIFAR-10 results are also interesting, but there is no comparison against other self-attention.\n\nThe paper is easy to follow, with detailed coverage of related works. \n\nThe ablation study demonstrates which components are important.\n\n# Weaknesses\n\n## Weak novelty, sigmoid, & regarding Wang et al. (2020)\n\nComponents that constitute GSA are already shown to be highly effective. The main novelty of the paper then seems to be that the softmax is not used at certain locations and that the two different types of attention are combined together. Thus, the novelty of the paper is limited, especially considering the fact that the gain in performance is marginal. Furthermore, while the focus is on the combination of the two types of attention, which does seem to help in the ablation study, I wonder whether the removal of the softmax had a larger effect. This makes the argument of the paper weak. This leads to my second concern about the paper.\n\nThe removal of the sigmoid seems to have had a large effect. At the end of section 3.1.1, it is mentioned that removing softmax increases performance by 0.9% absolute. Considering that the amount of gain from global attention is around 1.1% absolute, this means that without the softmax, there is almost no gain from introducing global attention. Furthermore, I wonder if Wang et al. (2020), without the softmax as in this paper would provide enhanced results as in this paper. \n\nThus, it is my current understanding that the method, with softmax and without content attention is similar, if not identical, to Wang et al. (2020). An additional ablation study where softmax is removed from Wang et al. (2020)'s method is necessary to clarify where the performance gain is really coming from.\n\nAlso, I am not sure why the paper has chosen the Conv-stem from Wang et al. (2020) and not the full version. Could the authors clarify this?\n\n## Not significantly better\n\nThe tone of the paper, regarding the experiments, also needs to be weakened. The method, compared to prior work, improves existing work only slightly. For example, in Table 3, the gap in top-1 performance 0.3% in the case of Resnet-50, 0.9% in ResNet-101, and 0.1% compared to the full version of Wang et al. (2020). I also wonder if this satisfies statistical significance, given that each training session will give slightly varying results. \n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Paper was reviewed by four expert reviewers. Unfortunately all reviewers, uniformly felt that paper fell marginally bellow bar and argue for rejection. A number of concerns have been identified by the reviewers in the review phase.  Those included: (1) lack of novelty [Reviewer3, Reviewer4], (2) lack of various ablations [Reviewer 2], (3)  issues with experimental setup [Reviewer4], and (4)  lack of significant improvements in performance [Reviewer 1, Reviewer 3]. While authors addressed some of the concerns with provided experiments and ablations during the rebuttal, Reviewers remained unconvinced on the main concerns of novelty and significance. As such, the reviewers are unanimous in their assessment and AC does not see a reason to overturn this consensus. 
Decision: Reject