Glance-and-Gaze Vision Transformer
Authors: Qihang Yu
Yingda Xia
Yutong Bai
Yongyi Lu
Alan Yuille
Wei Shen
Venue: NeurIPS 2021 Poster (2021)
Keywords: Vision Transformer
Image Classification
Semantic Segmentation
Object Detection

Abstract: Recently, there emerges a series of vision Transformers, which show superior performance with a more compact model size than conventional convolutional neural networks, thanks to the strong ability of Transformers to model long-range dependencies. However, the advantages of vision Transformers also come with a price: Self-attention, the core part of Transformer, has a quadratic complexity to the input sequence length. This leads to a dramatic increase of computation and memory cost with the increase of sequence length, thus introducing difficulties when applying Transformers to the vision tasks that require dense predictions based on high-resolution feature maps.

In this paper, we propose a new vision Transformer, named Glance-and-Gaze Transformer (GG-Transformer), to address the aforementioned issues. It is motivated by the Glance and Gaze behavior of human beings when recognizing objects in natural scenes, with the ability to efficiently model both long-range dependencies and local context. In GG-Transformer, the Glance and Gaze behavior is realized by two parallel branches: The Glance branch is achieved by performing self-attention on the adaptively-dilated partitions of the input, which leads to a linear complexity while still enjoying a global receptive field; The Gaze branch is implemented by a simple depth-wise convolutional layer, which compensates local image context to the features obtained by the Glance mechanism. We empirically demonstrate our method achieves consistently superior performance over previous state-of-the-art Transformers on various vision tasks and benchmarks.

Glance-and-Gaze Vision Transformer
Qihang Yu', Yingda Xia', Yutong Bai', Yongyi Lu', Alan Yuille', Wei Shen DDparrment Computer Science, Johns Hopkins University 2 MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University
(yucornetto, philyingdaxia, ytongbai, yylu1989, alan. man.1.1. yyl.yulllllmmaaaaae com wei. shen@sjtu. edu. cn
Abstract
Recently, there emerges a series of vision Transformers, which show superior performance with more compact model size than conventional convolutional neural networks, thanks to the strong ability of Transformers to model long-range dependencies. However, the advantages of vision Transformers also price: Self-attention, the core part of Transformer, has a quadratic complexity to the input sequence length. This leads to dramatic increase of computation and memory cost with the increase of sequence length, thus introducing difficulties when applying Transformers to the vision tasks that require dense predictions based on high-resolution feature maps.
with a
In this paper, we propose new vision Transformer. named Glance-and-Gand Trans- former TrGGTraasfforrr, to address the aforementioned issues. It is motivated by the Glance and Gaze behavior of human beings when recognizing objects in natural scenes, with the ability to efficiently model both aangraagee dependencies and local context. In GG- "G-Transformer, the Glance and Gaze behavior is realized by two parallel branches: The Glance branch is achieved by performing self-attention the caaptively-dilatee partitions of the input, which leads linear complexity while still enjoying a global receptive field; The Gaze branch is implemented by a simple depth-wiss wise convolutional layer, which compensates local image context the features obtained by the Glance mechanism. We empirically demonstrate our method achieves consistently superior performance over previous state-of-the-art Transformers on various vision tasks and benchmarks.
1 Introduction
Convolution Neural Networks (CNNs) have been dominating the field of computer vision, which have been de-facto standard and achieved tremendous success in various tasks, e.g., image clas- sification [16, object detection (15,, semantic segmentation [5] etc. CNNs model images from a local-to- -ocal-to-global perspective, starting with extracting local features such edges and textures, and forming high-level semantic concepts gradually. Although CNNs prove to be successful for various vision tasks, they lack the ability globally represent long-range dependencies. To compensate a global view to CNN, researchers explored different methods such as non-local operation [36). self-attention 33 Atrous Spatial Pyramid Pooling (ASPP) (5)
Recently, another type of networks with stacked Transformer blocks emerged. Unlike CNNs, Transformers naturally learn global features in parameter-free manner, which makes them stronger alternatives and raises questions about the necessity of CNNs in vision systems. Since the advent of Vision Transformer (VIT) [12). which applied Transformers to vision tasks by projecting and tokenizing natural images into sequences, various improvements have been introduced rapidly, e.g.,
"Corresponding Author.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

better training and distillation strategies 32 tokenization 41, position encoding 77. local feature learning [1]. Moreover, besides Transformers' success on image classification, many efforts have
been made to explore Transformers for various down-stream vision tasks 35 24
13
46
Nevertheless, the advantages of Transformers come at price. Since self-attention operates on the whole sequences, it incurs much more memory and computation costs than convolution, especially when it comes natural images, whose lengths are usually much longer than word sequences, treating each pixel as a token Therefore, most existing works have to adopt a compromised strategy to embed large image patch for each token, although treating smaller patches for tokens leads to better performance (e.g., ViT-32 compared to ViT-16 [2]. To address this dilemma, various strategies have been proposed. For instance, Pyramid Vision Transformer (PVT) [35] introduced a progressive shrinking pyramid to reduce the sequence length of the Transformer with the increase of network depth, and adopted spatial-reduction attention, where key and value in the attention module are down-sampled to a lower resolution. Siiii-Tranffomer (24] also adopted the pyramid structure, and further proposed to divide input feature maps into different fix-sized local windows, so that self-attention is computed within each window, which reduces the computation cost and makes it scalable to large image scales with linear complexity.
Nonetheless, we notice that these strategies have some limitations: Spatial-reduction attention can reduce memory and computation costs to learn high-rew nigh-resolution feature maps, yet with a price of losing details which are expected from the high-resolution feature maps. Adopting self-attention within local windows is efficient with línear complexity, but it sacrifices the most significant advantage of Transformers in modeling Long-range dependencies.
To address these limitations, propose Glance-and- Transformer (GG-Transformer), inspired by the Glance-and-Gaze human behavior when recognizing objects in natural scenes [[], which takes advantage of both the long-range dependency modeling ability of Transformers and locality of convolutions in complementary manner. A GG-Transformer block consists of two parallel branches: A Glance branch performs self-attention within adaptively-dilatee partitions of input images or feature maps, which preserves the global receptive field of the self eelf-attention operation, meanwhile reduces its computation cost to linear complexity as local window attention [24] does; Gaze branch compensates locality to the features obtained by the Glance branch, which is implemented by a geight depth-wise convolutional layer. merging operation finally re-arranges the points in each partition to their original locations, ensuring that the output of the NGG-Transformer block has the same size as the input. We evaluate GG-Transformer on several vision tasks and benchmarks including image classification on ImageNet [IO], object detection on COCO [23], and semantic segmentation on ADE20K (48), and show its efficiency and superior performance, compared to previous state-of-the-ar Transformers.
2
Related Work
Convolution has been the basic unit in deep neural networks for computer vision problems. Since standard CNN blocks were proposed in [22). researchers have been working on designing stronger and more efficient network architectures, egg., VGG 30, ResNet [66), Mo- bileNet (29). and EfficientNet [3 In addition to studying how to organize convolutional blocks into a network, several variants of the convolution layer have also been proposed, e.g., group convolu- tion (21), depth-wise convolution (6. and dilated convolution (40). With the development of CNN architectures, researchers also seeked to improve contextual representation of CNNs. Representative works, such as ASPP [5] and PPM 45 enhance CNNs with multi-scale context, and NI.Net [36] and CCNet [20] provided a non-local mechanism to CNNs. Morcover, instead of just using them as an add-on to CNNs, some works explored to use attention modules to replace convolutional
CNN and self-attention.
blocks I8 34 4F1
Vision Transformer. Recently, VIT 12 was proposed to adapt the Transformer 33 for image recognition by tokenizing and flattening 2D images into sequence of tokens. Since then, many works have been done to improve Transformers, making them more suitable for vision tasks. These works can be roughly categorized into three types: (1) Type I made efforts to improve the ViT design itself. For example, DeiT 332 introduced a training scheme to get rid of large-scale pre-training and distillation method to further improve the performance. TiT-ViT [41 presented token-to-token operation as alternatives to patch embedding, which keeps better local details. (2) Type II tried to
2

---

(a)
(b)
(c)
Figure Toy examples illustrating different methods reduce computation and memory cost of self-attention. (a) Spatial reduction [35] [13] spatially downsamples the feature map; (b) Local window [24 restricts self aelf-attention inside local windows; (c) Glance attention (ours) applies self- attention to adaptively-dilated partitions.
introduce convolution back into the VIT design. E.g., Chu et al. 7 proposed convolution for position encoding. Wu et al. (37) used convolution to replace the linear projection layers in Transformers. (3) Type III tried replace CNNs by building hierarchical Transformers backbone in many downstream tasks. Wang et al. (35) proposed a pyramid vision Transformer, which gradually downsamples the feature map and extract multi-scale features as common CNN backbones do. However, applying self-attention on high-resolution features is not affordable in terms of both memory and computation cost, thus they used spatial-reduction attention, which downsamples key and value in self-attention trade-off between efficiency and accuracy. Later, Liu er al. (24] proposed new hierarchical Transformer architecture, named Wwin-Trranffomer. To handle the expensive computation burden incurred with self-attention, they divided feature maps into several non-overlapped windows, and limited the self-attention operation to be performed within each window. By doing Swin-Transformer is more efficient and also scalable to large resolution input. Besides, to compensate the missing global information, a shifted window strategy is proposed to exchange information between different windows.
plug-in
Our method differs from aforementioned works the following aspects: Type I, methods usually utilize a large patch size and thus incompatible to work with high-resolution feature map. Type III methods proposed new attention mechanism to handle the extreme memory and computation burden with long sequences, but they sacrifices accuracy as trade-off with efficiency. In contrast, GG- Transformer proposes more efficient Transformer block with novel Glance-and-Gard mechanism, which not only enables it to handle long sequences and scalable to high-resolution feature maps, but also leads to a better performance than other efficient alternatives.
3 Method
The design of GG-Transformer draws inspiration from how human beings observe the world, which follows the Glance and Gaze mechanism. Specifically, humans will glance at the global view, and meanwhile gaze into local details obtain comprehensive understanding the environment. We note that these behaviors surprisingly match the property of self-attention and convolution, which models long-range dependencies and local context, respectively. Inspired from this, we propose OG-Transformer, whose Transformer block consists of two parallel branches: Glance branch, where self-attention is performed to adaptively-dikee dilated partitions of the input, and Gaze branch, west convolutional layer is adopted to capture the local patterns.
where a depth- wise
3.1 Revisit Vision Transformer
We start with revisiting the formulation of vision Transformer block, which consists of multi-head self-attention (MSA), layer normalization (LN), and multi-layer perceptron (MLP). A Transformer

---

Glance Branch
Softmax
Adaptively
K
Merging
Dilated
MLP
Splitting
Merging
Depthwise
Adaptively
Dilated
Splitting
Gaze Branch
Figure visual illustration GG Transformer block, where the Glance and Gaze branches parallely extract complementary information.
block processes input features as follows:
z MSA(LN(Z(-I) + Zc-1,
MLP(LN(Z))) + +zze
where z, is the encoded image representation at the l-th block. MSA gives Transformers the advantages of modeling global relationship in a parameter-free manner, which is formulated
MSA(X) = Softmax QKT
VC VV,
(3)
where K, V NxC are the query, key, and value matrices which are linear mappings of input X E R VxC C is the channel dimension of the input, and N is the length of input sequence. Note that for simplified derivation, assume the number of heads is 1 in the multi-head self attention, which will not affect the following complexity analysis and can be easily generalize to more complex cases.
For vision tasks, N is often related to the input height H and width W. In practice, a 2D image is often first tokenized based non-overlapping image patch grids, which maps 2D input with size H x W into a sequence of token embeddings with length size. In MSA, the relationships between token and all tokens computed. Such designs, though effectively capturing long-range features, incur computation complexity quadratic to N:
#xW p2 where (P, P) the grid
Q(MSA) 2 + 2.
For ViTs that only work on 16x down-sampled feature maps (i.e., P 16), the computation cost affordable, since this scenario N 14 14 196 (for typical ImageNet setting with input size 224 x 224). However, when it comes more general vision scenario with the need of dense prediction based on high-resolution feature maps (such as semantic segmentation), where cost dramatically increases by thousands of times even more. Naively applying MSA to such high-resolution feature maps can easily lead to the problem of oft-of-memory (OOM). and extremely high computational cost. Although some efficient alternatives [35] 24 were brought up recently, accuracy is often sacrificed as trade-off of efficiency. To address this issuc, we propose a new vision Transformer that can be applied to longer sequence while keeping high accuracy, inspired by the Glance-and-Gaan human behavior when recognizing objects in natural scenes [1]
3.2 Glance: Efficient Global Modeling with Adaptively-dilated Splitting
To address the efficiency problem of Transformers, existing solutions often adapt Transformers to high-resolution feature maps by down-sampling the key and value during the attention process [35), or limit self-attention to be computed in a local region then exchange information through shifting these

---

local regions to mimic global view [24]. But limitations exist in these methods. For down- wwwn-sampling methods, although the output feature maps keep to be high-resolution. they lose some details during the down-sampling processes. Besides, they still has quadratic complexity and thus may not scale up to a larger input size. For local-region based methods, though they successfully reduce the complexity to linear level, they cannot directly model a long-range dependency but instead are stuck within local context, which counters the design intuition of Transformer and self-attention for long-range dependency modeling. Besides, two consecutive blocks need to work together to mimic global receptive field, which may not achieve as good performance as MSA (see Table. 4b).
Thus, we propose Glance attention, which performs self-attention efficiently with global receptive field. It shares same time complexity as [24]. but directly models long-range dependencies, as shown in Fig. I Specifically, we first splits an input feature map to several dilated partitions, i.e., the points in partition are not from a local region but from the whole input feature map with a dilation rate adaptive to the feature map size and the token size. We name this operation Adaptively-dilatte Splitting For example, a partition contains M x M tokens and is obtained with an adaptive dilation rate w ). where h, is the height and width of current feature map respectively, and lww N. Here we assume all divisions have no remainder for simplicity. These partitions are easily to be split from the input feature map or merged back. Specifically, we formulate this process as follows:
Ze [z 2
(5)
where 1dd is the feature token at position (i, if reshaping the sequence of token embedding - back into 2D feature map and use the 2D coordinates accordingly. To reduce the memory and computation burden, while keeping global receptive field, 2 is split into several partitions:
z/_
AdapiivllyiillatadSpiiittig....
1.1.2 1.
| 2
Z
2--1
(7)
where t... - 1
i
(M
13h
CM 1)
2-1
where ق NxC, 2/ E R MCC Afterwards, MSA is applied to each partition, which substantially reduces the computation and memory cost yet does not lose the global feature represen- tation. And then all partitions are merged back into one feature map and go through the remaining modules:
z 2i MSA (z) (.11)) + Z 2
(9)
1. h w. :....:2. 2/ Merging(z4 o4 ;...;
(10)
Ze MLP(LN(z))) +
(11)
where Merging is an inverse operation of Adaptively DilateddSlittingg which re-arranges points in each partition back in their original orders.
This new self-attention module (formulated in Eq. to to 10), namely Glance multi-head self attention module (G-MSA), enables global feature learning with linear complexity:
M(G-MSA) = 4hwC2 + 222 = 4NC + NC.
(12)
3.3 Gaze: Compensating Local Relationship with Depthwise Convolution
Although the Glance branch can effectively capture long-range range representations, it misses the local connections across partitions, which can be crucial for vision tasks relying on local cues. To this end, we propose Gaze branch compensate the missing relationship and enhance the modeling power
at a negligible cost.
Specifically, to compensate the local patterns missed in the Glance branch, we propose to apply an additional depthwise convolution on the value in G-MSA:
Gaze( X) Depthhiseeonn (MeggMMnng()))
(13)
which has a neglectable computational cost and thus the overall cost is still significantly reduced:
2(GG-MSA) 2 + 2M NC + NC
(14)
5

---

where k is the Gaze branch kernel size, and M is the partition size set in Glance branch, both & and M are constants that are much smaller than N. We note that in this way, Long-range and -ranee features are naturally and effectively learned. Besides, unlike (24), GG-MSA does not require two consecutive blocks (e.g. W-MSA and SW-MSA) to be always used together, instead, it is a standalone module as the original MSA [2].
We propose two ways determine the kernel size for better compensating the local features:
Fixed Gazing. A straightforward way is to adopt the same kernel size (e.g., 3 x 3) for all Gaze branches, which can ensure same local feature learning regardless of the dilation rate.
Adaptive Gazing, Another way is implementing Gazing branch with adaptive kernels, where the kernel size should be the same as dilation rate (h/M, w/M). In this way, GG-MSA still enjoys complete view of the input.
By combining Glance and Gaze branches together, GG-MSA can achieve superior performance to other counterparts while remaining a low cost.
3.4 Network Instantiation
We build a hierarchical GG-TGrGTT TE--Trafffomrr with the proposed Glance-and-Gaud branches as shown in Fig. 2] For fair comparison, we follow the same settings as Twan-Transformer [24] in terms of network depth and width, with only difference in the attention methods used in Transformer blocks. Furthermore, we set M to be same as the window size [24], so that the model size and computation cost are also directly comparable. Note that GG- NG-Transformer has not been specifically tuned by scaling depth and width for a better accuracy-cost trade-off.
We build GG-T and GG-S, which share the same model size and computation costs as Swin-Ti T and Swin-S, respectively. For all GG-Transformers, we the fixed patch size M ratio of MLP a = 4. All GG-Transformer consists of 4 hierarchical stages, which corresponds to feature maps with sown-sampling ratio 4, 8, 16, 32, respectively. The first patch embedding layer projects input to a feature map with channel C = 96. When transitioning from one stage to the next one, we follow CNN design principles [16 to expand the channel by 2x when the spatial size is
expansion
down-sampled by 4x.
Experiments
In the following parts, we report results on ImageNet [10 classification, COCO 23 object detection, and ADE20K [48] semantic segmentation to compare GG-Transformer with those state-of-the CNNs and ViTs. Afterwards, we conduct ablation studies to verify the design of Glance and Gaze branches and also compare effectiveness of different alternative self-attention designs.
4.1 ImageNet Classification
We validate the performance of GG-Transformer ImageNet- IK [0] classification task, which contains 1.28M training images and 50K validation images for 1000 classes. We report top-1 accuracy with single 224 x 224 crop.
Implementation Details. To ensure fair comparison, we follow the same training settings of (24] Specifically, we use AdamW (25) optimizer for 300 epochs with cosine learning rate decay including 20 epochs for linear warm-up. The training batch size is 1024 with 8 GPUs. Initial learning rate starts
at 0.001, and weight decay is 0.05, Augmentations and regularizations setting follows [32] including
rand-augment [9], mixup 43). cutmix [42], random erasing [47), stochastic depth [19] but excluding
repeated repeated augmentation [17 and EMA (26].
Results. A summary results in Table D where we compare GG- GG-Transformer with various CNNs and ViTs. It is shown that GG- TG-Trrnsformer achieve better accuracy-cost trade-off com- pared to other models. Moreover, GG-T, weight model (28M/4.5G/82. comparable performance to those even much large models such as DeiT-B 186M/17.5G/8 1.8%), VIT-VVT-24 (64M/14. 1G/82 and PVT-Large HTMM//////1.... 1.7).. Furthermore, compared to Swin- Transformer. which we follows the architecture and ensures the same model size and computa-
can achieve

---

Table 1: Comparison of different models on ImageNet- IK classification,
image size
ImageNet
method
#param.
FLOPs
top-1 acc.
RegNetY-4G 27
224"
21M
4.0G
80.0
RegNetY-8G
27
224
39M
8.0G
81.7
RegNetY-166
27
224
84M
16.0G
EffNet-B3
31
300
12M
1.8G
81.6
EIfNet-B4
380
19M
4.2G
82.9
EltNet-B5
31
456
30M
9.9G
83.6
DeiT-T
32
224"
5M
1.3G
72.2
DeiT-S 32
224
22M
4.6G
79.8
DeiT-B
32
224
86M
17.5G
81.8
TNT-S 14
224
24M
5.2G
81.3
TNS-B 14
224
66M
14.1G
82.8
T2T-ViT-7
224
4M
1.2G
71.7
T2T-ViT-14
224
22M
5.2G
81.5
T2T-ViT-24
224
64M
14.1G
82.3
PVT-Tiny
224
13M
1.9G
PVT-Small
35
224
25M
3.8G
79.8
PVT-Medium
224
44M
6.7G
81.2
PVT-Large
351
224
61M
9.8G
81.7
Swin-T
24
224
28M
4.5G
81.2
Swin-S
24
224
50M
8.7G
83.2
GG-T
224
28M
4.5G
82.0
GG-S
224
50M
8.7G
83.4
tion costs to ensure fair comparison, our model consistently brings an improvement to baseline, with consistent improvement of 0.8% and 0.2% for T and S models respectively.
4.2 ADE20K Semantic Segmentation
is challenging semantic segmentation dataset, containing 20K images for training and 2K images for validation. We follow common practices use the training set for training and report mloU results on the validation sets. We use UperNet [38) the segmentation framework and replace the backbone with GG-Transformer.
ADE20K 4$
Implementation Details. We follow 24 and use MMSegmentation S to implement all related experiments. We use AdamW 25 with learning rate starting 0,01, batch size of 16, crop size of 512 x 512. The learning rate schedule contains a warmup of 1500 iterations and linear learning rate decay. The training is conducted with 8 GPUs and the training procedure lasts for 160K iterations in total. The augmentations follows the default setting of MMSegmentation, including random horizontal flipping, random re-scaling within ratio range [0.5, 2.0] and random photometric distortion. For testing, we follow [46] to utilize sliding window manner with crop size 512 and stride 341. ImageNet- 1K pretrained weights are used for initialization.
10-5, weight decay of
Results. We show results in Table 2 where results both w/ and w/o test-time augmentation are reported. Noticeably, GG-Transformer only achieves better results to baselines, but also obtain comparable single-scale testing performance to those with multi-scale testing results. Specifically, GG-T achieves 46.4% mloU with single-scale testing, which surpasses ResNet50. PVT-Small, Swin- T's multi-scale testing results by 3.6%, 1.6%, 0.6%. respectively. Moreover, our tiny model even can be comparable those much larger models (e.g.. 47.2% of GG-T compared 47.6% of Swin-S).
COCO Object Detection
We further verify the performance GG-Transformer when used as a plug-in backbone to object detection task COCO dataset (23), which contains 1 118K, 5K, 20K images for training, validation and test respectively. We use Mask-RCNN [55] and Cascaded Mask R-CNN (I as the detection frameworks, and compare GG-Transformer to various CNN and VIT backbones.

---

Table 2: Performance comparisons with different backbones on ADE20K validation dataset. FLOPs
tested on 1024 1024 resolution. All backbones are pretrained on ImageNet-Ik.
UperNet
Backbone
Prams (M)
FL.OPs (G)
mloU (%)
mloU(ms+flip) (%)
ResNet50 [6
67
952
42.8
PVT-Small 35
55
919
43.9
44.8
Swin-T 24
60
941
44.5
45.8
GG-T (ours)
60
942
46.4
47.2
ResNet101 16
86
1029
43.8
44.9
PVT-Medium 35
74
977
44.9
45.3
Swin-S 24
81
1034
47.6
49.5
GG-S (ours)
81
1035
48.4
49.6
Table Object detection and instance segmentation performance on the COCO va12017 dataset using the Mask R-CNN framework. Params/FLOPs is evaluated with Mask R-CNN architecture x280x800 image.
Params FLOPs
Mask R-CNN
Cascaded Mask R-CNN
Backbone
(M)
(G)
APb AP AP& APd
AP AP
APb AP3, AP AP AP AP
ResNet50 16
44
260
38.2 58.8 41.4 34.7 55.7 37.2
41.2 59.4 45,0
35.9 56.6 38.4
PVT-Small
44
245
40.4
62.9 43.8 37.8 60.1 40.3
Swin-T
48
264
43.7
66.6 47.7 39.8
63.3
42.7
48.1
67.1
52.2
41.7
64.4 45.0
GG-T (ours)
48
265
44.1
66.7
48.3 39,9
63.3 42.4
48.4
67.4
52.3
41.9
64.5
45.0
ResNet101 [6
63
336
40.0
60.6 44.0
36.1
57.5
38.6
42.9
61.0
46.6
37.3
58.2 40.1
ResNex(100-32xx4 33
63
340
41.9 62.5 45.9 37.5 59.4 40.2
44.3
62.8
48.4
38.3
59.7 41.2
PVT-Medium 35
64
302
42.0
64.4 45.6 39.0
61.6 42.1
Swin-S 24
69
354
45.4
67.9 49.6 41.4
65.1
44.6
49.7
68.8
42.8
46.4
GG-S (ours)
69
355
45.7 68.3 49.9
41.3
65.3
44.0
49.9
69.0
54.0
43.1 66.2 46.4
Implementation Details. We follow the setting of [24] and use MMDetection 4 to conduct all the experiments. We adopt multi-scale training (2). AdamW optimizer (25 with initial learning rate of 0.0001, weight decay of 0.05, batch size of 16. The training is conducted with 8 GPUs and a 1 1x schedule. All models initialized with ImageNet- 1K pretrained weights.
Results. As shown Table 3 GG-Transformer achieves superior performance other backbones in the two widely-used detection frameworks. Specifically, GG-T achieves 44.1 box AP and 39.9 mask AP, which surpasses both CNNs and other ViTs with a similar model size and computation costs. Compared with the state-of-the- art Swin- Transformer, GG-Transformer achieves better performance while keeping the same model size and computation costs for both and S models.
4.4 Ablation Studies
In this part, we conduct ablation studies regarding to the designs of OG-Transformer. Meanwhile, we also compare among different efficient alternatives to MSA. Besides, we verify GG-MSA on another ViT architecture [32] to compare its capacity to MSA directly. We conduct all these experiments based on Swin-T with 100 epochs training and DeiT 32) architectures with 300 epochs training.
Kernel Choice of Gaze Branch. We study the choice of Gaze branch in terms of fixed or adaptive mechanism. The kernel sizes for each stage and results are summarized in Table 4a, where we observe that both mechanisms work well. Using a larger kernel leads to non-significant improvement. In contrast, adaptive manner leads slightly better performance. Considering the adaptive manner provides a complete view as the original MSA has, we choose it in our final design.
Glance/Gaze Branch. We study the necessity of both Glance and Gaze branches. Meanwhile, comparison between different ways to conduct self- afff-tttenion is also studied. Results are in Table
serves as the baseline for all variants, which achieves 78.50% top-1 accuracy on ImageNet validation set. Firstly, we note that the local window attention and shifted window attention (W&SW-MSA) in [24] although significantly reduce the computation complexity and makes
Swin-T 24

---

Table Ablation studies regarding GG-Transformer design and comparison among different self-
attention mechanisms.
(b) Comparison among different self-attentions. Gaze (Conv) uses
(c) Applying GG-MSA DeiT backbone.
(a) Choices of Gaze Kernels.
kemels of F3x.3-(3,3.3.33.
Gaze Kernel
Top-1
Top-1
Top-1
Fixed- (3.3,33)
80.28%
W& SW-MSA 24
78.50%
DeiT-T GG-DeiT-T
72.2%
Fixed- (5,5,5()
80.31%
MSA
79.79%
73.8%
(d45.)ve-(9,53333)
80.38%
Glance Only
77.21%
DeiT-S
79.9%
Gaze Oaly
76.76%
GG-DeiT-S
80.5%
Glance+Gaze (Attn)
79.07%
Glance+Gaze (Conv)
80.28%
Transformer easier to scale-up, it sacrifices the accuracy and the combination of W&SW-MSA to mimic a global view is not as good as the original MSA. We replace the W&SW-MSA with MSA for all blocks in stage 3 and 4 (i.e., stages with down-sampling rate 16 and 32), which Ieads to a 1.29% performance improvement, indicating there exists a significant performance gap between MSA and its efficient alternative. Notably, when adopting the proposed Glance and Gaze mechanism instead, which shares a same complexity of W& SW-MSA, can achieves much better performance, where the Glance+Gaze (Attn) improves the performance by 0.57%, and Glance+Gaze (Conv) (t.e., GG-T) by 1.78%, which is even higher than MSA by 0.49%.
Besides using depthwise convolution, another natural choice is to also adopt self-attention for imple- menting the Gaze branch. Therefore, we conduct experiments by using local window attention 24 as the Gaze branch. Note that, unlike depthwise convolution, self-attention variant of the Gaze branch cannot be integrated with the Glance branch into the same Transformer block while keeping the overall model size and computation cost at the same level. To ensure fair comparison, we use two consecutive Transformer blocks where one is Glance attention and another is Gaze attention. Using either convolution self-attention to implement the Gaze branch can both improve the performance compared to 24),, illustrating the effectiveness of the Glance and Gaze designs. However, using self-attention is inferior to depth-wise convolution with degrade of 1.21% 1%, which may indicate that convolution is still better choice when it comes to learning local relationships. Besides, using depth-wise convolution as Gaze branch also naturally be integrated into the Transformer block with Glance attention, thus makes it more flexible terms of network designs.
We also note that Glance or Gaze branch alone is far from enough, while only combination of both can lead to a performance gain, which matches the behavior that we human beings can not rely on Glance or Gaze alone. For instance, using Glance alone can only lead to inferior performance with accuracy of 77.21 %6 and Gaze alone 76,, which is significantly lower than baseline with a degrade of 1.29% and 1.74%, respectively. Nevertheless,
note that this is because Glance and Gaze branches miss important local or global cues which can be compensated by each other. As a result, combination of both Glance and Gaze gives high accuracy of 80.28%, which improves the Glance alone and Gaze alone by 3.07% and 3.52% respectively.
Apply to other backbone. We verify the effectiveness of GG-Transformer another popular ViT architecture DeiT [32), as shown in Table 4c] We replace MSA with GG-MSA for two DeiT variants (2), DeiT-T and DeiT-S. We show that, although GG-MSA is an efficient alternative to MSA, it can also lead to a performance gain. Compared to DeiT-T and DeiT-S, GG-DeiT-T and GG-DeiT-S bring the performance up by 1.6% and 0.6% respectively, illustrating that it is not only efficient but also effectively even compared to a fully self-attention.
Network runtime. We follow [2 to report and compare work runtime measured by FPS: GG-T achieves 782.34 FPS compared 737.86 of Swin-T, and GG-S achieves 441.31 FPS compared to 423.51 of Swin-S. The evaluation is done with single Nvidia tesla v v100-sxm2-166g GPU.
5 Limitation
Although OG-Transformer provides a powerful and efficient solution to make Transformers scalable to large inputs, some limitations still exist and worth further exploring.

---

Firstly, over-fitting is common problem 12 in Vision Transformers and can be alleviated by large-scale pretraining [2] or strong augmentations and regularization [32]. This problem is more serious for stronger models (GG-Transformer) and in the tasks with relatively small dataset (e.g. semantic segmentation). Secondly, Transformers suffer from performance degradation in modeling longer-range dependencies, when there exists large discrepancy in the senningg-tsttngg image size. The limitations can come from position encoding, which has fixed size and need to be interpolated to different input sizes, self-attention itself, which may not adapt well when significant changes happen input size. Lastly, there is long-lasting debate on the impacts of Al on human world. As method improving the fundamental ability of deep learning, our work also advances the development of AI, which means there could be both beneficial and harmful influences depending on the users.
6 Conclusion
In this paper, we present GG- Transformer, which offers an efficient and effective solution to adapting Transformers for vision tasks. GG- Trannsssfrrmrr inspired by how human beings learn from the world, equipped with parallel and complementary Glance branch and Gaze branch, which offer long-range relationship and short-range modeling, respectively. The two branches can specialize in their tasks and collaborate with each other, which leads to a much more efficient VIT design for vision tasks. Experiments various architectures and benchmarks validate the advantages of GG- TG-Transformer.
Acknowledgment: This work was supported by ONR N00014-20-15 1-2000----2.,.--- ONR N00014-18-1 2119, NSFC 62176159 Natural Science Foundation of Shanghai 21ZR1432200 1432200, SJTU Explore X Foundation, and Shanghai Municipal Science and Technology Major Project 2021 SHZDZZX000
References
[1] Zhaowei Cai and Nuno Vasconcelos. Cascade r-cnn: Proceedingy of the IEEE conference computer vision and pattern recognition, pages 6154-6162, 2018.
Delving into high quality object detection. In
[2] Nicolas Carion, Francisco Massa, Gabriel Synnaeve. Nicolas Usumier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. European Conference on Computer Vision, pages 213-229. Springer, 2020.
[3] Jieneng Chen, Yongyi Lu, Qihang Yu, Xiangde Luo, Ehsan Adeli, Yan Wang. Le Lu, Alan L Yuille, and Yuyin Zhou. Transunet: Transformers make strong encoders for medical image segmentation. arXiv preprint XXXv:2200.04306, 2021.
[4] Kai Chen, Jiaqi Wang, Jiangmiao Pang, Yuhang Can, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, et al. Mmdetection: Open mmlab detection toolhox and henchmark. arXiv preprint arXiv: / 1906.07155 0550001555500 2019.
[5] Liang-Chieh Chen, George Papandreou, Florian Schrofff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. arXiv preprint urXiv:/706.05588 2017.
[6] François Chollet. Xception: Deep learning with depthwise separable convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1251-1258, 2017.
[7] Xiangxiang Chu. Zhi Tian, Bo Zhang. Xinlong Wang, Xiaolin Wei, Huaxia Xia, and Chunhua Shen. Conditional positional encodings for vision transformers. Arxiv preprimi 1082. 2021.
[8] MMSegmentation Contributors. MMSogmentation: Openmmlab semantic segmentation toolbox and benchmark. https:://githut com/
com/open-oeeeooo ope/oooennmmm mm1 ab/ htb////mmmmeentttionnn.. 2020.
[9] Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoe V Le. Randaugment: Practical automated data augmentation with reduced search space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 702-703, 2020.
[10] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. leee, 2009.
[11] Heiner Deubel and Werner X. Schneider. Saccade target selection and object recognition: Evidence for common attentional mechanism. Vision Research, 36 36(12) 1837, 1996.
10

---

[12] Alexey Dosovitskiy. Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai. Thomas Unterthiner. Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words:
Transformers for image recognition scale. arXiv preprint arXiv:2010.1199 2020.
[13] Haoqi Fan, Bo Xiong, Karttikeya Mangalam, Yanghao Li, Zhicheng Yan, Jitendra Malik, and Christoph Feichtenhofer. Multiscale vision transformers. arXiv preprint arXiv:2104. //227, 2021.
[14] Kai Han, An Xiao, Enhua Wu, Jianyuan Guo, Chunjing Xu, and Yunhe Wang. Transformer in transforme. arXiv preprint arXiv:2103.0011,, 2021.
[15] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference computer vision, pages 2961-2969, 2017.
[16] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pottern recognition, pages 770-778, 2016.
[17] Elad Hoffer, Tal Ben-Nun, Itay Hubara, Niv Giladi, Torsten Hoefler, and Daniel Soudry. Augment your batch: Improving generalization through instance repetition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8129-811 8138, 2020.
[18] Han Hu, Zheng Zhang, Zhenda Xic, and Stephen Lin. Local relation networks for image recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3464-3473, 2019.
[19] Gao Huang. Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Weinberger. Deep networks with stochastic depth. In European conference on computer vision, pages 646-661. Springer, 2016.
[20] Zilong Huang, Xinggang Wang, Lichao Huang, Chang Huang, Yunchao Wei, and Wenyu Liu. Cenet: Criss-cross attention for semantic segmentation. In Proceedings the IEEF/CVF International Conference on Computer Vision, pages 603-612, 2019.
[21] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 15:1097-1105, 2012.
[22] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied document recognition. Proceedings of the IEEE, 86(1 22:2228-3324, 1998.
[23] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Ziticck. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740-755. Springer, 2014.
[24] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. arXiv preprint arXiv:2103.14030. 2021.
[25] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.0510 2017.
[26] Boris T Polyak and Anatoli B Juditsky. Acceleration of stochastic approximation by averaging. SIAM journal on control and optimization, 30(4):838 8855, 1992.
[27] Ilija Radosavovic, Raj Prateek Kosaraju. Ross Girshick. Kaiming He, and Piotr Dollár. Designing network design spaces. In Proceedings the IEEE/CVF Conference on Computer Vision and Puttern Recognition. pages 10428-10436, 2020.
[28] Prajit Ramachandran, Niki Parmar, Ashish Vaswani, Irwan Bello, Anselm Levskaya, and Jonathon Shlens. Stand-alone self-attention in vision models. arXiv preprint arXiv: /006.:900000555 9099999000999.... 2019.
[29] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang Chich Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 45 4510-4520, 2018.
[30] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogni- tion. arXiv preprint arXiv: 404.1556., 2014.
[31] Mingxing Tan and Quoc L.. Efficientnet: Rethinking model scaling for convolutional neural networks. In International Conference on Machine Learning, pages 6105-6114, PMIRR 2019.
[32] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégon. Training data-efficient image transformers & distillation through attention. arXiv preprint arXiv:2012.12877.000 2020. 2020.
11

---

[33] Ashish Vaswani, Noam Shazeer. Niki Parmar, Jakob Uszkoreit, Llion Jones. Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. arXiv preprint urXiv://06.033722. 2017.
[34] Huiyu Wang. Yukun Zhu, Bradley Green, Hartwig Adam, Alan Yuille, and Liang-Chieh Chen. Axial- deeplab: Stand-alone axial axial-attention for panoptic segmentation. In European Conference Computer Vision, pages 106. Springer, 2020.
[35] Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformrr; versatile backbone for dense prediction without convolutions. arXiv preprint 1Xii:2102.122222 2021.
[36] Xiaolong Wang, Ross Girshick. Abhinav Gupta, and Kaiming He. Non-local neural networks. In Proceedingy of the IEEE conference on computer vision and pattern recognition, pages 7794-7803, 2018.
[37] Haiping Wu, Bin Xiao, Noel Codella, Mengchen Liu, Xiyang Dai, Lu Yuan, and Lei Zhang. Cvt: Introducing convolutions to vision transformers. arXiv preprint arXiv:2103. 2021.
[38] Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing scene understanding. In Proceedings of the European Conference on Computer Vision (ECCV), pages 418-434, 2018.
[39] Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transforma- tions for deep neural networks. In Proceedings of the IEEE conference computer vision and paltern recognition, pages 1492-1500, 2017.
[40] Fisher Yu and Vladlen Koltun. Multi-scale context aggregation by dilated convolutions. arXiv preprint arXiv:1511.0712, 2015.
[41] Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Francis EH Tay, Jiashi Feng, and Shuicheng Yan. fokens-to-token vit: Training vision transformers from scratch imagenet. arXiv preprint arXiv:2101.11986 2021.
[42] Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6023-6033 22232,03, 2019.
[43] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXiv:1710.0940 2017.
[44] Hengshuang Zhao, Jiaya Jia, and Vladlen Koltun. Exploring self-attention for image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10076- 10085, 2020.
[45] Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. In Proceedings of the IEEE conference on computer vision and patter recognition, pages 2881-2890, 2017.
[46] Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo. Yabiao Wang, Yanwei Fu, Jianfeng Feng, Tao Xiang. Philip HS Torr, al. Rethinking semantic segmentation from seuueecc-to-sqquene perspective with transformers. arXiv preprint arXiv:2012. 15840, 2020.
[47] Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang. Random crasing data augmentation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 13001- 2020.
[48] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler. Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 633-641, 2017.
12


Reviews:
[{"review_id": "qCekkxbOZQm", "reviewer": "Reviewer_xdt9", "summary": "The paper introduces a novel attention mechanism for the transformer architecture. Inspired by human gaze, authors propose to split the self-attention operation which is typically done across all patches in two parallel processes: one which reasons locally and another one that reasons globally. Authors evaluate their method in standard datasets for segmentation, classification and detection and ablate the main design decisions.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Summary: The paper introduces one single idea, describes it very well and show its effectiveness through evaluation in multiple tasks on standard benchmarks. For those reasons, I believe the paper should be accepted.\n\nStrengths:\n- I really like how the block splits the standard self-attention operation into local and global attention in a simple but effective way. I believe that as transformers are becoming a more relevant architecture over time, the community will need to move towards more efficient architectures to process images in transformers.\n\n- Authors evaluate their model in three tasks: detection, segmentation and classification. I believe this makes the paper very strong, as it shows that the proposed block is effective not only for the standard classification task (which is no affected by local changes in the image) but also works well for detection and segmentation.\n\n-   The results of the evaluation in the three task show that the performance of the proposed method improves previous proposal. Furthermore, authors report an extensive list of baselines which helps the reader put the result in perspective.\n\n- Authors ablate the most relevant design decisions of the paper. Specially, I like that the block also improves performance when used in other backbones Table 4.c.\n\n- The paper is very well written and explains very well the approach proposed. \n\nWeaknesses:\n\n- I am missing some qualitative evaluation on what each block is doing on the operation. I think this would be very interesting for the reader to gain a bit of understanding of the model. This could be archived by providing some qualitative results of the attention, or even examples where the proposed model improves the baselines. \n\n- In L58, the sentence in italic is a bit confusing and it appears to be the name of a paper (which is not). I would recommend the authors edit that part of the paper to make this clearer.\n\n- Have the authors considered processing the gaze part without a convolution and with a transformer-like approach? I would be curious to see if there a solution without involving convolution and how does that perform to the proposed method. \n\n- I think more discussion about the computational reduction would be also interesting and make the paper stronger. For instance, how would this approach scales to large-resolution images compared to traditional transformer approaches (ViT). \n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Authors discuss limitations and societal impact of their work. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3.5"}, "extra_text": {}}, {"review_id": "pRe_gIOiH_b", "reviewer": "Reviewer_E8bD", "summary": "This paper presents a glance and gaze module to replace standard self-attention with higher efficiency in terms of memory. The paper validated their methods in image classification, object detection, and semantic segmentation.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\n1. This paper achieved a better trade-off between FLOPs and accuracy than vanilla ViT.\n\n2. Runtime: Even though the proposed method has the same FLOPs as Swin; however, it seems that it needs more folding/unfolding operations which could slow down the network in runtime. \n\n\nQuestions\n\n1. See above point 2. What is the runtime speed compared to Swin? \n\n2. Why do the authors use stronger data augmentation in object detection? in PVT and Swin, for 1x schedule, they do not use multi-scale training. What is the performance if disabling multi-scale training?\n\n3. Why the ablation study does not use the same settings, do those observations still hold if using the same settings?\n\n4. Why the proposed model (Gaze + Glance) is 80.28% in Table 4b while 80.38% in Table 4a?\n\n5. In Table 4c, what is the performance of DeiT without repeated augmentation?\n\n6. Does the proposed method scale better? E.g. from Swin-S to Swin-B, the performance does not improve too much. How about the proposed method, GG-B?\n\n7. It seems that the proposed method achieved better improvement for the smaller model only (GG-T) while the gain over Swin-S is limited. I wonder what is the explanation.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"ethical_concerns": "No", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}, {"review_id": "j9jBT04uql", "reviewer": "Reviewer_y1o2", "summary": "This paper proposes a Glance and Gaze vision transformer to overcome the complexity problem common in vision transformers. The proposed transformer has two parallel branches - a glance branch that accounts for long-term dependencies and a gaze branch to account for local context. The authors compare their work on ImageNet-1k, COCO and ADE20K data sets for image classification, object recognition and semantic segmentation. Reference [24] (Swin Transformer) was the prior art on which the experimental set-up was taken from. Comparisons with a number of other vision transformer settings were made.", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The idea of Glance and Gaze has an intuitive appeal and this is appreciated by the reviewer. However, there are a number of issues with this paper:\n\n1) The paper has numerous grammatical errors and typos which made it hard to follow. \n2) The main issue with this paper is that it is unable to beat the accuracy of [24] which they have taken as baseline. For example, consider the last four rows of Table 1 which compares Swin with the proposed GG. The complexity is identical and accuracy similar. Also, results with ImageNet-22k are not shown.\n3) The Swin-T and Swin-S numbers listed in Table 2 for ADE20K task are lower than the ones reported in [24]. Why is this? The original paper [24] reports mIoU of 46.1% (Swin-T) and 49.3% (Swin-S) which are higher than those obtained by GG which further buttresses the comment in 2). FPS numbers are not reported. Not sure what is going on. \n4) Table 3 for object detection has the same issue as in Table 2 (see Comment 3). The numbers reported by [24] are higher that those cited and compared with. Also, since [24] is the main prior work for establishing a comparison framework, there needs to be comparison with other object detection frameworks such as ATSS, Rev Points V2...\n\nIt is important that the authors clarify the issue of numbers from prior art they are quoting in the paper (see comments 3 and 4) and expand the set of comparisons they are making to match up with reference [24]. The scope of the paper needs to cover at least the same data sets and metrics as in [24].\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "To some extent.", "review_text": "", "extra_scores": {"ethical_concerns": "There are no major ethical issues assuming there is a good explanation for the issue raised in comments 2 and 3.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1.5"}, "extra_text": {}}, {"review_id": "cvWu2iFG8K", "reviewer": "Reviewer_G6eL", "summary": "The paper presented a new Transformer-based model for visual recognition. The key idea is to use a checkerboard-like pattern (dilated splitting) to divide the 2D feature plan into multiple smaller partitions, where the standard self-attention can be efficiently computed on each of these partitions (glance branch). A local depthwise convolution was also added to link the partitions (gaze branch). To evaluate the proposed model, the authors presented extensive results across several visual recognition tasks, including image classification, semantic segmentation, and object detection. The results are promising.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Originality**\n\nThe key idea of the glance branch (dilated splitting) is well suited for 2D images and quite interesting. The combination of the glance / gaze branch is convincing.\n \nMy main concern is the missing link to previous works on efficient Transformer models in the ML / NLP community. Specifically, there has been quite some recent effort on developing linear time self-attention / Transformer blocks, e.g.,  Linformer [1], Longformer [2], Performer [3], Reformer [4], Nystromformer [5], Informer [6] and Big Bird [7]. As one of the main arguments of this paper is an efficient Transformer model, these previous works seem quite relevant and should be discussed and/or compared. My understanding is that the proposed method is tailored for 2D visual data, while previous approaches are more general and can also be applied to 2D data. Further, some of the ideas proposed in the paper can be found in these previous works.  For example, using sparse attention from different local pattern was discussed in [7] while using depthwise convolution in parallel to self-attention was presented in [5]. It thus remains unclear to me how the proposed method compares to these previous approaches.\n\n[1] Wang, S., Li, B. Z., Khabsa, M., Fang, H., & Ma, H. (2020). Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.04768.\n\n[2] Beltagy, I., Peters, M. E., & Cohan, A. (2020). Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150.\n\n[3] Choromanski, K., Likhosherstov, V., Dohan, D., Song, X., Gane, A., Sarlos, T., ... & Weller, A. (2020). Rethinking attention with performers. In ICLR.\n\n[4] Kitaev, N., Kaiser, Ł., & Levskaya, A. (2020). Reformer: The efficient transformer. In ICLR.\n\n[5] Xiong, Y., Zeng, Z., Chakraborty, R., Tan, M., Fung, G., Li, Y., & Singh, V. (2021). Nystr\\\" omformer: A Nystr\\\" om-Based Algorithm for Approximating Self-Attention. In AAAI.\n\n[6] Zhou, H., Zhang, S., Peng, J., Zhang, S., Li, J., Xiong, H., & Zhang, W. (2021, May). Informer: Beyond efficient transformer for long sequence time-series forecasting. In AAAI.\n\n[7] Zaheer, M., Guruganesh, G., Dubey, K. A., Ainslie, J., Alberti, C., Ontanon, S., ... & Ahmed, A. (2020, July). Big Bird: Transformers for Longer Sequences. In NeurIPS.\n\n\n**Quality**\n\nThe paper is technically sound. Experiments are thorough. And the results are solid.\n\nA main motivation of the model is the visual attention in the human visual system. Yet I don’t see a clear link between self-attention (computation of self similarity between features) and human visual attention (selection of relevant sensory information). I do think that the idea of dilated splitting stems from dilated kernels in 2D convolution. And that seems to be a better perspective. \n\n**Clarity**\n\nThe paper is reasonably well written. The key idea is nicely illustrated. \n\n**Significance**\n\nThe paper addressed the design of vision Transformers, a trendy topic in the field. The proposed model has the potential to serve as a common backbone for many vision tasks. The downside is that the proposed model seems to only have minor performance gains and similar computational costs when compared to Swin Transformer in [24], though I recognize that [24] is an arXiv paper and not yet published. \n\n**Post-rebuttal Update**\n\nThanks for the authors' response. I've read other reviews and responses. A main concern shared among reviewers is the comparison to Swin-Transformer [24]. The clarification of details and the additional results provided in the rebuttal have addressed my concern. Further, given that [24] appeared on arXiv around two months before the NeurIPS deadline, I don't think this paper should get penalized for not beating the results in [24]. That being said, I encourage the authors to clarify the experiment settings in the paper, and to improve the presentation quality. Overall, I think this is a paper with an interesting idea and good results and I am happy to increase my rating. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces an efficient transformer architecture with a “glance” branch to model long-range dependencies and a “gaze” branch to account for local context. Three reviewers recommend acceptance, highlighting that the idea is novel, the paper is well-written, and the results are solid. One reviewer, despite appreciating the intuitive appeal of the glance and gaze idea, recommends rejection primarily because of the discrepancy between the Swin baseline numbers being compared with and the actual numbers in the Swin paper referenced in the manuscript. However, the AC agrees with the other three reviewers and the authors based on their response that the comparison with Swin transformers is fair, as the same setting was used to compare both models. The paper is technically sound, has an interesting idea, and the results are promising. The authors should carefully proofread the paper and add the discussion in the rebuttal to the final version.
Decision: Accept (Poster)