Global Context Vision Transformers
Authors: Ali Hatamizadeh
Hongxu Yin
Jan Kautz
Pavlo Molchanov
Venue: Submitted to ICLR 2023 (2023)
Keywords: Vision Transformers
Classification
Detection
Instance Segmentation
Semantic Segmentation

Abstract: We propose global context vision transformer (GC ViT), a novel architecture that enhances parameter and compute utilization for computer vision tasks. The core of the novel model are  global context self-attention modules, joint with standard local self-attention, to effectively yet efficiently model both long and short-range spatial interactions, as an alternative to complex operations such as an attention masks or local windows shifting. While the local self-attention modules are responsible for modeling short-range information, the global query tokens are shared across all global self-attention modules to interact with local key and values. In addition, we address the lack of inductive bias in ViTs and improve the modeling of inter-channel dependencies by proposing a novel downsampler which leverages a parameter-efficient fused inverted residual block. The proposed GC ViT achieves new state-of-the-art performance across image classification, object detection and semantic segmentation tasks. On ImageNet-1K dataset for classification, the tiny, small and base variants of GC ViT with 28M, 51M and 90M parameters achieve 83.4%, 83.9% and 84.4% Top-1 accuracy, respectively, surpassing comparably-sized prior art such as CNN-based ConvNeXt and ViT-based Swin Transformer. Pre-trained GC ViT backbones in downstream tasks of object detection, instance segmentation, and semantic segmentation on MS COCO and ADE20K datasets outperform prior work consistently, sometimes by large margins.

Under review as a conference paper at ICLR 2023
GLOBAL CONTEXT VISION TRANSFORMERS
Anonymous authors
Paper under double-blind review
ABSTRACT
We propose global context vision transformer (GC ViT), novel architecture that enhances parameter and compute utilization for computer vision tasks. The core of the novel model are global context self-attention modules, joint with standard local self-attention, to effectively yet efficiently model both long and short-range spatial interactions, as an alternative to complex operations such as an attention masks or local windows shifting. While the local self-attention modules are responsible for modeling short-range information, the global query tokens are shared across all global self-attention modules to interact with local key and values. In addition, we address the lack of inductive bias in ViTs and improve the modeling of inter-channel dependencies by proposing novel downsampler which leverages a parameter-efin -efttrrefeffietn fused inverted residual block. The proposed GC ViT achieves new state-of-the-an performance across image classification, object detection and semantic segmentation tasks. On ImageNet- K dataset for classification, the tiny, small and base variants of GC ViT with 28M, 51M and 90M parameters achieve 83.4% %, 83.9% % and 84.4% Top-1 accuracy, respectively, surpassing comparably- sized prior art such as CNN-based ConvNeXt and ViT-based Swin Transformer. Pre-trained GC ViT backbones in downstream tasks of object detection, instance segmentation, and semantic segmentation on MS COCO and ADE20K datasets outperform prior work consistently, sometimes by large margins.
INTRODUCTION
During the recent years, Transformers (Vaswani et al., 2017) have achieved State- Oatt--Of-The-Ar (SOTA) performance in Natural Language Processing (NIP) benchmarks and became the de facto model for various tasks. A key element in the success of Transformers is the self-attention mechanism which allows for capturing contextual representations via attending to both distant and nearby tokens (Yin et al., 2021). 1). Following this trend, Vision Transformer (ViT) (Dosovitskiy et al., 2020) proposed to utilize image patches as tokens in monolithic architecture with minor differences comparing to encoder of the original Transformer. Despite the historic dominance of Convolutional Neural Network (CNN) in computer vision, ViT-based models have achieved SOTA competitive performance in various computer vision tasks. In essence, the self-attention mechanism in ViT allows for learning more uniform short and long-range information (Raghu et al., 2021) in comparison CNN. However, the monolithic architecture of ViT and quadratic computational complexity of self-attention baffle their swift application to high resolution images (Yang et al., 2021a) which capturing multi-scale long-range information is crucial for accurate representation modeling.
Several efforts (Liu al., 2021; Dong al., 2022; Chu et al., 2021a; Tu et al., 2022), most notably Swin Transformer (Liu et al., 2021), have attempted to address the balance between short- and long- range spatial dependencies by proposing multi-resolution architectures in which the computed in local windows. In this paradigm, cross-window connections such as window shifting are used for modeling the interactions across different regions. Despite the progress, the limited receptive field of local windows challenges the capability of self-attention and window-connection schemes such as shifting only cover a small neighborhood in the vicinity of each window. Subsequent efforts such Focal Transformer (Yang al., 2021b) attempted to address this issue by designing highly sophisticated self-attention modules with increased model complexity.
capture long-range information,
In this work, we introduce the Global Context (GC) ViT network address these limitations. Specifically, we propose a hicrarchical ViT architecture consisting of local and global self self-attention modules. At each stage, we compute global query tokens, using novel fused inverted residual blocks,

---

Under review as a conference paper at ICLR 2023
86
GC VTB
VIT-6
Foszl8
ConWeett
Focel
Com
Focal
ImageNer Top-1 Accuracy (50
6C VIT-T
Conceest
Swin-
ImageNet Top-1 Accuracy (%)
883
ConoNeis
Swind
Focal
Swm-
Swin-S
MTTX
Convhou-y
PVTL
Derr-2
GC VITATT
Convlestg
PVTL
PVT4
Swin-
.81
Swin
DelT.
80
GC VIT-XXT
DWiTs
00 VIT
PVT-8
79
Res/Nci152
Rashee-152
78
78
ReaNeet-121
77
77
RosNod.60
Resher-00
76
76
10 FLOPS (G)
15
25
65
85
Params (Millions)
Figure Top-1 accuracy vs. model FLOPs/parameter size on ImageNet-IK dataset. GC ViT achieves new SOTA benchmarks for different model sizes as well as FLOPs, outperforming competing approaches by significant margin. Best viewed in color.
Gas
Gat
Gel
Don
Coob USA
Architecture of the proposed GC VIT. At each stage, a query generator extracts global query tokens which captures long-range information by interacting with local key and value representations. We use alternating blocks of local and global context self attention layers. Best viewed in color.
Figure 2
which we refer to as modified Fused-MBCony blocks, that encompass global contextual information from different image regions. While the local self-attention modules are responsible for modeling short-range information, the global query tokens are shared across all global self-attention modules to interact with local key and value representations. The design of our proposed framework for global query generator and self-altention is intuitive and simple and can be efficiently implemented using major deep learning framework. Hence, it eliminates sophisticated and computationally expensive operations and ensures the effectiveness of self-attention when applied high-resolution images. In addition, we propose novel downsampling block with a parameter-efficien fused-MBConv layer to address the lack of inductive bias in ViTs and enhancing the modeling of inter-channel dependencies.
We have extensively validated the effectiveness of the proposed GC ViT using three publicly available datasets for various computer vision tasks. For image classification using ImageNet-IK IK dataset, GC ViT with 28M, 51M, 90M and 201M parameters, referred to as tiny, small, base and large variants, achieve new SOTA benchmarks of 83.4%, 83.9 %. 84.4% and 84.6% Top-1 accuracy. Hence, GC ViT consistently outperforms both Nen (Liu et al., 2022) and Swin Transformer (Liu et al., 2021) models by a significant margin (see Fig. 1). Using a pre-trained GC ViT base backbone with Cascade Mask RCNN (He et al., 2017) head, our model achieves box mAp of 52.9 for object detection and a mask mAP of 45.8 for instance segmentation on the MS COCO dataset. In addition, using an UPerNet (Xiao et al., 2018) head, our model achieves mloU of 49.0 ADE20K for semantic segmentation. Other variants of GC ViT with different learning capacities also demonstrate SOTA results when compared to similarly-sizee Hence, GC ViT demonstrates great scalability for high-resolution images on various downstream tasks, validating the effectiveness of the proposed framework in capturing both short and long-range information.
models on both MS COCO and ADE20K datasets.
The main contributions of our work are summarized as follows:
2

---

Under review as a conference paper at ICLR 2023
We introduce compute and parameter-optimizze hierarchical VIT with reparametrization of the design space (e.g., embedding dimension, number of heads, MLP ratio).
We design efficient CNN-like token generator that encodes spatial features at different resolutions for global query representations.
We propose global query tokens that can effectively capture contextual information in efficient manner and model both local and global interactions.
We introduce parameter-effiiiet downsampling module with modified Fused MB-Conv blocks that not only integrates inductive bias but also enables the modeling of inter-channel dependencies. We demonstrate new SOTA benchmarks for (1) ImageNet classification with Pareto fronts on ImageNet- K for model size and FLOPs (see Fig. 1), and (2) downstream tasks such as detection, instance segmentation and semantic segmentation on MS COCO and ADE20K, respectively.
GC VIT ARCHITECTURE
Architecture. Fig. 2 depicts the architecture of GC VIT. We propose a hierarchical framework obtain feature representations at several resolutions (called stages) by decreasing the spatial dimensions while expanding the embedding dimension, both by factors of 2. At first, given an input image with resolution of #xxx3 we obtain overlapping patches by applying 3 convolutional layer with stride of and appropriate padding. Then patches are projected into convolutional layer with stride Every
C-dimensional embedding space with another GC ViT stage is composed of alternating local and global self-attention modules to extract spatial features. Both operate in local windows like Swin Transformer (Liu al., 2021), however, the global self-attention has access to global features extracted by the global token generator. The token generator a CNN-like module that extracts features from the entire image only once at every stage. After each stage, the spatial resolution is decreased by 2 while the number of channels is increased by via a downsampling block. Resulting features are passed through average pooling and lincar layers to create embedding for downstream task.
The GC ViT architecture benefits from novel blocks such as a downsampling operator, a global query generator and global self- attention module described in the next sections.
Downsampler. We leverage an idea of spatial feature contraction from CNN models that imposes locality bias and cross channel interaction while reducing dimensions. We utilize a modified -used-MBCony block, followed by max pooling layer with kernel size of 3 and stride of as a downsampling operator, see Fig 3. The Fused-MBConv block in our work is similar the one in EfficientNetV2 (Tan & Le, 2021) with modifications as in
Conv 3x3, stnid 2
Fused MBConv
Layer Norm
x DW-Convaxa(X)
X GELU(&),
(1) Figure Downsampling block for dimension reduc- tion.
& = -SE(S),
Convi x (x) + x,
where SE, GELU and DW-Convax3 denote Squeeze and Excitation block (Hu al., 2018), Gaussian Error Linear Unit (Hendrycks Gimpel, 2016) and 3 depth-wise convolution, respectively. In our proposed architecture, the Fused- Mussed blocks provide desirable properties such as inductive bias and modeling of inter-channel dependencies. It is ablated in Table 5.
Attention. Multi-hese head self-attention is the the core computational operator in the GC ViT architecture, extracts semantic information from the image. GC ViT is composed of local and global self- attention modules illustrated in Fig 4. Similar to Swin Transformer (Liu et al., 2021), we benefit from splitting the image into windows and performing local self-attention within them, this leads to linear complexity scaling with image size. The local self-attention extracts local, short-range, information and in order to facilitate long range dependencies we propose to use a novel global self attention to allow cross- cross-patch communication with those far beyond the local window. Global self-attention attends other regions in the image via global query token that represents image embedding extracted with CNN-like module.
3

---

Under review as a conference paper at ICLR 2023
Local attention
Global Attention
Local attention with glocal query
Global query tokens
Generatorr
Qusry
Elobai
Local window aotentttoonn
Figure Attention formulation. Local attention is computed on feature patches within local window only (left). On the other hand, the global features extracted from the entire input features and then repeated to form global query tokens. The global query interacted with local key and value tokens, hence allowing to capture long-range information via cross-region interaction. Best viewed in color.
B h
Input
Extracted global
a w) x
H O
x w x C
Fused MBConv
Max pool 22
(h a no) x C
Stage sagge-wise
realaapo
repeat
x
X
X
20
global
(h
(
XK loga
times
dimension matching
Tokenized features
Spotially matched with local
Figure 5 Global query generator schematic diagram. is designed to (i) transform an input feature map to the current stage of dimension H, W, C' denoting height, width, and channel respectively, (ii) extract features via repeating the modified Fused MBConv block, joint with down wwwn-sampling, log, # times for dimension matching to local window size h (iii) output reshaped and repeated to (h number of local tokens that can attend global contextual information. denotes merged dimensions during reshaping.
2.1 GLOBAL QUERY GENERATOR
We propose to generate global query tokens that encompass information across the entire input feature maps for interaction with local key and value feature pairs. Specifically, as shown in Fig. 5, a layer in the the generator consists of Fused-MBBonv block followed by max pooling layer, similar to the one described in Sec. 2, and the final global query 9g.1 at stage i (i {1,2,3, 3, 4}) of GC ViT is computed according
F-M
(2)
MaxPool(x').
These query tokens computed once every stage of the model and shared across all global atten- tion blocks, hence decreasing number of parameters and FLOPs and improving the generalizability In addition, the global attention layers only learn local key and value features which will be used for interaction with global query tokens.
2.2 GLOBAL SELF-ATTENTION
Fig. 4 demonstrates the main idea behind our contribution. Local self-attention can only query patches within local window, whereas the global attention can query different image regions while still operating within the window. At each stage, the global query component is pre-computed as described in S.c.2.1. The global self-attention utilizes the extracted global query tokens, obtained according to Eq. 2 and shared across all blocks, to interact with the local key and value representations. In addition, GC ViT employs alternating local and global self-attention blocks to effectively capture
4

---

Under review as a conference paper at ICLR 2023
Iuil hoad
MLP
Cancal
Projecten
Loyer Noter
Concer
MLP
MLP
MLP
Figure 6 Local and global attention blocks. Global attention block does not compute query vector and reuses global query computed via Global Token Generation.
both local and global spatial information. Fig. 6 illustrates the difference between local and global self-attention. The global attention query 9g has size of B x C denote batch size, embedding dimension, local window height and width, respectively. Moreover, Og repeated along the batch dimension compensate for the overall number of windows and batch size B = B x N where N is the number of local windows. 9g is further reshaped into multiple heads. The value and key are computed within each local window using linear layer. The global self-attention query, key and value features are computed as follows
w, wherein B, C. h and w
Qg E RRT xxxxxxxxx. = =(q Axl. E RR BxCxhx
(3)
95 E CNNC reshapo
Q5 B' XCxhxe
(4)
kv X
(5)
Since the partitioned windows only contain local in- formation, interaction with rich contextual informa- tion embedded in the global query tokens provides effective way of enlarging the receptive field and attending to various regions in the input feature maps. The self-attention module computed as
Algorithm. 1 Global Attention Pseudocode in PyTorch Style
Input/cutpi shape: (B+, N, C) Be: Batchsize=Nis Windows; H: Height;
Midth; dim; g_g: Global Token
Attention( qx. k, v) Softmax(
4g
+ b)v,
(6)
F: Num Attention Head; N: Num Windows; def (nit())
Linear (C, 2+C)
where d is scaling factor and b is learnable rela- tive position bias term. Assuming position change between 1 p- along horizontal and vertical axes, b is sampled from the grid b E (222-1)x(2p-111 As shown in Sec. 5, relative position bias improves the performance, especially for dense prediction downstream tasks. In Algorithm 1, we present a PyTorch-like pseudocode for computing global self- attention in GC VIT. A complexity analysis of the global self-attention is presented the supplemen- tary materials.
softmax nn. n.. Softmax (dim=-1)
def forward ggg): Br, N, C x.shape h, g_g.
kv rex),reshape (B+, N, 2, F, C
//
kv kv. kv.permute C, 3, 1, 6)
k, split (kv, [2, 1), q_g. (1, Rt // B, 1, 1) g g_g.reshape (B+, F, N, /1
qk matmul (g_ trrasssste (-2,
att softmax (qk)
return matmul (attn, v) reshape (B*, N, C)
RELATED WORK
VIT. The ViT (Dosovitskiy et al., 2020) was first proposed as an alternative to CNNs with the advan- tage of enlarged receptive field, due to its self-attention layers. However, it lacked desirable properties of CNNs such as inductive biases and translation invariance and required large-scale training datasets to achieve competitive performance. Data-efficient Image Transformers (DeiT) Touvron et al. (2021b) introduced a distillation based training strategy which significantly improved the classification accu- racy. ViT (Graham et al., 2021) proposed hybrid model with re-designed multi- layer perceptron (MLP) and self-attention modules that are highly-ope -ogtyioop for fast inference. Cross-covanes covariacc Image Transformer (XCIT) (Ali et al., 2021) introduced transposed self-attention module for
5

---

Under review as a conference paper at ICLR 2023
modeling the interactions of feature channels. Convolutional vision Transformer (CvT) (Wu et al., 2021) introduced convolutional token embedding layer and Transformer block in a hierarchical architecture to improve the efficiency and accuracy of ViTs. Conditional Position encoding Vision Transformer (CPVT) (Chu et al., 2021b) demonstrated improved performance on various tasks such as image classification and object detection by conditioning the position encoding on localized patch token. Tokens- To-Tokor Vision Transformer (T2T- (Yuan et al., 2021) proposed a transformation layer for aggregating adjacent tokens and establishing image prior by exploiting spatial correlations. Pyramid Vision Transformer (PVT) (Wang et al., 2021) proposed a hierarchical architecture with patch embedding at the beginning of each stage and spatial dimension reduction to improve the com- putational efficiency. Independently, Swin Transformers (Liu et al., 2021) also proposed a hierarchical architecture in which self-attention is computed within local windows which are shifted for region interaction. Twins Transformer (Chu et al., 2021a) proposed a spatially separable self-attention with locally-grouped and global sub-sampling modules to improve the efficiency. Focal Transformer (Yang et al., 2021b) introduced the Focal self-attention to capture long-range spatial interactions. PVT- v2 (Wang et al., 2022) improved performance and efficiency comparing to PVT (Wang et al., 2021) by introducing overlapping patch embedding, convolutional ferd-forward network and linear attention. Edge VTT (Pan et al., 2022) introduced a lightweight ViT model, with global sparse attention and local aggregations and propagation modules for capturing short and long-range information
ConyNet. Since the advent of deep learning, CNNs (Krizhevsky et al., 2012; Simonyan & Zisserman, 2014; Howard et al., 2017; He et al., 2016; Szegedy et al., 2016: Huang et al., 2017: Hu al.. 2018) have dominated computer vision benchmarks with SOTA performance. Recently, inspired by ViTs, ConvMixer (Trockman & Kolter, 2022) introduced simple architecture with large-kernll depth-wise and point-wise convolutional layers and global pooling with competitive performance for classification. Furthermore, ConvNeXt (Liu et al., 2022) proposed modifications to the architecture of ResNet (He et al., 2016), and achieved competitive benchmarks for classification, detection and segmentation tasks.
4 EXPERIMENTS
Image classification. For image classification, we trained and tested our model on ImageNet- IK dataset (Deng et al., 2009). To allow for fair comparison, all GC VIT variants are trained by following training configurations of previous efforts (Liu et al., 2021; Yang et al., 2021b; Chu et al., 2021a). Specifically, all models are trained on nodes (32 A100 GPUs) with the AdamW (Loshchilov & Hutter, 2017) optimizer for 300 epochs with an initial learning rate of 0.001, weight decay of 0.05, cosine decay scheduler and 20 warm-up and cool-down epochs, respectively. We use total batch sizes of 4096 for GC ViT-XXT, GC ViT-XT and GC ViT-T models and 1028 for all other variants. See supplementary materials for more training details.
Object detection and semantic segmentation For object detection and instance segmentation, trained our model on MS COCO (Lin et al., 2014) with a Mask-RCNN (He et al., 2017) head, using x3 LR schedule with initial learning rate of 0.0001, a batch size of 16 and weight decay of 0.05. Following Liu et al. (2022), we compared against Tiny, Small and Base model variants using Cascade Mask-RCNN but only compared against Tiny variant using Mask-RCNN. For semantic segmentation, we used the ADE20K dataset (Zhou et al., 2017) with a UPerNet (Xiao et al., 2018) segmentation head. Following previous efforts, we used a random crop size of 512 512 for the input images. For fair assessment, we only compare against models with a pre-trained ImageNet- I backbone.
4.1 CLASSIFICATION
We present the ImageNet- IK classification benchmarks in Table and compare against CNN, ViT- based models across different model sizes. Our model achieves new SOTA benchmarks in all categories. Specifically, the proposed GC VIT surpasses similar-sized counterpart models by +0.5%
for GC ViT-XT 88.0%) compared to T2T-ViT-14 (Yuan et al., 2021) (8 11.5%) +0.7% for GC VIT-T (83.4 43) over CSwin-T (Dong et al., 2022) (82.7%), +0.3% for GC VIT-S (83.9%) over CSwin- S (Dong et al., 2022) (83.65 (3.6%)).oo +0.2%% +0.2% for GC ViT-B (84.4%) compared to CSwin-B (Dong et al., 2022) (84.2%) and +0.1% 1% for GC ViT-L (84.6%) over 3 (Dai et al., 2021) (84.5%), respectively. Furthermore, as shown in Fig. 1, GC VIT models have better or comparable computational efficiency in terms of number FL.Pss over the competing counterpart models.

---

Under review as a conference paper at ICLR 2023
Table Image classification benchmarks on ImageNet-IK dataset (Deng al., 2009).
Method
Param (M)
FLOPs (G)
Image Size
Top- (%)
ResMLP-SI (Touvron et al., 2021a)
15
224
76.6
PVT-v2-B1 (Wang et al., 2022)
13
2.1
224a
78.7
GC ViT-XXT
12
2.1
224
79.8
EdgeV (Pan et al., 2022)
11
1.9
224
81.0
DeiT-Small/16 (Touvron et al., 2021b)
224
79.9
T2T-ViT-14 (Yuan et al., 2021)
22
5.2
224
81.5
GC VIT-XT
20
2.6
224
82.0
ResNet50 al., 2016)
25
224
76.1
Swin-T (Liu al., 2021)
29
224
81.3
CoAtNet-0 (Dai et 2021)
25
4.2
224
81.6
PVT-v2-B2 (Wang et al., 2022)
25
4.0
224
82,0
ConvNeXi-T (Liu et 2022)
29
4.5
224
82.1
Focal-T (Yang et al., 2021b)
29
4.9
224
82.2
CSwin-T (Dong al., 2022)
23
4.3
224
82.7
GC VIT-T
28
224
83.4
ResNet-101 (He et al, 2016)
44
7.9
224
77.4
Swin-S (Liu al., 2021)
50
8.7
224
83.0
ConvNeXt-S (Liuu
2022)
50
224
83.1
CoANNet-1 (Dai al., 2021)
224
83.3
Focal-S (Yang et al., 2021b)
51
9.1
224
83.5
CSwin-S (Dong et al., 2022)
35
6.9
224
83.6
GC VTT-S
51
8.5
224
83.9
ResNet-152 (He al., 2016)
60
11.6
224
78.3
ViT-Base/16 (Dosovitskiy et al,, 2020)
86
17.6
224
77.9
DeiT-Base/16 (Touvron et al., 2021b)
86
17.6
224
81.8
Swin-B (Liu et al. 2021)
88
15.4
224
83.3
CoAtNet-2 (Dai al., 2021)
42
8.4
224
83.3
ConvNeXt-B (Liu al., 2022)
89
15.4
224
83.8
Focal-B (Yang et al., 2021b)
90
224
83.8
PVT-v2-B5 (Wang et al., 2022)
82
224
83.8
CSwin-B (Dong et al., 2022)
78
15.0
224
84.2
BoTNet (Dong et al., 2022)
79
19.3
256
84.2
GC VIT-B
90
14.8
224
84.4
ConvNeXt-L (Liu et al., 2022)
198
34.4
224
84.3
CoAINel-3 (Dai et al., 2021)
168
34.7
224
84.5
GC VIT-L
201
32.6
224
84.6
Table 2 Object detection and instance segmentation benchmarks using Mask R-CNN and Cascade Mask R-CNN MS COCO dataset (Lin al., 2014). All models employ 3x schedule.
Backhone
Param (M)
F1.PPs (G)
Aphm
APho
AP
Aprsl
AP
A
Mask-RCNN schedule
Swin (Liu et al., 2021)
48
267
46.0
68.1
50.3
41.6
65.1
44.9
CoX-- (Liu al., 2022)
GC VIT-
291
47.9
70.1
52.8
43.2
67.0
46.7
Mask schedulee
DeiT 6 (Touvron et al., 202
80
889
48.0
67.2
51.7
41.4
64.2
44.3
ResNet-50
2016)
64.3
Swin-T T (Liu et al., 2021)
86
745
50.4
69.2
$4.7
43.7
66.6
47.3
Con--NXXtT (Liu 2022)
69.1
66.5
GC VTTT
770
51.6
70.4
$6.1
44.6
67.8
48.3
X101-32
2017)
48.1
Swin-S (Liu al., 2021)
107
838
51.9
70.7
56.3
45.0
68.2
48.8
Con-NNXXtS (Lin 2022)
108
827
51.9
70.8
36.5
45.0
68.4
GC VIT-S
108
866
$2.4
71.0
$7.1
45.4
68.5
49.3
X101-64 (Xic 2017)
48.3
66.4
$2.3
41.7
64.0
Swin-R B (Lin al., 2021)
145
982
51.9
70.5
56.4
45.0
68.
48.9
B (Liu N
al, 2022)
146
964
52.7
71.3
57.2
45.6
68.9
49.5
GC VIT-B
146
1018
52.9
71.7
57.8
45.8
69.2
49.8

---

Under review as a conference paper at ICLR 2023
4.2 DETECTION
In Table we present object detection and instance segmentation benchmarks on MS COCO dataset. Using a Mask-RCNN head, the model with pre-trained GC VIT- outperform outperforms counter-
Backhome
(M) (G)
DeiT- 1eaT-SmalV16 1 (Touvron al., 202(b)
52
1099
44.0
Swin (Liu et al., 2021)
60
945
44.5
ResNet- 101 (He al., 2016)
1029
44.9
Twins-SVT-S (Chu ct.,, 2021a)
55
46.2
T (47. 9/4.2) backbone parts with pre-trained Co--NeXI-T (Liu et al., 2022) (46. (46.2/41.7) by +1.7 and +1.5 and Swin- T (Liu et al., 2021) (46.0/41.6) by +1.9 and +1.6 terms of box AP and mask AP, re- spectively. Using Cascade Mask-RCNN head, the models with pre-trained GC VIT-T (51.6/44.6) and GC ViT-S (52.4/45.4) back- bones outperform ConvNeXt-T (Liu et al., 2022) (50.4/43.7) by +1.2 and +0.9 and ConvNeXt-S (Liu et al., 2022) (51.9/45.0 by +0.5 and +0.4 in terms of box AP and mask AP, respectively. Furthermore, the model with GC ViT-B (52.9/45.8 5.) backbone outperforms the counterpart with ConvNeXt-B (Liu et al., 2022) (52.7/45.6) by +0.2 and +0.2 in terms of box AP and mask AP, respectively.
GC VIET
58
47.0
Swin-S (Liu et el., 2021)
81
1038
47.6
Twins SVT-B (Chu et al., 2021a)
89
47.7
GC VITS
84
1163
48.3
Swin (Liu al., 2021) 2022)
121
1188
48.1
Twins-SVT.I. (Chu al., 2021a)
GC VYT-B
125
1348
49.0
Table 3 Semantic segmentation benchmarks ADE20K validation set with UPerNet (Xiao al., 2018) and pre- trained ImageNet- IK backbone. All models use crop size of 512 x 512.
4.3 ADE20K SEMANTIC SEGMENTATION RESULTS
We present semantic segmentation benchmarks on ADE20K dataset in Table 3. The models using pre- trained GC ViT-T (47.0), GC ViT-S (48.3) and GC ViT-B (49.0) backbones outperform counterpart models with pre-trained Twins-SVT-S (Chu et al., 2021a) (46.2). Twins-SVT-B (Chu et al., 2021a) (47.7) and Twins-SVT-L (Chu et al., 2021a) (48.8) by +0.8, +0.6 and +0.2 in terms of mloU, respectively. In addition, models with GC VIT backbones significantly outperform counterparts with Swin Transformer backbones, hence demonstrating the effectiveness of the global self-attention.
5 ABLATION
5.1 COMPONENT-IISE ANALYSIS
As shown in Table 4, we study the role of each component in GC ViT model for classification, detection, instance and seman- tic segmentation. For simplic- ity, we start with Swin Trans- former the base model and progressively re-design the com- ponents to demonstrate their im- portance in improving the perfor- mance. Firstly, we remove the window shifting and predictably observe significant performance degradation across all tasks. Changing distribution of parameters to our design improves the performance by +1.7, +2.8, +2.2 and +1.7 in terms of accuracy, box AP, mask AP and mloU. Such reparametrization includes changing the window size, MLP ratio, number of layers to name but few. Adding the CNN-based stem of GC VIT to the model provides additional improvements +0.3, +0.2, +0.2 and +0.2 terms of accuracy, box AP, mask AP and mloU. In addition, using proposed downsampler further improves the accuracy, box AP, mask AP and +0.1 mloU by +0.4, +0.1 and +0.3, respectively. The last two changes demonstrate the importance of convolutional inductive bias and capturing the chter- dependencies in our model. Finally, leveraging the proposed global self-attention improves the performance by by +0.8, +0.8, +0.6 and +1.2 in terms of accuracy, box AP, mask AP and mloU. Hence, this validates the effectiveness of the
ImageNet top-1
COCO Aphic
ADE20K
Aprrok
mloU
Swin-T
81.3
50.4
43.7
44.5
Swin-T w/o Window Shifting
80.2
47.7
41.5
43.3
+ Reparam. (window, #blocks, ratio)
81.9
50.5
43.7
45.0
+ GC VIT-T Stem
82.2
50.7
43.9
45.2
+ GC VIT-T Down-sampler
82.6
44.0
+ GC VIT-T Global Self-attention
83,4
44.6
Table 4 Ablation study on the effectiveness of various components in GC VIT classification, detection and segmentation performance.
8

---

Under review as a conference paper at ICLR 2023
(b) Original inages from ImageNet- validation sel
Global atfention maps from GC VIT moxel (ours).
Corresponding Grad-CAM maps.
Figure 7 Visualization of (a) input images (h) global self-attention maps from GC VIT-T model (c) corresponding Grad-CAM attention maps. Both short and long-range spatial dependencies captured effectively. Please see the supplementary materials for illustration learned global query token feature maps.
proposed global self-attention, in particular for downstream tasks with high resolution images such semantic segmentation in which modeling long-range spatial dependencies is critical.
5.2 DOWNSAMPLER DESIGN
We studied the effectiveness of various downsampler blocks in Table 5. The simplest alternative to our design is a pair of convolutional and maxpooling layers. However, it results in a reduction of ImageNet Top-1 accuracy by -0.7. Patch merging is another variant which was introduced in Swin Transformers (Liu et al., 2021).
However, it reduces the accuracy by -0.5. Finally, our down-sampler which consists of modified MauedMMMCoovv block and strided convolution and shows the best re- sult. Importance of the former component explained by the SE operation which boosts cross channel interaction while keeping number of parameters and FLOPs low. We conclude that our proposed down-sampler is essential to achieve high accuracy as it introduces convolutional inductive bias.
Down-sampler
Architecture
Top-1
Conv
Conv (s=1), Maxpool
82.7
Swin
Linear
82.9
GC VIT
Modified Fused-MBConv (s=2)
83.4
Table Ablation study on the effectiveness of down- sampler in GC VIT architecture ImageNet Top-1 accuracy.
INTERPRETABILITY
To provide further insights on interpretability the proposed global self-attention and query tokens, we demonstrate visualization of the learned attention and Grad-CAM (Selvaraju et al., 2017) maps in Fig. The associated attention distributions uncovered by the global self-attention modules align with image semantics, and hence act an informative source for local attention modules. In addition, corresponding Grad-CAM maps demonstrate accurate object localization with most intricate details.
7 CONCLUSION
We introduced new vision transformer architecture named GC ViT that can efficiently capture global context by utilizing global query tokens and interact with local regions. Through extensive experiments show SOTA benchmarks for image classification ImageNet- 1K dataset, surpassing CNN/ViT-based counterparts by large margin. We also consistently achieved SOTA for downstream tasks of detection, instance and semantic segmentation MS COCO and ADE20K datasets.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Alaaeldin Ali, Hugo Touvron, Mathilde Caron, Piotr Bojanowski, Matthijs Douze, Armand Joulin, Ivan Laptev, Natalia Neverova, Gabriel Synnaeve, Jakob Verheek, et al. Xcit: Cross covorccovv image transformers. Advances in neural information processing systems, 34, 2021.
Chun-Fu Chen, Quanfu Fan, and Rameswar Panda. Crossvit:
Cross-attention multi-scale vision
transformer for image classification, 2021.
Kai Chen, Jiaqi Wang. Jiangmiao Pang. Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng. Ziwei Liu, Jiarui Xu, et al. Mmdetection: Open mmlab detection toolbox and benchmark. arXiv preprint arXiv: 091099900075555 2019.
Xiangxiang Chu, Zhi Tian, Yuqing Wang, Bo Zhang, Haibing Ren, Xiaolin Wei, Huaxia Xia, and Chunhua Shen. Twins: Revisiting the design of spatial attention in vision transformers. Advances in Neural Information Processing Systems, 34, 2021a.
Xiangxiang Chu, Zhi Tian, Bo Zhang, Xinlong Wang, Xiaolin Wei, Huaxia Xia, and Chunhua Shen. Conditional positional encodings for vision transformers. arXiv preprint arXiv:2102.10882 2021b.
MMSegmentation Contributors. MMSegmentation: Openmmlab semantic segmentation toolbox and benchmark. https: / //t::/giiit thub. com/ /pomm//pp--eeee mml
hamlab//e ab/ www.mmsggeentttiioomm 2020.
Zihang Dai, Hanxiao Liu, Quoc V Le, and Mingxing Tan. Coatnet: Marrying convolution and attention for all data sizes. Advances in Neural Information Processing Systems, 34:3965-3977, 2021.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp.
leee, 2009.
Xiaoyi Dong, Jianmin Bao, Dongdong Chen, Weiming Zhang, Nenghai Yu, Lu Yuan, Dong Chen, and Baining Guo. Cswin transformer: general vision transformer backbone with cross-shaped win- dows. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 12124-12134, 2022.
Alexey Dosovitskiy. Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image worth 16x16 words: Transformers for image recognition at scale. International Conference on Learning Representations, 2020.
Benjamin Graham, Alaaeldin El-Nouby, Hugo Touvron, Pierre Stock, Armand Joulin, Hervé Jégou, and Matthijs Douze. Levit: vision transformer convnet's clothing for faster inference. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 12259-12269, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-enn. In Proceedings of the IEEE international conference computer vision, pp. 2961-2969, 2017.
Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08411 2016.
Andrew G. Howard, Menglong Zhu, and Bo Chen. Mobilenets: Efficient convolutional neural networks for mobile vision applications. 2017.
Jie Hu, Li Shen, and Gang Sun. Squee7e-and-excitate networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7132-7141, 2018.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Weinberger. Densely connected convolutional networks. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4700-4708. 2017.
10

---

Under review as a conference paper at ICLR 2023
Alex Krizhevsky, llya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolu- tional neural networks. In Advances in neural information processing systems, pp. 1097-1105, 2012.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft COCO: Common objects in context. In ECCV, 2014.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows, In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10012-10022, 2021.
Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. convnet for the 2020s. arXiv preprint arXiv:2201.03545, 2022.
Ily Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint 17 2017.
Junting Pan, Adrian Bulat, Fuwen Tan, Xiatian Zhu, Lukasz Dudziak, Hongsheng Li, Georgios Tzimiropoulos, and Brais Martinez. Edgevits: Competing light-weight cnns on mobile devices with vision transformers. In European Conference Computer Vision, pp. 294-311. Springer, 2022.
Maithra Raghu, Thomas Unterthiner, Simon Korbliih, Chiyuan Zhang, and Alexey Dosovitskiy. Do vision transformers see like convolutional neural networks? Advances in Neural Information Processing Systems, 34, 2021.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based local- ization. In Proceedings of the IEEE international conference on computer vision, pp. 618-626, 2017.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for targe-scale image recognition. CoRR, abs/1409.1556. 1556, 2014. URL http: / arriv.. org/ abs///////// / 4 4494.0. 1556.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zhigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2818-2826, 2016.
Mingxing Tan and Quoe Le. Efficientnetv2: Smaller models and faster training. In International Conference on Machine Learning, pp. 10096-10106. PMLR, 2021.
Hugo Touvron, Piotr Bojanowski, Mathilde Caron, Matthieu Cord, Alaaeldin El-Nouby, Edouard Grave, Gautier Izacard, Armand Joulin, Gabriel Synnaeve, Jakob Verbeek, et al. Resmlp: Feedforward networks for image classification with data-efficient training. arXiv preprint 1oXiv:2/05.03404 2021a.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International Conference on Machine Learning, pp. 10347-10357. PMLR, 2021b.
Asher Trockman and J Zico Kolter, Patches are all you need? arXiv preprint arXiv:2201.09792. 2022.
Zhengzhong Tu, Hossein Talebi, Han Zhang, Feng Yang, Peyman Milanfar, Alan Bovik, and Yinxiao Li. Maxvit: Multi-axis vision transformer. arXiv preprint arXiv:2204.01699, 2022.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Wenhai Wang, Enze Xie, Xiang Li, Deng- Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer. A versatile backbone for dense prediction without convolutions. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 568-578, 2021.
11

---

Under review as a conference paper at ICLR 2023
Wenhai Wang. Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pvt v2: Improved baselines with pyramid vision transformer. Computational Visual Media, 8(3):415-43) 2022.
Ross Wightman. Pytorch image models. pytorch- image- 2019.
https://gitiie
honm/wiightman
Haiping Wu, Bin Xiao, Noel Codella, Mengchen Liu, Xiyang Dai, Lu Yuan, and Lei Zhang. Cvt: Introducing convolutions to vision transformers. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), pp. 22-31, October 2021.
Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understanding. In Proceedings of the European Conference on Computer Vision (ECCV), pp.
2018.
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1492-1500, 2017.
Huanrui Yang, Hongxu Yin, Pavio Molchanov, Hai Li, and Jan Kautz. NVTT: Vision transformer compression and parameter redistribution. arXiv preprint arXiv:2110.0488 2021a.
Jianwei Yang, Chunyuan Li, Pengchuan Zhang, Xiyang Dai, Bin Xiao, Lu Yuan, and Jianfeng Gao. Focal attention for long-range interactions in vision transformers. Advances in Neural Information Processing Systems, 34, 2021b.
Hongxu Yin, Arash Vahdat, Jose Alvarez, Arun Mallya, Jan Kautz, and Pavlo Molchanov. A-ViT: Adaptive tokens for efficient vision transformer. arXiv preprint arXiv:2112.07655 2021.
Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Zihang Jiang, Francis EH Tay, Jiashi Feng, and Shuicheng Yan. Tokens-to-token ViT: Training vision transformers from scratch on imagenet. In ICCV, 2021.
Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang. Li Yang, et al. Big bird: Transformers for longer sequences. Advances in Neural Information Processing Systems, 33: 33:17783-17297, 2020.
Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 633-641, 2017.
12

---

Under review as a conference paper at ICLR 2023
A APPENDIX
B GC VIT MODEL CONFIGURATIONS
GC ViT model configurations are presented in Table S.1 describing the choice of internal hyper parameters to obtain models with various compute load and parameter number.
Output Size (Downs. Rate)
VIT-XT
VIT-T
GC ViT-B
Conv, C:64, S:2, LNN
Conv, C:64, S:2, LN
Conv, C:96, S:2, L.N
Conv, C:128, S:2, I.N
x28x 128 (2x)
Stem
F-MBConV
F-MBConV
F-MBConY
F-MBCony
C:64
C:64
C:128
Conv, C:T28, S:2,
C:128, S.2.
CTT92 $22, LN
Conv, C:256,
56x56 (4x)
Stage 1
LG-SA,
LG-SA. C:64, head:2
LG-SA, C:96, head:3
LG-SA.
C:64, head::
3.
C:128, head:4
F-MRCony: C:128
F-MBConw. C:128
F-MBComw C:192
F-MBConN C:256
Conv, C:256, S:2. LN
Conv, C:256, S.2, LN
Conv, C.384, S:2. LN
Conv, C:512, S.2, LN
Stage 2
(8x)
L.GSA. head:4
LG--A. C:64, head:4
T.G-SA, C:96, head:6
TGG-A. C:128, head:8
F-MBConv, C:256
F-MBConv, C:256
F-MBComw, C:384
F-MBConv, C15
Conv, C:512, S22 LN
Coav. C:512. S.2. LN
Conv, C3768. $22 LN
Conv, C.1024. S22 LN
Stage
(16x)
C:64, head:8
LG-SA,
LG-SA, 19.
head:8
LG-SA, 19,
LG-SA, C:128, head:16
19.
F.MBCony, C:512
F-MBConw, C:512
F-MBComw, C:768
F-MBCony, C11024
Conv, C:1024, S:2, LN
Cony, C:1024, S:2, LN
Cony, C:1536, S:2, LN
Cony, C:204, 8:2, LN
7x7
Stage
(32x)
C:64, head:16
LGGSA,
5.
C:64, head:16
LG-SA,
5.
C:96, head:24
LG-SA,
5.
LG-SA, C:128, head:32 x
F-MBConv, C1024
F-MBCony, 1,22
T-MBConw, C31536
F-MBCony, C:2048
Table S.1 Architecture configurations for GC VIT. LG-SA and Conv denotes local, global self-attention and x convolutional layer, respectively. GC ViT-XT, GC VIT-T, GC VIT-S GC VIT-B denote XTiny, Tiny, Small and Base variants, respectively.
C ABLATION
C.1 GLOBAL QUERY
We performed ablation studies to validate the effectiveness of the proposed global query. Using the same architecture, instead of global query, we compute: (1) global key and value features and interact them with local query (2) global value features and interact with local query and key. As shown in Table S.2, replacing global query may significantly impact the performance for image segmentation and downstream tasks such as object detection, instance segmentation and semantic segmentation.
ImageNet
COCO
ADE20k
top-1
Aph
Apmaak
mIoU
Global KV
82.5
49.9
41.3
44.6
w. Global V
82.7
50.8
42.4
45.1
GC VIT-T
83.4
51.6
44.6
47.0
Table S.2 Ablation study on the effectiveness the proposed global query for classification, detection and segmentation.
C.2 EMA AND BATCH SIZE
We also used used Exponential Moving Averages (EMA) and observed slight improvement in terms of ImageNet TOp-1 accuracy. Furthermore, the performance of the model across different batch sizes were stable as we did not observe significant changes. Table S.3 demonstrates the effect of EMA and batch size the accuracy of GCVIT-T model.
13

---

Under review as a conference paper at ICLR 2023
Model
Local Batch Size Global Batch Size EMA
Top-1
GC VIIT
32
1024
No 83.37
GC VIT-T
128
No 83.38
GC VII-
32
1024
83.39
GC VIIT
128
4096
Yes 83.40
Table S.3 Ablation study on the effect of EMA and batch size on GC VIT-T ImageNet Top-1 accuracy.
(a) Original images from ImageNet- IK validation set.
(b) Learned global query tokens.
Figure S.1 Visualization of (a) input images (b) learned global query token feature maps.
D INTERPRETABILITY
In Fig. S.1, we illustrate the learned global query token maps and demonstrate their effectiveness in capturing long-range contextual representations from different image regions.
E TRAINING DETAILS
For image classification, GC ViT models were trained using four computational nodes with 32 NVIDIA A100 GPUs. The total training batch size is 1024 (32 GPU) for GC ViT-S, GC ViT-B, GC VIT-L and 4096 (128 per GPU) for GC VIT-XXT, GC VIT-XT and GC ViT-T. On average, each model required 32 hours of training with the specified hyper-parameters indicated in the paper. All classification models were trained using the t timm package (Wightman, 2019). Object detection and instance segmentation models well semantic segmentation models were trained using one computational node with NVIDIA A40 GPUs using a total batch size of 16, hence batch size of per GPU. Detection and instance segmentation models et al., 2019) package and on average required 56 hours of training. Semantic segmentation models were trained using mnsegment ition (Contributors, 2020) package, and on average required 34 hours of training.
trained using mindetecti on (Chen
F COMPLEXITY ANALYSIS
Given an input feature map of RxxWxC each stage with window size of h w, the computational complexity of GC ViT is as follows
O(GC viT) 2HW + hwC),
(7)
The efficient design of global query token generator and other components allows to maintain similar computational complexity in comparison to Swin Transformer Liu et al. (2021) while being able to capture long-range information and achieve better higher accuracy for classification and downstream tasks such as detection and segmentation.
G COMPARISON TO OTHER GLOBAL SELF-ATTENTIO MODULES
Other efforts such Edge ViT (Pan al., 2022) in computer vision and BigBird (Zaheer et al., 2020) NLP have proposed global self self-attention their respective applications. In this section, we discuss the differences between the proposed global self-attention in GC VIT and these efforts.
14

---

Under review as a conference paper at ICLR 2023
Model
Accuracy-Matches Matched Frequency Accuracy- Threshod--.7
GC VIT-XXT
69.3
77.2
GC VIT-XT
71.3
78.8
GC VIT-T
80.5
GC VIT-S
73.8
80.7
GC ViT-B
74.4
81.1
GC VIT-L
74.5
81.5
Table S.4 Classiication benchmarks of GC ViT models ImageNetV2 dataset.
Model
Added Component Top-1
Swin-T
None
81.3
Swin-T
GC Module
82.2
Swin-S
None
83.0
Swin-S
GC Module
83.7
Table S.5 - Ablation study on the effectiveness of Global Context (GC) module in Swin Transformers architecture ImageNet Top-1 accuracy.
ViT Edge and GC ViT completely different self-attention blocks. The Edge VIT uses series of local aggregation (convolution), sparse attention and local propagation convolution), whereas GC ViT only uses an interleaved pattern of local and global self celf-attention layers without convolution in order to compute self-attention. The proposed global sparse attention in Edge ViT and GCVIT are competently different. Edge samples representative tokens and only computes sparse self-attention between these representative tokens with reduced feature size. On the contrary, GC ViT computes self-attention between the global queries (not just the token) and local keys and values without any subsampling in their respective local regions. Furthermore, in Edge only subsampled representative tokens per region interact In the self-attention module; however, in GC VIT, the global queries interact with the entire local regions, instead of interacting with each other, and hence provide an effective mechanism for capturing both short and long-range spatial dependencies.
In addition, GC ViT generates global query tokens by using a series of modified Fused MB-Conv from the entire image and without subsampling. Note that the resolution of global query tokens the same as local query and values. However, in ViTe (A) the representative tokens are obtained per local window, not the entire image, and by subsampling and reducing the feature resolution. Hence, since generated tokens have a lower resolution compared to their respective local windows, this could result in loss of spatial information and impact the effectiveness of self-attention. Unlike Edge VIT, the downsampler in GCVIT also benefits from modified Fused-MBComt blocks which allows for modeling cross channel interactions and impose more locality and convolutional inductive bias.
BigBird Bigbird, which is primarily introduced for NLP applications with ID inputs, has significant differences compared GC ViT, which is proposed for computer vision with mainly 2D inputs. Firstly, BigBird uses a combination which is different from the proposed local and global self-attention scheme in GC ViT. In addition. BigBird does not have any specific mechanisms for extracting global tokens as the existing tokens or additional special tokens can be specified as global tokens. On the contrary, the global tokens in GC ViT are extracted by the proposed global query generator module which consists of a series of modified Fused MB-Conv blocks to extract contextual information from the entire input features. Lastly, BigBird employs set of global tokens which attend to the entire input sequence; in this case, select global query, key and values attend to local query, key and value tensors. However, as opposed to this formulation. in GC VIT. the global query tokens attend to local key and value tokens in partitioned windows. This is due to the fact that attending to the entire input sequence, as done in BigBird, is not feasible considering the larger size of input features in computer vision.
random, window and global attention mechanisms,
15

---

Under review as a conference paper at ICLR 2023
H IMAGENETV2 BENCHMARKS
In Table S.4, we have evaluated the performance of GC VIT on ImageNetV2 dataset (?) to further measure its robustness. Specifically, have used different sampling strategies of Matched Frequency and Threshold-0.7. These benchmarks demonstrate the competetive performance of GC ViT on ImageNetV2 dataset and validates its effectiveness in robustness and generalizability
EFFECT OF GLOBAL CONTEXT MODULE
In order demonstrate the effectiveness of Global Context (GC) module, use Swin Transformers as the base model and add our propoped GC module. In this analysis, remove the window shifting operation from Swin Transformers, since GC module is capable of modeling cross-region interactions. As shown in Table S.5, addition of GC module improves the ImageNet Top-1 accuracy by +0.9% and +0.77 7% for Swin Transformers Tiny and Small variants respectively.
IMAGENET CLASSIFICATION BENCHMARKS
In Table S.6, we provide a comprehensive benchmark terms of Top-1 accuracy for the models that are only trained on ImageNet- IK (Deng et al., 2009) dataset, and without additional data.
16

---

Under review as a conference paper at ICLR 2023
Table S.6 Image classification benchmarks on ImageNet-IK dataset (Deng al., 2009).
Method
Param (M)
FLOPs (G)
Image Size
Top-1 (%)
ResMLP-S12 (Touvron et al., 2021a)
15
3.0
224
76.6
PVT-v2-B1 (Wang et al., 2022)
13
2.1
224
78.7
GC ViT-XXT
12
2.1
224
79.8
EdgeViTss (Pan et al., 2022)
11
1.9
224
81.0
DeiT-Smaal/166 (Touvron al., 2021b)
22
4.6
224
79.9
T2T-ViT-14 (Yuan et al., 2021)
22
5.2
224
81.5
GC VIT-XT
20
2.6
224
82.0
ResNet50 (He et 2016)
25
4.1
224
76.1
PVT-Small (Wang et al., 2021)
24
3.8
224
79.8
Swin-T (Liu et al., 2021)
29
4.5
224
81.3
CoAtNet-O (Dai et al., 2021)
25
4.2
224
81.6
Twins-SVT-S (Chu et al., 2021a)
24
2.9
224
81.7
PVT-v2-B2 (Wang et al., 2022)
25
4.0
2242
82.0
ConvNeXi-T (Liu et al., 2022)
29
4.5
224
82.1
Focal-T (Yang al., 2021b)
29
4.9
224
82.2
CSwin-T (Dong et al., 2022)
23
4.3
2242
82.7
GC ViT-T
28
4.7
224
83.4
ResNet-101 (He et 2016)
44
7.9
224
77.4
ResMLP-S24 (Touvron et al., 2021a)
30
6.0
2242
79.3
PVT-Medium (Wang et al., 2021)
44
6.7
224
81.2
T2T-ViT-19 (Yuan et al., 2021)
39
8.9
224
81.9
Sains-PCPVT-BBBB (Chu et al., 2021a)
44
6.7
2242
82.7
Swin-S (Liu et al., 2021)
50
8.7
2242
83.0
Twins-SVT-B (Chu et al., 2021a)
56
8.6
224
83.2
ConvNeXI-S (Liu et al., 2022)
50
8.7
224
83.1
PVT-v2-B3 (Wang et al., 2022)
45
6.9
224
83.2
CoAtNet-1 (Dai et al., 2021)
42
8.4
2242
83.3
Focal-S (Yang et al., 2021b)
51
9.1
224
83.5
CSwin-S (Dong et al., 2022)
35
6.9
224
83.6
GC VIT-S
51
8.5
224
83.9
ResNet-152 (He et al., 2016)
60
11.6
224
78.3
ViT-Base/16 (Dosovitskiy et al., 2020)
86
17.6
224
77.9
ResMLP-B24 (Touvron et al., 2021a)
116
23.0
224
81.0
PVT-Large (Wang et al., 2021)
61
9.8
224
81.7
DeiT-Base/16 (Touvron al., 2021b)
86
17.6
224
81.8
Cross VTT-B (Chen et 2021)
104
21.2
2242
82.2
T2T-ViT-24 (Yuan et al., 2021)
64
14.1
2242
82.3
CPVT-B (Chu et al., 2021b)
88
17.6
224
82.3
Twins-PCPVT-LL (Chu al., 2021a)
61
9.8
2242
83.1
Swin-B (Liu et al., 2021)
88
15.4
224
83.3
CoAtNet-2 (Dai et al., 2021)
42
8.4
224
83.3
PVT-v2-B4 (Wang et al., 2022)
62
10.1
2242
83.6
Twins-SVT-L (Chu et al., 2021a)
99
15.1
224
83.7
ConvNeXt-B (Liu et al., 2022)
89
15.4
224
83,8
Focal-B (Yang et al., 2021b)
90
16.0
2242
83.8
PVT-v2-B5 (Wang et al., 2022)
82
11.8
2242
83.8
CSwin-B (Dong et al., 2022)
78
15.0
224
84.2
BoTNet (Dong et al., 2022)
79
19.3
2562
84.2
GC ViT-B
90
14.8
2242
84.4
ConvNeXi-L (Liu et al., 2022)
198
34.4
2242
84.3
CoAtNet-3 (Dai et al., 2021)
168
34.7
2242
84.5
GC VIT-L
201
32.6
224"
84.6
17


Reviews:
[{"review_id": "fRHrsYK3D_", "reviewer": "Reviewer_V9pP", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a novel global context vision transformer built with a simple combination of global context self-attention modules and local self-attention modules to model\nboth long-range and short-range interactions. Besides, the authors propose to model the inter-channel dependencies with an inverted residual block.\nThe proposed GC ViT shows encouraging results across image classification, object detection, instance segmentation, and semantic segmentation tasks.\n", "summary_of_the_review": "Designing new versatile architecture is always very important for the whole computer vision community. The overall quality is OK but the results are relatively WEAK to support the claim that the proposed method really outperforms the previous SoTA methods such as Swin-Transformer.\nPlease carefully address the above-listed weaknesses. I will increase the ratings if the authors could well address these concerns.", "strength_and_weaknesses": "> Strengths\n\n✅ The presented idea is very simple and easy to follow.\n\n✅ The overall writing is of high quality, such as Figure 2, Figure 3, Figure 4, Figure 5, and Figure 6 is all of GOOD quality and help the readers to understand the key\nidea easily.\n\n✅ The authors provide rich details and conduct rich experiments to verify the effectiveness of the proposed approach.\n\n> Weaknesses\n\n❎ The experimental results are relatively WEAK and the authors are encouraged to verify the advantages over Swin-T based on the very recent object detection and segmentation frameworks\nsuch as DINO-DETR[1], H-DETR[2], and Mask2Former[3]. For example, Mask2Former+Swin-B (pre-trained on ImageNet1K) achieves PQ=55.1, AP=46.7, and mIoU=52.4 on COCO panoptic segmentation, COCO instance\nsegmentation, and ADE20K semantic segmentation tasks respectively. I would like to expect the authors to present stronger results and release the related code upon acceptance.\n\n[1] https://github.com/IDEA-Research/DINO\n\n[2] https://github.com/HDETR/H-Deformable-DETR \n\n[3] https://github.com/facebookresearch/Mask2Former \n\n❎ According to Table 1 & Table 2, the proposed GC Vit backbones require more parameters and GFLOPs than the previous approaches such as CSWin and ConvNeXt. For example, GC ViT-B requires 1018 GFLOPs vs. ConvNeXt-B requires 964 GFLOPs and GC ViT-B only gains 0.2 measured by AP^box.\n\n❎ The authors are encouraged to verify the advantages of the proposed backbone under larger model scales. Specifically, the authors should construct a GC ViT-L and GC ViT-H instead of only comparing the models under moderate scales as scaling up is one of the most important aspects of an important backbone design.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "RD-aVJwq0HU", "reviewer": "Reviewer_t1Sn", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposed a new hirerchical transformers architecture. The paper revisit the attention mechanism and the downsampling layers.\nThe claim of the paper are the following: \n\n• We introduce a compute and parameter-optimized hierarchical ViT with reparametrization of the\ndesign space (e.g., embedding dimension, number of heads, MLP ratio).\n\n• We design an efficient CNN-like token generator that encodes spatial features at different resolutions\nfor global query representations.\n\n• We propose global query tokens that can effectively capture contextual information in an efficient\nmanner and model both local and global interactions.\n\n• We introduce a parameter-efficient downsampling module with modified Fused MB-Conv blocks\nthat not only integrates inductive bias but also enables the modeling of inter-channel dependencies.\n\n• We demonstrate new SOTA benchmarks for : (1) ImageNet classification with Pareto fronts on\nImageNet-1K for model size and FLOPs (see Fig. 1), and (2) downstream tasks such as detection,\ninstance segmentation and semantic segmentation on MS COCO and ADE20K, respectively.\n", "summary_of_the_review": "The idea of the paper is interesting but unfortunately the contribution does not seem to be justified experimentally. The paper seems to be incremental. There is a lack of discussion and comparison with very similar approaches in the literature.\n\n\n====== Post rebuttal ====\n\nThe rebuttal raised a very important concern: \n\nThe authors report maximum accuracy during training by evaluating at each epoch in order to maximise performance on the ImageNet-1k validation set (there is no separate test set for ImageNet). Although the authors argue that it is a common practice, there is no evidence in the mentioned papers that this is the approach used. Furthermore I consider this approach to be detrimental to the community as it puts scientific understanding behind the optimal performance.\n\nOn the 3 logs provided the impact of this practice is important on the log with the bigger model +0.2% and impact the conclusion. The authors no longer have the other logs of their experiments, especially those of the ablation, so it is impossible to know the impact of the method. There is no multi-seed evaluation that shows that 0.1% on ImageNet-v2 is significant although it is claimed in the rebuttal.\n\nIn the current state the experiments are mainly designed to optimise the performance of the GC ViT architecture at the expense of an understanding of the impact of each component of the proposed method.\n\nSo I will lower my score.", "strength_and_weaknesses": "Strength\n- The paper is well written and easy to follow\n- There is evaluation on different tasks\n\nWeaknesses:\n- Missing comparaison and discussion:\n\na) The global query tokens claim: The proposed global query token behaviour seem to be quite similar to the attention pattern mechanism in the BigBird paper[1] and also the local global attention in the paper EdgeViT[2]. It is necessary to discuss the contribution of the proposed method to these two papers. Because at present the contribution of the paper on this aspect is not clear.\n\n[1] Zaheer et al., Big Bird: Transformers for Longer Sequences\n[2] Pan et al., EdgeViTs: Competing Light-weight CNNs on Mobile Devices with Vision Transformers\n\nb) The compute and parameter-optimized hierarchical ViT and SOTA claim: in Figure 1 and Table 1: Indeed, the EdgeViT paper exploit similar ideas (local-global attention, Hierachical architecture) and seem to have better FLOPs-accuracy and Parameter accuracy trade-off (EdgeViT-S: 11.1M params, 1.9G FLOPs, 81.0 top-1 on ImageNet vs GC ViT-XXT 12M params, 2.1G FLOPs, 79.8 top-1 on ImageNet)  It is therefore necessary to add this architecture to the comparison. Moreover, the paper mention EfficientNetV2[1] because GC ViT exploit some similar blocks and seem also better/similar for the FLOPs-accuracy and parameter accuracy trade-off (EfficientNetV2-M 54M params, 24G flops, 85.1 top-1 on ImageNet  vs GC ViT-L  201M params, 32.6 G FLOPs, 84.6 top-1 ImageNet). So it's important to add also EfficinetNet-v2 in the comparaison.\n\nGiven these elements it is probably best to down tone the state-of-the art claims in the paper. Because it is not clear which state of the art it is.\n\n[1] Tan et al., EfficientNetV2: Smaller Models and Faster Training\n\nb) The efficient CNN-like token generator: Many papers have proposed convolution blocks for token generation like LeViT[1] or XCiT[2].\nThere is no discussion or comparison of performance with the previous approach. It is therefore impossible to evaluate the contribution of the paper on this point.\n\n[1] Graham et al., LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inference\n[2] El-Nouby et al., XCiT: Cross-Covariance Image Transformers\n\n- Overfitting evaluation: For the ImageNet dataset there is no separate test and validation set so it's important to evaluate the level of overfitting by doing evaluation of the models on ImageNet-v2[1].\n\n[1] Recht et al., are we done with ImageNet?\n\n- Missing metrics: There is only params and FLOPs in the different table. It's important to measure other trade-off like Latency and memory consumption. Indeed, the proposed architecture use MBconv which are known to be good on trade-off FLOPs accuracy and parameter accuracy.\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "QixU8ZHmRkj", "reviewer": "Reviewer_Fuf9", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduced GC ViT that can efficiently capture global context by utilizing global query tokens and interact with local regions. \nThis paper demonstrate through experiments that this simple modification on traditional ViTs could get SOTA results for image classification on ImageNet-1K dataset and downstream tasks.", "summary_of_the_review": "Overall this paper proposed a simple and yet effective module that could add reasonable improvement to current ViT. This work could be useful to the community once the code is released for verification/comparison.  ", "strength_and_weaknesses": "Strength:\n1. the idea is simple and straightforward, from the experiment it seems the improvement is reasonable.\n\nWeakness:\n1. The comparison of GC-VIT to other models clearly shows the quantitative improvements in Tab.1. In addition, the ablation study shows the proposed module could be quite plug-able into existing models. It will be more appealing to see how GC module improves existing models by simply plug them into them. This will inevitably add #param  and FLOPS, but it gives more intuitive evidence on the effectiveness. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "1OtJVS6_8I", "reviewer": "Reviewer_YBSt", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors propose a novel module to combine the local and global context. The key of this method is to introduce global query tokens into the local context module. The advantage of this method is that it can merge the context information from short and long-term ranges. The authors do experiments for image classification, detection, and segmentation tasks to show the effectiveness of the method.", "summary_of_the_review": "Please follow the previous parts.", "strength_and_weaknesses": "Strength\n1. The experiments show this method is effective and can be generalized to many different tasks.\n2. The problem is essential for network architecture design and the method is straightforward.\n3. The visualization results show the effectiveness of this method.\n\nWeakness\n1. The combination of local and global context is a long-standing problem and has been explored by many methods. The method for merging local and global contexts lacks novelty.\n2. The improvement between the proposed method and other SOTA methods is minor.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject