ResT: An Efficient Transformer for Visual Recognition
Authors: Qinglong Zhang
Yu-Bin Yang
Venue: NeurIPS 2021 Poster (2021)
Keywords: multi-scale vision Transformer
computation efficient
talking head
spatial attention

Abstract: This paper presents an efficient multi-scale vision Transformer, called ResT, that capably served as a general-purpose backbone for image recognition. Unlike existing Transformer methods, which employ standard Transformer blocks to tackle raw images with a fixed resolution, our ResT have several advantages: (1) A memory-efficient multi-head self-attention is built, which compresses the memory by a simple depth-wise convolution, and projects the interaction across the attention-heads dimension while keeping the diversity ability of multi-heads; (2) Positional encoding is constructed as spatial attention, which is more flexible and can tackle with input images of arbitrary size without interpolation or fine-tune; (3) Instead of the straightforward tokenization at the beginning of each stage, we design the patch embedding as a stack of overlapping convolution operation with stride on the token map. We comprehensively validate ResT on image classification and downstream tasks. Experimental results show that the proposed ResT can outperform the recently state-of-the-art backbones by a large margin, demonstrating the potential of ResT as strong backbones. The code and models will be made publicly available at https://github.com/wofmanaf/ResT.

ResT: An Efficient Transformer for Visual Recognition
Qing-Long Zhang, Yu-Bin Yang
State Key Laboratory for Novel Software Technology Nanjing University, Nanjing 21023, China
wofmana @@ nju. cdu yangyubinân ju edu. cn
Abstract
This paper presents efficient multi-scale vision Transformer, called ResT, that capably served as a general-purpose backbone for image recognition. Unlike existing Transformer methods, which employ standard Transformer blocks tackle raw images with fixed resolution, our ResT have several advantages: (I) A memory-efficien multi-head self-attention is built, which compresses the memory by a simple depth-wise convolution, and projects the interaction across the attention-heads dimension while keeping the diversity ability of multi-heads; (2) Positional encoding is constructed as spatial attention, which is more flexible and can tackle with input images of arbitrary size without interpolation or fine-tune; (3) Instead of the straightforward tokenization at the beginning of each stage, we design the patch embedding as stack of overlapping convolution operation with stride on the token map. We comprehensively validate ResTT on image classification and downstream tasks. Experimental results show that the proposed ResT can outperform the recently state-of-the-art backbones by large margin, demonstrating the potential of ResT as strong backbones. The code and models will be made publicly available at http://ghhuub.oo///wooaaaaa
Introduction
Deep learning backbone architectures have been evolved for years and boost the performance of computer vision tasks such as classification (5 26 33
object detection 18 25 and
instance segmentation [10 24. 31. etc.
There mainly two types of backbone architectures most commonly applied in computer vision:
convolutional network (CNN) architectures [I] 38) and Transformer ones [6 5 of them capture feature information by stacking multiple blocks. The CNN block is generally bottleneck structure 11, which can be defined as a stack of x 1, 3 x 3, and x convolution layers with residual learning (shown in Figure (h) The 1 layers responsible for reducing and then increasing channel dimensions, leaving the 3 x 3 layer bottleneck with smaller input/output channel dimensions. The CNN backbones are generally faster and require less inference time thanks to parameter sharing, local information aggregation, and dimension reduction. However, due to the limited and fixed receptive field, CNN blocks may be less effective scenarios that require modeling long-range dependencies. For example, in instance segmentation, being able to collect and associate scene information from a large neighborhood can be useful in learning relationships across objects (23]
B3 39 Both
To overcome these limitations, Transformer backbones recently explored for their ability to capture long-distance information 5 33 26
19. . Unlike CNN backbones, the Transformer ones first split an image into a sequence of patches (i.e., tokens), then sum these tokens with positional encoding to represent coarse spatial information, and finally adopt a stack of Transformer blocks to capture feature information. A standard Transformer block 28 comprises multi-head self-attention (MSA) that employs query-key-value decomposition to model global relationships between sequence
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Couv-1
Linear
Linear
Linear
Linear
Conv-3
Conv-1
MSA
EMSA
(a) ResNet Bottlenock
(b) Standard Transformer
Efficient Transformer
A standard ResNet Bottleneck Block [T. Middle: A Standard Transformer Block. Right: The proposed Efficient Transformer Block. The only difference compared with standard Transformer block is the replacement of the Multi-Head Self-Attention (MSA) with Efficient Multi-head Self-Attention (EMSA).
Figure 1: Examples backbone blocks. Left:
tokens, and feed-forward network (FFN) to learn wider representations (shown in Figure Lp) As result, Transformer blocks can dynamically adapt the receptive field according to the image content.
Despite showing great potential than CNNs, the Transformer backbones still have four major short- comings: (1) It difficult extract the low-level features which form some fundamental structures in images (e.g.. corners and edges) since existing Transformer backbones direct perform tokenization of patches from input images. (2) The memory and computation for MSA in Transformer blocks scale quadratically with spatial or embedding dimensions (i.e., the number of channels), causing vast overheads for training and inference. (3) Each head in MSA responsible for only subset of embedding dimensions, which may impair the performance of the network, particularly when the tokens embedding dimension (for each head) short, making the dot product of query and key unable to constitute informative function.
(4) The input tokens and positional encoding in existing Transformer backbones are all of a fixed scale, which are unsuitable for vision tasks that require dense prediction.
In this paper, we proposed an efficient general-purpose backbone ResT (named after ResNet IT) for computer vision, which can remedy the above issues. As illustrated in Figure ResT shares exactly the same pipeline of ResNet, i.e., a stem module applied for extracting low-level information and strengthening locality, followed by four stages to construct hierarchical feature maps, and finally head module for classification. Each stage consists of patch embedding, a positional encoding module, and multiple Transformer blocks with specific spatial resolution and channel dimension. The patch embedding module creates a multi-scale pyramid of features by hierarchically expanding the channel capacity while reducing the spatial resolution with overlapping convolution operations. Unlike the conventional methods which can only tackle images with fixed scale, our positional encoding module is constructed spatial attention which is conditioned on the local neighborhood of the input token. By doing this, the proposed method is more flexible and can process input images of arbitrary size without interpolation or fine-tune. Besides, to improve the efficiency of the MSA, we build an efficient multi-head self-attention (EMSA), which significantly reduce the computation cost by simple overlapping Depth-wiss Conv2d. In addition, compensate short-length limitations of the input token for each head by projecting the interaction across the attention- heads dimension while keeping the diversity ability of multi-heads.
We comprehensively validate the effectiveness of the proposed ResT on the commonly used bench- marks, including image classification on ImageNet- 1k and downstream tasks, such as object detection, and instance segmentation on MS COCO2017. Experimental results demonstrate the effectiveness and generalization ability of the proposed ResT compared with the recently state-of-the-art Vision Transformers and CNNs. For example, with a similar model size as ResNet-18 (69.7%) and PVT-Tiny (75.1%), our ResT-Small obtains Top-1 accuracy of 79.6% ImageNet-Ik.

---

P y x
x x
M x
BC
Staeel
Stage2
Suge3
Sugeel
It W x
Povition emhed
Patch & Position
stemn
Efficions Tmussormer Block
Efficient Tenosfomee Block
Efficient Tormeformer Block
Image
Trossorrcc Block
Punch &
Punch
x/4
Figure 2: The pipeline of the proposed ResT. Similar to ResNet [T],, ResT build stages with stacked blocks, making it flexible to as the backbone of downstream tasks, such Object detection, Person RelD, and Instance Segmentation, etc.
2 ResT
As illustrated Figure 2 ResT shares exactly the same pipeline as ResNet [[I, i.c., stem module applied to extract low-level information, followed by four stages to capture multi-scale feature maps. Each stage consists of three components, one patch embedding module (or stem module), one positional encoding module, and set of L efficient Transformer blocks. Specifically, the beginning of each stage, the patch embedding module adopted to reduce the resolution of the input token and expanding the channel dimension. The positional encoding module is fused restrain position information and strengthen the feature extracting ability of patch embedding. After that, the input token is fed to the efficient Transformer blocks (illustrated in Figure [b). In the following sections, we will introduce the intuition behind ResT.
2.1 Rethinking of Transformer Block
The standard Transformer block consists of two sub-layers of MSA and FFN. A residual connection employed around each sub-layer. Before MSA and FFN, layer normalization (LN [I] is applied. For token input R=xdm, where dm indicates the spatial dimension, channel dimension, respectively. The output for each Transformer block is:
y= x + FFN(LN(x')), and x' x MSA(LN(x))
MSA. MSA first obtains query Q, key K. and value V by applying three sets of projections to the input, each consisting of linear layers (i.e., heads) that map the dm dimensional input into dg dimensional space, where de dm /k is the head dimension. For the convenience of description, we assume 1, then MSA can be simplified to single-head self-attention (SA). The global relationship between the token sequence can be defined as
SA(Q, K, V) Softmax( QKT
V
The output values of each head are then concatenated and linearly projected to form the final output. The computation costs of MSA are O + 4d,,,), which scale quadratically with spatial dimension or embedding dimensions according the input token.
FFN. The FFN applied feature transformation and non-linearity. consists of two linear layers with non-linearity activation. The first layer expands the embedding dimensions of the input from dm to dy and the second layer reduce the dimensions from dy to dmm-
FFN(x) = (xW, + bb) + +b
where W, Id xdy and W Rdy xdm are weights the two Linear layers respectively. b, e Rd and b2 E Rdm are the bias terms, and (.)) is the activation function GELU [12). In standard Transformer block, the channel dimensions are expanded by a factor of 4, i.e., d, 4dm. The computation costs of FFN are Snd
2.2 Efficient Transformer Block
As analyzed above, MSA has two shorteomings: (1) The computation scales quadratically with d,. or according to the input token, causing vast overheads for training and inference; (2) Each head in
3

---

MSA only responsible for a subset of embedding dimensions, which may impair the performance of the network, particularly when the tokens embedding dimension (for each head) is short.
Add
YI X dyn
Lincur
X dsc
MatMul
k x de
k x n m
Conv Softmax IN
h n nxn""
kx x de nd x
MatMul Scale
x n de
Q
x de x
n da
Linear
Lincar
Linear
ol x dm
x dm
dm x H x w/
DWConv + LN
dm h
X: T X dm
Figure 3: Efficient Multi-Head Self-Attention.
To remedy these issues, propose efficient multi-head self-attention module (illustrated in Figure 3. Here, we make some explanations.
(1) Similar to MSA, EMSA first adopt set of projections to obtain query Q.
(2) To compress memory, the 2D input token Raxdm.. is reshaped to 3D along the spatial dimension (i.e., & E R xhxw) and then feed to a depth-wise convolution operation to reduce the height and width dimension by a factor To make simple, is adaptive set by the feature map size or the stage number. The kernel size, stride and padding are 8 + 1, 8, and 8/2 respectively.
(3) The new token map after spatial reduction R.om xp/mxh/ax//+ is then reshaped 2D one, i.e., X E R xdm n' = h/s x w/s. Then feed to two sets of projection to get key K and value V.
(4) After that, we adopt Eq. 4 to compute the attention function on query Q. K and value V.
EMSA(Q, K, V) IN(Sofmaax(onnv OK ))
Here, Conv(-) standard 1 x convolutional operation, which model the interactions among different heads. As result, attention function of each head can depend on all of the keys and queries. However, this will impair the ability of MSA to jointly attend to information from different representation subsets at different positions. To restore this diversity ability, we add an Instance Normalization 22 (i.e, IN(-) for the dot product matrix (after Softmax).
(5) Finally, the output values of each head are then concatenated and linearly projected to form the final output.
The computation costs of EMSA are O ( 2dmnn + 22 n1 + ( + dmn (a+1)) APp2) ), much lower
g than the original MSA (assume > 1), particularly lower stages, where is tend higher.
Also, we add FFN after EMSA for feature transformation and non-lincarity. The output for each efficient Transformer block is:
y x' FFN(LN(>))), x'= EMSA(LN(x))
(5)

---

2.3 Patch Embedding
The standard Transformer receives a sequence of token embeddings as input. Take ViT as an example, the input image x E I is split with a patch size of p x p. These patches are flattened into 2D ones and then mapped to latent embeddings with size of c, i.e x E R where n hw /p". However, this straightforward tokenization is failed to capture low-level feature information (such as edges and corners) [3]. addition, the length of tokens VIT are all of a fixed size in different blocks, making it unsuitable for downstream vision tasks such as object detection and instance segmentation that require multi-scale feature map representations.
Here, we build an efficient multi-scale backbone, calling ResT, for dense prediction. As introduced above, the efficient Transformer block in each stage operates on the same scale with identical resolution across the channel and spatial dimensions. Therefore, the patch embedding modules are required to progressively expand the channel dimension, while simultaneuusly reducing the spatial resolution throughout the network.
Similar to ResNet, the stem module (can be to shrunk both the height and width dimension with a reduction factor of 4. To effectively capture the low-feature information with few parameters, here introduce simple but effective way, i.e, stacking three 3 x 3 standard convolution layers (all with padding 1) with stride 2, stride I, and stride respectively. Batch Normalization (14] and ReLU activation 7 are applied for the first two layers. In stage 2. stage 3, and stage 4, the patch embedding module is adopted to down-sample the spatial dimension by 4x and increase the channel dimension by 2x. This can be done by a standard 3 x convolution with stride 2 and padding 1. For example, patch embedding module in stage 2 changes resolution from h/4 w/4 x to h/8 w/8 2c (shown in Figure
the first patch emhedding module) adopted
2.4 Positional Encoding
Positional encodings are crucial to exploiting the order of sequence. In ViT [5), a set of learnable parameters are added into the input tokens to encode positions. Let x E gaxe be the input, 0 E gaxe be position parameters, then the encoded input can be represent as
x 0
(6)
However, the length of positions is exactly the same the input tokens length, which limits the application scenarios.
To remedy this issue, the new positional encodings are required to have variable lengths according to input tokens. Let us look closer to Eq. 6 the summation operation is much like assigning pixel-wise weights to the input. Assume i.e. 0 Gx,(x), where GT.(.) is the group linear operation with the group number of c. Then 6 be modified to
Efficient
I,x
Transformer
Block
related with x,
Embedded Position
x= GL(x)
Spatial Attention
Embeeded
Besides Eq. 7 O can also be obtained by more flexible spatial attention mechanisms. Here, we propose a simple yet effective spatial attention module calling *A(pixel-attentin positions. Specifically, PA applies 3 x depth-wise convolu- tion (with padding 1) operation to get the pixel-wise weight and then scaled by sigmoid function (.). The positional encoding with PA module can then be represented
Patches
to encode
Figure 4: Patch and PE ResT.
X = PA(x)
(DWConv(x))
8
Since the input token in each stage is also obtained by a convolution operation, we can embed the positional encoding into the patch embedding module. The whole structure of stage can be illustrated in Figure 4 Note that PA can be replaced by any spatial attention modules, making the positional encoding flexible in ResT.
2.5 Classification Head
The classification head is performed by global average pooling layer on the output feature map of the last stage, followed by lincar classifier. The detailed ResT architecture for ImageNet- lk is shown
5

---

in Table which contains four models, i.e., ResT-Lite, ResT-Small and ResT-Base and ResT-Large which are
to ResNet-18, ResNet-18 18, ResNet-50, and ResNet-101, respectively.
Table I: Architectures for ImageNet-Ik. Here, we make some definitions. "Conv - c_c_s means convolution layers with kernel size k. output channel and stride 8. "MLP_c c" the FFN structure with hidden channel 4c and output channel And "EMSA n__r" is the EMSA operation with the number of heads n and reduction r. "C" is 64 for ResT-Lite and ResT-Small, and 96 for ResT-Base and ResT--Large "PA" is short for pixel-wise attention, which are introduced in Section 2.4
"PA"
Name
Output
Lite
Small
Base
Large
stem
56 x 56
patch patch_embed: Conv-3 2onv-3_C/2_2 C/2_2, Conv-3 ______2_1,,________11___1__, Conv-3 C_2,PA
EMSA_ 1_8
EMSA 1_8 MLP
EMSA 1_8 MLP __96
EMSA 1_8 MLP 96
stagel
x6
x2
x2
x2
x2
MLP_G 64
patch eriecd_emddd::- Conv-3_2C.1 PA
stage2
28
EMSA 2_4 MLP_ 128
EMSA 2 MLP_ _2_
EMSA 2 4 MLP_ 192
EMSA
x2
x2
MLP 192
patch_embed: Conv-3_4C22 PA
stage3
x4 14
EMSA 4 MLP 256
EMSA 4
EMSA 4 MLP _L4_
EMSA 4 2 MLP 384
x2
x6
x6
x18
MLP MLP_256
patch embed: Conv-3 8C 2, PA
stage4
7x7
EMSA_ 8_1 MLP_51
EMSA_8
EMSA_8 8_1 MLP_768
EMSA 8 MLP_ 768
x2
MLP_512
x2
x2
x2
Classifier
x
average pool, 1000d fully-connected
GFLOPs
1.4
1.94
4.26
7.91
Experiments
In this section, we conduct experiments on common-used benchmarks, including ImageNet- for classification, MS COCO2017 for object detection, and instance segmentation. In the following subsections, we first compared the proposed ResT with the previous state-of-the-arts on the three tasks. Then we adopt ablation studies to validate the important design elements of ResT.
3.1 Image Classification on ImageNet-1k
Settings. For image classification, we benchmark the proposed ResT on ImageNet-Ik, which contains 1. 1.28M training images and 50k validation images from 1,000 classes. The setting mostly follows (2 Specifically, we employ the Adam W (20 optimizer for 300 epochs using cosine decay learning rate scheduler and 5 epochs of linear warm-up. A batch size of 2048 (using GPUs with 256 images per GPU), an initial learning rate of 5e-4, weight decay of 0.05, and gradient clipping with max norm of are used. We include most the augmentation and regularization strategies of 26) in training. including RandAugment [.. Mixup (35], Cutmix K4. Random erasing 400, and stochastic depth [13]. An increasing degree of stochastic depth augmentation is employed for larger models, i... 0.1, 0.1, 0.2, 0.3 for ResT-Lite, Rest-Small, ResT- Base, and ResT-Large, respectively. For the testing on the validation set, the shorter side of an input image is first resized to 256, and a center crop of 224 x 224 is used for evaluation.
Results. Table 2 presents comparisons to other backbones, including both Transformer- based ones and ConvNet-based ones. We can see, compared to the previous tate-offthe tet-the-rr art Transformer-bant based architectures with similar model complexity, by large margin. For example, for smaller models, ResT noticeably surpass the counterpart PVT architectures with similar complexities: +4.5% for ResT-Small 79.6%) over PVT-T (75.1%). For larger models, ResT also significantly outperform the counterpart Swin architectures with similar
proposed ResT achieves significant improvement

---

Table 2: Comparison with state-of-the of- backbones ImageNet-Ik Ik benchmark. Throughput (images/ 8) is measured on a single V100 GPU, following [26]. All models are trained and evaluated on 224 resolution. The best records and the improvements over bench-marked ResNets marked in bold and blue, respectively.
Model
#Params (M)
FLOPs (G)
Throughput
Top-1 (%)
Top-5 (%)
ConvNet
ResNet-18
11.7
1.8
1852
69.7
89.1 I
ResNet-50
25.6
4.1
871
79.0
94.4
ResNet-101
44.7
7.9
635
80.3
95.2
RegNetY-4G [22
20.6
4.0
1156
79,4
94.7
RegNetY-8G 22
39.2
8.0
591
79.9
94.9
RegNetY-16G
83.6
15.9
334
80.4
95.1
Transformer
DeiT-S
26
22.1
4.6
940
79.8
94.9
DeiT-B
26
86.6
17.6
292
81.8
95.6
PVT-T
29
13.2
1.9
1038
75.1
92.4
PVT-S 29
24.5
3.7
820
79.8
94.9
PVT-M
29
44.2
6.4
526
81.2
95.6
PVT-L
29
61.4
9.5
367
81.7
95.9
Swin-T
19
28.29
4.5
755
81.3
95.5
Swin-S
49.61
8.7
437
83.3
96.2
Swin-B 19
87.77
15.4
278
83.5
96.5
ResT-Lite (Ours)
10.49
1.4
1246
77.2 (1 7.5)
93.7 (1 4.6)
ResT-SSmal (Ours)
13.66
1.9
1043
79.6 (1 9.9)
94.9 (1 5.8)
ResT-Base (Ours)
30.28
4.3
673
81.6 (1 2.6)
95.7 (1 1.3)
Res Resg-Large (Ours)
51.63
7.9
429
83.6 (1 3.3)
96.3 (1 1.1)
complexities: +0.3% for ResT-Base (81.6%) over Swin-T (81.3%), and +0.3% for ResT-Large (83.6%) over (iinn-((3.33%) using 224 224 input.
Compared with the state-of-the-art ConvNets, i.e., RegNet, the ResT with similar model complexity also achieves better performance: average improvement of 1.7% in terms of Top-1 Accuracy. Note that RegNet is trained via thorough architecture search, the proposed Res' is adapted from the standard Transformer and has strong potential for further improvement.
3.2 Object Detection and Instance Segmentation on COCO
Settings. Object detection and instance segmentation experiments are conducted on COCO 2017, which contains 118k training, 5k validation, and 20k test-dev images. We evaluate the performance of ResT using two representative frameworks: RetinaNet [18) and Mask RCNN [10]. For these two frameworks, we utilize the same settings: multi-scale training (resizing the input such that the shorter side is between 480 and 800 while the longer side is ar most 1333), Adam W (20 optimizer (initial learning rate Ic-4, weight decay 0.05, and batch size of 16), and 1x schedule (12 epochs). Unlike CNN backbones, which adopt post normalization and can directly apply downstream tasks. ResT employs the pre-normalization strategy to accelerate network convergence, which means the output of each stage is not normalized before feeding to FPN [77. Here, we add layer normalization (LN I for the output of each stage (before FPN )) similar to Swin 19 Results are reported on the validation split.
Object Detection Results. Table lists the results of RetinaNet with different backbones. From these results, it can be seen that for smaller models, ResT-Smmall is +3.6 box AP higher (40.3 vs. 36.7) than PVT-T with similar computation cost. For larger models, our ResT-Base surpassing the PVT-S by +1.6 box AP.

---

Table 3: Object detection performance the COCO va12017 split using the RetinaNet framework.
Backbones
AP50:95
AP50
AP75
APs
APm
API
Param (M)
R18 I
31.8
49.6
33.6
16.3
34.3
43.2
21.3
PVT-T 29
36.7
56.9
38.9
22.6
38.8
50.0
23.0
(esrs))allOurs)
40.3
61.3
42.7
25.7
43.7
51.2
23.4
R50 [1]
37.4
56.7
40.3
23.1
41.6
48.3
37.9
PVT-S 29
40.4
61.3
43.0
25.0
42.9
55.7
34.2
Swin-T T
19
41.5
62.1
44.1
27.0
44.2
53.2
38.5
ResT-Base (Ours)
42.0
63.2
44.8
29.1
45.3
53.3
40.5
RI01 [
38.5
57.8
41.2
21.4
42.6
51.1
56.9
PVT-M 229
41.9
63.1
44.3
25.0
44.9
57.6
53.9
Swin-S [9
44.5
65.7
47.5
27.4
48.0
59.9
59.8
ResT-Large (Ours)
44.8
66.1
48.0
28.3
48.7
60.3
61.8
Instance Segmentation Results. Table 4 compares the results of ResT with those of previous state-of-the-arr models on the Mask RCNN framework. Rest-Small exceeds PVT-T by +2.9 box AP and +2.1 mask AP on the COCO val2017 split. As for larger models, ResT-Base brings consistent +1.2 and +0.9 gains over PVT-S in terms of box AP and mask AP, with slightly larger model size.
Table 4: Object detection and instance segmentation performance on the COCO val2017 split using Mask RCNN framework.
Backbones
Apbox
APbox APbox
A Amnssk
50
Pmusk
Param (M)
R18 [11
34.0
54.0
36.7
31.2
51.0
32.7
31.2
PVT-T 29
36.7
59.2
39.3
35.1
56.7
37.3
32.9
ResT-Small(Ours)
39.6
62.9
42.3
37.2
59.8
39.7
33.3
R50 [II
38.6
59.5
42.1
35.2
56.3
37.5
44.3
PVT-S 29
40.4
62.9
43.8
37.8
60.1
40.3
44.1
ResT--Baee(Ours
41.6
64.9
45.1
38.7
61.6
41.4
49.8
3.3 Ablation Study
In this section, we report the ablation studies of the proposed ResT, using ImageNet-Ik image classification. To thoroughly investigate the important design elements, only adopt the simplest data augmentation and papammerrseeterss settings in [II] Specifically, the input images are randomly cropped to 224 224 with random horizontal flipping. All the architectures of ResT-Lite with SGD optimizer (with weight decay le-4 and momentum 0.9) for 100 epochs, starting from the initial learning rate of 0.1 batch batch_siiee/a /522 (with linear warm-up of 5 epochs) and decreasing by a factor of 10 every 30 epochs. Also, a batch size of 2048 (using 8 GPUs with 256 images per GPU) is used.
trained
Different types of stem module, Here, we test three type of stem modules: (I) the first patch embedding module in PVT (29), i.c., 4 convolution operation with stride 4 and padding; (2) the stem module in ResNet [T]] i.c., one 7 x convolution layer with stride and padding followed by one x 3 max-pooling layer; (3) the stem module in the proposed ResT, i.e., three 3 x 3 convolutional layers (all with padding 1) with stride 2, stride 1, and stride 2, respectively. We report the results in Table 5 The stem module in the proposed ResT is more effective than that in PVT and ResNet: +0.92% and +0.64% improvements terms of Top- accuracy, respectively.
Ablation study on EMSA. As shown in Figure 3 adopt Depth-wise Conv2d to reduce the computation of MSA. Here, we provide the comparison of more strategies with the same reduction stride Results are shown in Table 6 As can be seen, average pooling achieves slightly worse

---

Table 5: Comparison of various stem modules ResT-Lite. Results show that the proposed stem module is more effective than existing ones in PVT and ResNet.
Table 6: Comparison of different reduction strate- gies of EMSA on ResT-Lite. Results show that Average Pooling be alternative to Depth- wise Conv2d to make a trade-off.
Stem
Top-1 (%)
Top-5 (%)
Reduction
Top-1 (%)
Top-5 (%)
PVT
29
71.96
89.87
DWConv
72.88
90.62
ResNet
72.24
90.17
Avg Pooling
72.64
90.41
ResT (Ours)
72.88
90.62
Max Pooling
72.20
89.97
results (-0.24%) compared with the original Depth-wise Conv2d, while the results of the Max Pooling strategy are the worst. Since the pooling operation introduces no extra parameters, therefore, average pooling can be an alternative Depth-wise Conv2d in practice.
Table 7: Ablation study results on the important design elements of EMSA on ResT-Lite, includ- ing the 1 convolution operation and Instance Normalization in Eq. 4
Table 8: Comparison of various positional encod- ing (PE) strategies on ResT-Lite.
Encoding
Top-1 (%)
Top-5 (%)
Methods
Top-1 (%)
Top-5 (%)
w/o position
71.54
89.82
origin
72.88
90.62
LE
71.98
90.32
w/o IN
71.98
90.32
GL
72.04
90.41
w/o Conv-I&IN
71.72
89.93
PA
72.88
90.62
In addition, EMSA also adding two important elements the standard MSA, i.e., one 1 convolu- tion operation to model the interaction among different heads, and the Instance Normalization(IN) to restore diversity of different heads. Here, validate the effectiveness of these two settings. Results are shown in Table 7 We can see, without IN, the Top-1 accuracy is degraded by 0.9%, we attribute to the destroying of diversity among different heads because the 1 convolution operation makes all heads focus on all the tokens. In addition, the performance drops 1. 1.16% without the convolution operation and IN. This can demonstrate that the combination of long sequence and diversity are both important for attention function.
introduced 3 types of positional encoding types, i.e., the original learnable parameters with fixed lengths [5] (LE), the proposed group linear mode(GL), and PA mode. These encodings are added/multiplied beginning of each stage. Here, we compared the proposed GL and PA with LE, results are shown in Table We can see, the Top-1 accuracy degrades from 72.88% to 71.54% when the PA encoding removed, this means that positional encoding is crucial for ResT. T. The LE and GL. achieve similar performance, which means possible construct variable length of positional encoding. Moreover, the PA mode significantly surpasses the GL, achieving 0.84% Top-1 accuracy improvement, which indicates that spatial attention can also be modeled as positional encoding.
Different types of positional encoding. In section 2.4
the input patch token at the
Conclusion
In this paper, we proposed ResT, new version of multi-scale Transformer which produces hierarchi- cal feature representations for dense prediction. We compressed the memory of standard MSA and model the interaction between multi-heads while keeping the diversity ability. To tackle input images with arbitrary, we further redesign the positional encoding spatial attention. Experimental results demonstrate that the potential of ResT as strong backbones for dense prediction. We hope that our approach will foster further research in visual recognition.

---

Acknowledgments and Disclosure of Funding
This work is funded by the Natural Science Foundation of China (No. 12176119) and the program for Outstanding PhD candidate of Nanjing University.
References
[1] Lei Jimmy Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization. CoRR, 0tss/1607.06450
2016.
[2] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usumier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, Computer Vision ECCV 2020 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part I, volume 12346 of Lecture Notes in Computer
Science, pages 213 229. Springer, 2020.
[3] Hanting Chen, Yunhe Wang. Tianyu Guo, Chang Xu, Yiping Deng, Zhenhua Liu, Siwei Ma, Chunjing Xu, Chao Xu, and Wen Gao. Pre-trained image processing transformer. CoRR, abs/201 2.00644 2020.
[4] Ekin Dogus Cubuk, Barret Zoph, Jon Shlens, and Quoc Le. Randaugment: Practical automated data augmentation with a reduced search space. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria- Florina Balcan, and Hsuan-Tiee Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
[5] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer. Georg Heigold, Sylvain Gelly. Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. 9th International Conference on Learning Reprexentations. ICLR 2021, Virtual Event, Austria, May 3-7. 2021. OpenReview.net, 2021.
[6] Peng Gao, Jiasen Lu, Hongsheng Li, Roozbeh Mottaghi, and Aniruddha Kembhavi. Container: Context
aggregation network. CoRR. abs/2106.01 2021.
[7] Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Deep sparse rectifier neural networks. In Geoffrey J. Gordon, David B. Dunson, and Miroslav Dudík, editors, Proceedings of the Fourteenth International Conference Artificial Intelligence and Statistics, AISTATS 2011, Fort Lauderdale, USA, April I 11-13, 2011, volume 15 of JMLR Proceedings, pages 315-323. JMLR. 2011.
[8] Kai Han, An Xiao, Enhua Wu, Jianyuan Guo, Chunjing Xu, and Yunhe Wang. Transformer in transformer. CoRR, abs/2103.001 12, 2021.
[9] Karttikeya Mangalam Yanghao Li Zhicheng Yan Jitendra Malik Christoph Feichtenhofer Haoqi Fan,
Bo Xiong. Multiscale vision transformers. arXivv2104.1 1227, 2021.
[10] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross B. Girshick. Mask R-CNN. In IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017, pages 2980-2988, IEEE Computer Society, 2017.
[1I] Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016. pages 770 770.778. IEEE Computer Society, 2016.
In 2016 IEEE Conference
[12] Dan Hendrycks and Kevin Gimpel. Bridging nonlincarities and stochastic regularizers with gaussian error linear units. CoRR, ahs//600..00 60..08415, 2016.
[13] Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q. Weinberger. Deep networks with stochastic depth. In Bastian Leibe, Jiri Matas. Nicu Sebe, and Max Welling, editors, Computer Vision - ECCV 2016 14th European Conference, Amsterdan, The Netherlands, October 11-14, 2016, Proceedings, Part IV, volume 9908 of Lecture Notes in Computer Science, pages 646 (46-661. Springer, 2016.
[14] Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Francis R. Bach and David M. Blei, editors, Proceedings of the 32nd International Conference on Machine Learning. ICML 2015, Lille, France, 6-11 July 2015, volume 37 of JMLR Workshop and Conference Proceedings, pages
pages 448 456. JMLR.or,. 2015.
[15] Md. Amirul Islam, Sen Jia, and Neil D. B. Bruce. How much position information do convolutional neural networks encode? In Sth International Conference on Learning Representations, ICLR 2020, Addis Ababa. Ethiopia, April 26-30, 2020. OpenReview.net 2020.
[16] Xiang Li, Wenhai Wang, Xiaolin Hu, and Jian Yang. Selective kernel networks. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pages
510-519. Computer Vision Foundation IEEF, 2019.
[17] Tsung- Yi Lin, Piotr Dollár, Ross B. Girshick, Kaiming He. Bharath Hariharan, and Serge J. Belongie. Feature pyramid networks for object detection. 2017 IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pages 936-944, 2017.
[18] Tsung- Yi Lin, Priya Goyal, Ross B. Girshick, Kaiming Hc, and Piotr Dollár, Focal loss for dense object detection. In IEEÉ International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29. 2017, pages 2999-3007. IEEE Computer Society, 2017.
[19] Ze Liu, Yutong Lin. Yue Cao. Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer:
Hierarchical vision transformer using shifted windows. CoRR, abs/2103. 14030, 2021.
10

---

[20] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In 7th International Conference Learning Representations, ICLR 2019, New Orleans. LA. USA. May 6-9, 2019. OpenReview.net, 2019. [21] Niki Parmar, Prajit Ramachandran, Ashish Vaswani, Irwan Bello, Anselm Levskaya, and Jon Shlens. Stand-alone self-attention in vision models. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Bue, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, December 8- 14, 2019, Vancouver, BC, Canada, pages 68 80, 2019.
[22] Iija Radosavovic, Raj Prateek Kosaraju, Ross B. Girshick, Kaiming He, and Piotr Dollár. Designing network design spaces. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pages 0445-10433. IEEE, 2020.
[23] Aravind Srinivas, Tsung-Yi Yi Lin, Niki Parmar, Jonathon Shlens, Pieter Abbeel, and Ashish Vaswani.
Bottleneck transformers for visual recognition. CoRR, abs/2101.11605 2021.
[24] Zhi Tian, Chunhua Shen, and Hao Chen. Conditional convolutions for instance segmentation. In Andrea
Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, 16th European Conference on
Computer Vision ECCV 2020, Glasgow, UK, August 23-28, 2020, Proceedings, Part I, volume 12346 of Lecture Notes in Computer Science, pages 282-298. Springer, 2020.
[25] Zhi Tian, Chunhua Shen, Hao Chen, and Tong Hc. FCOS: A simple and strong anchor-free object detector.
CoRR,
2020.
[26] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. CoRR, abs/2012.12877. 2020.
[27] Dmitry Ulyanov, Andrea Vedaldi, and Victor S. Lempitsky. Instance normalization: The missing ingredient
for fast stylization. CoRR, abs/1607 08722,002,2222 2016.
[28] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike von Luxburg. Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 5998-6008, 2017.
[29] Wenhai Wang. Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song. Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: versatile backbone for dense prediction without convolutions.
arXiv preprint arXiv: 12122, 2021.
[30] Xiaolong Wang. Ross B. Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. CoRR,
abs/1711.00 1.09971. 2017.
2017.
[31] Xinlong Wang. Rufeng Zhang. Tao Kong, Lei Li, and Chunhua Shen. Solov2: Dynamic, faster and stronger.
CoRR, abs/2003. 10152. 2020.
[32] Saining Xie, Ross B. Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming Hc. Aggregated residual transfor- mations for deep neural networks. In 2017 IEEE Conference Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pages 5987-5995. IEEE Computer Society, 2017. [33] Li Yuan, Yunpeng Chen, Tao Wang. Weihao Yu, Yujun Shi, Francis E. H. Tay, Jiashi Feng, and Shuicheng Yan. Tokens-to-token vit: Training vision transformers from scratch on imagenet. CoRR, abs/2 101. 1986. 2021.
[34] Sangdoo Yun. Dongyoon Han, Sanghyuk Chun, Seong Joon Oh, Youngjoon Yoo, and Junsuk Choe. Cutmix: Regularization strategy to train strong classifiers with localizable features. In 2019 JEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 - -Noember
2, 2019. pages 1022-6031. IEBE, 2019.
[35] Hongyi Zhang, Moustapha Cissé, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018.
[36] Hang Zhang. Chongruo Wu, Zhongyue Zhang. Yi 7hu, Zhi Zhang, Haibin Lin, Yue Sun, Tong Hc, Jonas Mueller, R. Manmatha, Mu Li, and Alexander Smola. Resnest: Split-attention networks. CORR, abs/2004.08955, 2020.
[37] Qing-Long Zhang, Lu Rao, and Yubin Yang. Group-cam: Group score-weighted visual explanations for
deep convolutional networks. CoRR, 0oss2103,118555 2021.
[38] Qing-Long Zhang and Yu-Bin Yang. Sa-net: Shuffle attention for deep convolutional neural networks.
CoRR, abs/2102.00240, 2021.
[39] Zelin Zhao, Karan Samel, Binghong Chen, and Le Song. Proto: Program- guided transformer for program-
guided tasks. CoRR, abs/21 10.00804, 2021.
[40] Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang. Random erasing data augmentation. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAl 2020, New York, NY, USA, February 7-12, 2020, pages 13001- 13008. AAAI Press, 2020.
[41] Xizhou 7hu, Waijie Su, Lewei Lu, Bin Li, Xiangang Wang, and Jifeng Dai. Deformable DETR: deformable transformers for end-to-end object detection. CoRR, abs/2010.041,5 2020.
11


Reviews:
[{"review_id": "nHdhn0n6-9", "reviewer": "Reviewer_Qyop", "summary": "This paper presents an efficient multi-head self-attention by reducing the tokens of keys and values; furthermore, this paper also proposed another way to encode position information in the ViT. The authors validated their methods in image classification and object detection.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\n1. This paper improves the vanilla ViT by integrating many useful techniques, e.g. pyramid structure, EMSA, PA, etc.\n\n2.  The proposed components are very similar to the previous work, making the novelty of this paper limited.\n\n    - EMSA, it is similar to PVT. Except for the authors use depthwise convolution to reduce spatial tokens; and the instance normalization and 1d Conv on the attention map. Does IN and 1d conv play an important role for the large model?\n\n    - Using a stronger patch embedding could lead to better performance had been mentioned in the original ViT paper for small-scale data, e.g. ImageNet1K.\n\nQuestions\n\n1. See point 2 above\n\n2. Why do the authors use stronger data augmentation (multi-scale) in object detection? in PVT, for 1x schedule, they do not use multi-scale training. What is the performance if disabling multi-scale training?\nFurthermore, why authors do not compare to Swin in the detection as they are in the classification. \n(Others reproduce Swin results with ImageNet1K pretrained weights. \"Twins: Revisiting the Design of Spatial Attention in Vision Transformers.\")\n\n3. Why the ablation study does not use the same settings, do those observations still hold if using the same settings? Especially, why only use simpler data augmentation?  \n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"ethical_concerns": "No", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}, {"review_id": "_9hOLXoToD", "reviewer": "Reviewer_Wueu", "summary": "This work presents a computationally efficient vision transformer network for image recognition. In particular, this work propose memory-efficient self-attention module to replace the original one in the original transformer. Second, it modifies the inflexible position encoding (that only allows fixed length of tokens) to accept variable size of input images. Lastly, the patch embedding is achieved by stacking overlapping convolution operations with strides. Extensive experiments are conducted to show the superiority of the proposed ResT.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n- The writing is good. The paper is easy to follow.\n- The proposed methods are intuitive and reasonable, including efficient transformer block and the flexible position encoding.\n- The proposed ResT is a general backbone and is expected to benefit many vision tasks.\n- The code of this work is provided.\n\nCons:\n- The proposed methods seem to apply some CNN components/designs (eg. depthwise convolution and stem) to the transformers, which might not be considered as major novelty and is more like incremental improvements over ViT.\n- The abstract says ResT is a general-purpose backbone, so I would expect some experiments on one or two more vision tasks, such as semantic segmentation or pose estimation.\n- The improvements over Swin transformer in Table 2 are not so obvious. \n- Can the author provide some detection results on Swin transformer on COCO??", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1.5"}, "extra_text": {}}, {"review_id": "PfU8Kfrqu73", "reviewer": "Reviewer_WKZ1", "summary": "In this paper, the authors propose ResT, which is an efficient multi-scale vision Transformer backbone. Comparing with ViT, ResT has the following three modifications: 1) A DWconv-based efficient self-attention. 2) A DWconv-based position encoding. 3) Overlapped token embedding. The experiment results show that ResT outperforms previous methods.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n1) The paper is well written and easy to follow.\n2) The results seem good.\n3) The idea of using Instance Normalization after softmax is interesting and its corresponding ablation study results seem good.\n\nWeaknesses\n1) The analysis about the second shortcoming of ViT is weird. The authors think the multi-head with a low subset head dimension may influence the performance. But no experimental results are provided to prove it. It also contradicts with common sense that more head brings better model capability. Meanwhile, if the low dimension influences the performance, why not reduce the head number directly? \n2) The definition of s=8/k is weird and does not make sense. Why the spatial reduce factor is defined by k (number of heads), but not the feature map size or stage number.\n3) The authors propose several modifications about ViT, but most of them are already proposed by several papers. For example, mixing attention map with Conv and normalization is used in DeepViT[1]. The spatial reduction of K and V is already used in PvT[2] and CvT[3]. The DWConv-based position embedding is proposed in CPVT[4]. Overlapped token embedding is also used in CvT[3] and T2T[5]. This limits the contribution of this paper.\n4) The experiment is not convincing enough. Although the authors mentioned many ViT-based methods in related works, the experiment part only compared with a few of them. For example, for the ImageNet part, there are many strong methods, such as T2T, CvT. But the authors only compare with DeiT, PvT, and Swin. The detection and segmentation experiment on COCO also misses the comparison with Swin, which is the SOTA method at the NeurIPS submission deadline.\n5) The experiment setting on COCO is not fair. The PvT and ResNet use single-scale training but the authors use multi-scale training. So the comparison is totally unfair and does not make sense. Meanwhile, I notice the multi-scale training resizes the short side between 480 and 800, which is the setting from Swin, but not PvT (640 to 800). Why not compare with Swin while using the setting from Swin?\n6) The experiment lacks a large model comparison. This paper's ``Large'' model is only 50M and 8G FLOPs, which is quite small. This also makes the experiment results not convincing enough.\n7) Some citation is not correct, [7] in the paper is ViT, but not PvT.\n\n\nOverall, this paper proposes several modifications to ViT and improved its performance. But most of the modifications are already proposed in previous papers, and the experiment lacks comparison, some of the comparisons are even not fair. Due to the above analysis, I give a ``5: Marginally below the acceptance threshold'' to this paper. \n\n[1] Zhou, Daquan, et al. \"Deepvit: Towards deeper vision transformer.\"\n\n[2] Wang, Wenhai, et al. \"Pyramid vision transformer: A versatile backbone for dense prediction without convolutions.\"\n\n[3] Wu, Haiping, et al. \"Cvt: Introducing convolutions to vision transformers.\"\n\n[4] Chu, Xiangxiang, et al. \"Conditional positional encodings for vision transformers.\"\n\n[5] Yuan, Li, et al. \"Tokens-to-token vit: Training vision transformers from scratch on imagenet.\"\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors did not point out the limitations and potential negative societal impact of their work.", "review_text": "", "extra_scores": {"ethical_concerns": "No", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "Fz_Nqfv2TPX", "reviewer": "Reviewer_3XRr", "summary": "The authors propose a new ViT-based architecture for image recognition. The architecture is based on 3 main ideas: 1) using channel-wise conv to reduce the spatial dimension of feature maps to save computations. 2) using a stack of convs as \"patch embedding\", which is used at the beginning of each stage to shrink spatial size while increasing channel dimensions. 3) A depth-wise conv to replace the \"position encoding\" in the original ViT. \n\nOverall the model achieves strong performance, outperforming Swin-Transformer at comparable computational cost.\n\n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "In the following I review this paper in the following axes.\n\n[originality: medium]\n - Reducing the spatial dimension to save computational costs and memory has been used in, e.g., \"non-local neural networks\" (Wang et al., CVPR 18). \n\n - Using convs with stride to reduce spatial dimensions and increase channel dimensions has been the standard practice in many conv nets (e.g., ResNet). \n\n - Replacing positional encoding with the depth-wise conv is original/novel to my knowledge. I find this part quite interesting.\n\n[quality: medium-high]\n - The experiments are satisfactory, covering important ablation studies that I'd expect. The results are impressive --- high performance with low computational costs compared to recent, strong ViT-based methods.\n\n[clarity: low-medium]\nI find some descriptions of this paper confusing:\n1. L159: The description of \"GL\" is vague. Should be mathematically defined.\n2. \"pixel-wise attention (PA)\" turns out to be a \"conv\" operator. Something like \"conv-based re-weighting\" might be clearer.\n3. I found 2.4 confusing. If I understand it correctly, in (7) GL(x) operates on features \"x\", instead of the \"positions\". Thus this operator doesn't explicitly encode any position. Why is it then called a \"positional encoding\"? (I understand that if we implement GL as conv (PA), it can model some spatial layout to a certain extent, but this is not positional encoding. This is using conv nets to model spatial patterns.)\n\n[significance: medium]\n - The area of ViT based models have many interesting open questions to be explored. This work advances the model design through solid improvement in accuracy. I think this work can be useful for future research.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I've listed the limitations in my review above. I don't see anything particularly concerning regarding societal impact, compared to other deep network architecture. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1.5 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Three reviewers recommend borderline-acceptance, one reviewer recommends borderline-rejection. Most remaining concerns stem from a lack of novelty compared to a very rapidly moving (partially concurrent) work on vision transformers. However, the AC agrees with the majority of reviewers that there is enough merit in the presented work to justify acceptance.
Decision: Accept (Poster)