ResT V2: Simpler, Faster and Stronger
Authors: Qinglong Zhang
Yu-Bin Yang
Venue: NeurIPS 2022 Accept (2022)
Keywords: multi-scale vision Transformer
downsampling
upsampling
computation density

Abstract: This paper proposes ResTv2, a simpler, faster, and stronger multi-scale vision Transformer for visual recognition. ResTv2 simplifies the EMSA structure in ResTv1 (i.e., eliminating the multi-head interaction part) and employs an upsample operation to reconstruct the lost medium- and high-frequency information caused by the downsampling operation. In addition, we explore different techniques for better applying ResTv2 backbones to downstream tasks. We find that although combining EMSAv2 and window attention can greatly reduce the theoretical matrix multiply FLOPs, it may significantly decrease the computation density, thus causing lower actual speed. We comprehensively validate ResTv2 on ImageNet classification, COCO detection, and ADE20K semantic segmentation. Experimental results show that the proposed ResTv2 can outperform the recently state-of-the-art backbones by a large margin, demonstrating the potential of ResTv2 as solid backbones. The code and models will be made publicly available at \url{https://github.com/wofmanaf/ResT}.

ResT V2: Simpler, Faster and Stronger
Qing-Long Zhang, Yu-Bin Yang
State Key Laboratory for Novel Software Technology Nanjing University, Nanjing 21023, China wofmana @@ juu.. edu.cn, yangyubinân ju edu. cn
Abstract
This paper proposes ResTv2, simpler, faster, and stronger multi-scale vision Transformer for visual recognition. Res'Tv2 simplifies the EMSA structure ResTvl (i.e., eliminating the multi-head interaction part) and employs upsample operation to reconstruct the lost medium- and high-frequency information caused by the downsampling operation. In addition, explore different techniques for better applying ResTv2 backbones to downstream tasks. We find that although combining EMSAv2 and window attention can greatly reduce the theoretical matrix multiply FLOPs, it may significantly decrease the computation density, thus causing lower actual speed. We comprehensively validate ResTv2 on ImageNet classification, COCO detection, and ADE20K semantic segmentation. Experimental results show that the proposed ResTvv can outperform the recently state-of of- the-art backbones by large margin, demonstrating the potential of ResTv2 as solid backbones. The code and models will be made publicly available https: / /gg thub. com/ wo fmana ResT.
Introduction
Recent advances Vision Transformers (ViTs) have created new state-of-the-art results on many computer vision tasks. While scaling up ViTs with billions of parameters [22, 45, 40, 13] is a well- proven way to improve the capacity of the ViTs, it is more important to explore more energy-efficient approaches to build simpler ViTs with fewer parameters and less computation cost while retaining high model capacity.
Toward this direction, there
few works that significantly improve the efficiency of ViTs [35, 10, 12, 23, 5]. The first kind is reintroducing the "sliding window" strategy to ViTs. Among them, Swin Transformer [23] is milestone work that partitions the patched inputs into non-overlapping windows and computes multi-head self-attention (MSA) independently within each window. Based on Swin, Focal Transformer [41] further splits the feature map into multiple windows in which tokens share the same surroundings to effectively capture short- and long-range dependencies. The second type improve efficiency is downsampling one several dimension of MSA. PVT [35] is pioneer work in this area, which adopts another non-overlapping patch embedding module to reduce the spatial dimension of keys and values MSA. ResTvl [47] further explores three types of overlapping spatial reduction methods (i.c., max pooling, average pooling, and depth-wise convolution) in MSA to balance the computation and effectiveness different scenarios. However, the downsampling operation in MSA will inevitably impair the model's 's performance since it destroys the global dependency modeling ability MSA certain extent (shown Figure 1).
In this paper, we propose EMSAv2, which explores different upsample strategies adding to EMSA to compensate for the performance degradation caused by the downsampling operation. Surprisingly, the "downsample-upsampl" combination builds an independent convolution hourglass architecture. which can efficiently capture the local information that complementary to long-distance dependency with fewer extra parameters and computation costs. Besides, EMSAv2 eliminates the multi-head
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

764
Top-I 1 Aecuracy (%)
75
Up
MSA
74-
ResTvI SerTVv--Lie L-Lite
73
+ Up
72
MSA
PV 1- Tiny
3
4
5
GFLOPs
Top-1 Accuracy of ResT-Lite [47] and PVT-Tiny [35] under 100 epochs training settings. Results show that downsampling operation will impair the performance while adding upsampling operation can address this issue. Detailed comparisons are shown in Appendix A.
Figure I:
interaction module in EMSA to simply the self-attention structure. Based EMSAv2, we build simpler, faster, and stronger general-purpor methods of applying ResTv2 backbones to downstream tasks. We found that combining EMSAv2 and window attention is not that good when the inputs' resolution high (c.g., 800 x 1333), although it can significantly reduce the theoretical matrix multiply FLOPs. Due the padding operation in window partition and grouping operation of window attention, the computation density of EMSAv2 will be significantly decreased, causing lower actual inference speed. We hope the observations and discussions can challenge some common beliefs and encourage people to rethink the relations between theoretical FLOPs and actual speeds, particularly rumning GPUs.
backbones, ResTv2. In addition, we explore four
We evaluate ResTv2 on various vision tasks such as ImageNet classification, object detec- tion/segmentation on COCO, and semantic segmentation on ADE20K. Experimental results reveal the potential of ResTv2 as strong backhones. For example, our ResTVv2-L yields 84.2% Top-1 accuracy (with size 2 on ImageNet- Ik, which is significantly better than Swin-B [23] (83.5% and ConyNeXt-B [24] (83.8%) while ResTv2-L has fewer parameters (87M vs. 88M vs. 89M) and much higher throughput (415 vs. 278 vs. 292 images/s).
2 Related Work
Efficient self-attention structures. MSA has shown great power to capture global dependency in computer vision tasks [11, 2, 3, 43, 43, 50]. However, the computation complexity of MSA quadratic to the input size, which might be acceptable for ImageNet classification, but quickly becomes intractable with higher-resolution inputs. One typical way to improve efficiency is partition- ing the patched inputs into non-overlapping windows and computing self-attention independently within each of these windows (i.e., windowed self-attention). To enable information communicate across windows, rescarchers have developed several integrate techniques, such as shift window [23], spatial shuffle [17]. alternately running global attention and local attention [5, 42] between successive blocks, Other ways are trying to reduce spatial dimension of the MSA. For example, PVT [35] and ResTvl [47] designed different downsample strategies to reduce the spatial dimension of keys and values in MSA. MVIT [12] proposed pooling attention to downsample queries, keys, and values spatial resolution. However, either the windowed self-atte attentienio downsampled self-attention will impair the long-distane distance modeling ability to some content, i.e., surrendering some important information for efficiency. Our target in this paper reconstruct the lost information in a light way.
Convolution enhanced MSA. Recently, designing transformer models with convolution operations has become popular since convolutions introduce inductive biases, which is complementary to

---

MSA. ResTvl [47] and [38] reintroduce convolutions at the early stage to achieve stabler training. CoAtNet [9] and UniFormer [19] replace MSA blocks with convolution blocks in the former two stages. CvT [36] adopts convolution in the tokenization process and utilizes stride convolution to reduce the computation complexity of self-attention. CSwin Transformer [10] and CPVT [6] adopt a convolution-base positional encoding technique and show improvements on downstream tasks. Conformer [28] and (obile-Former(4] combine Transformer with an independent ConvNet model to fuse convolutional features and MSA representations under different resolutions. ACmix [26] explores closer relationship between convolution and self-attention by sharing the x 1 convolutions and combining them with the remaining lightweight aggregation operations. The "downsample- upsample" branch in ResTv2 happens to build an independent convolutional module, which can effectively reconstruct information lost by the MSA module.
3 Proposed Method
3.1 A brief review ResTv1
[esTv1[47] is an efficient multi-scale vision Transformer, which can capably serve as general- purpose backbone for image recognition. ResTvl effectively reduces the memory of standard MSA [34, 11] and models the interaction between multi-heads while keeping the diversity ability. To tackle input images with an arbitrary size, ResTvl constructs the positional embedding as spatial attention, which models absolute positions between pixels with the help of zero paddings in the transformation function.
EMSA is the critical component in ResTvl [47] (shown in Figure 2(a) ). Given a ID input token E Raxdm where n is the token length, dm is the channel dimension. EMSA first projects x using a linear operation to get the query: Q = 2W, g + by where W; g and by are the weights and bias of linear projection. After that, Q is split into groups (i.e., heads) to prepare for the next step, i.e. Q E R*xnxd, where d2 = dm./k is the head dimension. To compress memory, x is reshaped to its 2D size and then are downsampled by a depth-wise convolution to reduce the height and width. After that, the output x' is reshaped to the 1D size, and then a Layer Norm [1] is added. Then the author employs the same way as to obtain Q to get key K and value V on x'. The output of EMSA can be calculated by
QKT Vdh ))
EMSA(Q, K) = Norm(Softmax( (Conv(
where "Conv" is applied model the interactions among different heads. "Norm" can be Instance Norm [33] or Layer Norm [1], which is applied to re-weight the attention matrix captured by different heads.
3.2 ResTv2
As shown in Figure 1, although downsample operation in EMSA can significantly reduce the computation cost, it will inevitably lose some vital information, particularly in the earlier stages, where the downsampling ratio is relatively higher, e.g., 8 in the first stage. To address this issue, one feasible solution is to introduce spatial pyramid structural information. That is, setting different downsampling rates for the input, calculating the corresponding keys and values respectively, and then combining these multi-scale keys and values along the channel dimension. The obtained new keys and values are then sent to the EMSA module to model global dependencies or directly calculate multi-scale self-attention with the original multi-scale keys and values.
However, the multi-path calculation of keys and values will greatly reduce the computational density of self-attention. although the theoretical FLOPs do not seem to change much. For example, the multi-path Focal-T [41] and the single-path Swin-T [23] have comparable theoretical FLOPs (4.9G vs. 4.5G), but the actual inference throughput of Focal-T is only 0.42 times of Swin-T (319 vs. 755 images/s).
In order to effectively reconstruct the lost information without having a large impact on the actual running speed, this paper, we propose to execute an upsampling operation on the values directly. There are many upsampling strategies, such as "nearest", "bilinear", "pixel-shuffle", etc. We find that all of them can improve the model's performance, but "pixel-shuffle" (which first leverages one DoCony to extend the channel dimension and then adopts pixel-shuffle operation to upscale the
3

---

cm
Linear
Linear
x X
x
x
Scale & Conv Softmax
Scale Softmax
Pixcl-Shuffle
xx x de
n x
x
DWConv
X
x
x de V X de
Q h R x de
xpx
do
V A x
Linear
Linear
Lingar
Lincar
Linear
Linear
dm x h/ x
dm
x wr!
DWConv
DWConv
dmn Xxxx
thp
X das
X da (b) EMSAV2 module
(a) EMSA module
Figure 2: Comparison of EMSA in ResTvl and EMSAv2 ResTv2. To simplify, all normalization operators in EMSA and EMSAv2 not displayed.
spatial dimension) works better. We call this new self-attention structure EvS The detailed structure is shown in Figure 2(b).
Surprisingly. the "downsample- "oossaalpleuussamplee" combination EMSAv2 happens to build an indepen- dent convolution hourglass architecture. which can efficiently capture the local information that is complementary to long-dise onng-iitance dependency with fewer extra parameters and computation costs. Besides, we find that the multi-head interaction module of the self-attention branch in EMSAv2 will decrease the actual inference speed of EMSAv2, although it can increase the final performance. And the performance improvements will be decreased as the channel dimension for each head increases. Therefore, we remove it for faster speed under default settings. However, if the head dimension small (e.g., d2 64 smaller), the multi-head interaction module will make difference (Detailed Results can be found in Appendix B). By doing so, we can also increase the training speed since the computation gaps between the self-attention branch and the upsample branch are bridged. The mathematical definition of the EMSAv2 module can be represented as
QKT Vdg
EMSAv2(Q, K, K) Softmax(
JV + Up(V)
3.3 Model configurations.
We construct different ResTv2 variants based on EMSAv2. ResTv2-T/B/LL to of similar complex- ities to Swin-T/S/B. We also build ResTv2-S to make better speed- trade-off. The four variants only differ in the number channels, heads' number EMSAv2, and blocks in each stage. Other hyper-parameter are the same as [45Tv Note that the upsampling module in ResT'v2 introduces extra parameters and FLOOss To make fair comparison, the block number in the first stage of ResTv2-T/S/B is set to 1, half of the one in ResTvl. Assume C' is the channel number of hidden layers in the first stage. We summarize the configurations below:
ResTv2-T: C 96, heads (1, 2, 4, 8), blocks number 21, 6, 2]
ResTv2-S: C 96, heads 21, 4, 8), blocks number (1,2, 12, 2
ResTv2-B: C = 96, heads 21, 4, 8), blocks number = (1,3, 16,
. ResTv2-L: C' = 128, heads = (2. 8, 16), blocks number= (2. 16. 2]
Detailed model size, theoretical computational complexity (FI.Oss,, and hyper-parameters of the model variants for ImageNet image classification are listed in Appendix D.

---

3.4 Explanation of upsample branch
To better explain the role of the upsample branch in EMSAv2, we plot the Fourier transformed feature maps of EMSAv2, the separate self-attention branch, and upsample branch of ResTvv-T following [27]. Here, we give some explanations: (1) different coloured polylines represent 11 blocks in ResTv2-T, and the bottom one is the first block; (2) We only use half-diagonal components of shift Fourier results. Therefore, for each polyline, 0.0T, 0.5x, and 1.0x can also represent low-, medium-, and high-frequency, respectively.
Compared with Figure 3(a) and 3(b), earlier blocks, the average value of the upsampling branch is higher than the self-attention branch, particularly in 0.5x and 1.0T, which means the upsample branch can capture more medium- and figh-freeuency information. Compared with Figure 3(b) and 3(c), almost all value of the combined branch is higher than the self-attention branch, particularly in earlier blocks, demonstrating the upsample module's effectiveness.
1.00
Log amplitude
A Log amplitude
Log amplitude
-2
-2
-2
0.75
Normalized depth
0.50
4
0.25
-6
0.00
0.0%
0.5xt
1.0%
0.0x
0.5x Frequency
1.0t
0.0x
0.5x Frequency
1.0z
Frequency
(a) upsample
(b) self-attention
(c) combine
Figure 3: Relative log amplitudes of Fourier transformed feature maps. A Log amplitude is the difference between the log amplitude at normalized frequency 0.0r (center) and 1.0x (boundary).
4
Empirical Evaluations on ImageNet
4.1
Settings
The ImageNet- Ik dataset consists of 1.28M training images and 50k validation images from 1,000 classes. We report the Top-1 and Top-5 accuracy the validation set. We summarize our training and fine-tuning setups below. More details can be found in Appendix C.I.
We train ResTv2 for 300 epochs using AdamW [25], with cosine decay learning rate scheduler and 50 epochs of linear warm-up. An initial learning rate of 1.5e-4x batch size 256, weight decay of 0.05, and gradient clipping with max norm of 1.0 are used. For data augmentations. we adopt common schemes including Mixup [46], Cutmix [44], RandAAgment [8], and Random Erasing [48]. We regularize the networks with Stochastic Depth [16] and Label Smoothing [32]. We use Exponential Moving Average (EMA) [29] as we find it alleviates larger models' over-fitting. The default training and testing resolution is 224 Additionally, we fine-tune at large resolution of 384?, adopting Adam for 30 epochs, with learning rate 1.5e-5x batch_size size 256, a cosine decaying schedule afterward, no warm up, and weight decay of 1e-8.
4.2 Main Results
Table 1 shows the result comparison of the proposed ResTy2 with three recent Transformer variants, ResTvl [47], Swin Transformer [23], and Focal Transformer [41], well two strong ConvNets:
RegNet [30] and ConvNeXt [24].
We can ResTv2 competes favorably with them in terms of speed-accuracy trade-off. Specifically, ResTv2 outperforms ResTvl of similar complexities across the board, sometimes with a substantial margin, c.g., +0.7% (82.3% vs. 81.6%) in terms of Top-1 accuracy for ResTv2-T. Besides, ResTv2
5

---

Table 1: Classification accuracy on ImageNet-lk. Inference throughput (images / s) is measured on a V100 GPU, following [47].
Model
Image Size
Params
FLOPs
Throughput
Top-1 (%)
Top-5 (%)
RegNetY-4G [30]
2242
21M
4.0G
1156
79.4
94.7
ConvNeXt-T [24]
2242
29M
4.5G
775
82.1
95.9
Swin-T [23]
224
28M
4.5G
755
81.3
95.5
Focal-T [41]
224
29M
4.9G
319
82.2
95.9
ResTv1-B [47]
2242
30M
4.3G
673
81.6
95.7
ResTv2-T
224²
30M
4.1G
826
82.3
95.5
ResTv2-T
3842
30M
12.7G
319
83.7
96.6
RegNetY-8G [30]
224
39M
8.0G
591
79.9
94.9
ResTv2-S
224
41M
6.0G
687
83.2
96.1
ResTv2-S
3842
41M
18.4G
256
84.5
96.7
ConvNeXt-S [24]
224
50M
8.7G
447
83.1
96.4
Swin-S [23]
2242
50M
8.7G
437
83.2
96.2
Focal-S [41]
2242
51M
9.4G
192
83.6
96.2
ResTv1-L [47]
2242
52M
7.9G
429
83.6
96.3
ResTv2-B
224
56M
7.9G
582
83.7
96.3
ResTv2-B
3842
56M
24.3G
210
85.1
97.2
RegNetY-16G [30]
224
84M
15.9G
334
80.4
95.1
ConvNeXt-B [24]
2242
89M
15.4G
292
83.8
96.7
Swin-B [23]
224
88M
15.4G
278
83.5
96.5
Focal-B [41]
224
90M
16.4G
138
84.0
96.5
ResTv2-L
224
87M
13.8G
415
84.2
96.5
ConvNeXt-B [24]
3842
89M
45.0G
96
85.1
97.3
Swin-B [23]
3842
88M
47.1G
85
84.5
97.0
ResTv2-L
384²
87M
42.4G
141
85.4
97.1
outperforms the Focal counterparts with an average X188 inference throughput acceleration, although both of them share similar FLOPs. A highlight from the results is ResTv2-B: it outperforms Focal-S by +0.1% (83.7% vs. 83.6%), but with +203% higher inference throughput (582 vs. 192 images/s). ResTv2 also enjoys improved accuracy and throughput compared with similar-sized Swin Transformers, particularly for tiny models, the Top-1 accuracy improvement is + +1.0% and (82.3% vs. 81.3%).
Additionally, we observe a highlight accuracy improvement when the resolution increases from 224 to 3842. An average + +1.4% Top-1 accuracy is achieved. We can conclude that the proposed ResTv2 also possesses the ability to scale up capacity and resolution.
4.3 Ablation Study
Here, ablate essential design elements in ResTv2-T using ImageNet-Ik image classification. To save computation energy, all experiments this part are trained for 100 epochs, and 10 of them are applied for linear warm-up. with other settings unchanged.
Upsampling Targets. There are three options for upsampling, the output of down-sample operation r', K. and V. Table 2(a) shows the results of upsampling these targets. Undoubtedly, upsampling K or achieves better results than since and V are obtained from x' via linear projection, enabling the communication of information between different features. Upsampling V works best. This can be attributed to the fact that unified modeling of the same variable (i.e., V) can better enhance the feature representation.
Upsampling Strategies. Table 2(b) varies the upsampling strategies. can see that all of the three upsample strategies increase the Top-1 accuracy, which means the upsample operation can

---

Table 2: Ablation experiments with ResTv2-T on ImageNet-lk. If not specified, the default is: upsampling V using pixel-shuffle operation and applying PA as positional embedding. Default settings are marked in gray.
(a) Upsampling Targets. Upsampling V works the best.
(b) Upsampling Strategies. Pixel-Shuffle achieves
better speed-accuracy trade-off.
Targets
Top-1 (%)
Top-5 (%)
Upsample
Params
FLOPs
Top-1 I (%)
w/o
79.04
94.61
w/o
30.26M
4.08G
79.04
x'
79.64
94.90
nearest
30.26M
4.08G
79.16
K
80.03
94.95
bilinear
30.26M
4.08G
79.28
V
80.33
95.06
pixel-shufin
30.43M
4.10G
80.33
(c) ConvNet or EMSA? Both of them can boost the performance.
(d) Positional Embedding. Both RPE and PA work well, but PA more flexible.
Branches
Params
FLOPs
Top-1 (%)
PE
Params
Top-1 (%)
EMSA
30.26M
4.08G
79.04
w/o
30.42M
79.94
ConvNet
26.11M
3.56G
77.18
APE [1]]
30.98M
79.99
ConvNetv2
26.67M
4.09G
77.91
RPE [31]
30.48M
80.32
ConvNetv3
30.43M
4.54G
78.63
PEG [6]
30.43M
80.17
EMSAv2
30.43M
4.10G
80.33
PA [47]
30.43M
80.33
provide information not captured by self-attention. In addition, pixel-shuffle operation obtains much stronger feature extraction capabilities with a few parameters and FLOPSs increase.
ConvNet EMSA?
As mentioned in Scction 3.2, we point out that the wowwsaapling-upsaappe
pipeline in EMSAv2 can constitute complete ConvNet block for extracting fea- tures. Here, we separate it (i.e., a ResTv2-T variant without self-attention) to see whether can replace the MSA module in ViTs. Table 2(c) shows that with the same number blocks, the performance of the ConvNet version is guite poor. In order show that insufficient parameters and computation do not predominantly cause this issue, constructed ConvNety2 (block numbers in the four stages are (2, 3, 6, 2)) and ConvNetv3 (block numbers are (2, 3, 6, 3)) that the model complex- ity of the ConvNetv2 and EMSA versions (without up- sample) is equivalent. Experimental results show that 2onvNetv2 and ConvNetv3 still perform inferior to the EMSA version (77.91 78.63 79.04 in terms of Top- accuracy). This observation indicates that ConvNet does not like EMSA. Thus, not reasonable to replace MSA with ConvNet in ViTs.
1.0
up alt
0.8
up
CKA (Linear)
0.6
0.4
0.2
0.0
0
2
4
10
Block ID)
Figure 4: Linear CKA Similarity be- tween EMSA, Upsample and EMSAv2 with ResTv2-T. Higher value means higher similarity.
However, combining the upsample module and EMSA (i.e., EMSAv2) indeed improves the overall performance.
We conclude that the downsampling operation of EMSAv2 will lead the loss of input informa- tion, resulting in insufficient information extracted by the EMSA module constructed on this basis, and the upsampling operation reconstruct the lost information.
We further plot the linear CKA [18] curves measure which is more critical for EMSAv2 (i.e., the combination variant, short for "com"), the self-attention branch (i.c., EMSA, short for "attn")

---

or the upsample module (short for "up")? As shown in Figure 4 (the red polyline, i..., "up attn"), "). in earlier blocks, feature representations extracted by self-attention and upsample module show relatively low similarity, while in deeper blocks, they exhibit a surprisingly high similarity. We can conclude that features in earlier blocks extracted by self-attention and upsample modules are complementary. Combining them can boost the final performance. In deeper blocks, particularly the last block, self-attention behaves like the upsample module (linear CKA > 0.8), although shows a higher similarity with EMSAv2 (linear CKA > 0.9, shown in the purple polyline, i.e., comtnn These observations could provide a guide for designing hybrid models, i.e., integrating ConvNets and MSAs in the early stages significantly improve the performance of ViTs.
Positional Embedding. We also validate whether Positional Embedding (short for PE) still works in ResTv2. Table 2(d) shows PE can still improve the performance, but not that obvious as ResTvl [47]. Specifically, both RPE and PA work well, but PEG and PA are more flexible and can process input images of arbitrary size without interpolation or fine-tuning, Besides, PA outperforms PEG with the same model complexity. Therefore, we apply PA as the default PE strategy. Detailed settings about these positional embedding can be found in Appendix E.
Empirical Evaluation on Downstream Tasks 5
5.1 Object Detection and Segmentation on COCO
Settings. Object detection and instance segmentation experiments are conducted on COCO 2017, which contains 1 118K training, 5K validation, and 20K test-dev images. We report results using the validation set. We fine-tune Mask R-CNN [14] with ResTv2 backbones. Following [24], adopt multi-scale training, Adam W optimizer, "x1 schedule" for ablation study, and "x3 schedule" for main results. Further details and hyper-paun parameter settings can be found in Appendix C.2.
Ablation Study. There are several ways to fine-tune ImageNet pre-trained ViT backbones. The conventional one the global style, which directly adopts ViTs into downstream tasks. The recent popular one is window-style (short for Win), which constrained part or all MSA modules of ViTs into computation overhead. However, performing all MSA into limited-sized window will lose the MSA's long-range dependency ability. To alleviate this issue, we add a 7 x 7 depth-wise convolution layer after the last block in each stage to enable information communicate across windows. We call this style CWin. In addition, [20] provides a hybrid approach (HWin) to integrate window information, i.e., computes MSA within a window in all but the last blocks in each stage that feed into FPN [21]. Window sizes in Win, CWin, and HWin are set as [64, 32, 16, 8] for the four stages.
fixed window
Table 3: Object detection results of fine-tuning styles on COCO val2017 with ResTv2-T using Mask RCNN. Inference "ms/iter" is measured on a V100 GPU, and FLOPs are calculated with Ik validation images.
(a) Object detection results.
(b) Detailed GFLOPs Analysis
Style
Params.
FLOPs ms/iter
Apbox
A P.maak
Style
Conv
Linear
Matmul
Others
Win
49.94MM 205.2G 149.6
43.95
40.42
Win
119.09
82.00
3.69
0.47
CWin
|49.66M 212.5G 150.7
44.07
40.44
CWin
126.29
82.00
3.69
0.47
HWin
49.94MM 218.9G
135.9
45.02
41.56
HWin
118.57
79.71
20.17
0.45
Global
.9,94M 229.7G 79.9
46.13
42.03
Global 116.95
75.70
36.66
0.42
Table 3(a) shows that although restricted EMSAv2 into fixed windows can effectively reduce theoreti- cal FLOPs, the actual inference speed is almost double the global style, and the box/mask AP is lower than the global one. Therefore, we adopt the Global fine-tuning strategy as default in downstream tasks to get better accuracy and inference speed.
There are predominantly two reasons for the decrease in inference speed: (1) padding to inputs is required to satisfy the divisible non-overlapped window partition. In our settings, the theoretical upper limit of padding in the first stage is 63 x 63, close to the lower bound of the input features'
8

---

size (i.e., 64 x 64), (2) the process of window partition is similar to feature grouping, which reduces the computational density of GPUs.
Table 3(b) shows the detailed FLOPs of different modules. We can see that window-based fine-tune methods can effectively reduce the "Matmul" (short of matrix multiply) FLOOSs with the cost of introducing extra "Linear" FLOPs, demonstrating that window partition padding is common in detection tasks. In addition, the "Matmul" operation not the most time-consuming part of the four settings (S 16%). Therefore, it is reasonable to speculate that window attention will reduce computational density.
We hope the observations and discussions can challenge some common beliefs and encourage people to rethink the relations between theoretical FLOPs and actual speeds, particularly running on GPUs.
Main Results. Table 4 shows main results of ResTv2 comparing with ConvNeXt [24], Swin Transformer [23], and traditional ConvNet such as ResNet [15]. Across different model complexities, ResTv2 outperforms Swin Transformer and ConvNeXt with higher mAP and inference FPS (frames second), particularly for tiny models. The mAP improvements over Swin Transformer are +1.6 box AP (47.6 vs. 46.0), and +1.6 mask AP (43.2 vs. 41.6). When comparing with ConvNeXt, the improvements are + +1.4 box AP (47.6 vs. 46.2), and +1.5 mask AP (43.2 vs. 41.7).
Table 4: COCO object detection and segmentation results using -CNN. We measure FPS on one V100 GPU. FLOPs are calculated with image size (1280, 800).
Backbones
Apoux
AP pmmskk
Params.
FLOPs
FPS
ResNet-50 [15]
41.0
37.1
44.2M
260G
24.1
ConvNeXt-T [24]
46.2
41.7
48.1M
262G
23,4
Swin-T [23]
46.0
41.6
47.8M
264G
21.8
ResTv2-T
47.6
43.2
49.9M
253G
25.0
ResNet-101 [15]
42.8
38.5
63.2M
336G
13.5
Swin-S [23]
48.5
43.3
69.1M
354G
17.4
ResTvv-S
48.1
43.3
60.7M
290G
21.3
ResTv2-B
48.7
43.9
75.5M
328G
18.3
5.2 Semantic Segmentation ADE20K
Settings. We also evaluate ResTv2 backbones on the ADE20K [49] semantic segmentation task with UperNet [39]. ADE20K contains broad range of 150 semantic categories. It has 25K images in total, with 20K for training, 2K for validation, and another 3K for testing. All model variants trained for 160k iterations with batch size of 16. Other experimental settings follow [23] (see Appendix C.2 for more details).
Table 5: ADE20K validation results using UperNet. Following Swin, we report mloU results with multiscale testing. FLOPs are based input sizes of (2048, 512).
Backbones
input crop.
mloU
Params.
FLOPss
FPS
ResNet-50 [15]
5122
42.8
66.5M
952G
23.4
ConvNeXt-T [24]
5122
46.7
60.2M
939G
19.9
Swin-T [23]
5122
45.8
59.9 M
941G
21.1
ResTv2-T
5122
47.3
62.1M
977G
22.4
ResNet- 101 [15]
5122
44.9
85.5M
1029G
20.3
ConvNeXt-S [24]
5122
49.0
81.9M
1027G
15.3
Swin-S [23]
512"
49.2
81.3M
1038G
14.7
ResTvv-SS
512"
49.2
72.9M
1035G
20.0
ResTy2-B
5122
49.6
87.6M
1095G
19.2
Results. Table
we report validation mloU with multi-scale testing. ResTv2 models can achieve competitive performance across different model capacities, further validating the effectiveness of

---

our architecture design. Specifically, ResTv2-T outperforms Swin-T T and ConvNeeXi-T with +1.5 and +0.7 mIoU improvements, respectively (47.3 vs. 45.8 vs. 46.7) with much higher FPS (22.4 vs. 21.1 vs. 19.9 images/s). As for larger models, the mloU improvements of ResTvv-B over Swin-S and ConyNeXt-B are +0.4 and +0.6 (49.6 vs. 49.2 vs. 49.0). The inference speed improvements are +30.6% and +25.5% (19.2 vs. 14.7 vs. 15.3 images/s).
6 Conclusion
In this paper, we proposed ResTv2, a simpler, faster, and stronger multi-scale vision Transformer for image recognition. ResTv2 adopts pixel-shuffle in EMSAv2 to reconstruct the lost information due to the downsampling operation. In addition, we explore different techniques for better apply ResTv2 to downstream tasks. Results show that the theoretical FLOPss is not a good reflection of actual speed, particularly running on GPUs. We hope that these observations could encourage people to rethink architecture design techniques that can actually prompt the network's ' efficiency.
Acknowledgments and Disclosure of Funding
This work is funded by the Natural Science Foundation of China under Grant No. 621761 19. We also greatly appreciate the help provided by our colleagues at Nanjing University, particularly Rao Lu, Niu Zhong-Hope Han, and Xu Jian.
References
[1] Jimmy Lei Ba. Jamie Ryan Kiros, and Geoffrey Hinton. Layer normalization. arXiv:1607.0644 2016.
arXiv preprint
[2] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usumier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European conference on computer vision,
ECCV2020, pages 213-229. Springer, 2020.
[3] Hanting Chen, Yunhe Wang. Tianyu Guo, Chang Xu, Yiping Deng, Zhenhua Liu, Siwei Ma, Chunjing Xu, Chao Xu, and Wen Gao. Pre-trained image processing transformer. In IEEE Conference Computer Vision and Pattern Recognition, CVPR 2021, pages 12299 12310. Computer Vision Foundation IEEE,
2021.
[4] Yinpeng Chen, Xiyang Dai. Dongdong Chen, Mengchen Liu, Xiaoyi Dong. Lu Yuan, and Zicheng Liu. Mobile-former: Bridging mobilenet and transformer. In International Conference on Computer Vision and Puttern Recognition, CVPR 2022, 2022.
[5] Xiangxiang Chu, Zhi Tian, Yuqing Wang, Bo Zhang, Haibing Ren, Xiaolin Wei, Huaxia Xia, and Chunhua Shen. Twins: Revisiting the design of spatial attention in vision transformers. In A. Beygelzimer, Y. Dauphin, P. Liang. and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems,
NewrPPS 2021, 2021.
[6] Xiangxiang Chu, Bo Zhang, Zhi Tian, Xiaolin Wei, and Huaxia Xia. Conditional positional encodings for vision transformers. arXiv preprint arXiv:2 10882, 2021.
[7] MMSegmentation Contributors. MMSegmentation:: Openmmlab semantic segmentation toolbox and tunsegment at ito. 2020.
benchmark. https: : /ttgtggtt/t tiithuu.b.. com/
[8] Ekin Dogus Cubuk, Barret Zoph, Jon Shlens, and Quoe Lc. Randaugment: Practical automated data augmentation with a reduced search space. In Advances in Neural Information Processing Systems,
NewrlPS 2020, 2020.
[9] Zihang Dai, Hanxiao Liu, Quoc V Le, and Mingxing Tan. Coatnet: Marrying convolution and attention for all data sizes. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, NeurlPS 2021, 2021.
[10] Xiaoyi Dong. Jianmin Bao, Dongdong Chen, Weiming Zhang, Nenghai Yu, Lu Yuan, Dong Chen, and Baining Guo. Cswin transformer: A general vision transformer backbone with cross-shaped windows. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, 2022.
[11] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly. Jakoh Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In 9th International Conference on Learning Representations. ICLR 2021. OpenReview.ne 2021.
[12] Haoqi Fan, Bo Xiong, Karttikeya Mangalam, Yanghao Li, Zhicheng Yan, Jitendra Malik, and Christoph Feichtenhofer. Multiscale vision transformers. In 2021 IEEE/CVF International Conference on Computer
Vision, ICCV 202/, pages 6804-6815. IEEE, 2021.
[13] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In International Conference on Computer Vision and Puttern Recognition. CVPR 2022, 2022.
10

---

[14] Kaiming He. Georgia Gkioxari, Piotr Dollár, and Ross B. Girshick. Mask R-CNN. In IEEE International Conference on Computer Vision, ICCV 2017, pages 2980-2988. IEEE Computer Society, 2017.
[15] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, pages 770-778. IEEE Computer Society, 2016.
[16] Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q. Weinberger. Deep networks with stochastic depth. In 14th European Conference on Computer Vision, ECCV 2016, volume 9908, pages 646-661.
Springer, 2016.
[17] Zilong Huang, Youcheng Ben, Guozhong Luo, Pei Cheng, Gang Yu, and Bin Fu. Shuffle transformer:
Rethinking spatial shuffle for vision transformer. arXiv preprint wttiv::206..066663 2021.
[18] Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey E. Hinton. Similarity of neural network representations revisited. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, volume 97. pages 3519-3529. PMLR, 2019.
[19] Kunchang Li. Yali Wang. Gao Peng, Guanglu Song, Yu Liu, Hongsheng Li, and Yu Qiao. Uniformer: Unified transformer for efficient spatial-temporal representation learning. In International Conference on Learning Representations, ICLR 2022, 2022.
[20] Yanghao Li, Chao- Yuan Wu, Haoqi Fan, Karttikeya Mangalam, Bo Xiong. Jitendra Malik, and Christoph Feichtenhofer. Improved multiscale vision transformers for classification and detection. arXiv preprint
arXiv:2 2021.
[21] Tsung- Yi Lin. Piotr Dollár, Ross B. Girshick, Kaiming He. Bharath Hariharan, and Serge J. Belongie. Feature pyramid networks for object detection. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 20/7, pages 936-944, 2017.
[22] Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning, Yue Cao. Zheng Zhang, Li Dong, et al. Swin transformer v2: Scaling up capacity and resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, 2022.
[23] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang. Stephen Lin, and Baining Guo. Swin Hierarchical vision transformer using shifted windows. In 2021 IEEF/CVF International Conference on Computer Vision, ICCV 2021, pages
transformer:
IEEE, 2021.
[24] Zhuang Liu, Hanzi Mao, Chao-Yuse Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xic. A convnet for the 2020s. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, 2022.
[25] Iya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019. OpenReview.net, 2019.
[26] Xuran Pan, Chunjiang Ge, Rui L.,, Shiji Song, Guanfu Chen, Zeyi Huang, and Gao Huang. On the integration self-attention and convolution. In Proceedings of the IEEE/CVF Conference on Computer
Vision and Pattern Recognition, CVPR 2022, 2022.
[27] Namuk Park and Songkuk Kim. How do vision transformers work? In International Conference on Learning Representations, ICLR 2022, 2022.
[28] Zhiliang Peng, Wei Huang. Shanzhi Gu, Lingxi Xie, Yaowei Wang, Jianbin Jiao, and Qixiang Ye. Con- former: Local features coupling global representations for visual recognition. In 2021 IEEF/CVF Interna- tional Conference on Computer Vision, ICCV 2021, pages 357-366. IEEE, 2021.
[29] Boris T Polyak and Anatoli B Juditsky. Acceleration of stochastic approximation by averaging. SIAM journal on control and optimization, 30(4):838-855 1992.
[30] Iija Radosavovic, Raj Prateek Kosaraju. Ross B. Girshick, Kaiming He, and Piotr Dollár. Designing network design spaces. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. CVPR 2020, pages 10425-10433. IEEE, 2020.
[31] Aravind Srinivas, Tsung-Yi Yi Lin, Niki Parmar, Jonathon Shlens, Pieter Abbeel, and Ashish Vaswani. Bottleneck transformers for visual recognition. In IEFE Conference on Computer Vision and Pattern Recognition, CVPR 2021. pages 16519-16529, Computer Vision Foundation / IEEE, 2021.
[32] Christian Szegedy, Vincent Vanhoucke, Sergey loffe. Jonathon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, pages 2818-2826. IEEE Computer Society, 2016
[33] Dmitry Ulyanvv, Andrea Vedaldi, and Victor S. Lempitsky. Instance normalization: The missing ingredient
for fast stylization. arXiv preprint IwXiv:1607.0000
2016.
[34] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing
Systems, NeuriPS 2017, pages 5998-6008. 2017.
[35] Wenhai Wang, Enze Xic, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, pages 548 558. IEEE,
2021.
[36] Haiping Wu, Bin Xiao, Noel Codella, Mengehen Liu, Xiyang Dai, Lu Yuan, and Lei Zhang. Cvt: Introducing convolutions to vision transformers. In 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, pages 22-31. IEEE, 2021.
[37] Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan- Yen Lo, and Ross Girshick, Detectron2, ht https: 2019.
11

---

[38]
Tete Xiao, Piotr Dollar, Mannat Singh, Eric Mintun, Trevor Darrell, and Ross Girshick. Early convolutions help transformers see better. In A. Beygelzimer. Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advancey in Neural Information Processing Systems, NeurlPS 2017, 2021.
[39] Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understanding. In 15th European Conference on Computer Vision, ECCV 2018, volume 11209, pages
432-448. Springer, 2018.
[40]
Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: A simple framework for masked image modeling. In International Conference on Computer Vision and Partern Recognition, CVPR 2022, 2022.
[41] Jianwei Yang. Chunyuan Li, Pengchuan Zhang, Xiyang Dai, Bin Xiao, Lu Yuan, and Jianfeng Gao. Focal attention for long-range interactions in vision transformers. In A. Beygelzimer, Y. Dauphin, P. Liang, and Wortman Vaughan, editors, Advances in Neural Information Processing Systems, NeurlPS 2021, 2021. [42] Qihang Yu, Yingda Xia, Yutong Bai, Yongyi Lu,, Alan Yuille, and Wei Shen. Glance-and- gaze vision transformer. In A. Beygelzimer,
Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Newral Information Processing Systems, NeurlPS 2021, 2021.
[43] Li Yuan, Yunpeng Chen, Tao Wang, WeiHao Yu, Yujun Shi, Zihang Jiang, Francis E. H. Tay, Jiashi Feng, and Shuicheng Yan. Tokens-to-token vit: Training vision transformers from scratch on imagenet. In 202/ IEEE/CVF International Conference Computer Vision, ICCV 2021, pages 538-547. IEEE, 2021. [44] Sangdoo Yun. Dongyoon Han, Sanghyuk Chun, Seong Joon Oh, Youngjoon Yoo. and Junsuk Choe. Cutmix: Regularization strategy train strong classifiers with localizable features. In 2019 JEEE/CVF International Conference on Computer Vision, ICCV 2019, pages 6022 6031. IEEE. 2019.
[45] Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers. arXiv
preprint or Xiv:22100 04560, 2021.
[46] Hongyi Zhang. Moustapha Cissé, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In 6th International Conference on Learning Representations, ICLR 2018. OpenReview.net,
2018.
[47] Qinglong Zhang and Yu hin Yang. Rest: An efficient transformer for visual recognition. In A. Beygelzimer, Y. Dauphin. P. Liang. and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, NeurlPS 2021, 2021.
[48] Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang. Random crasing data augmentation. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, pages 13001 10000-13000 13008. AAAI Press, 2020.
[49] Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao. Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ADE20K dataset. Int. J. Comput. Vis., 127(3):302-322) 2019.
[50] Xizhou Zhu, Waijie Su, Lewei Lu, Bin Li, Xiangang Wang, and Jifeng Dai. Deformable DETR: deformable transformers for end-to-end object detection. In 9th International Conference on Learning Representations, ICLR 2021. OpenReview.net, 2021.
Checklist
The checklist follows the references. Please read the checklist guidelines carefully for information on how to answer these questions. For each question, change the default [TODO] [Yes], [No] or [N/A]. You are strongly encouraged to include a justification to your answer, either by referencing the appropriate section of your paper or providing a brief inline description. For example:
Did you include the license to the code and datasets? [Yes] See Section.
Did you include the license the code and datasets? [No] The code and the data proprietary.
Did you include the license to the code and datasets? [N/A]
Please do not modify the questions and only use the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questionssansswrs below.
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [N/A]
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them? [Yes]
12

---

2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)? [Yes] In https: //t////// /gtthub. com/ wo fmanaf/Res:
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they chosen)? [Yes] In the Appendix.
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [N/A]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] In the Appendix.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "x5Ths3AKScc", "reviewer": "Reviewer_TbMq", "summary": "This paper proposes a vision transformer network based on the previously proposed ResT-V1 architecture by involving a new element that performs upsampling (of a token sequence). The upsampling module is actually attached to the projected output of value to recover the dimensionality of the original sequence length. The authors additionally refine the baseline ResT-V1 network 1) removing the multi-head interaction components; and 2) reconfiguring the network stage due to the appearance of the upsampling module.  The modifications seem to work well but the key design choice still remain. The authors provide experimental results on ImageNet and some downstream tasks to justify the effectiveness of the proposed network architecture.", "questions": "- What is the meaning of the computational density? at line 109 in p3. \n\n- Is the LN after DwConv in EMSAv1 removed in EMSAv2? \n\n- How does the upsampling change the latency? Please specify the latency or throughput in Table 2 (b)\n\n- The difference in the results between the target of K and V in Table 2 (a) is slight, so it should be verified with the results from multiple runs.\n\n- Table 2 (c) should contain the results with the models having a similar number of parameters of the baselines; can the authors provide the models have similar parameters with EMSA and EMSAv2? ", "limitations": "The authors did not provide the limitations and potential negative societal impact of their work.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n- The paper is easy to follow\n- The exploration of using pixel-shuffle as the upsampling model for recovering the original signal is a nice idea, and interpreting it as the convolutional hourglass architecture also seems to make sense.\n\n\n### Weaknesses\n- Novelty is incremental. The major change over the baseline ResTv1 is the pixel-shuffle only, and the rest of the modifications are not new and cannot be one of the contributions. \n- Any intuitions or insights of why the architecture should be designed like this are missing: why the upsampling module should be involved? What can we learn from the architectural modifications from ResTv1 to RestV2 such as the block number at the first stage is halved.\n- Experimental justifications in Sections 3.4 and 4.3 do not seem to be enough backups for the explanation of the proposed architectural design. For example, Figure 3 tells us the upsampling module seemingly reduces the difference in the log amplitude between particular frequencies and the center frequency. However, this does not indicate the upsampling module is necessarily used. Furthermore, one may naturally ask the questions: 1) why do some specific frequencies only benefit from information recovery?; 2) If the upsampling module really helps information flow, shouldn't the entire frequency have the same effect?; 3) why the output-side layers do not benefit from it? Furthermore, Figure 4 is not clearly illustrated.\n\n- The details of Pixel-shuffle are not clearly presented. Is it the pixel-shuffle operation used in the super-resolution field? Then, why the dimensionality remains the same after upsampling in Figure 2. (b)?\n\n ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "s6dwonP-qQJ", "reviewer": "Reviewer_cW5j", "summary": "The paper modifies the original attention block in ResT paper by 1. Rewind conv+IN based multi-head attention in ResT to original multi-head attention form (no conv, IN) in ViT. 2. Add upsampling operation to the downsampled V. Image classification performance is validated on ImageNet. Besides, the paper conducted ablation studies on how different window-based attention (Win, Cwin, Hwin, Global) affect AP, FLOPs, Throughput when transfer to object detection task. \n", "questions": "1.\tThe target is to recover the lost information in the K/V downsampling process by using the proposed upsampling operation. We already have the residual path that adds the original X (no information loss from resolution change), difference here is that this method adds conv/linear transformed, downsampled-then-upsampled X back. Curious to see the performance if we just add back non-downsampled linear/conv projected V instead of the downsampled-then-upsampled one?\n2.\tIf applying the proposed upsampling to other work that downsample K/V for saving computation (e.g PVT), can we see performance improvements as well? \n\n\n---post-rebuttal---\nThe authors' response has partially resolved my concerns, specific they have presented the upsampling operation can generalize to other ViT variants, I changed my rating towards accept. ", "limitations": "The paper proposes specific change to ResT, the change might benefit other methods, but not validated in the work. ", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\tThe paper presents results across a range of different vision tasks, including ImageNet classification, object detection and segmentation on COCO, semantic segmentation on ADE20k and achieves competitive results. \n\tBesides, FLOPs and parameters, Throughput is also benchmarked to make give one more practical aspect of model performance. \n\nWeaknesses:\n\tNovelty is too limited. At core, what the paper proposed is to upsample k/v, where k/v are downsampled for saving computations. Current presentation is more like a patch fix to ResT. If the generalization abilities of the proposed upsampling operation beyond ResT is validated, as there exist many other ViT variants saving computations by downsampling (e.g. PVT), the proposed method can be much strengthened. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Je7SdHDvcSi", "reviewer": "Reviewer_21zc", "summary": "This paper proposes an improved vision Transformer architecture based on the ResTv1. To address the issue that the downsample operation in ResTv1 will impair the long-distance modeling ability, an upsample operation is employed to reconstruct the lost information. Besides, authors apply the proposed ResTv2 to different downstream tasks. Extensive results demonstrate the effectiveness and efficiency of this Transformer backbone.", "questions": "Please explain the aforementioned three weakness issues (especially the 1-st issue about the generalizability)", "limitations": "Yes, the authors address the efficiency to some extent. They point out the gap between the theoretical FLOPs and the actual speed, and consider the actual running speed more when designing the model.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n1) The proposed architecture is simple but seems effective for different downstream tasks. \n2) It is interesting that the authors find the gap between the theoretical FLOPs and the actual speed, and they consider the actual speed when designing the model. \n3) The experiments and theoretical analysis are sufficient.\n\nWeaknesses:\n1) The main contribution of this paper is to introduce the upsample operation into the ResTv1 to compensate for the lost information from the downsample. Though this simple design can provide the performance benefit, the generalizability of this design seems narrow, does it only work well for the specific efficient Transformer model with the downsample operation? \n2) For Equ.2, is the Norm operation also eliminated along with the Conv? If so, the paper does not mention it, which is somewhat confusing. 3) I think a more detailed description of Fig.3 is needed for a better understanding, e.g., the meaning of coordinates and curves.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept