Expediting Large-Scale Vision Transformer for Dense Prediction without Fine-tuning
Authors: Weicong Liang
Yuhui Yuan
Henghui Ding
Xiao Luo
Weihong Lin
Ding Jia
Zheng Zhang
Chao Zhang
Han Hu
Venue: NeurIPS 2022 Accept (2022)
Keywords: Transformer
High-Resolution
Semantic Segmentation
Depth Estimation
Classification
Efficient Architecture

Abstract: Vision transformers have recently achieved competitive results across various vision tasks but still suffer from heavy computation costs when processing a large number of tokens. Many advanced approaches have been developed to reduce the total number of tokens in the large-scale vision transformers, especially for image classification tasks. Typically, they select a small group of essential tokens according to their relevance with the [\texttt{class}] token, then fine-tune the weights of the vision transformer. Such fine-tuning is less practical for dense prediction due to the much heavier computation and GPU memory cost than image classification.

In this paper, we focus on a more challenging problem, \ie, accelerating large-scale vision transformers for dense prediction without any additional re-training or fine-tuning. In response to the fact that high-resolution representations are necessary for dense prediction, we present two non-parametric operators, a \emph{token clustering layer} to decrease the number of tokens and a \emph{token reconstruction layer} to increase the number of tokens. The following steps are performed to achieve this: (i) we use the token clustering layer to cluster the neighboring tokens together, resulting in low-resolution representations that maintain the spatial structures; (ii) we apply the following transformer layers only to these low-resolution representations or clustered tokens; and (iii) we use the token reconstruction layer to re-create the high-resolution representations from the refined low-resolution representations. The results obtained by our method are promising on five dense prediction tasks including object detection, semantic segmentation, panoptic segmentation, instance segmentation, and depth estimation. Accordingly, our method accelerates $40\%\uparrow$ FPS and saves $30\%\downarrow$ GFLOPs of ``Segmenter+ViT-L/$16$'' while maintaining $99.5\%$ of the performance on ADE$20$K without fine-tuning the official weights.

Expediting Large-Scale Vision Transformer for Dense Prediction without Fine-tuning
Weicong Liang'
Yuhui Yuan**i Henghui Ding'
Xiao Luo Han Hu
Weihong Lin Ding Jia'
Zheng Zhang Chao Zhang'
Key Laboratory of Machine Perception (MOE) School of Intelligence Science and Technology, Peking University School of Mathematical Sciences, Peking University EETH Zurich *Microsoft Research Asia
Abstract
Vision transformers have recently achieved competitive results across various vision tasks but still suffer from heavy computation costs when processing a large number of tokens. Many advanced approaches have been developed reduce the total number of tokens in large-scale vision transformers, especially for image classification tasks. Typically, they select small group of essential tokens according to their relevance with the [class] token, then fine-tune the weights of the vision transformer. Such fine-tuning is less practical for dense prediction due to the much heavier computation and GPU memory cost than image classification. In this paper, we focus vision transformers for dense prediction without any additional re-training or fine- tuning. In response the fact that high-resolution representations for dense prediction, we present two non-parametric operators, a loken clustering layer decrease the number of tokens and token reconstruction layer increase the number of tokens. The following steps are performed to achieve this: (i) we use the token clustering layer to cluster the neighboring tokens together, resulting in low-resolution representations that maintain the spatial structures; (ii) we apply the following transformer layers only these low-resolution representations clustered tokens; and (iii) we use the token reconstruction layer to re-create the high-resolution representations from the refined low-resolution representations. The results obtained by our method are promising on five dense prediction tasks, including object detection, semantic segmentation, panoptic segmentation, instance segmentation, and depth estimation. Accordingly, our method accelerates 40% + FPS and saves 30% J. GFLOPs of Seggmentee+VVi--... while maintaining 99.5% of the performance on ADE20K without fine-tuning the official weights.
challenging problem, i.e., accelerating large-scale
necessary
Introduction
Transformer (66] has made significant progress across various challenging vision tasks since pi-
oneering efforts such as DETR [4) Vision Transformer (ViT) [17], and Swin Transformer [46]. By removing the local inductive bias [18 from convolutional neural networks 27 63 59), vi-
sion transformers armed with global self-attention show superiority scalability for large-scale
models and billion-scale dataset [17 82 60), self-supervised learning 226 74 []. connecting vi-
sion and language [52 33),
etc. We can find from recent developments of SOTA approaches that
"qqual contribution.
R yuhui. yuan @ microsoft.com
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

vision transformers have dominated various leader-boards, including but not limited to image clas-
sification 72, 16 82), object detection
84 45 37 semantic segmentation 31 pose
estimation 75 image generation 83 and depth estimation
[1
Although vision transformers have achieved more accurate predictions in many vision tasks, large- scale vision transformers are still burdened with heavy computational overhead, particularly when processing high-resolution inputs 19 451, thus limiting their broader application more resource- constrained applications and attracting efforts on re-designing light-weight vision transformer ar- chitectures [10 50, 85 In addition to this, several recent efforts have investigated how to decrease the model complexity and accelerate vision transformers, especially for image classification, and introduced various advanced approaches accelerate vision transformers. Dynamic ViT [54] and EVIT [43]. for example, propose two different dynamic token sparsification frameworks to reduce the redundant tokens progressively and select the most informative tokens according to the scores predicted with an extra trained prediction module or their relevance with the [class] token. To- kenLearner [57] learns to spatially attend over a subset of tokens and generates a set of clustered tokens adaptive to the input for video understanding tasks. Most of these token reduction approaches are carefully designed for image classification tasks and require fine-tuning or retraining. These approaches might not be suitable to tackle more challenging dense prediction tasks that need to process high-resolution input images, e.g., 1024 x 1024, thus, resulting in heavy computation and GPU memory cost brought. We also demonstrate in the supplemental material the superiority of our method over several representative methods on dense prediction tasks.
Rather than proposing new lightweight architecture for dense prediction or token reduction scheme for only image classification, we focus on how to expedite well-trained large-seale vision transformers and use them for various dense prediction tasks without fine-tuning re-training. Motivated by these two key observations including (i) the intermediate token representations of a well-trained vision transformer carry a heavy amount of local spatial redundancy and (ii) dense prediction tasks require high-resolution representations, we propose simple yet effective scheme to convert the "high-resolution" path of the vision transformer to "high-to-low-to-hhe non-pan con-paramttric layers including a token clustering layer and token reconstruction layer. Our method can produce wide range of more efficient models without requiring further fine-tuning or re-training. We apply our approach to expedite two main-stream vision transformer architectures, c.g., VITs and Swin Transformers, for five challenging dense prediction tasks, including object detection, semantic segmentation, panoptic segmentation, instance segmentation, and depth estimation. We have achieved encouraging results across several evaluated benchmarks and Figure illustrates some representative results on both semantic segmentation and depth estimation tasks.
resolution" path via two
2 Related work
Pruning Convolutional Neural Networks. Convolutional neural network pruning 2 29 69 is a task that involves removing the redundant parameters to reduce the model complexity without a significant performance drop. Pruning methods typically entail three steps: (i) training a large, wver-parameterized model to convergence, (ii) pruning the trained large model according to a certain criterion, and (iii) fine-tuning the pruned model to regain the lost performance [48]. The key idea is to design an importance score function that capable of pruning the less informative parameters. We follow [7] to categorize the existing methods into two main paths: (i) unstructured pruning (also named weight pruning) and (ii) structured pruning. Unstructured pruming methods explore the absolute value of each weight the product of each weight and its gradient estimate the importance scores. Structured pruning methods, such as layer-level pruning [70]. filter-level pruning 47 77). and
image-level pruning 24 64) removes the model nub-structures. Recent studies 5 6 30 further
extend these pruning methods to vision transformer. Unlike the previous pruning methods, we explore how to expedite vision transformers for dense prediction tasks by carefully reducing & increasing the number of tokens without removing or modifying the parameters.
The success of vision transformers has incentivised many recent 43 56] to exploit the spatial redundancies of intermediate token representations. For example, TokenLearner [57] learns to attend over a subset of tokens and generates a set of clustered tokens adaptive to the input. They empirically show that very few clustered tokens are sufficient for video understanding tasks. Token Pooling [49] exploits a nonuniform data-aware down-sampling operator based on K-Means or K-medoids to cluster similar
Efficient Vision Transformer.
efforts 57
61
49 28
54
36
23
67
9 34
2

---

1.00
NCCALLCaano
INCCLL-Canoo
(a) mlo Segmenter
(b) FPS@ Segmenter
(c) GFLOPs& Segmenter
20
1004
114
(d) RMSE@DD DPT
(e) FPS @ DPT
(I) GFLLOP@DDP
Figure 1: Illustrating the improvements of our approach: we report the results of applying our approach to Segmenter [62] for semantic segmentation and DPT [33] for depth estimation on the 1-st and 2-ed row respectively. Without any fine-tuning, our proposed method reduces the GFLOPs and accelerates the FPS significantly with slight performance drop on both dense prediction tasks.
and J. represent higher is better and lower is better respectively. Refer to Section 4 for more details.
tokens together to reduce the number of tokens while minimizing the reconstruction error. Dynamic ViT 544 observes that the accurate image recognition with vision transformers mainly depends on subset of the most informative tokens, and hence it develops dynamic token sparsification framework for pruning the redundant tokens dynamically based on the input. EViT (expediting vision transformers) 43) proposes to calculate the attentiveness of the [class] token with respect to each token and identify the top-k attentive tokens according to the attentiveness score. Patch Merger (55] uses a learnable attention matrix to merge and combine together the redundant tokens, therefore creating a much more practical and cheaper model with only a slight performance drop. Refer to 665 for more details on efficient transformer architecture designs, such as Performer [12] and Reformer 35 In contrast to these methods that require either retraining or fine-tuning the modified transformer architectures from scratch the pre-trained weights, approach reuse the once-trained weights for free and produce lightweight models with a modest performance drop.
Vision Transformer for Dense Prediction. In the wake of success the representative pyramid
vision transformers 46 68 for object detection and semantic segmentation, more and more efforts
have explored different advanced vision transformer architecture designs [38
39
40
20
79 76
86 22 42
78
15
81
80 25 suitable for various dense prediction tasks. For example, MVIT (39
focuses more on multi-scale representation learning, while Former (79] examines the henefits of combining multi-scale representation learning and high-resolution representation learning. Instead of designing a novel vision transformer architecture for dense prediction, we focus on how to accelerate a well-trained vision transformer while maintaining the prediction performance as much as possible.
Our approach. The contribution of our work lies in main aspects: (i) we the first study how to accelerate state-of-the-art large-scale vision transformers for dense prediction tasks without fine-tuning (c.g., "Mask2Former Swin-L" and "SwinV2-L- + HTC++"), Besides, our approach also achieves much better accuracy and speedup trade-off when compared to the very recent ACT [1] which based on a clustering attention scheme; (ii) our token clustering and reconstruction layers are capable of maintaining the semantic information encoded in the original high-resolution representations. This the very most important factor to avoid fine-tuning. We design an effective combination of token clustering function and a token reconstruction function to maximize the cosine similarity between the reconstructed high-resolution feature maps and the original ones without fine-tuning. The design of our token reconstruction layer is the key and not straightforward essentially. We also show that our token reconstruction layer can be used to adapt the very recent EVIT 43 and Dynamic VIT (54) for dense prediction tasks in the supplementary.
3

---

Repeat Lx (a)
Repeat ax..
Repeat
Regost 7X
(b)
Layes
Toles Clastering
Transf Transfomme cormer Layer
Token Recomstrnction
Transformet Layer
Tadesspeciii Head
Ingut Imagge
hxwx x
W
W
Repoat ox
Repear 8x
Repeat 7XX
(c)
Figure 2: (a) Plain high-resolution vision transformer with L layers. (b) U-shape High-to-low- to-high-resolution vision transformer with a, B, and ry layers respectively (L = a + 8 7)- (c) Illustrating the details of using our approach to plain ViTs: insert token clustering layer and token reconstruction layer into a trained vision transformer in order to decrease and then increase the spatial resolution, respectively. The weights of modules marked with are trained once based on the configuration (a). The token clustering layer and token reconstruction layer are marked with r are non-parametric, thus do not require any fine-tuning and can be included directly during evaluation.
Our Approach
Preliminary. The conventional Vision Transformer [17 first reshapes the input image X E RHxWx3 into sequence of flatten patches Xp (P233) where (P, P) represents the resolution of each patch, (H, W) represents the resolution of the input image, N (H x W)/P² represents the number of resulting patches or tokens, i.e., the input sequence length. The Vision Transformer consists al- ternating layers of multi-head self-attention (MHSA) and feed-forward network (FFN) accompanied with layer norm (LN) and residual connections:
Z = MHSA (LN (Z(-1)) + Z
(1)
Z = FFN [NN (z)) + ZZj
where E (1....,,L
L} represents the layer index, Z; E NxC and Zo is based on X,- The computa- tion cost (LLNC(N + C)) mainly depends on the number of layers L, the number of tokens N, and the channel dimension
Despite the great success of transformer, its computation cost increases significantly when handling high-resolution representations, which are critical for dense prediction tasks. This paper attempts to resolve this issue by reducing the computation complexity during the inference stage, and presents very simple solution for generating a large number efficient vision transformer models directly from single trained vision transformer, requiring further training fine-tuning.
We demonstrate how our approach could be applied to the existing standard Vision Transformer in Figure 2 The original Vision Transformer is modified using two non parametric operations, namely token clustering layer and a token reconstruction layer. The proposed token clustering layer is utilized to convert the high-resolution representations to low-resolution representations by clustering locally semantically similar tokens. Then, we apply the following transformer layers on low-resolution representations, which greatly accelerates the inference speed and saves computation resources. Last, token reconstruction layer proposed reconstruct the feature representations back to high- high-resolution.

---

Token Clustering Layer. We construct the token clustering layer following the improved SLIC scheme 32). which performs local k-means clustering follows:
Initial superpixel center: We apply adaptive average pooling (AAP) over the high-resolution representations from the a-th layer compute the h x initial cluster center representations:
Sa = (Z. (h x w)),
where S. E hR Za RNC and hw <
-Iterative local clustering: (i) Expectation step: compute the normalized similarity between each pixel p and the surrounding superpixel (we only consider the neighboring À positions), (ii) Maximization step: compute the new superpixel centers:
exp (-llza.p Sa.:11/TT)
N Sa,i Σ p=1
Pp.
Op.i
Σ5=1 exp (isp--1zzon) -liZzo.p Sa.119/T)
where we iterate the above Expectation step and Maximization step for A times, T is a temperature hyper-parameter, and i f1,2, {}. We apply the following 3 transformer layers S. instead of Z.. thus results in Sa+8 and decreases the computation cost significantly.
Token Reconstruction Layer. We implement the token reconstruction layer by exploiting the relations between the high-resolution representations and the low-resolution elustered representations:
exp(-I|Za.p Sa.i //T
Z Z.+
Es Es..,cckNN(Z. exp( (up--IlZa) Za.p - 11/////T) Sat3.i
EL,NNN.
where 7 is the same temperature hyper-parameter in Equation 3 k-NN(Za.,) represents set of the k nearest, a.k.a, most similar, superpixel representations for Za We empirically find that choosing the same neighboring positions in Equation achieves close performance as the k-NN scheme while being more easy to implementation.
In summary, we estimate their semantic relations based on the representations before refinement with the following 3 transformer layers and then reconstruct the high-resolution representations from the refined low-resolution clustered representations accordingly.
Finally, we apply the remained y transformer layers the reconstructed high-resolution features and the task-specific head on the refined high-resolution features to predict the target results such as semantic segmentation maps monocular depth maps.
Extension to Swin Transformer. We further introduce the window token clustering layer and window token reconstruction layer, which are suitable for Swin Transformer [45]
46 Figure 3 illustrates an example usage of the proposed window token clustering layer and window token reconstruction layer. We first cluster the K x K window tokens into k x k window tokens and then reconstruct K K window tokens according to the refined k x window tokens. We apply the swin transformer layer equipped with smaller window size k x k on the clustered representations, where we need to bi-linear interpolate the pre-trrieed weights relative position embedding table from (2K - 1)x (2K - 1) to (2k x)x (2k- 1) when processing the clustered representations. In summary, efficiency of Swin Transformer by injecting the window token clustering layer and the window token reconstruction layer into the backbones seamlessly without fine-tuning the model weights.
improve the
Why our approach can avoid fine-tuning? The reasons include the following two aspects: (i) our token beastrrinnreeoosstuutiie layers are non- non-parametric. thus avoiding retraining any additional parameters, (ii) the reconstructed high-resolution representations maintain high semantic similarity with the original high-resolution representations. We take Segmenter+ ViT-L/11 (on ADEK, a=10) as an example and analyze the semantic similarity between the reconstructed high-resolutie (with our approach) and the original high-resolution feature (with the original ViT-L/16) in Table [] Accordingly, we can see that the cosine similarities are consistently high across different transformer layers between the reconstructed high-resolution feature (with our approach) and the original high- resolution feature. In other words, our approach well maintains the semantic information carried in the original high-resolution feature maps and thus is capable of avoiding fine-tuning.
feature
5

---

Tokmm Chuslaring
apply Swiul Swisl Tannt Praaaaarma Layer
Tolor Rocomater www.onaatttiiii Too
SwicTTanndorun Lapvr
Tokm Chaneinng
Swih manforntoommm
Token Clusteting
Talen Clastring
Tobm Recousteuriinn
kxkxx
Repratax 0x
Kxkxc
Repost
KxKxC
Repeot
Figure 3: Illustrating the details of using our approach for Swin Transformer 45 46 four groups token clustering layer and token reconstruction layer within the four non-overlapped windows marked with clustering layer and window token reconstruction layer. We apply the intermediate swin transformer layers equipped with window size k k) the clustered window tokens. Window sizes before and after token clustering layer are K x K and k x k, and vice versa, for the token reconstruction layer.
and respectively, in the example referred to window token
Table 1: Ablation of the cosine similarity between the reconstructed high-resolution feature maps and the original high-resolution feature maps, where Zoriginal and Za+ represents the original and reconstructed high-resolution feature maps, respectively.
a+3
12
14
16
18
20
22
24
cos(Za+ Zate
0.94
0.95
0.96
0.96
0.96
0.96
0.96
Experiment
We verify the effectiveness of our method across five challenging dense prediction tasks, including object detection, semantic segmentation, instance segmentation, panoptic segmentation, and monoc- ular depth estimation. We carefully choose the advanced SOTA methods that build the framework based on either the plain ViTs [17 the Swin Transformers
We integrate the proposed token clustering layer and token reconstruction layer seamlessly with the provided official trained checkpoints with no further fine-tuning required. More experimental details are illustrated follows.
45 46
4.1 Datasets
This dataset consists of 123K images with 896K annotated bounding boxes belonging to 80 thing classes and 53 stuff classes, where the train contains 118K images and the val set contains 5K images. We report the object detection performance of SwinV2 + HTC++ and the instance/panoptic segmentation performance of Fask2Former [1]] on the val set.
COCO
ADE20K This dataset contains challenging scenes with fine-grained labels and one of the most challenging semantic segmentation datasets. The train set contains 20, 210 images with 150 semantic classes. The val set contains 2, 0.0 images. We report the segmentation results with
Segmenter 622 on the val set.
PASCAL-Context [5I). This dataset consists of 59 semantic classes plus background class, where the train set contains 4, 996 images with and the val set contains 5, 104 images. We report the segmentation results with Segmenter (62) on the val set.
This dataset is an urban scene understanding dataset with 30 classes while only 19 classes used for parsing evaluation. The train and val set contains 975 and 500 images respectively. We report the segmentation results with Segmenter 62 on the val set.
Cityscapes 13
KITTI 21 This dataset provides stereo, optical flow, visual odometry (SLAM), and 3D object detection of outdoor scenes captured by equipment mounted on a moving vehicle. We choose the monocular depth prediction tasks, which consists of around 26K images for train set and 698 images for val set, where only 653 images have the ground-truth depth maps and the image resolution is of 1, 241 376.
DPT (53 as baseline conduct experiments
This dataset consists of 1, 449 RGBD images with resolution 640 x 480, which captures 464 diverse indoor scenes and contains rich detailed dense annotations such surface normals, segmentation maps, depth, 3D planes, and so on. We report the depth prediction results of DPT [53 evaluated on 655 val images.
NYUv2 58

---

Table 2: Influence of hyyeer-parameterr of token clusterringgreccosstutciii layer.
k
Parameter
3
x
X
10
15
20
10
25
50
75
10
20
50
GFLOPs
438.6
438.9
439.6
438.9
439.6
440.2
440.8
438.9
438.9
438.9
439.0
mloU
51.46
51.56
51.55
51.56
51.59
51.62
51.62
51.18
51.35
51.56
51.07
51.34
51.56
51.56
Table
Influence the cluster size h x w.
Cluster size
8 8
12 T2
16 x 16
20 x 20
24 x 24
28 28
40 x 40 (bascline)
GFLOPs
274.0
290.9
315.1
347.2
388.2
438.9
659.0
mloU
32.13
44.01
48.21
50.17
51.32
51.56
51.82
Table Comparison with adap-Table Table 5: Comparison with bi-Tab 6: Combination with lighter vision transformer backbone.
tive average pooling(AAP)
linear upsample.
Cluster size 20 x 20
28 28
Cluster size 20 x 20
28
Cluster size
mlou
GFLOPs
AAP
46.45
46.54
Bi-linear
44.68
44.74
Segmenter+ ViT-B/16 ViT--B//
48.48
124.7
Ours
50.17
51.56
Ours
50.17
51.56
Segmenter+ ViT-B/16++Urss
48.40
91.9
(a) mloti
(h) GALOPS
(c)
FPS
(a)
mloU
(b)
GRLOP
(c) FPS
Figure 4: Influence of the inserted position of token Figure 5: Influence of the inserted position + of token reconstruction layer.
clustering layer.
4.2 Evaluation Metrics
We report the numbers of AP (average precision), mask AP (mask average precision), PQ (panoptic quality), mloU (mean intersection-over-nnin)) and RMSE (root mean squared error) across object detection, instance segmentation, panoptic segmentation, semantic segmentation, and depth estimation tasks respectively. Since the dense prediction tasks care less about the throughput used in image classification tasks [43]. we report FPS to measure the latency and the number of GFLOPs to measure the model complexity during evaluation. FPS is tested on a single V100 GPU with Pytorch 1.10 and CUDA 10.2 by default. More details are provided in the supplementary material.
4.3 Ablation Study Experiments
We conduct the following ablation experiments on ADE20K semantic segmentation benchmark with the official checkpoints of Vegmenter+ViTT-L/16 62 by default if not specified.
Hyper-parameters token chustering/reconsttuctoon layer. We first study the influence of the hyper-parameters associated with the token clustering layer, i.e., the number of neighboring pixels X used Equation 3 the number EM iterations Æ, and the choice of the temperature 7 in Table 2 see that method is relatively less sensitive to the choice of
According the results, both À and x compared to т. In summary, we choose À as 5 x 5, A as 5, and T as 50 considering both performance and efficiency. Next. we also study the influence of the hyper-parametest within the token clustering layer, i.c., the number of nearest neighbors k within k-NN. We do not observe obvious differences and thus set k as 20. More details provided in the supplementary material.
Influence of cluster size choices. We study the influence of different cluster size x w choices based input feature map of size H x results, we can see that choosing too small cluster sizes significantly harms the dense prediction performance, and setting x W as 28 28 achieves the better trade-off between performance drop and model complexity. Therefore, we choose 28 x 28 on Seenenter+ViT-L//N/ by default. We also
40 x 40 (N 1, 600 in Table 3 According to the
pattpa://giti tht.. cowp//sttrualll/gaaaattttat
MIT License
We choose H x W 640 640 and 16, thus, F 40 40 N -,1 600, ADE20K.
7

---

empirically find that selecting the cluster size h x w around N/4 ~ N/2 performs better on most of the other experiments. We conduct the following ablation experiments under two typical settings, including 20 x 20 (~ N/4) and 28 x 28 (~ N/2).
Comparison with adaptive average pooling and bi-linear upsample. We report the comparison results between our proposed token clustering/reconsttutiii scheme and adaptive average pooling/bi- linear upsample scheme Table and Table 5 under two cluster size settings respectively. We choose to compare with adaptive average pooling and bi-linear upsample instead of strided convolution or deconvolution as the previous ones are non non-parametric and the later ones require re-training or fine-tuning, which are not the focus of this work. Specifically. we keep the inserted position choices the same and only replace the token cluster or token reconstruction layer with adaptive average pooling or bi- bi-linear upsampling under the same cluster size choices. According to the results, we can see that our proposed token clustering and token reconstruction consistently outperform adaptive average pooling and bi-linear upsampling under different cluster size choices.
Influence of inserted position of token clustering/reconstruuce
layer. We investigate the influence of the inserted position of both token clustering layer and token reconstruction layer and summarize the detailed results in Figure 4 and Figure 3 under two different cluster size choices. According to the results shown in Figure 4 our method achieves better performance when choosing larger than 10, therefore, we choose
10 as it achieves a better trade-off between model complexity and segmentation accuracy. Then we study the influence of the inserted positions of the token reconstruction layer by fixing a 10. According to Figure [ we can see that our method achieves the best performance when setting a + B = 24, in other words, we insert the token reconstruction layer after the last transformer layer of ViT-L/16. We choose a = 10, a + 3 24,
and y = 0 for all ablation experiments on ADE20K by default if not specified.
Combination with lighter vision transformer architecture. We report the results of applying our method to lighter vision transformer backbones such as ViT-B/16, in Table Our approach consistently improves the efficiency of pignnenterVVT-BBB/
the cost of slight performance drop without fine-tuning. Specifically speaking, our approach saves more than 26% GFLOPs of a trained "Segmenter+ ViT-B/16" with only a slight performance drop from 48.48 48% to 48.40%, which verifies our method also generalizes to lighter vision transformer architectures.
Comparison with uniform downsampling. We compare our method with the simple uniform downsampling scheme, which directly downsamples the input image into a lower resolution. F summarizes the detailed comparison results. For example, on ADE20K, we downsample the input resolution from 640 x 640 to smaller resolutions (e.g., 592 x 592, 576 x 576, 560 x 560, and 544 x 544) and report their performance and GFLOPs Figure We also plot the results with our method and we can see that our method consistently outperforms uniform sampling on both ADE20K and PASCAL-Context under multiple different GFLOPs budgets.
4.4 Object Detection
We use the recent SOTA object detection framework Swin + HTC++ 45 as our baseline. We summarize the results of combining our method with Swin HTC++ COCO object detection and instance segmentation tasks in Figure 7
Implementation details. The original S2in consists of (2,2,18,2] shifted window transformer blocks across the four stages. We only apply our method to the 3-rd stage with 18 blocks considering it dominates the computation overhead, which we also follow in the "Mask2Former + Swin-1." experiments. We insert the window token clustering/reonsstuutionn layer after the 8-th/18-th block within the 3-rd stage, which are based on the official checkpoints of V2 + HTC++ In other words, we set a = 12 and a a+ 3 22 for V2inV2-LL The default window size is x K=32 x 32 and we set the clustered window size as k x k=23 x 23. We choose the values of other hyperparameters following the ablation experiments. According to the results summarized in Figure 7 compared to SwinV2-LL + HTC++, our method improves the FPS by 21% f and saves the GFLOPs by nearly 20% 1 while maintaining around 98% of object detection & instance segmentation performance.
bstps://githut wwt.///////oot//eeee/o// oaterooootttiiii--raaaaeeeeeee Transformer
MIT License
8

---

1000
00
(a) ADF20K
(b) PASCAL Cantext
(a)
MAP
(h) FPS
(c) GALOPS
Figure 6: Comparison uniform downsam- pling on ADE20K and PASCAL-Conie with Segmenter+ ViT-L/16.
Figure 7: Illustrating the improvements of approach object detection task Swin V2-L+ HTC++ and represent higher is better and lower is better respectively.
D0000 Putigit ADCINE
COCD
Damaroa
(a)
MAP
(b)
(c) GFI OPL
Figure 8: Illustrating the improvements our approach on panoptic segmentation, semantic segmentation, and instance segmentation tasks with Mask2Former-Swin-L.
RIG HST
1,00
1,80
80
(a) Top-1 Accuracy
(b)
(c) GFLOPs
Figure 9: Illustrating the improvements our approach and the comparisons with EVIT on ImageNet
classification tasks with SWAG+ViT-H/14 and SWAG+VIT-L/IEE
4.5 Senantic/nnsancc/Paapppi Segmentation
We first apply our method to plain ViT-based segmentation framework Segmenter 622 and illustrate the semantic segmentation results across three benchmarks including ADE20K, PASCAL-Contesr and Cityscapes on the first row Figure Then, we apply our method to very recent framework Mask2Former [IT] that is based on Swin Transformer and summarize the semantic segmentation, instance segmentation, and panoptic segmentation results on COCO in Figure X
Implementation details. The original ViT-L first splits an image into a sequence of image patches of size 16 16 and applies patch embedding layer to increase the channel dimensions 1024, then applies 24 consecutive transformer encoder layers for representation learning. To apply our method the ViT-L backbone of Segmenter, we use the official checkpoints of "Segmenter ViT-L/16" and insert the token clustering layers and token reconstruction layer into the ViT-L/16 backbone without fine-tuning.
For the Mask2Former built on Swin-L with window size as 12 x 12, we use the official checkpoints / of FFrmee + Swin-L" and insert the window token clustering layer and the window token reconstruction layer into the empirically chosen positions, which first cluster 12 x 12 tokens into 8 x 8 tokens and then reconstruct 12 x 12 tokens within each window. Figure & summarizes the detailed comparison results. Accordingly, we can see that our method significantly improves the FPS by more than 35% 1 with slight performance drop COCO panoptic segmentation task.
MIT License
CC-BY-NC 4.0

---

Table 7: Depth estimation results based on DPT [53] with ResNet-550-ViT-B/LEE
Dataset
Method
GFLOPs
FPS
0>1.25
8>1.25'
8>1.25'
AbsRel SqRel RMSE RMSElog SILog log10
DPT
810
11.38
0.959
0.995
0.999
0.062
0.222 2.573
0.092
8.282 0.027
KITTI
DuT+Ours
627
14.75
0.958
0.995
0.999
0.062
0.226 2.597
0.093
8.341 0.027
DPT
560
17.38
0.904
0.988
0.998
0.110
0.054 0.357
0.129
9.522 0.045
NYUv2
DPT+Ours
404
24.03
0.900
0.987
0.998
0.113
0.056 0.363
0.132
9.532 0.046
4.6
Monocular Depth Estimation
To verify the generalization of our approach, we apply our method to depth estimation tasks that measure the distance of each pixel relative to the camera. We choose the DPT (Dense Prediction [53] that builds on the hybrid vision transformer, i.c., ViT--BB-/, following [17].
Transformer)
Implementation details. The original R50+VIT-B/16 consists of a ResNet50 followed by ViT- B/16, where the ViT-B/16 consists of 12 transformer encoder layers that process 16x downsampled representations. We insert the token clustering layer & token reconstruction layer into ViT-B/16 and summarize the results on both KITTI and NYUv2 on the second row of Figure [ We also report their detailed depth estimation results in Table 7 where we can see that our method accelerates DPT by nearly 30%/37% 1 on KITTI/NYUv2, respectively.
consisss
4.7 ImageNet-IK Classification
Finally, we apply our method to the ImageNet-IK classification task and compare our method with a very recent SOTA method EVTT [43. The key idea of EVIT is to identify and only keep the top-k tokens according to their attention scores relative to the [class] token. We empirically find that applying EVTT for dense prediction tasks directly suffers from significant performance drops. More details are illustrated in the supplementary material.
Implementation details. We choose the recent SWAG 660 our baseline, which exploits 3.6 billion weakly labeled images associated with around 27K categories (or hashtags) to pre-train the large-scale vision transformer models, i.e., VIT-L/16 and ViT-H/14. According to their official implementations SWAG + ViT-H/14 and SWAG + ViT-L/16 achieve 88.55 58%55% and 88.07% top-1 accuracy on ImaegNet- IK respectively. We apply our approach and EVIT to both baselines and summarize the comparison results in Figure 9 According to the results, our method achieves comparable results as EViT while being more efficient, which further verifies that our method also generalizes to the image classification tasks without fine-tuning.
Conclusion
In this paper, we present simple and effective mechanism to improve the efficiency of large-scale vision transformer models for dense prediction tasks. In light of the relatively high costs associated with re-training or fine-tuning large vision transformer models on various dense prediction tasks, our study provides very lightweight method for expediting the inference process while requiring no additional fine-tuning, We hope our work could inspire further research efforts into exploring how to accelerate large-scale vision transformers for dense prediction tasks without fine-tuning.
Acknowledgement
This work is partially supported by the National Nature Science Foundation of China under Grant 62071013 and 6167 1027, and National Key R&D Program of China under Grant 2018AAA0100300,
References
[1] Hangbo Bao, Li Dong, and Furu Wei. Beit: Bert pre-training of image transformers. arXiv preprint 108.::210060082.... 2021.
MTT License
attpe: // /ttp//gttthhhhtheeeeeeeeeeeheeet wwt.ggttt.....eeeeeeeeeeeeeeeeee cow//ecebookreeesrc///5s
CC-BY-NC 4.0
10

---

[2] Davis Blalock, Jose Javier Gonzalez Ortiz, Jonathan Frankle, and John Guttag, What is the state of neural network pruning? Proceedings of machine learning and systems, 2:129-146 146, 2020.
[3] Walid Bousselham, Guillaume Thibault, Lucas Pagano, Archana Machireddy, Joe Gray, Young Hwan Chang, and Xubo Song. Efficient self-ensemble framework for semantic segmen- tation. arXiv preprint 1et::2211.1133 11.11:2111.13280, 2021.
[4] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with Transformers. In ECCV, 2020.
[5] Arnay Chavan, Zhiqiang Shen, Zhuang Liu, Zechun Liu, Kwang-Ting Cheng, and Eric Xing. Vision transformer slimming: Multi-dimension searching in continuous optimization space. arXiv preprint wrXiv:2201.008//1 2022.
[6] Minghao Chen, Houwen Peng, Jianlong Fu, and Haibin Ling. Autoformer: Searching trans- formers for visual recognition. In ICCV, pages 12270-12280 2021.
[7] Tianlong Chen, Yu Cheng, Zhe Gan, Lu Yuan, Lei Zhang, and Zhangyang Wang. Chasing sparsity in vision transformers: An end-to-end exploration. NeurlPS, 34, 2021.
[8] Wuyang Chen, Xianzhi Du Fan Yang. Lucas Beyer, Xiaohua Zhai, Tsung- Yi Lin, Huizhong Chen, Jing Li, Xiaodan Song, Zhangyang Wang, et al. A simple single-scale vision transformer for object localization and instance segmentation. arXiv preprint arXiv:2112.0974 2021.
[9] Wuyang Chen, Wei Huang, Xianzhi Du, Xiaodan Song, Zhangyang Wang, and Denny Zhou. Auto-scaling vision transformers without training. In International Conference on Learning Representations, 2021.
[10] Yinpeng Chen, Xiyang Dai, Dongdong Chen, Mengchen Liu, Xiaoyi Dong, Lu Yuan, and Zicheng Liu. Mobile-former: Bridging mobilenet and transformer. arXiv preprint arXiv:2 05895, 2021.
[11] Bowen Cheng, Ishan Misra, Alexander G. Schwing, Alexander Kirillov, and Rohit Girdhar. Masked-attention mask transformer for universal image segmentation. 2022.
[12] Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, et al. Rethinking attention with performers. arXiv preprint arXiv:2009. 14794, 2020.
[13] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The Cityscapes dataset for semantic urban scene understanding. In CVPR, 2016.
[14] Zihang Dai. Hanxiao Liu, Quoc V Lc, and Mingxing Tan. Coatnet: Marrying convolution and attention for all data sizes. NeurIPS, 34:3965-3977,, 2021.
[15] Henghui Ding, Chang Liu, Suchen Wang, and Xudong Jiang. Vision-language transformer and query generation for referring segmentation. In ICCV, pages 16321-16330, 2021.
[16] Mingyu Ding, Bin Xiao, Noel Codella, Ping Luo, Jingdong Wang, and Lu Yuan. Davit: Dual attention vision transformer. arXiv preprint arXiv:2204.036655, 2022.
[17] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 1929, 2020.
[18] Stéphane d' Ascoli, Hugo Touvron, Matthew L Leavitt, Ari Morcos, Giulio Biroli, and Levent Sagun. Convit: Improving vision transformers with soft convolutional inductive biases. In ICML, pages 2286-2294 PMLR, 2021.
[19] Alaaeldin El-Nouby, Hugo Touvron, Mathilde Caron, Piotr Bojanowski, Matthijs Douze, Armand Joulin, Ivan Laptev, Natalia Neverova, Gabriel Synnaeve, Jakob Verbeek, et al. Xcit: Cross-covariance image transformers. arXiv preprint arXiv:2106.09681, 2021.
[20] Haoqi Fan, Bo Xiong, Karttikeya Mangalam, Yanghao Li, Zhicheng Yan, Jitendra Malik, and Christoph Feichtenhofer. Multiscale vision transformers. In ICCV, pages 6824-6835, 2021.
[21] Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In 2012 IEEE conference on computer vision and pattern recognition, pages 3354-3361. IEEE, 2012.

---

[22] Jiaqi Gu, Hyoukjun Kwon, Dilin Wang, Wei Ye, Meng Li, Yu-Hsin Chen, Liangzhen Lai, Vikas Chandra, and David Z Pan. Hrvit: Multi-scale high-resolution vision transformer. arXiv preprint urXiv:211.0022 2021.
[23] John Guibas, Morteza Mardani, Zongyi Li, Andrew Tao, Anima Anandkumar, and Bryan Catanzaro. Efficient token mixing for transformers via adaptive fourier neural operators. In International Conference on Learning Representations, 2021.
[24] Kai Han, Yunhe Wang. Qiulin Zhang, Wei Zhang, Chunjing Xu, and Tong Zhang. Model rubik's cube: Twisting resolution, depth and width for tinynets. NeurIPS, 33:19353-19364, 2020.
[25] Haodi He, Yuhui Yuan, Xiangyu Yue, and Han Hu. Mlseg: Image and video segmen- tation as multi-label classification and selected-label pixel classification. arXiv preprint N.KV:22003.01877, 2022.
[26] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. arXiv preprini arXiv:2111.06377 2021.
[27] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. In CVPR, 2016.
[28]
Byeongho Heo, Sangdoo Yun, Dongyoon Han, Sanghyuk Chun, Junsuk Choe, and Seong Joon Oh. Rethinking spatial dimensions of vision transformers. In ICCV, pages 11936-11945, 2021.
[29] Torsten Hoefler, Dan Alistarh, Tal Ben-Nun, Nikoli Dryden, and Alexandra Peste. Sparsity in deep learning: Pruning and growth for efficient inference and training in neural networks. JMLR, 22(241):1-124, 2021.
[30] Zejiang Hou and Sun-Yuan Kung. Multi-dimennional model compression of vision transformer. arXiv preprint arXiv:2201.00045 2021.
[31] Jitesh Jain, Anukriti Singh, Nikita Orlov, Zilong Huang, Jiachen Li, Steven Walton, and Humphrey Shi. Semask: Semantically masked transformers for semantic segmentation. arXiv preprint arXiv 2021.
[32] Varun Jampani, Deqing Sun, Ming- Yu Liu, Ming-Hsuan Yang, and Jan Kautz. Superpixel sampling networks. In ECCV, pages 352-368, 2018.
[33] Chao Jia, Yinfei Yang. Ye Xia, Yi-Ting Chen, Zarana Parekh, Hicu Pham, Quoc Lc, Kun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and with noisy text supervision. In ICML. pages 4904-4916. PMLR, 2021.
representation learning
[34] Kumara Kahatapitiya and Michael S Ryoo. Swat: Spatial structure within and among tokens.
arXiv preprint 1rXiv:21/1.136677 2021.
[35] Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. arXiv preprint arXiv:2001.04451, 2020.
[36] Zhenglun Kong, Peiyan Dong, Xiaolong Ma, Xin Meng, Wei Niu, Mengshu Sun, Bin Ren, Minghai Qin, Hao Tang, and Yanzhi Wang. Spvit: Enabling faster vision transformers via soft token pruning. arXiv preprint arXiv:2112.1388
2021.
[37] Liunian Harold Li** Pengehuan Zhang*. Haotian Zhang*, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang. Lu Yuan, Lei Zhang, Jenq- Neng Hwang, Kai-Wei Chang, and Jianfeng Gao. Grounded language-image pre-training. In CVPR, 2022.
[38] Yanghao Li, Hanzi Mao, Ross Girshick, and Kaiming He. Exploring plain vision transformer backbones for object detection. arXiv preprint arXiv:2203. /6527, 2022.
[39] Yanghao Li, Chao- Yuan Wu, Haoqi Fan, Karttikeya Mangalam, Bo Xiong, Jitendra Malik, and Christoph Feichtenhofer. Improved multiscale vision transformers for classification and detection. arXiv preprint arXiv:2
2021.
[40] Yanghao Li, Saining Xie, Xinlei Chen, Piotr Dollar, Kaiming He, and Ross Girshick. Bench- marking detection transfer learning with vision transformers. arXiv preprint arXiv:2111.11429, 2021.
[41] Zhenyu Li, Zehui Chen, Xianming Liu, and Junjun Jiang. Depthformer: Depthformer: Ex- ploiting long-range correlation and local information for accurate monocular depth estimation. arXiv preprint arXiv:2203. 142111 2022.
[42] Jingyun Liang, Jiezhang Cao, Guolei Sun, Kai Zhang, Lue Van Gool, and Radu Timofte. Swinir: Image restoration using swin transformer. arXiv preprint 20Xiv:2108.10253 2021.
12

---

[43] Youwei Liang, GE Chongjian, Zhan Tong, Yibing Song, Jue Wang, and Pengtao Xie. Evit: Expediting vision transformers via token reorganizations. In International Conference on Learning Representations, 2022.
[44] Tsung-Yo Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, pages 740-755. Springer, 2014.
[45] Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning. Yue Cao, Zheng Zhang, Li Dong, et al. Swin transformer v2: Scaling up capacity and resolution. arXiv preprint arXiv:2111.09883 2021.
[46] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, pages 10012-10022, 2021.
[47] Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. Learning efficient convolutional networks through network slimming. In Proceedings of the IEEE international conference on computer vision, pages 2736-2744, 2017.
[48] Zhuang Liu, Mingjie Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell. Rethinking the value of network pruning. arXiv preprini arXiv: 181000520,0 2018.
[49] Dmitrii Marin, Jen-Hao Rick Chang, Anurag Ranjan, Anish Prabhu, Mohammad Rastegari, and
Oncel Tuzel. Token pooling in vision transformers. arXiv preprint arXiv:21 2021.
[50] Sachin Mehta and Mohammad Rastegari. Mobilevit: light-weight general-purpose, and mobile- friendly vision transformer. arXiv preprint arXiv:2110.021770 2021.
[51] Roozbeh Mottaghi, Xianjie Chen, Xiaobai Liu, Nam-Gyu Cho, Seong-Whan Lee, Sanja Fidler, Raquel Urtasun, and Alan L. Yuille. The role of context for object detection and semantic segmentation in the wild. In CVPR, 2014.
[52] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In /CML, pages 1748-8763. PMLR, 2021.
[53] René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. Vision transformers for dense prediction. In ICCV, pages 12179-121 288, 2021.
[54] Yongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu, Jie Zhou, and Cho-Jui Hsieh. Dy- namievit: Efficient vision transformers with dynamic token sparsification. arXiv preprint arXiv:2106.02034, 2021.
[55] Cedric Renggli, André Susano Pinto, Neil Houlsby, Basil Mustafa, Joan Puigcerver, and Carlos Riquelme. Learning to merge tokens in vision transformers. arXiv preprint arXiv:2202.12015, 2022.
[56] Aurko Roy, Mohammad Saffar, Ashish Vaswani, and David Grangier. Efficient content- content-based sparse attention with routing transformers. ACL, 9:53-68, 2021.
[57] Michael Ryoo, AJ Piergiovanni, Anurag Arnab, Mostafa Dehghani, and Anelia Angelova. Tokenlearner: Adaptive space-time tokenization for videos. NeurIPS, 34, 2021.
[58] Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus. Indoor segmentation and support inference from rgbd images. In ECCV, pages 746-760. Springer, 2012.
[59]
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint 1409. 1et9. 2014.
[60] Mannat Singh, Laura Gustafson, Aaron Adcock, Vinicius de Freitas Reis, Bugra Gedik, Raj Pra- teek Kosaraju, Dhruv Mahajan, Ross Girshick, Piotr Dollár, and Laurens van der Maaten. Revisiting weakly supervised pre-training of visual perception models, 2022.
[61] Lin Song. Songyang Zhang, Songtao Liu, Zeming Li, Xuming He, Hongbin Sun, Jian Sun, and Nanning Zheng. Dynamic grained encoder for vision transformers. NeurIPS, 34, 2021.
[62] Robin Strudel, Ricardo Garcia, Ivan Laptev, and Cordelia Schmid. Segmenter: Transformer for semantic segmentation. arXiv preprint arXiv:2105.0566 2021.
[63] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In CVPR, pages 1-9,2015 2015.
13

---

[64]
Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In ICML, pages 6105-6114 14. PMLR, 2019.
[65] Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Metzler. Efficient transformers: A survey. arXiv preprint arXiv:2009.06732, 2020.
[66] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. NeurIPS, 30, 2017.
[67] Tao Wang. Li Yuan, Yunpeng Chen, Jiashi Feng, and Shuicheng Yan. Pnp-detr: towards efficient
visual analysis with transformers. In ICCV, pages 4661-4670, 2021.
[68] Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In ICCV, pages 568-578, 2021.
[69] Wenxiao Wang. Minghao Chen, Shuai Zhao, Long Chen, Jinming Hu, Haifeng Liu, Deng Cai, Xiaofei He, and Wei Liu. Accelerate cnns from three dimensions: A comprehensive pruning framework. In ICML, pages 10717-10726. PMLR, 2021.
[70] Wenxiao Wang, Shuai Zhao, Minghao Chen, Jinming Hu, Deng Cai, and Haifeng Liu. Dbp: discrimination based block-level pruning for deep model acceleration. arXiv preprint 19111:1912.10178, 2019.
[71] Yulin Wang, Rui Huang, Shiji Song, Zeyi Huang, and Gao Huang. Not all images are worth 16x16 words: Dynamic transformers for efficient image recognition. NeurIPS, 34, 2021.
[72] Mitchell Wortsman, Gabriel Ilharco, Samir Yitzhak Gadre, Rebecca Roelofs, Raphael Gontijo- Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Komblith, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. arXiv preprint arXiv:2203.055: 05482, 2022.
[73] Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Segformer: Simple and efficient design for semantic segmentation with transformers. NeurIPS, 34, 2021.
[74] Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu, Simmim: A simple framework for masked image modeling. arXiv preprint arXiv:2111.09800 2021.
[75] Yufei Xu, Jing Zhang, Qiming Zhang, and Dacheng Tao. Vitpose: Simple vision transformer baselines for human pose estimation. arXiv preprint arXiv: 12484, 2022.
[76] Jianwei Yang, Chunyuan Li, Pengchuan Zhang, Xiyang Dai, Bin Xiao, Lu Yuan, and Jianfeng Gao. Focal self-attention for local-global interactions in vision transformers. arXiv preprint arXiv:2107.0064/ 2021.
[77] Jiahui Yu, Linjie Yang, Ning Xu, Jianchao Yang, and Thomas Huang. Slimmable neural networks. arXiv preprint /8/2.088 2018.
[78] Yuhui Yuan, Xilin Chen, and Jingdong Wang. Object-contextual representations for semantic segmentation. In ECCV, pages 173-190. Springer, 2020.
[79] Yuhui Yuan, Rao Fu, Lang Huang, Weihong Lin, Chao Zhang, Xilin Chen, and Jingdong Wang. Hrformer: High-resolution transformer for dense prediction. arXiv preprint arXiv:2 10.09408, 2021.
[80] Yuhui Yuan, Lang Huang, Jianyuan Guo, Chao Zhang, Xilin Chen, and lingdong Wang. Ocnet: Object context network for scene parsing. arXiv preprint 18Xiv: /6, 2018.
[81] Yuhui Yuan, Jingyi Xie, Xilin Chen, and Jingdong Wang. Segfix: Model-agnostic boundary refinement for segmentation. In ECCV. Springer, 2020.
[82] Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transform-
ers. CVPR, 2022.
[83] Bowen Zhang. Shuyang Gu, Bo Zhang, Jianmin Bao, Dong Chen, Fang Wen, Yong Wang, and Baining Guo. Styleswin: Transformer-based gan for high-resolution image generation. arXiv preprint arXiv:2
2021.
[84] Hao Zhang, Feng Li, Shilong Liu, Lei Zhang, Hang Su, Jun Zhu, Lionel M Ni, and Heung- Yeung Shum. Dino: Detr with improved denoising anchor boxes for end-to-end object detection. arXiv preprint trXiv:2203.03600 2022.
14

---

[85] Haokui Zhang, Wenze Hu, and Xiaoyu Wang. Edgeformer: Improving light-weight convnets by learning from vision transformers. arXiv preprint arXiv:2203.03952. 2022.
[86] Pengchuan Zhang, Xiyang Dai, Jianwei Yang, Bin Xiao, Lu Yuan, Lei Zhang, and Jianfeng Gao. Multi-scale vision longformer: A new vision transformer for high-resolution image encoding. In ICCV, pages 2998-3008, 2021.
[87] Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torallb. Semantic understanding of scenes through the ADE20K dataset. IJC,, 2019.
15

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Have you read the ethics review guidelines and ensured that your paper conforms to
them? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Did you describe the limitations of your work? [N/A]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either the supplemental material or as a URL)? [TODO]We will release the code soon.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [TODO]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] We use the official checkpoints provided by:
attpe://gttabb.cm/rrttuudddddddd
http:://ggabb...c.s.rrru////ssts zoo MIT License
zo0.ad, CC-BY-NC 4.0
http:://ggtaa.ccmo/ttaaaatttttts Pramarrrrr
MIT License
CC-BY-NC 4.0
https://gitbub..u com/isl- oxg/DPT MIT License.
(b) Did you mention the license of the assets? [Yes] We mark the license of these assets as
above.
(c) Did you include any new assets either in the supplemental material or as URL? [Yes] We add the URL of these assets in the footnote.
(d) Did you discuss whether and how consent was obtained from people whose data you't 're using/curating? [N/A]
(c) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
16


Reviews:
[{"review_id": "s-kzfNBDej7", "reviewer": "Reviewer_kZ4Z", "summary": "This paper introduces two non-parametric operators to efficiently down-sampling some intermedia tokens to accelerate the large-scale vision transformers for dense prediction. One is a token clustering layer  to decrease the number of tokens and the other one is a token reconstruction layer to recover the number of tokens. These two operators need no training or finetuning, while the related experimental results prove the effectiveness of them in typical benchmarks. ", "questions": "None.", "limitations": "See the above discussion about weaknesses. ", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nThe motivation of the paper is clear and solid. While the current existing works for efficient transfomer need training or finetuning, this work introduces two non-parametric operators and brings some meaningful insight for this topic.\nThe experimetal results are sufficient to support the proposed work. \nThe paper is well written and easy to follow.\n\nWeaknesses:\nFrom Table 2, I see that the proposed operators actually have limited performance when increasing the downsampling stride. I think this limitation is normal for non-paramettic modules, but I also think the author should also compare with the other parametric methods such as [47,52,55] here to give a comprehensive discussion. \n\nOverall, I think this paper is technically sound and achieve good performance. Thus I vote for borderline accept.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "r6AT4zJCIEq", "reviewer": "Reviewer_Ldum", "summary": "The authors observed that the presence of local spatial representation redundancy and high definition representation in the vision transformer contribute to dense prediction. Therefore, this paper proposes the token clustering layer and the token reconstruction layer to reduce computation and memory cost. The proposed modules are non-parametric operations and are applicable to different models and multiple dense prediction tasks.", "questions": "I suggest that authors provide visualizations of the original features and clustering features in the paper or in the appendix, if possible.", "limitations": "I do not see any potential negative societal impact for this work.", "rating": 5, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The paper is clearly motivated, well written and with sufficient experimentation.\n2. The proposed token clustering and reconstruction are non-parametric operations and therefore can be implemented directly on the well-trained model without additional fine-tuning. Taking existing trained models, the proposed method significantly improves the efficiency of the large vision transformer on the dense prediction task at a low cost in terms of precision.\n3. The authors' observations are insightful, and the methods proposed are clever and easy to implement. In addition to the five dense prediction tasks in the paper, I think the proposed approach also has the potential to be extended to other vision transformer-based tasks.\n4. The authors also provides the extension strategy and experiments for Swin-Transformer in addition to ViT.\n\nWeaknesses:\n1. Since the authors claimed that there is redundancy in the tokens of the vision transformer and that the proposed method is based on clustering, some visualizations could better represent this observation.\n2. Clustering attention has been explored by previous approaches such as clustering attention[1] and adaptive clustering attention[2]. Specifically, ACT can slim the high-resolution transformer without retraining. The author is highly encouraged to discuss relationship between previous works. \n\nOverall, I think the proposed approach is straightforward and effective. However, the paper lack comparison with previous similar approaches. The author is encouraged to discuss related works especially on clustering attention topic. \n[1] Fast transformers with clustered attention\n[2] End-to-end object detection with adaptive clustering transformer\n[3] SMYRF: Efficient Attention using Asymmetric Clustering", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "QFf9S0S2A8K", "reviewer": "Reviewer_Wt6a", "summary": "This paper accelerates vision transformers for dense prediction without finetuning. This is done by (i) using the token clustering layer to cluster the neighboring tokens; (2) using the token reconstruction layer to re-create the high-resolution representations.   The result of proposed method achieves state-of-the-art performance on five dense prediction tasks ", "questions": "According to my understanding, author speed up Transformer by reducing token number and restore feature representation from these clustered tokens.  However, I do not understand why this approach could avoid finetune.", "limitations": "Could authors provide detailed discussion for reason why their acceleration method can avoid finetune. ", "rating": 5, "confidence": 1, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This paper proposes two layers (token clustering layer and token reconstruction layer) and apply them into Swin Transformer. The two ideas are simple and easy to follow. The setting of experiments is clear and the number of experiments is adequate.\n\n\nThis paper fulfills with texts and numbers. I think that authors should provide visual comparison for  detection, segmentation and depth estimation\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "2-KNyTq5j_", "reviewer": "Reviewer_sJzo", "summary": "When using the Transformer for dense image prediction, detection, and segmentation, the large number of tokens can impose a heavy computational burden. Therefore, the core of this paper is to propose clustering tokens to obtain a low-resolution representation to reduce the computational burden, and to perform attention on the low-resolution token cluster, followed by a token reconstruction layer to reconstruct the high-resolution token representation. The authors' experiments verify that the proposed approach significantly improves the FPS and reduces the GFLOPS index for segmentation, detection, and depth estimation tasks when using Transformer with only a small loss of AP.", "questions": "The use of the superpixel method, while reducing the size of the token, must lead to a loss of semantic information. And after the reconstruction of the layers, there must be noise in the regained token. My question is how the loss of semantic information in each layer compares to that of the original ViT after using the two modules proposed by the authors", "limitations": "I personally believe that the work in this article has contributed to reducing the computational burden of the Transformer. However, the two modules proposed in the article should be further verified and reasoned in detail to show the reader more details.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nThis paper introduces the superpixel scheme into the Transformer structure, which really reduces the computational burden on the Transformer. The author's solution ensures that the entire computational process is differentiable and is a practical solution.\n\nWeaknesses\nThe Token Clustering Layer in the paper is similar to the algorithm cited in [30], where the center of the superpixel is obtained and then iterated until the result of the latest clustering is obtained. Although better results are obtained, the method is only introduced in and the innovation point is not sufficient. And I think the author's description of the two modules is not detailed enough and should be further illustrated with pictures. The experimental part should be more experimental on multiple datasets for each task. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept