AiluRus: A Scalable ViT Framework for Dense Prediction
Authors: Jin Li
Yaoming Wang
XIAOPENG ZHANG
Bowen Shi
Dongsheng Jiang
Chenglin Li
Wenrui Dai
Hongkai Xiong
Qi Tian
Venue: NeurIPS 2023 poster (2023)
Keywords: vision transformer
dense prediction

Abstract: Vision transformers (ViTs) have emerged as a prevalent architecture for vision tasks owing to their impressive performance. However, their complexity dramatically increases when handling long token sequences, particularly for dense prediction tasks that require high-resolution input. Notably, dense prediction tasks, such as semantic segmentation or object detection, emphasize more on the contours or shapes of objects, while the texture inside objects is less informative. Motivated by this observation, we propose to apply adaptive resolution for different regions in the image according to their importance. Specifically, at the intermediate layer of the ViT, we select anchors from the token sequence using the proposed spatial-aware density-based clustering algorithm. Tokens that are adjacent to anchors are merged to form low-resolution regions, while others are preserved independently as high-resolution. This strategy could significantly reduce the number of tokens, and the following layers only handle the reduced token sequence for acceleration. At the output end, the resolution of the feature map is recovered by unfolding merged tokens for task prediction. Consequently, we can considerably accelerate ViTs for dense prediction tasks. The proposed method is evaluated across three different datasets and demonstrates promising performance. For instance, "Segmenter ViT-L" can be accelerated by 48\% FPS without fine-tuning, while maintaining the performance. Moreover, our method can also be applied to accelerate fine-tuning. Experiments indicate that we can save 52\% training time while accelerating 2.46$\times$ FPS with only a 0.09\% performance drop.

AiluRus: A Scalable ViT Framework for Dense Prediction
Jin Li
Yaoming Wang' Xiaopeng Zhang Bowen Shi!
Dongsheng Jiang? Chenglin Li Wenrui Dai
Hongkai Xiong' Qi Tian
Shanghaii Jiao Tong University Huuaei Cloud
[deserve_lj, wang_yaoming, sjtu_shibowen Ic11985, daiwenrui, xionghongkai jtsikkaiiisssuueuuuuuu :2hangxiaopeng12, jiangdongshengl, tian. gi1/ htw.ww/hhwwweccccm com
Abstract
Vision transformers (ViTs) have emerged as prevalent architecture for vision tasks owing to their impressive performance. However, when it comes to handling long token sequences, especially dense prediction tasks that require high-resolution put, the complexity of ViT's increases significantly. Notably, dense prediction tasks, such as semantic segmentation object detection, emphasize more on the contours or shapes of objects, while the texture inside objects is less informative. Motivated by this observation, we propose to apply adaptive resolution for different regions in the image according to their importance. Specifically, at the intermediate layer the ViT, we utilize a spatial-aware density-based clustering algorithm to select representative tokens from the token sequence. Once the representative tokens are determined, we proceed to merge other tokens into their closest representative token. Consequently, semantic similar tokens merged together to form owwresolution regions, while semantic irrelevant tokens are preserved independently as high- resolution regions. This strategy effectively reduces the number of tokens, allowing subsequent layers to handle a reduced token sequence and achieve acceleration. At the output layers, the resolution of the feature map restored by unfolding the merged tokens for task prediction. As a result, our method significantly accelerates ViT's for dense prediction tasks. We evaluate our proposed method on three different datasets and observe promising performance. For example, the "Segmenter ViT-L." model can be accelerated by 48% FPS without fine-tuning, while maintaining the performance. Additionally, our method can be applied to accelerate fine-tuning as well. Experimental results demonstrate that we can save 52% training time while accelerating 2.46x FPS with only a 0.09% performance drop. The code is available at http:///gggaaaacccc/o/c////////a
mmain
Introduction
Transformers have shown significant advancements in various vision tasks such image classification 2, 33 18), object detection [38 34, and semantic segmentation[[6] 4 Despite their impressive performance across various visual tasks, the complexity of these models poses challenges for fine- tuning and deployment, particularly as their capacity continues to grow 11
This complexity issue particularly relevant for dense prediction tasks that require high-resolution input. Efforts have been made to address this challenge by designing efficient VIT models [36] 6, 20,
13
39
19 [6] 23 B1] [7]. However, most of these works are primarily focused on classification tasks and applicable to dense prediction tasks. Recently, [15] proposed to expedite well-trained ViTs for dense
24
:: Corresponding author. t: Equal contribution. This work was done when Jin Li interned at Huawei Cloud.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

prediction tasks through super-pixel clustering. Nevertheless, the clustering operation can be only conducted in the relatively deep layers, resulting in limited acceleration ratio and scalability.
In this paper, we introduce a novel approach, namely Adaptive resolution with spatial-awaRe clustering (AiluRus), accelerate ViT's for dense prediction tasks. We find that dense prediction tasks focus more on the shape or contour of objects rather than the texture. For instance, in segmentation maps, the contour of objects carries crucial information, while the interior regions are filled with the same prediction values, indicating their lower informativeness compared to the boundaries. Motivated by this observation, we propose token pruning technique that incorporates adaptive resolution to represent different regions in an image. Specifically, we allocate more tokens to critical regions that contribute to recision-makinng and fewer tokens less informative regions. The main challenge is to determine a reasonable assignment of resolutions. To address this issue, we utilize density-basee clustering algorithms 10 21 to generate the assignment of each token, where spatial information is incorporated to encourage neighboring tokens to have the same assignment. Tokens that have the same assignments are averaged to produce the representative tokens. These representative tokens could be the original ones, which correspond to informative regions, or the average of several tokens, which correspond to less informative regions. This approach enables us to reduce the length of the token sequence at intermediate layers, thereby accelerating the model. At the output stage, we restore the original resolution for prediction tasks by assigning the value of the representative token to its corresponding regions.
We provide compelling visualizations in Figure Fig. 2 to support the reasonableness of the generated assignments and their limited impact on visual perception. To assess the effectiveness of our proposed method, we adopt the benchmark of [15 and integrate our method into well-trained models without fine-tuning. Our experimental results demonstrate that AiluRus effectively accelerates ViTs and outperforms previous methods, particularly in scenarios with high acceleration ratios. Specifically, AiluRus achieves a 48% increase in FPS for Segmenter ViT-L while maintaining the performance. Moreover, we further apply AiluRus to accelerate the fine-tuning process. Experiments show that AiluRus reduces training time by 52% while achieving 2.46 x increase in FPS with only 0.09% drop in performance. These findings demonstrate the effectiveness of AiluRus in accelerating ViTs.
In summary, we list our contributions as follows:
We propose apply adaptive resolution the feature map of ViT-based dense prediction tasks for acceleration without fine-tuning.
We propose to generate the resolution assignments through the proposed spatial-aware DPC algorithm. Visualizations demonstrate that the produced assignments have little influence on visual perception and thus could expedite models without fine-tuning.
Our proposed AiluRus can be used to accelerate well-trained models or pre-trained models for inference or fine-tume tne-tuning. Experiments show that AiluRus could significantly accelerate models with a negligible performance drop.
2 Related Work
Vision transformer for dense prediction tasks. Transformers have gained immense popularity in Natural Language Processing (NLP) tasks, and there have been considerable efforts extend their success to computer vision tasks. DETR (38 introduced transformers as the detection head in a convolutional neural network, opening up new avenues for utilizing transformers dense prediction tasks. This work has inspired the development of hybrid-transformer architectures aimed at facilitating dense prediction tasks [3 30). Other works have proposed pure transformer architectures, which have achieved significant progress in recent advances [26] 4 29]. In this paper, instead of proposing a new architecture or framework for dense prediction tasks, we focus on accelerating existing dense prediction methods.
Efficient vision transformers. One of the primary strategies for improving the efficiency of Vision Transformers (ViTs) is to reduce the complexity of the self-attention operation. The conventional self- attention operation involves establishing interactions between any two tokens, resulting in quadratic complexity with respect to the number of tokens. To address this challenge, recent approaches aim to approximate the self-attention results through clustering based on the sparse and low-rank properties of sclf-attention [36] 6 28).

---

Imput
Recovery output
Sportal-monn
Rewdight
Resohttinn
Fieek
DPC
Tokeen
Elodk
Head
Deodar
Representative
Tokens
Spatial Mask
Local Density
Spatial-aware DPC
Token Reweight
Recovery Resolution
Tokens attributed one cluster center
exp(yi kg/s) En Zaem Cf
Attention:
ay
Color Bar
Color Bar
Calculate
Cluster
Cluster Centers
Centers
Using Cluster C Centers
Spatial Mask
Distance Indicator
Higher Density
Longer
The framework of AiluRus. focus specific intermediate layer of the ViT and apply our AiluRus using the spatial-aware DPC algorithm. This algorithm searches for cluster centers based local density and distance indicators, and then averages the tokens in each cluster obtain a representative token for the following layers. To ensure that each representative token is weighted appropriately, re-weight them based on the number of tokens they represent. Finally, unfold the representative tokens at the output end to recover the original resolution. For a more detailed explanation of our method, please refer to Section 3
Figure 1:
These methods aim to gradually discard less informative tokens at different layers and retain only subset of tokens at the output end to accelerate the model. However, most of these approaches are designed for classification tasks and are less practical for dense prediction tasks. For instance, EVTT [16], EvvoVVTT (3), and PCAE 14] select informative tokens based on a single criterion, such the attention weight to the class token or similarity to the mean token, which is not suitable for dense prediction tasks where there are many objects belonging to various categorics in the image. Dynamic ViT (20) and Ada- ViT [19] rely specific architectures and require re-training, which may lead to additional computational overhead in practical applications. ToMe (I progressively merges a percentage the most similar tokens between bipartite tokens but has only been verified in classification tasks.
Another line of works focuses on token pruning 20
Some works aim to design more efficient VITs by introducing learnable token merging or token For example, STVIT [2] replaces redundant tokens with a few semantic tokens, which can be regarded as cluster centers for acceleration. PaCa-Va ViT [12 proposes patch-to-cluster attention to address the semantic gap between patch-to sorch-tt-ptcch attention in visual tasks and its NL. counterpart. TCFormer 322 merges tokens from less informative regions via elustering to emphasize critical regions for human-centric tasks. Although these methods achieve promising results for efficient ViTs, some of them rely on specific architectures [32] [2] and all require fine-tuning. Recently, Liang et al. [15] proposed method to expedite well-trained large ViTs for dense prediction tasks using superpixel clustering, which does not require fine-tuning. However, this strategy can only be applied in relatively deep layers, resulting in limited improvements in efficiency.
pruning modules P 12 32 25
3 Methodology
3.1 Preliminary
typical ViT requires sequential input and thus reshapes the input image E Rxx as a token sequence Xp E ( x3=p², where indicates the patch size. As the patch size is fixed for a given ViT, the number of tokens depends on the resolution of the input image, and thus high-resolution images, which are usually required for dense prediction tasks, inevitably suffer from high computational complexity. One intuitive way toward efficient ViTs reduce the number of tokens. However, reducing tokens inevitably accompanies information loss and results in performance degradation. notice that dense prediction tasks such as detection and segmentation mainly focus on the shape and contour of objects while less caring about the texture inside objects, or irrelevant background. Based on this observation, we propose an adaptive resolution strategy for accelerating dense prediction tasks. Our framework is illustrated in Fig. I
3

---

3.2 Adaptive Resolution
For an imput token sequence Z E NxD, the target to generate M representative tokens according to Z where . < N. These representative tokens can be either the original tokens in Z that correspond to informative regions in the image or the average of several tokens in Z that correspond to less important regions for decision-making. In this way, different regions are represented by different numbers of tokens, i.e., resolutions based on their importance. The main challenge is to generate a proper assignment of Z that minimizes information loss.
To generate this assignment, we propose to apply density- clustering algorithms, specifically DPC [211[10). We are motivated by two reasons. For one thing. DPC does not rely on iterative updates like traditional clustering methods such as K-means, making more suitable for latency-sensitive scenarios. For another thing, DPC searches for cluster centers among the input data, and thus specific tokens can be independently preserved, which enables to preserve details for informative regions. We find that over 20% cluster centers are independently preserved when selecting 400 cluster centers from 600 tokens (Please refer to supplementary for details). In contrast, the cluster centers in K-means are linearly weighted by input, which will lead to information distortion and affect decision- making for fine-gre linegrained regions. However, conventional DPC algorithms only consider the relations of data points in the feature space but ignore their intrinsic spatial structure. Since image patches have clear spatial relationships that are critical for dense prediction, directly applying DPC may lead to unreasonable assignments and performance degradation. To address this issue, we propose to incorporate spatial information into clustering.
Spatial-aware DPC. DPC selects cluster centers from input data points based on their product of local density P and distance indicator 6, where large p * indicates the high potential to be the cluster center. We will explain how we calculate p. 6, and incorporate the spatial information in the following. Specifically, we calculate the local density of each token by:
1 Σ
P: exp(
(zi, si,,
K
EXC
(1 - r)rankk()))))))))) (j/A+ rank(f)) 1 X inn
(1,
rank(j) 2
where -.:--) denotes the distance metric, K. = (2;) and we apply the Euclidean distance here, indicates the number of neighbors used to calculate the local density, s(;,)) is the introduced spatial information where is the hyperparameter to control the strength of the spatial constraint, and X is the number of spatial neighbors. s(..)) assigns different weights for o2s: 2,) according to their spatial relation. Specifically, for tokens that are not À nearest, s(i. j) assigns the maximum weight for them while assigning the value of a to for the À nearest tokens. s(.,-) encourages spatially adjacent tokens to be merged first. With the introduced spatial information, the local density of each token only depends on the X spatial neighbors, and each cluster at most merges À tokens. These properties enable the produced assignments to avoid extreme cases where tokens far away in space are merged or too many tokens are merged together.
The distance indicator ò is calculated by:
min (21, sii,s), 3pj > pi jP:ppp inf,
6:-
otherwise
With P and d,, we rank tokens according to P; ō; and select top M tokens as the cluster centers. The remaining tokens are assigned to the closest cluster centers, and tokens belonging to the same cluster centers are merged together as the representative token.
Token reighttht. As the produced representative tokens correspond to different numbers of original tokens (from dozens to only one), there is a distribution gap between the representative tokens and original tokens. For example, tokens corresponding to large objects may be merged into a few representative tokens, which results in inconsistent self-attention results. To minimize this gap, assign different weights for each representative token during self-attention. In conventional ViTs, the attention of token to token j is given as:
(xplufFk//)) aij = Σ, e.pppppp k5/s)
(4)
4

---

Original
Low Resolution
Assignments
Original
Low Resolution
Assignments
Figure 2: Visualization to clustering results. The first and fourth columns display the original image, the third and sixth columns show the produced assignments, where tokens with the same assignment are marked with the same color, and the reconstructed low-resolution images are presented in the second and fifth columns.
where qi is the query token i, ky is the key of token and is the scale constant. To minimize the differences brought by token reduction, the same token is expected to have similar attention values in both the original token sequence and the representative token sequence. To this end, we group tokens belonging to the same representative token in self-attention. As only similar tokens merged, can be assumed that their value of q² k is also similar, and thus Eq. (4) can be written as:
exp(g kj/s) aij Σn Eiem, 9 kj
exp ohe ks/s)
En, mn exp(g k2)s)
where m, denote the nuh representative token. We notice that this trick is also used in [
3.3 Visualization
To evaluate the effectiveness of the clustering algorithm, we visualize the assignments and the reconstructed low-resolution images. Specifically, we apply our spatial-award density-based clustering method with 400 cluster centers to the output of the second layer of Segmenter ViT-L, which consists of 1600 tokens. We also reconstruct the entire image using 400 patches corresponding to the cluster centers based on the assignments. The visualizations are shown in Fig. 2 Our results indicate that the reconstructed images have a similar appearance the original images, suggesting that the produced assignments are reasonable and that 1/4 of the tokens are capable of capturing most of the shape and contour information of the original images. Please note that although some regions may appear to have the same color in the visualization of assignments due to the use of 400 different colors, they may actually have different assignments. These findings provide strong evidence for the effectiveness of our proposed method in generating high-qualipy representative tokens for dense prediction tasks.
5

---

Table 1: GFLOPs, FPS and mloU of AiluRus under different accelcration ratio. The baseline results
refer Segmenter 26
ViT-L. The best results are marked in bold.
Slight
Mild
Extreme
Methods
GFLOPs
FPS
mloU
GFLOPs
FPS
mIoU GFLOPs
FPS
mloU
Results on ADE20K
Baseline
659.0
6.55
51.82
659.0
6.55
51.82
659.0
6.55
51.82
ACT [36
611.1
6.01
51.69
545.2
6.16
51.24
533.5
6.33
48.03
ToMe
516.2
6.97
51.66
448.7
8.31
50.96
321.3
10.75
47.12
EVIT [16
572.0
7.58
51.52
500.2
8.50
50.37
351.8
12.03
38.89
Expedite
15
529.8
7.92
51.93
443.8
9.51
51.56
309.4
13.51
47.96
AiluRus
478.8
8.72
52.17
427.8
9.53
51.79
300.8
14.14
50.21
Results CityScapes
Baseline
995.6
4.20
79.14
995.6
4.20
79.14
995.6
4.20
79.14
ACT 36
906.3
4.76
79.00
742.7
4.49
78.71
730.4
5.32
75.42
ToMe
760.8
5.20
78.37
651.5
5.50
77.81
448.5
7.84
71.23
EVIT [16
822.7
5.27
79.03
707.2
5.96
78.49
506.2
8.68
68.14
Expedite 15
840.9
4.82
78.82
691.0
5.89
78.38
529.6
8.02
76.20
AiluRus
710.9
5.88
78.83
669.8
6.65
78.73
461.5
9.36
77.38
Results on Pascal Context
Baseline
338.7
14.7
58.07
338.7
14.7
58.07
338.7
14.7
58.07
ACT 33
306.7
11.1
58.04
299.0
11.7
57.88
298.3
11.9
56.08
ToMe 1
269.8
13.9
57.67
236.5
17.0
57.24
172.4
18.2
54.25
EVIT [16
271.7
16.0
57.94
261.0
17.7
56.99
184.4
23.5
48.57
Expedite 11
251.2
18.2
58.27
201.3
21.6
57.85
161.0
25.0
55.08
AiluRus
241.2
19.8
57.95
224.3
21.9
57.91
157.7
28.4
57.02
Table GFLOPs, FPS and mIoU of AiluRus under different acceleration ratio. The baseline results
refer to Segmenter 26
ViT-B. The best results are marked in bold.
Slight
Mild
Extreme
Methods
GFLOPs
FPS
mloU
GFLOPs
FPS
mloU
GFLOPs
FPS
mloU
Results on ADE20k
Baseline
124.7
32.2
48.48
124.7
32.2
48.48
124.7
32.2
48.48
ACT 36
105.1
26.0
48.39
105.1
25.9
47.55
105.1
26.1
44.01
ToMe 1
100.2
31.2
47.99
88.6
32.4
46.96
66.4
35.6
40.85
EVIT [16
109.4
34.1
48.44
96.5
37.8
48.05
69.4
46.5
38.27
Expedite 15
110.2
29.8
46.74
97.4
34.6
46.14
87.1
39.4
45.17
AiluRus
62.3
37.9
48.59
50.3
45.9
48.38
40.2
55.3
47.32
4 Experiments
In this section, we begin by comparing AiluRus with recent SOTA methods on semantic segmentation tasks in Section 4.] which includes a more detailed comparison with the reminiscent method (15) Subsequently, in Section 4.2 we evaluate the performance of AiluRus on object detection and instance segmentation tasks assess its generalization ability across various dense prediction tasks. Moving on to Section 4.3 we investigate the applicability of AiluRus in the fine-tuning process to enable acceleration. Additionally,
Section 4.4, we conduct ablation experiments to study the impact of different saper-parameters of AiluRus on the overall performance. Furthermore, we provide supplementary experiments in the appendix, including the application AiluRus in expediting classification tasks and text-based video generation tasks. We also delve into the reasons behind the superior performance of AiluRus compared Expedite these tasks. For more detailed information, please refer to our appendix.
4.1 Comparison to other methods
We follow the benchmark [5] to adapt the proposed method to the Segmenter 26 framework built on ViT 19. Specifically, we load the parameters of the officially released models and integrate

---

0.5
0.50
0.4
0.45
mlou
Allurus Expedit
mlou
0.3
0.40
Expedite
Baseline
Baseline
0.2
0.35
0.1
0.30
256
400
1024 Num clusters (a)
784
1296
196
324
676
900
Num clusters
(b)
Figure 3: The ablation study on the number of clusters with a fixed cluster location equal to 2. All results are obtained from the officially released checkpoints, and the ablation study ViT-L and ViT-B are shown in (a) and (b) respectively.
0.50
0.45
miou
miou
0.3
0.40
0.2
0.35
Alluns
Ailurus
Expedit
0.1
Expedite
0.30
Baseline
B Baseline
0.0
0
2
8
10
12
14
16
0
2
6
8
Location
Location
(a)
(b)
Figure 4: The ablation study the cluster location with fixed cluster number equal to 400. All results are obtained from the officially released checkpoints, and the ablation study on ViT-L and ViT-B are shown in (a) and (b) respectively.
AiluRus into the backbone. The adaptive resolution is applied to the output of certain intermediate layer, produced representative tokens are processed by the following layers. With the reduced feature size, the inference speed of models can be significantly improved. We find that the decoder in Segmenter is robust to the reduced feature size, and thus the resolution is recovered after the decoder instead of the backbone.
Comparisons to SOTA methods. We conduct experiments compare AiluRus with recent SOTA efficient ViT methods across different datasets and architectures. The results are presented in Tab. and Tab. Note that Segmenter (26 only provides ViT-B for the ADE20K dataset. Hence, we only compare different methods using VIT-B on the ADE20K dataset. We follow the benchmark by [S and report the GFLOPS, FPS, and mIoU under three different acceleration ratios. The results demonstrate that AiluRus consistently achieves the best trade-off between performance and efficiency compared to other methods across various datasets and acceleration ratios. The advantages of AiluRus are particularly evident in the extreme scenario. For instance, as presented in Tab. with higher FPS, AiluRus outperforms Expedite by 2.251 mloU on ADE20K. 1.18 mloU on CityScapes, and 1.94f mloU Pascal Context, indicating that AiluRus is much more robust to high acceleration ratios. Such a property enables AiluRus achieve more significant acceleration ratios at an acceptable performance drop.
More comparisons to Expedite. As both AiluRus and Expedite accelerate VITs by reducing the feature size at the intermediate layer, they can be further compared under the same configuration. Specifically, we fix either the number of clusters or the cluster location and vary the other parameters across different configurations for both VIT-I. and VIT-B models. Initially, we fix the cluster location and experiment with different numbers of clusters. As illustrated in Fig. AiluRus consistently outperforms Expedite under all settings. The advantage of AiluRus is especially evident for ViT-B,

---

Allurus
Expedit
Expedite
65
65
Baseline
55
FPS
FPS
45
45
35
35
25
25
196
324
484
676
900
0
2
8
Num clusters
Location
(a)
(b)
Figure With the officially released 'Segmenter 26 between AiluRus and Expedite under the same configuration.
ViT-B', we illustrate the FPS comparison
0.55
0.50
0.50
0.45
miou
mloU
0.40
0.40
0.35
0.35
Ours
0.30
Expedite Baseline
Expedite
Baseline
0.25
30
50
60
FPS
FPS
(a)
(b)
Figure 6: We run different configurations of AiluRus and Expedit (i.c. the cluster number and cluster location) and illustrate the optimal mloU-FPS curve for VIT-L and ViT-B in figure (a) and (b).
where AiluRus maintains performance even with only 484 clusters, while Expedite almost does not work. Besides the performance, as illustrated in Fig. []] AiluRus demonstrates higher efficiency than Expedite under the same configuration. This can be attributed to the lower complexity of AiluRus. Unlike Expedite, which requires multiple iterations for accurate cluster centers and inevitably involves high latency, AiluRus selects cluster centers from candidate tokens in step, making it more suitable for latency-sensitive scenarios. This advantage becomes more pronounced accelerating relatively lightweight architectures.
Next, we fix the number of clusters 400 and analyze the impact cluster location. As depicted in Fig. 3] the performance of AiluRus is close to Expedite when clustering is applied at deeper layers but significantly exceeds Expedite when clustering performed at shallower layers. This suggests that AiluRus can achieve higher acceleration ratios by reducing the feature map size at shallower layers. contrast, Expedite suffers serious performance degradation when clustering applied at shallower layers, limiting its potential in accelerating ViT's.
To make more comprehensive comparison between AiluRus and Expedite, run various con- figurations for AiluRus andExpedite and illustrate the optimal mioU-FPS curve in Fig. 6 The results consistently demonstrate that AiluRus achieves superior trade-off's between performance and efficiency compared to C, particularly under high acceleration ratios.
4.2 Acceleration for object detection and instance segmentation.
To further validate the generalization of AiluRus in dense prediction tasks, we deploy on object detection and instance segmentation tasks achieve instant acceleration. Since there are no well- trained models available directly, we follow the setup of CAE and train detection models based on ViT-I. and ViT-B within the Mask-RCNN framework. Specifically, we use the CAE pre-trained model as initialization and perform 1x schedule training. The training hyper- parameters strictly

---

Table 3: The results of AiluRus accelerating Mask-RCNN based on ViT-L.
Methods
Location Clusters
FPS
Object Detection
Instance Segmentation
Baseline
1.9
54.7
47.8
7
1600
2.9 (153%)
53.0 (L1.7)
46.5 ({1.3)
AiluRus
9
1764
2.6 (137%)
54.0 (J0.7)
47.2 (10.6)
11
1764
2.5 (132%)
54.3 (J0.4)
47.4 (10.4)
15
1600
2.3 (121%)
54.5 (J0.2)
47.6 (10.2)
Table 4: The results of AiluRus accelerating Mask-RCNN based ViT-B.
Methods
Location Clusters
FPS
Object Detection
Instance Segmentation
Baseline
4.4
50.1
44.0
3
1296
6.3 (143%)
48.3 (11.8)
42.4 (11.6)
AiluRus
3
1764
5,9 (134%)
49.4 (10.7)
43.4 (10.6)
6
1444
5.3 (120%)
49.8 (10.3)
43,9 (10.1)
7
1444
5.1 (116%
49.9 (J0.2)
43,9 (J0.1)
Table 5: The results of AiluRus accelerating fine-tuning based on ViT-L.
Methods
Location
Clusters
Training Time
FPS
mIoU
Baseline
26.76
6.55
52.16
2
400
14.75 (145%)
18.44 (1182%)
48.98 (43.18)
2
484
15.37 (143%)
17.53 (116))
49.63 (12.53)
2
576
15.62 (142%)
16.39 (1150%)
50.41 (11.75)
Expedite
15
2
900
18.46 (131%)
10.77 (164%)
50.62 (11.54)
8
400
19.19 (128%)
12.92 (197%)
50.74 (L1.42)
10
784
21.27 (121%)
9.02 (138%)
51.98 (10.18)
2
400
11.80 (156%)
19.24 (1194%)
50.82 (41.34)
2
484
12.28 (154%)
17.30 (1164%)
51.82 (J0.34)
AiluRus
2
576
12.81 (152%)
16.11 (1146%)
52.07 (10.09)
2
900
16.37 (139%)
11.24 (172%)
52.42 (10.26)
follow the CAE setup, and the reproduce results are also close to the officially reported ones (reproduce official: 50.1 1 vs 50.3 for ViT-B, 54.7 54.5 for VIT-L). Subsequently, integrate AiluRus into the well-trained models without fine-tuning, and the hyper-parameters (i.e. a, À and ke) of AiluRus remained the same in previous experiments. We vary the cluster location and the number of clusters to achieve different acceleration ratios, these results are presented in Tab, 3 and Tab. 4 The results demonstrate that AiluRus generalizes well in object detection and instance segmentation tasks, indicating that AiluRus can effectively expedite various dense prediction tasks with minor modifications.
4.3 Acceleration for fine-tuning
The growing capacity of ViTs leads satisfactory performance but remains major challenge in fine-tuning the pre-trained models. Since AiluRus does not rely on any specific architecture or learnable parameter, can be seamlessly integrated into the fine-tuning phase for acceleration. We adopt AiluRus to Segmenter [26] and fine-tune the pre-trained model following the official schedule. Our code base MMsegmentation [5], and the results reported by MMsegmentation are used as the baseline. We fine-tune the ire-trained modes on 8 V100-32G and evaluate the FPS on single V-100 32G. Both VYT-L and ViT-B are evaluated, and the results are reported Tab. 5 and Tab. 6 We surprisingly find that AiluRus can largely reduce the overheads in fine-tuning while slightly degrading performance or even enjoying better performance. Specifically, take ViT-L as example, AiluRus reduces 52% training time(.) and improves 146% FPS(1) with only 0.09 mloU drop when remaining 576 clusters. Besides, AiluRus even achieves better performance(f 0.26) with improved FPS(1 72%) when remaining 900 clusters.

---

Table 6: The results of AiluRus accelerating fine-tuning based on ViT-B.
Methods
Location
Clusters
Training Time
FPS
mloU
Baseline
6.89
32.2
49.60
2
324
6.12 (111%)
53.8 (167%)
39.66 (19.94)
2
400
6.13 (111%)
48.2 (150%)
40.20 (J9.40)
2
484
6.16 (111%)
43.3 (134%)
41.27 (18.33)
Expedite 15
2
576
6.21 (110%)
38.9 (121%)
41.94 (U7.66)
6
400
6.79 (111)
37.7 (117%)
47.00 (42.60)
7
400
6.94 (11%)
35.8 (111%)
48.88 (J0.72)
2
324
4.94 (128%)
61.5 (191%)
48.06 (11.54)
2
400
5.14 (125%%
54.1 (168%)
49.04 (10.56)
AiluRus
2
484
5.33 (123%)
49.7 (154%)
49.39 (J0.21)
2
576
5.48 (120%)
44.0 (137%)
49.57 (10.03)
Table 7: Ablation study of the hyper-parameters.
Parameter
Ox
0.6
0.7
0.8
0.9
1.0
0
20
30
50
70
mloU
50.59
50.76
50.75
75
50.81
50.67
50.47
50.48
50.58
50.81
50.73
50.81
50.45
50.29
50.19
We also run Expedite under the same configurations for a comprehensive comparison. The results in Tab. [ and Tab, iiidiicte that Expedite still works worse when reducing feature size at shallower layers even with fine-tuning, We further run Expedite under official configurations and find that its performance is obviously improved. However, the acceleration ratios are also significantly decreased as these configurations reduce feature size at deeper layers. This comparison shows that the advantage of AiluRus over Expedite more obvious in accelerating fine-tuning. We attribute this to the good robustness of AiluRus for feature size reduction and shallower cluster locations. This property enables AiluRus maintain considerable performance at high acceleration ratios with well-trained models and compensates for the performance drop during fine-tuning. In contrast, find that the low tolerance to shallow clustering layers Expedite cannot addressed by fine-tuning, and ultimately results in limited efficiency improvement.
4.4 Ablation Study
We conducted hyper-parameter ablation experiments on the adaptive resolution strategy presented in Section 3.2 using the ADE20K semantic segmentation benchmark and the officially released Segmenter ViT-L/16 26 checkpoint. For the neighbor weight hyper- parameter a, we searched its value from 0.6 to 1.0 (1.0 indicates disabling this hyper-parameter), and the results showed that = 0.9 performed best. Similarly, we searched the value of from 0 to 70 (0 indicates not using spatial information), and the results showed that À 50 performed best. The ablation results of indicated that A 1, i.e., choosing the closest token to calculate the local density, performed best.
Conclusion
The emergence of ViTs has empowered numerous vision tasks also brought increasing overheads in fine-tuning and inference models. In this paper, we proposed plug-in strategy, called AiluRus that can be integrated into well-trained models to immediately accelerate inference without any fine-tuning to pre-trained models to expedite both fine-tuning and inference. Our experiments demonstrated the advantages of AiluRus over previous methods, particularly in cases with high accelcration ratios. For example, with Segmenter ViT-L (26), AiluRus could accelerate FPS by 45% for the well-trained model with negligible performance drop (L 0.03). For the pre-trained model, AiluRus could reduce fine-tuning time by 52% and accelerate FPS 146% with a minor performance drop (L 0.09). These impressive results lead us to believe that current dense prediction tasks contain significant redundancy that unnecessarily benefits performance and can be removed for significant efficiency improvements. We hope that this consideration could inspire future work in the design of dense prediction tasks.
10

---

Acknowledgement
This work was supported in part by the National Natural Science Foundation of China under Grant 62250055, Grant 61931023, Grant T2122024, Grant 62125109, Grant 61932022, Grant 61972256, Grant 61971285, Grant 61831018, Grant 62120106007, Grant 62320106003, Grant 62371288, and in part by the Program of Shanghai Science and Technology Innovation Project under Grant 20511100100
References
[1] Daniel Bolya, Cheng- Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token merging: Your ViT but faster. In International Conference on Learning Representations, 2023.
[2] Shuning Chang, Pichao Wang, Ming Lin, Fan Wang, David Junhao Zhang, Rong Jin, and Mike Zheng Shou. Making vision transformers efficient from a token sparsification view. arXiv preprint 230.23333.08885 2023.
[3] Jieneng Chen, Yongyi Lu, Qihang Yu, Xiangde Luo, Ehsan Adeli, Yan Wang, Le Lu, Alan L Yuille, and Yuyin Zhou. Transunet: Transformers make strong encoders for medical image segmentation. arXiv preprint arXiv:2 2021.
[4] Bowen Cheng, Ishan Misra, Alexander G. Schwing, Alexander Kirillov, and Rohit Girdhar. Masked- mask transformer for universal image segmentation. 2022.
[5] MMSegmennation Contributors. MMSeggentatiin: Openmmlab semantic segmentation toolbox and benchmark. httpp::/ com open 1in---mmn mm l
atinn 2020.
[6] Giannis Daras, Nikita Kitaev, Augustus Odena, and Alexandros G Dimakis. Smyrf-efficient attention using asymmetric clustering. Advances in Neural Information Processing Systems, 33:6476-6498 6489, 2020.
[7] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248-255, 2009.
[8] Shuangrui Ding, Peisen Zhao, Xiaopeng Zhang, Rui Qian, Hongkai Xiong, and Qi Tian. Prune spatio-temporal tokens by semantic-aware temporal accumulation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 16945-16956, 2023.
[9] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition scale. In ICLR 2021: The Ninth International Conference Learning Representations, 2021.
[10] Mingjing Du, Shifei Ding, and Hongjee Jia. Study on density peaks clustering based on k-nearest neighbors and principal component analysis. Knowledge-Baud -Bssed Systems, 19:135-145, 2016.
[11] Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva: Exploring the limits of masked visual representation learning scale. arXiv preprint arXiv:2211.076333 2022.
[12] Ryan Grainger, Thomas Paniagua, Xi Song, and Tianfu Wu. Learning patch-to-cluster attention in vision transformer. arXiv preprint arXiv:2203.1 19877 2022.
[13] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, et al. Segment anything. arXiv preprint arXiw:2304.026443 2023.
[14] Jin Li, Yaoming Wang. XIAOPENG ZHANG, Yabo Chen, Dongsheng Jiang, Wenrui Dai, Chenglin Li, Hongkai Xiong, and Qi Tian. Progressively compressed auto-encoder for self- supervised representation learning. In The Eleventh International Conference on Learning Representations, 2023.

---

[15] Weicong Liang, Yuhui Yuan, Henghui Ding, Xiao Luo, Weihong Lin, Ding Jia, Zheng Zhang, Chao Zhang, and Han Hu. Expediting large-scale vision transformer for dense prediction without fine-tuning. arXiv preprint arXiv:2210.01035, 2022.
[16] Youwei Liang, Chongjian GE, Zhan Tong, Yibing Song, Jue Wang, and Pengtao Xie. EVit: Expediting vision transformers via token reorganizations. In International Conference on Learning Representations, 2022.
[17] Zhihao Lin, Yongtao Wang, Jinhe Zhang, and Xiaojie Chu. Dynamicdet: A unified dynamic architecture for object detection. In Proceedings of the IEEE/CVE Conference on Computer Vision and Puttern Recognition (CVPR), 2023.
[18] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference computer vision, pages 10012-10022, 2021.
[19] Lingchen Meng, Hengduo Li, Bor-Chun Chen, Shiyi Lan, Zuxuan Wu, Yu-Gang Jiang, and Ser- Nam Lim. Adavit: Adaptive vision transformers for efficient image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12309-12318, 2022.
[20] Yongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu, Jie Zhou, and Cho-Jui Hsieh. Dynamicvit: Efficient vision transformers with dynamic token sparsification. Advances in neural information processing systems, 34: 13937-111394, 2021.
[21] Alex Rodriguez and Alessandro Laio. Clustering by fast search and find of density peaks. science, 344(6191):1492- 1490, 2014.
[22] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10684-10695, June 2022.
[23] Michael S Ryoo, AJ Piergiovanni, Anurag Arnab, Mostafa Dehghani, and Anelia Angelova. Tokenlearner: What can learned tokens do for images and videos? arXiv preprint 1XXiv:2106.11291 2021.
[24] Lin Song, Songyang Zhang, Songtao Liu, Zeming Li, Xuming He, Hongbin Sun, Jian Sun, and Nanning Zheng. Dynamic grained encoder for vision transformers. Advances in Neural Information Processing Systems, 34:5770-5783, 2021.
[25] Lin Song, Songyang Zhang, Songtao Liu, Zeming Li, Xuming He, Hongbin Sun, Jian Sun, and Nanning Zheng. Dynamic grained encoder for vision transformers. Advances in Neural Information Processing Systems, 34:5770-5783, 2021.
[26] Robin Strudel, Ricardo Garcia, Ivan Laptev, and Cordelia Schmid. Segmenter: Transformer for semantic segmentation. arXiv preprint arXiv22105.06633 2021.
[27] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data- aita-efficient image transformers & distillation through attention. In International conference on machine learning, pages 10347-10357. PMLR, 2021.
[28] Cong Wei, Brendan Duke, Ruowei Jiang, Parham Aarabi, Graham W Taylor, and Florian Shkurti. Sparsifiner: Learning sparse mustance-dependee attention for efficient vision transformers. arXiv preprint arXiv:2303. 13755, 2023.
[29] Enze Xie, Wenhai Wang. Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Segformer: Simple and efficient design for semantic segmentation with transformers. Advances in Neural Information Processing Systems, 34: 24177-1220000 2021.
[30] Yutong Xie, Jianpeng Zhang, Chunhua Shen, and Yong Xia. Cotr: Efficiently bridging cnn and transformer for 3d medical image segmentation. In Medical Image Computing and Com- puter Assisted Intereenion-MCCCA 2021: 24th International Conference, Strashourg, France, September 27-October 1, 2021, Proceedings, Part III 24, pages 17 1-1-1 Springer, 2021.
12

---

[31] Yifan Xu, Zhijie Zhang, Mengdan Zhang, Kekai Sheng. Ke Li. Weiming Dong, Liqing Zhang, Changsheng Xu, and Xing Sun. Evo-vit: Slow-fast token evolution for dynamic vision trans- former. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 2964-299 2972, 2022.
[32] Wang Zeng. Sheng Jin, Wentao Liu, Chen Qian, Ping Luo. Wanli Ouyang, and Xiaogang Wang. Not all tokens equal: Human-centrr visual analysis via token clustering transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11101-11111, 2022.
[33] Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transform- ers. In Proceedings of the IEEE/CVE Conference on Computer Vision and Puttern Recognition, pages 12104-121 12111211122 13, 2022.
[34] Hao Zhang, Feng Li, Shilong Liu, Lei Zhang, Hang Su, Jun Zhu, Lionel M Ni, and Heung- Yeung Shum. Dino: Detr with improved denoising anchor boxes for end-to-end object detection. arXiv preprint 2rXiv:2203.03605, 2022.
[35] Yabo Zhang, Yuxiang Wei, Dongsheng Jiang, Xiaopeng Zhang, Wangmeng Zuo, and Qi Tian. Controlvideo: Training-free controllable text-to-video generation. arXiv preprint arXiv:2335..3377, 2023.
[36] Minghang Zheng, Peng Gao, Renrui Zhang, Kunchang Li, Xiaogang Wang, Hongsheng Li, and Hao Dong. End-to-end object detection with adaptive clustering transformer. arXiv preprint arXiv:2011.09315. 2020.
[37] Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ade20k dataset. International Journal of Computer Vision 127(3):302-33)) 321, 2019.
[38] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. arXiv preprint 1rXi:20010004159 2020.
[39] Xueyan Zou, Jianwei Yang, Hao Zhang, Feng Li, Linjie Li, Jianfeng Gao, and Yong Jae Lee. Segment everything everywhere all at once. arXiv preprint drXiv:230..0671, 2023.
13

---

APPENDIX
A Statistics of Assignments.
The produced assignments of each token play erucial role in the trade-off between performance and throughput. Therefore, we present the assignment statistics in Fig. 7a where we deploy AiluRus on Segmenter VİT-L and perform clustering on the output of the second layer. The produced assignments are collected across the ADE20K (37 validation set. For a given cluster center, the number of tokens belonging to it is denoted by æ, while y/ (x) and y2(x) indicate the percentage or the number of cluster centers that dominate x tokens. We run this configuration with varying numbers of cluster
centers and illustrate their corresponding y1 (z) and (2)r) ) in Fig. 7a
cluster centers 400
600
0.6
cluster centers 576
cluster centers 576
centers 900
500
cluster centers 900
Percentage
Amount
400
300
0.2
200
100
0.0
0
10
20
30
40
50
10
20
30
40
50
Num token
Num token
(a)
(b)
Figure 7: The statistics of the assignments. The horizontal axis indicates how many tokens each cluster center has. The vertical axis denotes the percentages and amount of certain kind of cluster centers in (a) and (b) respectively.
The results reveal that cluster centers containing only one token are the most in all cases, and their percentage increases with an increase in the number of cluster centers. As shown in Fig. (bb the number cluster centers that dominate more tokens is similar for different total numbers of cluster centers. This suggests that the assignments of the low-resolution regions (dominating more tokens) are similar across different total numbers of cluster centers and the extra cluster centers are assigned the high-resolution areas. This phenomenon justifies AiluRus because the redundant regions within an image remain fixed, while the areas containing details require more tokens to describe them. Therefore, when increasing the number of cluster centers, the additional cluster centers should be assigned to high-resolution regions to reduce the error.
B Acceleration for Classification
Table The results of AiluRus accelerating classification tasks based on DEiT-B (27..
Methods
Location
Clusters
Throughput (imgs/s)
FLOPs (G)
Aces
Baseline
268.2
17.7
81.8
2
121
390.4 (146%)
12.1 (132%)
78.1 (J3.7)
Expedite 15
4
144
314.3 (117%)
14.7 (L17%)
79.3 (J2.5)
6
144
299.7 (112
15.5 (112%)
81.0 (Q0.8)
0
121
419.8 (157%)
11.4 (136%)
80.4 ($1.4)
AiluRus
0
144
350.5 (131%)
13.3 (125%)
81.3 (10.5)
4
144
313.7 (11%%
14.9 (.16%)
81.7 (1O.1)
While we focus dense prediction tasks in this paper, AiluRus can be easily integrated into classification tasks for instant acceleration. Specifically, we take the typical ImageNet (7) supervised pre-trained DEiT-B as the base model and integrate AiluRus into it without fine-tuning. We run the
14

---

same configurations for Expedite [[]] in comparison. As presented in Tab. AiluRus consistently outperforms Expedite in various settings.
C
Acceleration for Video Generation
Row
Allurus
Figure 8: The frames of generated videos. The frames in the first generated using the original ControlVideo method 35 while the frames in the second row are generated by integrating AiluRus with ControlVideo.
To demonstrate the versatility of AiluRus, we seamlessly integrate it into the Control Video method for text-basee based video generation. Specifically, we incorporate AiluRus into the StableDiffusion model [22 to reduce the size of intermediate features and execute the model using the prompt "A man wanders the ocean" with depth condition. We also run the original model with the same configuration for comparison. As shown above, AiluRus produces similar content to the original model. However, AiluRus accomplishes this task in only half the time required by the original model (2:11 vs 4:24). This observation highlights the capability of AiluRus to instantancously accelerate large models with negligible overheads, which is of critical importance for the recent emergence of large models.
D Why AiluRus outperforms Expedite?
As described in the main text, AiluRus exhibits significant performance superiority over Expedite 15). particularly when comes to token reduction at shallow layers. Shallow layer token reduction poses more significant challenges due to increased information loss. In order to validate this claim, we conduct detailed experiments to explore the intrinsic mechanism.
i) The reconstruction errors of AiluRus and Expedite. With the frozen model, the reconstruction errors between the output features and the original features are highly correlated with performance. To provide comprehensive comparison, we employ various configurations of AiluRus and Expedite on ViT-B and ViT-I. models, respectively, and calculate the cosine similarity between the reconstructed features and the original features. The results presented in Tab. 9 and Tab. [0] demonstrate that AiluRus generates output features that are more similar to the original features compared to Expedite across different settings, providing explanation for the superior performance of AiluRus.
Table 9: The reconstruction similarity comparison between AiluRus and Expedite over VIT-L. X x
indicates the number of clusters.
20x20
22x22
24x24
26x26
Layer
Method
mloU
Similarity
mloU
Similarity
mloU
Similarity
mloU
Similarity
AiluRus
47.36
0.8401
48.84
0.8699
50.05
0.8941
50.56
0.9143
Expedite
42.88
0.7706
44.26
0.7926
47.19
0.8308
47.74
0.8504
AiluRus
48.80
0.8724
49.47
0.8967
50.45
0.9164
50.98
0.9326
Expedite
45.32
0.8162
46.05
0.8304
47.80
0.8626
48.37
0.8789
AiluRus 49.97
0.9039
50.39
0.9213
51.06
0.9356
51.37
0.9480
Expedite 49.73
0.8934
50.03
0.9048
50.77
0.9219
51.06
0.9324
ii) The reconstruction errors different cluster locations. We further observe that AiluRus ex- hibits a more pronounced advantage in shallow clustering, while Expedite's 's performance deteriorates significantly. Since both AiluRus and Expedite perform clustering only once and reuse the clustering results at the output layer, it is evidently easier perform clustering in deeper layers compared to
15

---

Table 10: The reconstruction similarity comparison between AiluRus and Expedite over ViT-B. X Xx indicates the number of clusters.
18x18
20x20
22x22
24x24
Layer
Method
mloU
Similarity
mloU
Similarity
mloU
Similarity
mloU
Similarity
AiluRus
46.45
0.8971
47.12
0.9291
47.70
0.9436
48,26
0.9602
2
Expedite
4.32
0.2882
5.21
0.3342
6.19
0.3652
7.57
0.4065
AiluRus
47.69
0.9339
48.27
0.9503
48.66
0.9636
48,88
0.9746
4
Expedite
12.66
0.4714
15,53
0.5218
18.03
0.5594
21.57
0.6017
AiluRus
48. 19
0.9527
48.83
0,9642
48,90
0.9736
48.99
0.9817
6
Expedite
44.07
0.8782
45.17
0.8972
45.44
0.9068
46.14
0.9161
Table 11: The reconstruction similarity at different layers.
Methods
2
4
5
6
7
9
10
11
AiluRus
0.9210
0.8959
0.8863
0.8906
0.8972
0.9092
0.9238
0.9312
0.9361
0.9291
Expedite
0.7334
0.6228
0.5524
0.4824
0.4293
0.3694
0.3778
0.4219
0.4433
0.3342
Table 12: The probability of preserving token pair similarity for different similarity intervals.
Intervals
0.0-0.1
0.1-0.2
0.2-0.3
0.3-0.4
0.4-0.5
0.5-0.6
0.6-0.7
0.7-0.8
0.8-0.9
0.9-1.0
Probability 0.0137
0.1246 0.2179 0.1439
0.0629
0.0347
0.0255 0.0193 0.0212 0.9903
shallow layers. To investigate the reasons behind the performance gap in shallow clustering, we analyze representative scenario (VīT-B, layer index=2, num 2luster=20*20) and calculate the feature reconstruction quality in subsequent layers. The results presented in Tab. [1 inndiaaee that Expedite experiences increasingly severe reconstruction distortion during the forward pass, while AiluRus maintains reconstruction quality across layers. This elucidates the poor performance of Expedite when applied shallow layers.
iii) Why AiluRus has lower reconstruction errors and maintains its advantage across layers? We delve into the advantages of AiluRus over Expedite in terms of the intrinsic mechanism. Both methods aim to reduce the number of tokens while preserving the relationships between the retained tokens and the reduced ones, and subsequently utilize the preserved relationships for recovery at the output layer. The quality of recovery determines the overall performance, and it depends on whether the preserved relationships can be maintained after passing through several Transformer blocks. Expedite and AiluRus differ in the nature of the relationship they aim to preserve.
Expedite employs K-means clustering generate super-pixel features and aims to preserve the relationship between the super-pixel features and original tokens. Since the recovery for each token relies on several super-pixel features, the recorded relationships become complex, and the similarity between the removed token and corresponding super-pixel features is relatively low. In contrast, AiluRus selects representative tokens from the original ones and aggregates the remaining tokens into the nearest representative tokens, which allows AiluRus to recover tokens only relying on the corresponding aggregated token. Therefore, the relationship preserved by AiluRus is simpler, and the similarity between the removed token and the aggregated token is higher. To empirically verify this, we conducted a study on the similarity between the original tokens and the super-pixel features in Expedite, as well as the similarity between the aggregated tokens and the original tokens in AiluRus during the inference process. Specifically, we run Expedite and AiluRus with the same configuration and report the average similarity over the ADE20K validation set. The results confirm that the average similarity in Expedite is 0.6728, while in AiluRus, it is 0.9210, aligning with our expectations.
We further investigate the probability of token pairs in different similarity intervals to maintain their respective intervals at the output end. A higher probability indicates robust preservation of the relationship within that interval, while lower probability suggests a higher propensity for distortion. Experiments are conducted on the entire validation set of ade20k, and the results are recorded in Tab. 12 The results demonstrate that token pairs with a similarity between 0.9 and 1.0 have a probability of over 99% of maintaining this similarity at the output layer. However, token pairs in other similarity intervals suffer from significant similarity distortion. This finding elucidates why Expedite experiences serious distortion while AiluRus consistently achieves low reconstruction enrors across layers.
16

---

E Latency Analysis
As our motivation is to expedite vision transformers, it is necessary to ensure that the operations introduced by AiluRus are sufficiently efficient, avoiding significant latency that slows down the whole model. To investigate the latency brought by AiluRus, we decompose the running time of models into three components, i.e., time for transformer blocks, time for clustering, and time for recovering, and compare the elapsed time of Expedite and AiluRus. For fair comparisons, we run them under the same configuration over ViT-L and ViT-B respectively. For ViT-L, we start clustering at the 4-th layer and set the number of clusters as 576. For ViT-B, we start clustering at the 2-nd layer and set the number of clusters as 400. We report the average latency of each component on the ADE20K validation set in Tab. 13
Table 13:
The latency analysis. The percentage in parentheses indicates the percentage of latency occupied by the current operation.
Methods
Total (ms)
blocks (ms)
clustering (ms)
recovering (ms)
mIoU
Results on VIT-I.
Expedite [15]
31.4
26.5 (84.4%)
3.9 (12.4%)
1.0 (3.2%)
47.19 50.27
AiluRus
26.6
25.3 (95.1%)
1.2 (4.5%)
0.09 (0.3%)
Results on ViT-B
Expedite [5]
11.1
8.4 (75.7%)
2.1 (18.9%)
0.6 (5.4%)
5.21
AiluRus
10.0
8.7 (87.0%)
1.2 (12.0%)
0.08 (0.8%)
47.35
The analysis results indicate that AiluRus exhibits significantly lower clustering and recovery costs compared to Expedite. As we aforementioned. Expedite replaces raw tokens with super-pixel features for forward passes. Consequently, Expedite relies on multiple iterations to achieve sufficiently accu- rate super-pixel features and necessitates the computation of reconstruction coefficients for recovering the original features. Thus, Expedite consumes more time in both clustering and reconstruction. In contrast, AiluRus directly operates tokens in the original feature space and thus could efficiently generate token assignments in single iteration. The produced assignments can be directly used for subsequent reconstruction, thereby avoiding additional recovery costs.
F
Prediction Visualizations
Allurus
Original
Figure 9: The prediction visualizations of the original Segmenter ViT-L model and the AiluRus integrated one.
We conducted a comparative analysis of the prediction results between AiluRus and the original model in order to elucidate the factors contributing to the performance improvement of AiluRus in Tab. 5 Specifically, we visualize the predictions generated by both AiluRus and the original models over randomly selected ADE20K [37] images. As illustrated in the figure, the predictions generated by AiluRus exhibit enhanced smoothness, thereby avoiding undesirable predictions such as voids or spikes. Consequently, in certain scenarios, AiluRus demonstrates superiority over the original model. This improved smoothness can be attributed the application of the adaptive resolution strategy. By introducing a smoothness constraint in regions with low information density, such as backgrounds,
17

---

this strategy effectively incorporates prior for generating smooth predictions in these areas. When retained an adequate number of tokens, this prior aligns well with the true information distribution of the images, leading to performance improvements.
G TCFormer for Dense Prediction.
Both TCFormer 32 and AiluRus employ DPC clustering. However, they have very different motivations and implementations. TCFomer is designed for human-centric tasks while AiluRus focuses on expediting dense prediction tasks. In this section, we show that it is untrivial to apply TCFormer for dense prediction tasks.
Table 14:
The latency analysis for TCFormer and AiluRus.
Method
Total (ms)
blocks (ms)
extra (ms)
mloU
TCFormer
33.8
8.6 (25%)
25.2 (75%)
0.14
AiluRus
10.0
8.7 (87%)
1.3 (13%)
47.35
Table 15:
The performance comparison between AiluRus and ViT-CTM.
Method
Num clusters
FPS
mIoU
Baseline
1024
32.2
49.60
ViT-CTM
410 (1024 0.4)
29.1 (1 10%)
33.48 (L 16.12)
AiluRus
400
53.7 (1 67%)
49.04 0.56)
Initially, we intend integrate TCFormer into the mmseg framework for semantic segmentation. We carefully implement the TCFormer under the mmseg framework according to the officially released code. However, we find that TCFormer an extremely computationally intensive backbone. Even with its lightest configuration, the FPS for inferring ADE20K images at resolution of 312x512 under the Segmenter framework is only 2.03. In comparison, typical ViT-Base model achieves an FPS of 32.2, while AiluRus can surpass 50 without sacrificing performance. Upon analyzing the TCFormer code, we discover that it heavily relies on sparse matrix multiplication. These operations may not have high FLOPs yet introducing significant latency. Therefore, it is nearly impossible and impractical to apply TCFormer for dense prediction tasks given its high computational complexity.
Furthermore, we attempt to integrate the core TCFormer design, the CTM module, into ViT for acceleration. This would allow us compare the performance TCFormer with AiluRus using the same backbone and initialization. However, we encountered the same issue of excessive complexity for the CTM module. In a similar setup, the time required for the CTM module execute even higher than the inference time of the model itself. The details are presented in Tab. 14
Despite these issues, proceeded to train the VIT model with the integrated CTM module in the Segmenter framework for comparison with AiluRus. We followed the instructions provided in the TCFormer paper parameters and initialization as AiluRus. However, as demonstrated in Tab. [5] we find that even with only the integration of CTM into ViT, it raises significant negative impact training and causes very poor performance. Thus, is untrivial to integrate the CTM module into ViT' for acceleration.
up the CTM module and trained ViT-CTM with the same optimization
In conclusion, TCFormer is specifically designed for human-centric tasks, and adapting it to dense prediction tasks poses significant challenges. Additionally, TCFormer exhibits high computational complexity, rendering it unsuitable for accelerating ViTs. In contrast, AiluRus can be seamlessly integrated into well-trained models, providing immediate acceleration or expediting training without the need for additional syper-parameter adjustments. These two methods differ significantly in terms of motivation, implementation, and application. AiluRus stands out for its flexibility and lightweight nature, enabling its deployment in various tasks that TCFormer is incapable of addressing.
18


Reviews:
[{"review_id": "0NMvs1X6Od", "reviewer": "Reviewer_gX5G", "summary": "This paper studies the problem of accelerating vision transformers for dense prediction without fine-tuning and proposes a spatial-aware density-based clustering algorithm to select anchors from the token sequence. The proposed method is evaluated across three different\ndatasets and demonstrates promising performance.  ", "questions": "Please refer to Weaknesses", "limitations": "None", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ The idea is reasonable and interesting, and the performance is good.\n+ The abundant ablation studies are provided to verify the effectiveness of the proposed method.", "weaknesses": "- As said in the title, the proposed method is for dense prediction, of course, segmentation is typical, But it is better to include another dense prediction task for further verifying the effectiveness of the proposed method, for example, instance segmentation, depth estimation.\n- It is better to calculate the reconstruction error between the reconstructed feature map and the original feature map. Meanwhile, make a comparison to the other SOTA method, e.g. Expedite.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Other dense prediction tasks like instance segmentation and depth estimation**\n\n**A1:** Thanks for your suggestion! We have presented the performance on object detection and instance segmentation in the appendix and summarize the results here for your reference. Specifically, we integrate Ailurus into well-trained Mask-RCNN models for both object detection and instance segmentation and conduct experiments over various configurations and backbones. The results are presented below.\n\nResults for ViT-B:\n\n|    | Location   | Clusters   | FPS                  | Object Detection        | Instance Segmentation   |\n|:----------|:-----------|:-----------|:---------------------|:------------------------|:------------------------|\n| Baseline  | -          | -          | 4.4                  | 50.1                    | 44.0                    |\n|    | 3          | 1296       | 6.3 ($\\uparrow$ 43%) | 48.3 ($\\downarrow$ 1.8) | 42.4 ($\\downarrow$ 1.6) |\n|    | 3          | 1764       | 5.9 ($\\uparrow$ 34%) | 49.4 ($\\downarrow$ 0.7) | 43.4 ($\\downarrow$ 0.6) |\n|    | 6          | 1444       | 5.3 ($\\uparrow$ 20%) | 49.8 ($\\downarrow$ 0.3) | 43.9 ($\\downarrow$ 0.1) |\n|    | 7          | 1444       | 5.1 ($\\uparrow$ 16%) | 49.9 ($\\downarrow$ 0.2) | 43.9 ($\\downarrow$ 0.1) |\n\nResults for ViT-L:\n\n|    | Location   | Clusters   | FPS                  | Object Detection        | Instance Segmentation   |\n|:----------|:-----------|:-----------|:---------------------|:------------------------|:------------------------|\n| Baseline  | -          | -          | 1.9                  | 54.7                    | 47.8                    |\n|    | 7          | 1600       | 2.9 ($\\uparrow$ 53%) | 53.0 ($\\downarrow$ 1.7) | 46.5 ($\\downarrow$ 1.3) |\n|    | 9          | 1764       | 2.6 ($\\uparrow$ 37%) | 54.0 ($\\downarrow$ 0.7) | 47.2 ($\\downarrow$ 0.6) |\n|    | 11         | 1764       | 2.5 ($\\uparrow$ 32%) | 54.3 ($\\downarrow$ 0.4) | 47.4 ($\\downarrow$ 0.4) |\n|    | 15         | 1600       | 2.3 ($\\uparrow$ 21%) | 54.5 ($\\downarrow$ 0.2) | 47.6 ($\\downarrow$ 0.2) |\n\nAs presented above, Ailurus still performs well in object detection and instance segmentation tasks, which demonstrates its generalization ability.\n\n**Q2: The reconstruction error and comparison to Expedite**\n\n**A2:** We sincerely appreciate your suggestion. \nFor a comprehensive comparison, we run various configurations for Ailurus and Expedite on ViT-B and ViT-L, respectively, and calculate the cosine similarity between the reconstructed features and the original ones, where, 18 $\\times$ 18, 20 $\\times$ 20, 22 $\\times$ 22, and 24 $\\times$ 24 are different cluster numbers. The results are presented below. Ailurus consistently outperforms Expedite in terms of reconstruction errors across various settings, which partially explains why Ailurus performs better than Expedite. We also post more analysis for the reconstruction comparison between Ailurus and Expedite in the general response. Please refer to it for more details.\n\nResults for ViT-B\n\n| layer index | Method  | 18 $\\times$ 18 |            | 20 $\\times$ 20 |            | 22 $\\times$ 22 |            | 24 $\\times$ 24 |            |\n| :---------- | :------ | -------------: | ---------: | -------------: | ---------: | -------------: | ---------: | -------------: | ---------: |\n|             |         |           mIoU | Similarity |           mIoU | Similarity |           mIoU | Similarity |           mIoU | Similarity |\n| 2           | Ailurus |          46.45 |     0.8971 |          47.12 |     0.9291 |          47.70 |     0.9436 |          48.26 |     0.9602 |\n|             | Expedit |           4.32 |     0.2882 |           5.21 |     0.3342 |           6.19 |     0.3652 |           7.57 |     0.4065 |\n| 4           | Ailurus |          47.69 |     0.9339 |          48.27 |     0.9503 |          48.66 |     0.9636 |          48.88 |     0.9746 |\n|             | Expedit |          12.66 |     0.4714 |          15.53 |     0.5218 |          18.03 |     0.5594 |          21.57 |     0.6017 |\n| 6           | Ailurus |          48.19 |     0.9527 |          48.83 |     0.9642 |          48.90 |     0.9736 |          48.99 |     0.9817 |\n|             | Expedit |          44.07 |     0.8782 |          45.17 |     0.8972 |          45.44 |     0.9068 |          46.14 |     0.9161 |\n\nResults for ViT-L\n\n| layer index   | Method   |   20 $\\times$ 20 |        |   22 $\\times$ 22 |        |   24 $\\times$ 24 |        |   26 $\\times$ 26 |        | \n|:--------------|:---------|-----------------:|-------:|-----------------:|-------:|-----------------:|-------:|-----------------:|-------:|\n|               |          |  mIoU            |Similarity|   mIoU           |Similarity|   mIoU           |Similarity|    mIoU           |Similarity|  \n| 4             | Ailurus  |            47.36 | 0.8401 |            48.84 | 0.8699 |            50.05 | 0.8941 |            50.56 | 0.9143 |    \n|               | Expedit  |            42.88 | 0.7706 |            44.26 | 0.7926 |            47.19 | 0.8308 |            47.74 | 0.8504 |     \n| 6             | Ailurus  |            48.80 | 0.8724 |            49.47 | 0.8967 |            50.45 | 0.9164 |            50.98 | 0.9326 |      \n|               | Expedit  |            45.32 | 0.8162 |            46.05 | 0.8304 |            47.80 | 0.8626 |            48.37 | 0.8789 |      \n| 8             | Ailurus  |            49.97 | 0.9039 |            50.39 | 0.9213 |            51.06 | 0.9356 |            51.37 | 0.948  |    \n|               | Expedit  |            49.73 | 0.8934 |            50.03 | 0.9048 |            50.77 | 0.9219 |            51.06 | 0.9324 |   \n"}, {"review_id": "3rcntB2ZBj", "reviewer": "Reviewer_ZFoB", "summary": "This paper proposes a new approach to improving the efficiency of Vision Transformers (ViTs) for dense prediction tasks such as semantic segmentation. Recognizing that these tasks emphasize the contours or shapes of objects more than the textures within, the authors introduce a method to apply adaptive resolution to tokens in ViTs through density-based clustering. At the intermediate layer of the ViT, a spatial-aware density-based clustering algorithm is used to select anchors from the token sequence. Tokens near these anchors are merged to form low-resolution regions, while others are kept as high-resolution, thereby significantly reducing the number of tokens processed in subsequent layers. This strategy aims to accelerate processing without loss of critical information.\n\nThe proposed method showcased promising results on different semantic segmentation datasets compared to a couple of previous works, including a baseline method [13] which also conducts token clustering. The proposed method provides higher speed up with less performance drop on both well-trained and fine-tuning scenarios.", "questions": "- Does clustering location refers to layer number? If so, please declare. \n- Any idea why the new clustering method provides more robust clustering than [13]? \n- The ablation study seems to indicate that alpha makes little difference, any thoughts on that?", "limitations": "I don't think there are any outstanding concerns about any negative societal impacts in this work.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper presents promising empirical results compared to a closely related baseline [13], especially in reducing fine-tuning training time. \n- The paper is well-intuitive and easy to follow.\n- The paper provides extensive analysis and ablation on parameters, cluster numbers, and clustering layers. \n- The paper provides visual analysis to demonstrate that the clustering of tokens towards the segmentation task is reasonable.", "weaknesses": "- The manuscript suffers from a substantial number of typographical errors. Some are actually critical. E.g., the abusing usage of delta and sigma (Ln138-142).\n- The paper can be improved by better structure and writing. E.g., The figures and Tables are all over the place instead of being roughly around the page when they are mentioned. \n- While I recognize the promising performance of the paper, I am concerned that the paper might not contain enough novelty and theory contribution for publication in the underlying conference. The idea to decrease the number of tokens has been proposed in related works, as mentioned in the paper itself. The clustering idea is also proposed by [13].  The critical contribution of the paper is adapting a different existing density-based clustering method to the semantic segmentation task. In addition, I don't think the paper provides enough concrete analysis on why the proposed clustering method is better than the one in [13], despite empirical results. \n- The paper refers to the text's segmentation and object detection, while only semantic segmentation results are presented in the main experiment section. For the sake of fairness, I would recommend removing the misleading reference to the object detection task or using it for future work. \n- The visualization example doesn't contribute much to the paper as the idea of decreasing the token for tasks that do not need local fine-gran information has been proposed and demonstrated in related works. Plus, the figures are not quite self-evident. For E.g. the clustering validity can depend on ontology.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Typographical errors**\n\n**A1:** We are sorry for these typos and have corrected them in the revised manuscript. Many thanks for your patience.\n\n**Q2: The placements of figures and tables**\n\n**A2:** We sincerely appreciate your suggestion and will rearrange them to make it easier to refer to.\n\n**Q3: The motivation**\n\n**A3:** We have discussed our motivation in our main text (Line 33-42) and the general response. Please refer to the general response.\n   \n\n**Q4: Why does Ailurus perform better than previous methods?**\n\n**A4.**  As illustrated in our experiments and **the general response**, Ailurus applies a more reasonable approach for feature reconstruction. This ensures the superior performance of Ailurus compared to Expedite, especially in challenging scenarios. Please refer to the general response for more details\n\n\n**Q5: Object detection results**\n\n**A5:** You may have missed the object detection and the instance segmentation experiments in our appendix, for which we apologize. We summarize the results here for your reference and have added them to the main text in the revised manuscript. Specifically, we integrate Ailurus into well-trained Mask-RCNN models for both object detection and instance segmentation and conduct experiments over various configurations and backbones. The results show that Ailurus still performs well in other dense prediction tasks and are presented below.\n\n\nResults for ViT-B:\n\n|    | Location   | Clusters   | FPS                  | Object Detection        | Instance Segmentation   |\n|:----------|:-----------|:-----------|:---------------------|:------------------------|:------------------------|\n| Baseline  | -          | -          | 4.4                  | 50.1                    | 44.0                    |\n|    | 3          | 1296       | 6.3 ($\\uparrow$ 43%) | 48.3 ($\\downarrow$ 1.8) | 42.4 ($\\downarrow$ 1.6) |\n|    | 3          | 1764       | 5.9 ($\\uparrow$ 34%) | 49.4 ($\\downarrow$ 0.7) | 43.4 ($\\downarrow$ 0.6) |\n|    | 6          | 1444       | 5.3 ($\\uparrow$ 20%) | 49.8 ($\\downarrow$ 0.3) | 43.9 ($\\downarrow$ 0.1) |\n|    | 7          | 1444       | 5.1 ($\\uparrow$ 16%) | 49.9 ($\\downarrow$ 0.2) | 43.9 ($\\downarrow$ 0.1) |\n\nResults for ViT-L:\n\n|    | Location   | Clusters   | FPS                  | Object Detection        | Instance Segmentation   |\n|:----------|:-----------|:-----------|:---------------------|:------------------------|:------------------------|\n| Baseline  | -          | -          | 1.9                  | 54.7                    | 47.8                    |\n|    | 7          | 1600       | 2.9 ($\\uparrow$ 53%) | 53.0 ($\\downarrow$ 1.7) | 46.5 ($\\downarrow$ 1.3) |\n|    | 9          | 1764       | 2.6 ($\\uparrow$ 37%) | 54.0 ($\\downarrow$ 0.7) | 47.2 ($\\downarrow$ 0.6) |\n|    | 11         | 1764       | 2.5 ($\\uparrow$ 32%) | 54.3 ($\\downarrow$ 0.4) | 47.4 ($\\downarrow$ 0.4) |\n|    | 15         | 1600       | 2.3 ($\\uparrow$ 21%) | 54.5 ($\\downarrow$ 0.2) | 47.6 ($\\downarrow$ 0.2) |\n\nAs presented above, Ailurus still performs well in object detection and instance segmentation tasks, which demonstrates its generalization ability.\n\n**Q6: The visualizations**\n\n**A6:** We respectfully disagree with this comment. As mentioned in the general response, Ailurus focuses on dense prediction tasks, which receive less attention in previous related works. Dense prediction tasks quite rely on local fine-grain information, especially for semantic segmentation tasks which are required to produce pixel-level prediction. Thus, this paper is not so-called \"decreasing the token for tasks that do not need local fine-grain information\". On the contrary, this paper focuses on how to preserve local fine-grain information with as less as possible tokens. To this end, we propose the adaptive resolution strategy, and the most critical component is the resolution assignments. This visualization shows that Ailurus manages to preserve local fine-grain information with only 1/4 patches, which clearly demonstrates the effectiveness of the generated assignments. The visualizations strongly support our motivation and explain why Ailurus could accelerate vision transformers with negligible performance loss. As commented by Reviewer4 2MFv, \"This visualization in Fig 2. is one of the most important ones in this paper\". \n\n\nBesides, the provided visualizations contain different scenarios, and some of them are quite complicated such as 4-th, 6-th, and 8-th. These diverse and complicated scenarios lend credence to the visualization results.\n\n**Q7: Clustering location**\n\n**A7:** Yes, the clustering location indicates the layer index where we perform clustering. We have declared it in our revised manuscript. Thanks for your suggestion!\n\n**Q8: Why does Ailurus provide more robust clustering than Expedite?**\n\n**A8:** Thanks for your question! As we discussed in **the general response**, Ailurus reconstructs tokens depending on similar tokens, and their similarity could be preserved even after several transformer blocks. In contrast, Expedite reconstructs tokens using the super-pixel features, which are less similar to the original tokens. The limited similarity makes it hard to preserve their relations when passing through transformer blocks and causes more serious performance degradation compared with Ailurus. \n\n**Q9: The impact of $\\alpha$.**\n\n**A9:** Please note that, as described in the context, the variable $\\alpha$ represents the penalty strength applied to tokens within the neighborhood. A smaller value of $\\alpha$ indicates a larger penalty strength, indicating a preference for prioritizing closer tokens. Since the tokens are already constrained within a neighborhood, they inherently exhibit relatively high similarity, resulting in relatively low variance for different values of $\\alpha$. "}, {"review_id": "V6jOGQBBmz", "reviewer": "Reviewer_PvMg", "summary": "This paper proposed a spatial-aware density-based clustering algorithm for ViT-based dense prediction tasks. The proposed method can merge the original tokens into fewer representative tokens to accelerate the transformer inference without finetuning. The experiments on different datasets show that the proposed method can accelerate the ViT inference significantly with a slight performance drop.", "questions": "1. Why does the proposed method perform better than existing methods when applied to shallower layers? \n2. How does the proposed method recover the output of each transformer block?", "limitations": "The authors have adequately addressed the limitations.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. This paper focuses on an important topic of ViT inference acceleration, which may facilitate further applications. \n2. The experiments on different datasets prove that the proposed method can speed up both training and inference significantly. The experiments also show that the proposed method can outperform the Expedite method, especially at shallow layers.\n3. The visualization results are also reasonable and meaningful. ", "weaknesses": "1. The technical contribution is incremental compared with the previous method. The DPC-based token clustering module has been proposed in the previous method[29] for transformer-based dense prediction tasks. The only difference between the proposed method in this paper and the previous method is introducing spatial information into the DPC module. \n\n2. The experiments are not sufficient to justify the algorithm design. \n\n    a. The experiment results in Table 3 show that the different settings of α only result in a negligible accuracy variance. The results suggest that the proposed spatial information is incremental to the original DPC.\n\n    b. The DPC module may also introduce computation overhead for each transformer block. It is mandatory to present the break-down time consumption for each block, e.g. the DPC module, the attention module, the feature map restore module, etc.\n\n3. The DPC module will generate representative tokens adaptively for each image. It is not clear whether it will affect the batch inference performance. It is better to show a comparison in the batch inference mode.\n\n4. Some typos and format errors\n\n    a. In L.204, Fig.5 and Fig.4 should be Fig.3 and Fig.4.\n\n    b. In L.217, Fig. 7a and Fig. 7 should be Fig. 7a and Fig. 7b.\n\n    c. The line space between L.154 and L.155 is not sufficient.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: The relation to [29]**\n\n**A1:** We respectfully disagree. TCFormer and Ailurus exhibit significant differences in terms of motivation, implementation, and application scenarios. \n\n1) The motivation behind TCFormer lies in the utilization of DPC to incorporate saliency information in human-centric scenarios, leading to enhanced performance. Conversely, Ailurus is specifically designed for more general scenarios, with a focus on dense prediction tasks. Ailurus draws motivation from the heterogeneous information distribution within images and applies adaptive resolution to feature maps (please refer to the general response for our detailed motivation), resulting in acceleration for ViTs.\n  \n2) TCFormer relies on specific designs that pose challenges for integration into existing methods. Additionally, the proposed CTM module necessitates training and, therefore, cannot be integrated into well-trained models. Moreover, TCFormer requires the execution of DPC at every layer, rendering it an inefficient approach unsuitable for accelerating ViTs. In contrast, Ailurus can be seamlessly integrated into well-trained models without the need for fine-tuning, enabling instant acceleration.\n\n3) TCFormer primarily focuses on improving performance in human-centric tasks such as pose estimation, mesh reconstruction, and key point localization. Conventional dense prediction tasks like semantic segmentation have not been adequately explored. In contrast, Ailurus centers its efforts on accelerating well-trained models for dense prediction tasks.\n\n**Q2 (a): The impact of $\\alpha$.** \n\n**A2 (a):** As described in the context, the variable $\\alpha$ represents the penalty strength applied to tokens within the neighborhood. A smaller value of $\\alpha$ indicates a larger penalty strength, indicating a preference for prioritizing closer tokens. Since the tokens are already constrained within a neighborhood, they inherently exhibit relatively high similarity, resulting in relatively low variance for different values of $\\alpha$. \n\n**Q2 (b): The break-down time consumption**\n\n**A2 (b):** We decompose the running time into three components, i.e., time for transformer blocks, time for clustering, and time for recovering, and compare the elapsed time of Expedite and Ailurus. \nFor fair comparisons, we run Expedite and Ailurus under the same configuration over ViT-L. For ViT-L, we start clustering at 4-th layer and set the number of clusters as 576. \nThe results are presented below:\n\nResults for ViT-L\n|Method   | Total (ms) | blocks (ms) | clustering (ms)| recovering (ms)  | mIoU |\n|:------| :---------:|:----------:|:--------:|:-----:|:-----:|\n|Expedite   | 31.4   |   26.5      | 3.9        | 1.0  | 47.19 |\n|Ailurus     | 26.6      | 25.3        | 1.2        | 0.09  | 50.27 |\n\nThe above results indicate that Ailurus exhibits significantly lower clustering and recovery costs compared to Expedite. As we analyzed in the general response, Expedite replaces raw tokens with super-pixel features for forward passes. Consequently, Expedite relies on multiple iterations to achieve sufficiently accurate super-pixel features and necessitates the computation of reconstruction coefficients for recovering the original features. Thus, Expedite consumes more time in both clustering and reconstruction. In contrast, Ailurus directly operates tokens in the original feature space and thus could efficiently generate token assignments in a single iteration. The produced assignments can be directly used for subsequent reconstruction, thereby avoiding additional recovery costs.\n\n**Q3: The batch inference performance of Ailurus**\n\n**A3:** It is important to note that our approach employs adaptive resolution for different regions within an image, rather than adapting the number of tokens for different images. Consequently, each image is converted into an equal number of tokens, enabling efficient batch processing. In order to showcase the potential of Ailurus for batch inference, we conducted an investigation to determine the maximum throughput of both Ailurus and the original model on a single V-100 GPU by progressively increasing the batch size. The results revealed that Ailurus achieves its maximum throughput with a batch size of 32, whereas the original model attains its maximum throughput with a batch size of 16. The detailed findings are presented below:\n\n|Method   | Num clusters | Batch size |  FPS | Throughput  | mIoU |\n|:------| :---------:|:----------:|:--------:|:-----:|:-----:|\n|Baseline   | 1024   | 16 |   32.2      | 35.6        | 48.48  |\n|Ailurus     | 324          | 32 | 61.2 ($\\uparrow$ 90%)       | 81.6 ($\\uparrow$ 129%)       | 46.45  |\n|     | 400      | 32 | 53.7 ($\\uparrow$ 67%)       | 69.6 ($\\uparrow$ 96%)        | 47.12  | \n\nThe results presented above demonstrate that by increasing the batch size, we can achieve a more substantial acceleration in terms of throughput compared to the baseline model.\n\n**Q4: Why Ailurus performs better than existing methods when applied to shallower layers?**\n\n**A4:** Thanks for your question! As we discussed in the general response, Ailurus reconstructs tokens using the most similar ones, and their similarity could be preserved even after several transformer blocks. In contrast, Expedite reconstructs tokens using the super-pixel features, which are less similar to the original tokens. The limited similarity makes it hard to preserve their relations when passing through transformer blocks (the more passed blocks, the less similarity is preserved) and causes more serious performance degradation compared with Ailurus. Thus, Ailurus performs much better than Expedite when applied to shallower layers as passing more blocks. \n\n**Q5: How does the proposed method recover the output of each transformer block?**\n\n**A5:** We recover the spatial resolution of the feature map through the unmerge operation. Specifically, we store the produced assignments and unfold merged tokens at output layers."}, {"review_id": "rNnKOaalJ2", "reviewer": "Reviewer_2MFv", "summary": "This paper first points out that when handling long token sequences, ViTs particularly suffer in terms of computation. Motivated by an observation that dense prediction tasks focus more on the contours or shapes of object, the authors propose to leverage adaptive resolution strategy for different regions depending on each regions' importance. Specifically, regions where repetitive patterns are present, e.g., walls, they are merged to predefined anchors to form low resolution regions, while other \"important\" regions are preserved for further processing. Thanks to this design, the number of tokens have been greatly reduced, and their shapes are recovered via unfolding operation at the output end. This method has shown to its effectiveness using three different datasets. Especially, for the variant that uses Segmenter ViT-L, almost 48% FPS acceleration is accomplished, which can also benefit to reduce the training time required to learn the models. ", "questions": "See above weaknesses. I do not have any major issue.  ", "limitations": "\"We attribute this to the complex designs used in object detection, such as the FPN and the complicated decoder. AiluRus can be improved by mitigating the performance drop caused by these complex design\" \n\nIf this was also addressed in this paper, it would have made this paper much stronger. However, this paper already shows good results and analysis in main task (segmentation). ", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. I really like the motivation and idea of this paper. It is a well known issue that the number of tokens fed to ViTs will inevitably require expensive computations and memory footprints, and to me, this paper nicely addresses this issue with a novel approach. \n\n2. This paper is very easy to read and understand.\n\n3. Significant improvements are made when this method is applied. ", "weaknesses": "1. Figure 2 visualization looks good. I also found the authors saying in L177 that some regions looking like they are assigned to a same group (same color) is not actually in the same group. I believe this visualization in Fig 2. is one of the most important one in this paper, which I would recommend looking for different visualization to make this figure look more clear and easy to catch the idea without reading the texts. Maybe scale the values then visualize? or change the colors? \n\n \n\n2. Adding ViT-B results in Table 1 would make this table more informative. \n \n3. What could be the reason for performance improvements in Table 2 last row (AiluRus clusters 900) ?\n\n\n\nMinor comments : typo in L200. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\n**Q1: Improve visualization**\n\n**A1:** Thank you for your suggestion! We greatly appreciate it. Based on the concept of coloring a map, we have made improvements to our visualization. Specifically, we use highly distinguishable colors to differentiate adjacent patches belonging to different assignments, while not distinguishing non-adjacent patches. This approach allows us to represent our assignments using a small number of highly distinguishable colors. We have included the updated visualization in the supplementary PDF file. The results demonstrate that our algorithm effectively adheres to our motivation, which is to preserve more tokens in high-information-density regions and fewer tokens in low-information-density regions. This provides a compelling explanation for the outstanding performance of Ailurus. We have incorporated these findings into our revised manuscript.\n\n**Q2: Adding ViT-B results in Table 1**\n\n**A2:** As Segmenter only provide well-trained ViT-B for ADE20K, we add the comparison between different efficient vision transformer methods on ADE20K over ViT-B.\n\n|Method   | GFLOPs | FPS | mIoU | GFLOPs | FPS | mIoU | GFLOPs | FPS | mIoU |\n|:------| :---------:|:----------:|:--------:|:-----:|:-----:|:----------:|:--------:|:-----:|:-----:|\n|Baseline   | 124.7   | 32.2      | 48.48        |124.7   | 32.2      | 48.48         |124.7   | 32.2      | 48.48        |\n|ACT       | 105.1          | 26.0        | 48.39        | 105.1  | 25.9 |    47.55     |    105.1     |  26.1 | 44.01 |\n|ToMe     | 100.2      | 31.2        | 47.99        | 88.58  | 32.4 |   46.96      | 66.4        | 35.6  | 40.85 |\n|EViT       | 109.4          | 34.1        | 48.44        | 96.5  | 37.8 |   48.05      | 69.4        | 46.5  | 38.27 |\n|Expedite     | 110.2      | 29.8        | 46.74        | 97.4  | 34.6 |   46.14      |  87.1        | 39.4  | 45.17 |\n|Ailurus       | **62.3**          | **37.9**        | **48.59**        | **50.3**  | **45.9** |   **48.38**      | **40.2**        | **55.3**  | **47.32** |\n\nAs presented above, Ailurus demonstrates even more obvious advantages over previous methods. We owe it to two folds.\n\nFirst, Ailurus introduces negligible extra overheads. As posted in the response to Reviewer3 PvMg Q2(b), the clustering and recovering operations of Ailurus only take around 1.3 ms while our counterpart, i.e., Expedite takes 2.7 ms which is double our cost. For other methods, ACT and ToMe require to deploy their algorithm at each transformer block, and EViT requires drop tokens three times. As a result, these methods introduce more extra overheads and cause negligible or even negative efficiency improvement at the slight acceleration scenario.\n\nSecond, Ailurus is more effective in recovering raw features from reduced ones. As posted in the response to Reviewer gX5G, compared with the recent SOTA method Expedite, Ailurus performs much better in recovering features across various configurations, which empowers Ailurus to accelerate vision transformers with much less performance drop.\n\nThanks for your suggestion! We have added it to our revised manuscript.\n\n**Q3: The reason for performance improvement**\n\n**A3:** As stated in our paper, Ailurus employs adaptive resolution for input images. This strategy introduces a smoothness constraint to low information density regions, such as backgrounds, effectively incorporating a prior for smooth predictions in these areas. When retained an adequate number of tokens, this prior aligns well with the true information distribution of the images, resulting in performance gains for the model. We provide visualization of the prediction results of Ailurus and the original one in Figure 1 in supplemented pdf. As depicted in the figure, the predictions generated by Ailurus are more smooth, thereby avoiding terrible predictions such as voids or spikes. Consequently, in certain scenarios, Ailurus demonstrates superiority over the original model."}, {"review_id": "QSKFejPrlY", "reviewer": "Reviewer_viNG", "summary": "ViTs process input images as a sequence of tokens, and as the token size increases, the computational complexity escalates. The paper aims to accelerate the training and inference of ViTs by reducing the number of tokens. To achieve this, the authors introduce an adaptive resolution strategy, leveraging the observation that dense prediction tasks primarily emphasize object shape and contour. They present a density-based clustering algorithm, spatial-aware DPC, which effectively prunes tokens. The proposed method is extensively evaluated on three diverse datasets, demonstrating its successful acceleration of ViT-based models while preserving performance.", "questions": "- Can the proposed algorithm be applied to classification tasks?\n- In Table. 2, the mIoU of AiluRus with 900 clusters outperforms the baseline, I’m interested in the explanation for this. ", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- This paper presents an efficient ViT that reduces the complexity of the self-attention operation with a novel dense-based clustering algorithm.\n- The paper is well written, and the motivation and technical details are clearly presented.\n- The experiments show good qualitative and quantitative results compared to baselines. ", "weaknesses": "- The idea of merging tokens to speed up ViT is not new (STViT, PaCa-ViT, etc.), and the proposed algorithm relies on DPC, which also reduces the novelty.\n- Comparisons are only conducted with Expedite, while there are many other efficient ViTs as introduced in the related works. Though some of them requires fine-tuning, but as least can be compared with AiluRus in table 4.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\n**Q1: The novelty**\n\n**A1:**  \n1. Please note that both STViT and PaCa-ViT rely on training and specific designs, which limits their ability to accelerate well-trained models. In contrast, Ailurus can be seamlessly integrated into pre-trained models for instant acceleration. To illustrate the versatility of Ailurus, we use the recent video generation work, ControlVideo [1], as an example. ControlVideo employs StableDiffusion and ControlNet to generate videos from text. We successfully integrated Ailurus into ControlVideo, reducing the number of tokens during inference by half. The generated results are included in the supplementary PDF file. Notably, Ailurus achieves comparable quality to the original model while significantly reducing the processing time (2:11 vs 4:24). This demonstrates the effectiveness of Ailurus in accelerating large models instantly. Such instant acceleration is unattainable with methodologies like STViT and PaCa-ViT. Moreover, training a new StableDiffusion or other large models using their approaches is nearly impractical due to the architecture gap (Diffusion models mostly use U-nets while STViT and PaCa-ViT are difficult to migrate into U-nets due to their unique designs) and computational overheads. Therefore, it is unreasonable to criticize the novelty of Ailurus based on these methods, as Ailurus offers more promising applications, especially in the current landscape of emerging large models.\n\n2. As discussed in our main text (Line 37-42) and the general response, Ailurus is motivated by **the heterogeneous distribution of information within images**. Based on this observation, we propose to apply adaptive resolution for the features of different regions. To this end, we need to measure **the information density in different regions** within the image, and DPC, which is based on **local spatial density**, is one of the ways to approach our motivation. \n\n3. We provide concrete analysis and abundant experiments to show the differences and advantages of Ailurus to previous methods. The results show that Ailurus significantly outperforms previous SOTA methods.\n\n4. As posted in the general response, DPC is one of the ways to measure information density. Under the framework of Ailurus, it can be replaced with other strategies while keeping satisfactory performance. For example, we replace the DPC clustering with a naive strategy where the information density is measured by the average distance to other tokens. This strategy, termed Ailurus-Dist, still outperforms Expedite significantly as posted below, which demonstrates the effectiveness of Ailurus.\n\n| Method   |   18 * 18 |   20 * 20 |   22 * 22 |   24 * 24 |\n|:---------|----------:|----------:|----------:|----------:|\n| Layer index = 2  |     |    |     |     | \n| Expedite  |    4.32 |    5.21 |    6.19 |    7.57 | \n| Ailurus-Dist  |    36.54  |  40.33 |    43.43 |   45.97 | \n| Layer index = 4  |     |    |     |     |  \n| Expedite  |    12.66 |    15.53 |    18.03 |    21.57 |\n| Ailurus-Dist  |    41.89  |  44.14 |    45.79 |   47.29 | \n\n**Q2: Comparison to other efficient ViTs**\n\n**A2：** Please note that we have compared various efficient ViTs in Table 1, and most of them exhibit poor performance on dense prediction tasks. We take Expedite as our main counterpart because it is the SOTA method. Besides, we also demonstrate the versatility of Ailurus by integrating it into ControlVideo for acceleration, which is inaccessible for the mentioned STViT or PaCa-ViT.\n\n**Q3: Classification tasks**\n\n**A3:**\nWhile we focus on dense prediction tasks in this paper, Ailurus can be easily integrated into classification tasks for instant acceleration. Specifically, we take well-trained DEiT as the base model and integrate Ailurus into it **without finetuning**. We run the same configurations for Expedite \n\n|Method   | Location | Clusters  | Throughput (imgs/s) | FLOPs (G) | Acc |\n|:------| :---------:|:----------:|:--------:|:-----:|:-----:|\n|Baseline   | -   | 196      |    268.2 |  17.7   |81.8   |\n|Expedite     | 6      | 144        |   299.7 ($\\uparrow$ 12%) |   15.5 ($\\downarrow$ 12%)  | 81.0 ($\\downarrow$ 0.8)  |\n|Ailurus       | 4          | 144        |    313.7 ($\\uparrow$ 17%) |  14.9 ($\\downarrow$ 16%)   | **81.7** ($\\downarrow$ 0.1)  |\n|Expedite     | 4      | 144        |  314.3 ($\\uparrow$ 17%)   | 14.7 ($\\downarrow$ 17%)  | 79.3 ($\\downarrow$ 2.5)  |\n|Ailurus       | 0          | 144        |   350.5 ($\\uparrow$ 31%) |   13.3 ($\\downarrow$ 25%)  | **81.3** ($\\downarrow$ 0.5)  |\n|Expedite     | 2      | 121        |   390.4 ($\\uparrow$ 46%)     | 12.1 ($\\downarrow$ 32%)  |  78.1 ($\\downarrow$ 3.7)   |\n|Ailurus       | 0          | 121        |   419.8 ($\\uparrow$ 57%) |   11.4 ($\\downarrow$ 36%)   | **80.4** ($\\downarrow$ 1.4)  |\n\nAs posted above, Ailurus still outperforms Expedite on classification tasks.\n\n**Q4: Why Ailurus performs better than baseline?**\n\n**A4:** As stated in our paper, Ailurus employs adaptive resolution for input images. This strategy introduces a smoothness constraint to low information density regions, such as backgrounds, effectively incorporating a prior for smooth predictions in these areas. When retained an adequate number of tokens, this prior aligns well with the true information distribution of the images, resulting in performance gains for the model. We provide visualization of the prediction results of Ailurus and the original one in Figure 1 in supplemented pdf. As depicted in the figure, the predictions generated by Ailurus are more smooth, thereby avoiding terrible predictions such as voids or spikes. Consequently, in certain scenarios, Ailurus demonstrates superiority over the original model.\n\n[1]. Zhang, Yabo, et al. \"ControlVideo: Training-free Controllable Text-to-Video Generation.\" arXiv preprint arXiv:2305.13077 (2023).\n\n"}]

Author rebuttal:
We sincerely appreciate all reviewers for their efforts, and first give general responses for some common issues.

**Why does Ailurus performs better than Expedite?**

As presented in the original paper, Ailurus significantly outperforms Expedite, especially in cases **when performing token reduction at shallow layers**, which is more challenging as it suffers more serious information loss. We explain the reasons with detailed experiments to confirm the claim. 

i) **Comparing the reconstruction errors of Ailurus and Expedite**. As posted in the response to Reviewer1 gx5G, Ailurus enjoys much lower reconstruction errors compared with Expedite across various settings, which partially explains why Ailurus performs better than Expedite.

ii) We further observe that **Ailurus exhibits more pronounced advantages in shallow clustering while Expedite's performance deteriorates a lot**. Since both Ailurus and Expedite perform clustering only once and reuse the clustering results at the output layer, it is evidently easier to perform clustering in deeper layers compared to shallow layers.  To investigate the reasons behind the performance gap in shallow clustering, we take a representative scenario (ViT-B, layer index=2, num cluster=20*20), and calculate the feature reconstruction quality in subsequent layers. The results are presented below. The results indicate that Expedite suffers from serious reconstruction distortion during the forward pass while Ailurus preserves the reconstruction quality across layers.

| Method  | Layer-2 | Layer-3 | Layer-4 | Layer-5 | Layer-6 | Layer-7 | Layer-8 | Layer-9 | Layer-10 | Layer-11 |
| :------ | ------: | ------: | ------: | ------: | ------: | ------: | ------: | ------: | -------: | -------: |
| Ailurus |  0.9210 |  0.8959 |  0.8863 |  0.8906 |  0.8972 |  0.9092 |  0.9238 |  0.9312 |   0.9361 |   0.9291 |
| Expedit |  0.7334 |  0.6228 |  0.5524 |  0.4824 |  0.4293 |  0.3694 |  0.3778 |  0.4219 |   0.4433 |   0.3342 |

iii) We analyze **the advantages of Ailurus over Expedite in terms of intrinsic mechanism**: both methods target at reducing the number of tokens while preserving the relationships between the retained tokens and the reduced ones, and then utilizing these relationships for recovery at the output layer. The quality of the recovery process determines the performance, and **the recovery quality depends on whether the preserved relationships can be maintained after passing through several Transformer blocks**. Expedite and Ailurus employ different methods to preserve these relationships.

**Compared with Expedite, the similarity between the aggregated tokens and the original tokens in Ailurus are higher.** For Expedite makes use of K-means clustering, while Ailurus directly aggregates low information density regions within the image (i.e., regions with high redundancy). To validate this empirically, we conducted a study on the similarity between the original tokens and the super-pixel features in Expedite, as well as the similarity between the aggregated tokens and the original tokens in Ailurus during the inference process. **The results showed that the average similarity in Expedite was 0.6728, while in Ailurus, it was 0.9210, which aligns with our expectations**.

We also investigate the probability of preserving the relationships at different levels of similarity at the output layer. The results are as follows. The results demonstrate that for tokens with a similarity between 0.9 and 1.0, there is a probability of over 99% that they will maintain this similarity at the output layer. However, tokens in other similarity ranges suffer from significant similarity distortion. This explains why Ailurus outperforms Expedite.

|   Intervals | 0.0-0.1 | 0.1-0.2 | 0.2-0.3 | 0.3-0.4 | 0.4-0.5 | 0.5-0.6 | 0.6-0.7 | 0.7-0.8 | 0.8-0.9 | 0.9-1.0 |
| ----------: | ------: | ------: | ------: | ------: | ------: | ------: | ------: | ------: | ------: | ------: |
| Probability |  0.0137 |  0.1246 |  0.2179 |  0.1439 |  0.0629 |  0.0347 |  0.0255 |  0.0193 |  0.0212 |  0.9903 |

**The motivation of Ailurus.**
1.  While there have been studies on accelerating Vision Transformers (ViTs) through token reduction, most of them have focused on classification tasks. This is because reducing the number of tokens affects the spatial resolution of feature maps, making them unsuitable for dense prediction tasks. In contrast, our method focuses on accelerating dense tasks, which is crucial for the efficient deployment of ViTs.

2.  Expedite attempts to accelerate dense prediction tasks by **replacing original features with fewer superpixel features**. However, as presented in our main text (Line 30-32), Expedite's acceleration is limited because it can only be deployed in relatively deep ViT layers.
We provide a concrete analysis of this issue in the general response 1, and our findings suggest that this limitation is due to the similarity between superpixel features and original features becoming lower and lower after several transformer blocks.

3.   The limitation of Expedite inspires us to consider **operating tokens in the original feature space**. Based on the **heterogeneous information distribution** in the image, we propose to apply adaptive resolution for features of different regions. Specifically, we use fewer tokens for regions with lower information density, while more tokens for regions containing local fine-grain information. Thus we can minimize the number of tokens while preserving performance in downstream dense prediction tasks. To achieve this goal, the core is to measure the information density. Therefore, we have chosen an algorithm based on **local spatial density**, i.e., DPC, to allocate the resolution in different regions of the image.

Meta-review: This paper received mixed reviews.

The reasons to accept the paper:

- Target an important problem: speeding up dense prediction of ViT models for both training and inference.
- Good performance on standard benchmarks across different tasks: object detection, instance segmentation, and semantic segmentation.
- Abundant ablation studies with both quantitative and visual analysis.

The reason to reject the paper is because the idea is not novel enough. Specifically, the reviewers ZFoB, PvMg, and viNG pointed out that the DPC-based token clustering module has been proposed in the previous method[29] for transformer-based dense prediction tasks. The reviewer ZFoB is concerned the clustering idea is also proposed by [13] and not enough analysis has been shown why the proposed clustering method is better than [13].

In the rebuttal, the authors provided more empirical results showing

- Better performance and analysis of the proposed apporach than Expedite [13]. The proposed approach exhibits more advantages in shallow clustering while Expedite's performance deteriorates a lot.
- TCFormer [29] is specifically designed for human-centric tasks, and adapting it to dense prediction tasks poses significant challenges.

After the rebuttal, reviewers PvMg and viNG recommend Borderline Reject (viNG did not participate in the discussions with the authors).

The AC acknowledges that the DPC-based token clustering module has been studied in [29]. But the rebuttal shows that adapting it to dense prediction task is not trivial, making the proposed approach valuable. Overall, the AC finds the positive side of reviews outweights the negative side and thus recommends an acceptance as a poster.
Decision: Accept (poster)