SlowGuess/NeurIPS_2023_OCR huggingface dataset:
More comparisons to Expedite. As both AiluRus and Expedite accelerate ViTs by reducing the feature size at the intermediate layer, they can be further compared under the same configuration. Specifically, we fix either the number of clusters or the cluster location and vary the other parameters across different configurations for both ViT-L and ViT-B models. Initially, we fix the cluster location at 2 and experiment with different numbers of clusters. As illustrated in Fig. 4, AiluRus consistently outperforms Expedite under all settings. The advantage of AiluRus is especially evident for ViT-B,
(a)
Figure 6: We run different configurations of AiluRus and Expedit (i.e. the cluster number and cluster location) and illustrate the optimal mIoU-FPS curve for ViT-L and ViT-B in figure (a) and (b).
Figure 5: With the officially released 'Segmenter[26] ViT-B', we illustrate the FPS comparison between AiluRus and Expedite under the same configuration.
(b)
where AiluRus maintains performance even with only 484 clusters, while Expedite almost does not work. Besides the performance, as illustrated in Fig. 5, AiluRus demonstrates higher efficiency than Expedite under the same configuration. This can be attributed to the lower complexity of AiluRus. Unlike Expedite, which requires multiple iterations for accurate cluster centers and inevitably involves high latency, AiluRus selects cluster centers from candidate tokens in one step, making it more suitable for latency-sensitive scenarios. This advantage becomes more pronounced in accelerating relatively lightweight architectures.
Next, we fix the number of clusters at 400 and analyze the impact of cluster location. As depicted in Fig. 3, the performance of AiluRus is close to Expedite when clustering is applied at deeper layers but significantly exceeds Expedite when clustering is performed at shallower layers. This suggests that AiluRus can achieve higher acceleration ratios by reducing the feature map size at shallower layers. In contrast, Expedite suffers serious performance degradation when clustering is applied at shallower layers, limiting its potential in accelerating ViTs.
To make a more comprehensive comparison between AiluRus and Expedite, we run various configurations for AiluRus and Expedite and illustrate the optimal mIoU-FPS curve in Fig. 6. The results consistently demonstrate that AiluRus achieves superior trade-offs between performance and efficiency compared to C, particularly under high acceleration ratios.
4.2 Acceleration for object detection and instance segmentation.
To further validate the generalization of AiluRus in dense prediction tasks, we deploy it on object detection and instance segmentation tasks to achieve instant acceleration. Since there are no well-trained models available directly, we follow the setup of CAE and train detection models based on ViT-L and ViT-B within the Mask-RCNN framework. Specifically, we use the CAE pre-trained model as initialization and perform 1x schedule training. The training hyper-parameters strictly
Table 3: The results of AiluRus accelerating Mask-RCNN based on ViT-L.
Methods	Location	Clusters	FPS	Object Detection	Instance Segmentation
Baseline	-	-	1.9	54.7	47.8
AiluRus	7	1600	2.9 (↑53%)	53.0 (↓1.7)	46.5 (↓1.3)
9	1764	2.6 (↑37%)	54.0 (↓0.7)	47.2 (↓0.6)
11	1764	2.5 (↑32%)	54.3 (↓0.4)	47.4 (↓0.4)
15	1600	2.3 (↑21%)	54.5 (↓0.2)	47.6 (↓0.2)
Table 4: The results of AiluRus accelerating Mask-RCNN based on ViT-B.
Methods	Location	Clusters	FPS	Object Detection	Instance Segmentation
Baseline	-	-	4.4	50.1	44.0
AiluRus	3	1296	6.3 (↑43%)	48.3 (↓1.8)	42.4 (↓1.6)
3	1764	5.9 (↑34%)	49.4 (↓0.7)	43.4 (↓0.6)
6	1444	5.3 (↑20%)	49.8 (↓0.3)	43.9 (↓0.1)
7	1444	5.1 (↑16%)	49.9 (↓0.2)	43.9 (↓0.1)
Table 5: The results of AiluRus accelerating fine-tuning based on ViT-L.
Methods	Location	Clusters	Training Time	FPS	mIoU
Baseline	-	-	26.76	6.55	52.16
Expedite [15]	2	400	14.75 (↓45%)	18.44 (↑182%)	48.98 (↓3.18)
2	484	15.37 (↓43%)	17.53 (↑168%)	49.63 (↓2.53)
2	576	15.62 (↓42%)	16.39 (↑150%)	50.41 (↓1.75)
2	900	18.46 (↓31%)	10.77 (↑64%)	50.62 (↓1.54)
8	400	19.19 (↓28%)	12.92 (↑97%)	50.74 (↓1.42)
10	784	21.27 (↓21%)	9.02 (↑38%)	51.98 (↓0.18)
AiluRus	2	400	11.80 (↓56%)	19.24 (↑194%)	50.82 (↓1.34)
2	484	12.28 (↓54%)	17.30 (↑164%)	51.82 (↓0.34)
2	576	12.81 (↓52%)	16.11 (↑146%)	52.07 (↓0.09)
2	900	16.37 (↓39%)	11.24 (↑72%)	52.42 (↑0.26)
follow the CAE setup, and the reproduce results are also close to the officially reported ones (reproduce vs official: 50.1 vs 50.3 for ViT-B, 54.7 vs 54.5 for ViT-L). Subsequently, we integrate AiluRus into the well-trained models without fine-tuning, and the hyper-parameters (i.e.  $\alpha$ ,  $\lambda$  and  $k$ ) of AiluRus remained the same as in previous experiments. We vary the cluster location and the number of clusters to achieve different acceleration ratios, these results are presented in Tab. 3 and Tab. 4. The results demonstrate that AiluRus generalizes well in object detection and instance segmentation tasks, indicating that AiluRus can effectively expedite various dense prediction tasks with minor modifications.
4.3 Acceleration for fine-tuning
The growing capacity of ViTs leads to satisfactory performance but remains a major challenge in fine-tuning the pre-trained models. Since AiluRus does not rely on any specific architecture or learnable parameter, it can be seamlessly integrated into the fine-tuning phase for acceleration. We adopt AiluRus to Segmenter[26] and fine-tune the pre-trained model following the official schedule. Our code base is MMsegmentation [5], and the results reported by MMsegmentation are used as the baseline. We fine-tune the pre-trained modes on 8 V100-32G and evaluate the FPS on single V-100 32G. Both ViT-L and ViT-B are evaluated, and the results are reported in Tab. 5 and Tab. 6. We surprisingly find that AiluRus can largely reduce the overheads in fine-tuning while slightly degrading performance or even enjoying better performance. Specifically, take ViT-L as example, AiluRus reduces  $52\%$  training time  $(\downarrow)$  and improves  $146\%$  FPS  $(\uparrow)$  with only  $0.09\mathrm{mIoU}$  drop when remaining 576 clusters. Besides, AiluRus even achieves better performance  $(\uparrow 0.26)$  with improved FPS  $(\uparrow 72\%)$  when remaining 900 clusters.
Table 6: The results of AiluRus accelerating fine-tuning based on ViT-B.
Methods	Location	Clusters	Training Time	FPS	mIoU
Baseline	-	-	6.89	32.2	49.60
Expedite [15]	2	324	6.12 (↓11%)	53.8 (↑67%)	39.66 (↓9.94)
2	400	6.13 (↓11%)	48.2 (↑50%)	40.20 (↓9.40)
2	484	6.16 (↓11%)	43.3 (↑34%)	41.27 (↓8.33)
2	576	6.21 (↓10%)	38.9 (↑21%)	41.94 (↓7.66)
6	400	6.79 (↓1%)	37.7 (↑17%)	47.00 (↓2.60)
7	400	6.94 (↑1%)	35.8 (↑11%)	48.88 (↓0.72)
AiluRus	2	324	4.94 (↓28%)	61.5 (↑91%)	48.06 (↓1.54)
2	400	5.14 (↓25%)	54.1 (↑68%)	49.04 (↓0.56)
2	484	5.33 (↓23%)	49.7 (↑54%)	49.39 (↓0.21)
2	576	5.48 (↓20%)	44.0 (↑37%)	49.57 (↓0.03)
Table 7: Ablation study of the hyper-parameters.
Parameter	α	λ	k
0.6	0.7	0.8	0.9	1.0	0	20	30	50	70	1	2	3	4
mIoU	50.59	50.76	50.75	50.81	50.67	50.47	50.48	50.58	50.81	50.73	50.81	50.45	50.29	50.19
We also run Expedite under the same configurations for a comprehensive comparison. The results in Tab. 5 and Tab. 6 indicate that Expedite still works worse when reducing feature size at shallower layers even with fine-tuning. We further run Expedite under official configurations and find that its performance is obviously improved. However, the acceleration ratios are also significantly decreased as these configurations reduce feature size at deeper layers. This comparison shows that the advantage of AiluRus over Expedite is more obvious in accelerating fine-tuning. We attribute this to the good robustness of AiluRus for feature size reduction and shallower cluster locations. This property enables AiluRus maintain considerable performance at high acceleration ratios with well-trained models and compensates for the performance drop during fine-tuning. In contrast, we find that the low tolerance to shallow clustering layers of Expedite cannot be addressed by fine-tuning, and ultimately results in limited efficiency improvement.
We conducted hyper-parameter ablation experiments on the adaptive resolution strategy presented in Section 3.2 using the ADE20K semantic segmentation benchmark and the officially released Segmenter ViT-L/16 [26] checkpoint. For the neighbor weight hyper-parameter  $\alpha$ , we searched its value from 0.6 to 1.0 (1.0 indicates disabling this hyper-parameter), and the results showed that  $\alpha = 0.9$  performed best. Similarly, we searched the value of  $\lambda$  from 0 to 70 (0 indicates not using spatial information), and the results showed that  $\lambda = 50$  performed best. The ablation results of  $k$  indicated that  $k = 1$ , i.e., choosing the closest token to calculate the local density, performed best.
The emergence of ViTs has empowered numerous vision tasks but also brought increasing overheads in fine-tuning and inference models. In this paper, we proposed a plug-in strategy, called AiluRus, that can be integrated into well-trained models to immediately accelerate inference without any fine-tuning or to pre-trained models to expedite both fine-tuning and inference. Our experiments demonstrated the advantages of AiluRus over previous methods, particularly in cases with high acceleration ratios. For example, with Segmenter ViT-L [26], AiluRus could accelerate FPS by  $45\%$  for the well-trained model with a negligible performance drop  $(\downarrow 0.03)$ . For the pre-trained model, AiluRus could reduce fine-tuning time by  $52\%$  and accelerate FPS by  $146\%$  with a minor performance drop  $(\downarrow 0.09)$ . These impressive results lead us to believe that current dense prediction tasks contain significant redundancy that unnecessarily benefits performance and can be removed for significant efficiency improvements. We hope that this consideration could inspire future work in the design of dense prediction tasks.
This work was supported in part by the National Natural Science Foundation of China under Grant 62250055, Grant 61931023, Grant T2122024, Grant 62125109, Grant 61932022, Grant 61972256, Grant 61971285, Grant 61831018, Grant 62120106007, Grant 62320106003, Grant 62371288, and in part by the Program of Shanghai Science and Technology Innovation Project under Grant 20511100100.
[1] Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token merging: Your ViT but faster. In International Conference on Learning Representations, 2023.
[2] Shuning Chang, Pichao Wang, Ming Lin, Fan Wang, David Junhao Zhang, Rong Jin, and Mike Zheng Shou. Making vision transformers efficient from a token sparsification view. arXiv preprint arXiv:2303.08685, 2023.
[3] Jieneng Chen, Yongyi Lu, Qihang Yu, Xiangde Luo, Ehsan Adeli, Yan Wang, Le Lu, Alan L Yuille, and Yuyin Zhou. Transunet: Transformers make strong encoders for medical image segmentation. arXiv preprint arXiv:2102.04306, 2021.
[4] Bowen Cheng, Ishan Misra, Alexander G. Schwing, Alexander Kirillov, and Rohit Girdhar. Masked-attention mask transformer for universal image segmentation. 2022.
[5] MMSegmentation Contributors. MMSegmentation: Openmmlab semantic segmentation toolbox and benchmark. https://github.com/open-mmlab/mmsegmentation, 2020.
[6] Giannis Daras, Nikita Kitaev, Augustus Odena, and Alexandros G Dimakis. Smyrf-efficient attention using asymmetric clustering. Advances in Neural Information Processing Systems, 33:6476-6489, 2020.
[7] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248–255, 2009.
[8] Shuangrui Ding, Peisen Zhao, Xiaopeng Zhang, Rui Qian, Hongkai Xiong, and Qi Tian. Prune spatio-temporal tokens by semantic-aware temporal accumulation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 16945-16956, 2023.
[9] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR 2021: The Ninth International Conference on Learning Representations, 2021.
[10] Mingjing Du, Shifei Ding, and Hongjie Jia. Study on density peaks clustering based on k-nearest neighbors and principal component analysis. Knowledge-Based Systems, 99:135-145, 2016.
[11] Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva: Exploring the limits of masked visual representation learning at scale. arXiv preprint arXiv:2211.07636, 2022.
[12] Ryan Grainger, Thomas Paniagua, Xi Song, and Tianfu Wu. Learning patch-to-cluster attention in vision transformer. arXiv preprint arXiv:2203.11987, 2022.
[13] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. arXiv preprint arXiv:2304.02643, 2023.
[14] Jin Li, Yaoming Wang, XIAOPENG ZHANG, Yabo Chen, Dongsheng Jiang, Wenrui Dai, Chenglin Li, Hongkai Xiong, and Qi Tian. Progressively compressed auto-encoder for self-supervised representation learning. In The Eleventh International Conference on Learning Representations, 2023.
[31] Yifan Xu, Zhijie Zhang, Mengdan Zhang, Kekai Sheng, Ke Li, Weiming Dong, Liqing Zhang, Changsheng Xu, and Xing Sun. Evo-vit: Slow-fast token evolution for dynamic vision transformer. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 2964-2972, 2022.
[32] Wang Zeng, Sheng Jin, Wentao Liu, Chen Qian, Ping Luo, Wanli Ouyang, and Xiaogang Wang. Not all tokens are equal: Human-centric visual analysis via token clustering transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11101-11111, 2022.
[33] Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12104-12113, 2022.
[34] Hao Zhang, Feng Li, Shilong Liu, Lei Zhang, Hang Su, Jun Zhu, Lionel M Ni, and Heung-Yeung Shum. Dino: Detr with improved denoising anchor boxes for end-to-end object detection. arXiv preprint arXiv:2203.03605, 2022.
[35] Yabo Zhang, Yuxiang Wei, Dongsheng Jiang, Xiaopeng Zhang, Wangmeng Zuo, and Qi Tian. Controlvideo: Training-free controllable text-to-video generation. arXiv preprint arXiv:2305.13077, 2023.
[36] Minghang Zheng, Peng Gao, Renrui Zhang, Kunchang Li, Xiaogang Wang, Hongsheng Li, and Hao Dong. End-to-end object detection with adaptive clustering transformer. arXiv preprint arXiv:2011.09315, 2020.
[37] Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ade20k dataset. International Journal of Computer Vision, 127(3):302-321, 2019.
[38] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. arXiv preprint arXiv:2010.04159, 2020.
[39] Xueyan Zou, Jianwei Yang, Hao Zhang, Feng Li, Linjie Li, Jianfeng Gao, and Yong Jae Lee. Segment everything everywhere all at once. arXiv preprint arXiv:2304.06718, 2023.
A Statistics of Assignments.
The produced assignments of each token play a crucial role in the trade-off between performance and throughput. Therefore, we present the assignment statistics in Fig. 7a, where we deploy AiluRus on Segmenter ViT-L and perform clustering on the output of the second layer. The produced assignments are collected across the ADE20K [37] validation set. For a given cluster center, the number of tokens belonging to it is denoted by  $x$ , while  $y_{1}(x)$  and  $y_{2}(x)$  indicate the percentage or the number of cluster centers that dominate  $x$  tokens. We run this configuration with varying numbers of cluster centers and illustrate their corresponding  $y_{1}(x)$  and  $y_{2}(x)$  in Fig. 7a.
(b)
Figure 7: The statistics of the assignments. The horizontal axis indicates how many tokens each cluster center has. The vertical axis denotes the percentages and amount of a certain kind of cluster centers in (a) and (b) respectively.
The results reveal that cluster centers containing only one token are the most in all cases, and their percentage increases with an increase in the number of cluster centers. As shown in Fig. 7b, the number of cluster centers that dominate more tokens is similar for different total numbers of cluster centers. This suggests that the assignments of the low-resolution regions (dominating more tokens) are similar across different total numbers of cluster centers and the extra cluster centers are assigned to the high-resolution areas. This phenomenon justifies AiluRus because the redundant regions within an image remain fixed, while the areas containing details require more tokens to describe them. Therefore, when increasing the number of cluster centers, the additional cluster centers should be assigned to high-resolution regions to reduce the error.
B Acceleration for Classification
Table 8: The results of AiluRus accelerating classification tasks based on DEiT-B [27].
Methods	Location	Clusters	Throughput (imgs/s)	FLOPs (G)	Accs
Baseline	-	-	268.2	17.7	81.8
Expedite [15]	2	121	390.4 (↑46%)	12.1 (↓32%)	78.1 (↓3.7)
4	144	314.3 (↑17%)	14.7 (↓17%)	79.3 (↓2.5)
6	144	299.7 (↑12%)	15.5 (↓12%)	81.0 (↓0.8)
AiluRus	0	121	419.8 (↑57%)	11.4 (↓36%)	80.4 (↓1.4)
0	144	350.5 (↑31%)	13.3 (↓25%)	81.3 (↓0.5)
4	144	313.7 (↑17%)	14.9 (↓16%)	81.7 (↓0.1)
While we focus on dense prediction tasks in this paper, AiluRus can be easily integrated into classification tasks for instant acceleration. Specifically, we take the typical ImageNet [7] supervised pre-trained DEiT-B as the base model and integrate AiluRus into it without fine-tuning. We run the
same configurations for Expedite [15] in comparison. As presented in Tab. 8, AiluRus consistently outperforms Expedite in various settings.
C Acceleration for Video Generation
Figure 8: The frames of generated videos. The frames in the first row are generated using the original ControlVideo method [35], while the frames in the second row are generated by integrating AiluRus with ControlVideo.
To demonstrate the versatility of AiluRus, we seamlessly integrate it into the ControlVideo [35] method for text-based video generation. Specifically, we incorporate AiluRus into the StableDiffusion model [22] to reduce the size of intermediate features and execute the model using the prompt "A man wanders on the ocean" with depth condition. We also run the original model with the same configuration for comparison. As shown above, AiluRus produces similar content to the original model. However, AiluRus accomplishes this task in only half the time required by the original model (2:11 vs 4:24). This observation highlights the capability of AiluRus to instantaneously accelerate large models with negligible overheads, which is of critical importance for the recent emergence of large models.
D Why AiluRus outperforms Expedite?
As described in the main text, AiluRus exhibits significant performance superiority over Expedite[15], particularly when it comes to token reduction at shallow layers. Shallow layer token reduction poses more significant challenges due to increased information loss. In order to validate this claim, we conduct detailed experiments to explore the intrinsic mechanism.
i) The reconstruction errors of AiluRus and Expedite. With the frozen model, the reconstruction errors between the output features and the original features are highly correlated with performance. To provide a comprehensive comparison, we employ various configurations of AiluRus and Expedite on ViT-B and ViT-L models, respectively, and calculate the cosine similarity between the reconstructed features and the original features. The results presented in Tab. 9 and Tab. 10 demonstrate that AiluRus generates output features that are more similar to the original features compared to Expedite across different settings, providing an explanation for the superior performance of AiluRus.
Table 9: The reconstruction similarity comparison between AiluRus and Expedite over ViT-L.  $\mathrm{X} \times  \mathrm{X}$  indicates the number of clusters.
Layer	Method	20×20	22×22	24×24	26×26
mIoU	Similarity	mIoU	Similarity	mIoU	Similarity	mIoU	Similarity
4	AiluRus	47.36	0.8401	48.84	0.8699	50.05	0.8941	50.56	0.9143
Expedite	42.88	0.7706	44.26	0.7926	47.19	0.8308	47.74	0.8504
6	AiluRus	48.80	0.8724	49.47	0.8967	50.45	0.9164	50.98	0.9326
Expedite	45.32	0.8162	46.05	0.8304	47.80	0.8626	48.37	0.8789
8	AiluRus	49.97	0.9039	50.39	0.9213	51.06	0.9356	51.37	0.9480
Expedite	49.73	0.8934	50.03	0.9048	50.77	0.9219	51.06	0.9324
ii) The reconstruction errors for different cluster locations. We further observe that AiluRus exhibits a more pronounced advantage in shallow clustering, while Expedite's performance deteriorates significantly. Since both AiluRus and Expedite perform clustering only once and reuse the clustering results at the output layer, it is evidently easier to perform clustering in deeper layers compared to
Table 10: The reconstruction similarity comparison between AiluRus and Expedite over ViT-B. X × X indicates the number of clusters.
Layer	Method	18×18	20×20	22×22	24×24
mIoU	Similarity	mIoU	Similarity	mIoU	Similarity	mIoU	Similarity
2	AiluRus	46.45	0.8971	47.12	0.9291	47.70	0.9436	48.26	0.9602
Expedite	4.32	0.2882	5.21	0.3342	6.19	0.3652	7.57	0.4065
4	AiluRus	47.69	0.9339	48.27	0.9503	48.66	0.9636	48.88	0.9746
Expedite	12.66	0.4714	15.53	0.5218	18.03	0.5594	21.57	0.6017
6	AiluRus	48.19	0.9527	48.83	0.9642	48.90	0.9736	48.99	0.9817
Expedite	44.07	0.8782	45.17	0.8972	45.44	0.9068	46.14	0.9161
Table 11: The reconstruction similarity at different layers.
Methods	2	3	4	5	6	7	8	9	10	11
AiluRus	0.9210	0.8959	0.8863	0.8906	0.8972	0.9092	0.9238	0.9312	0.9361	0.9291
Expedite	0.7334	0.6228	0.5524	0.4824	0.4293	0.3694	0.3778	0.4219	0.4433	0.3342
Table 12: The probability of preserving token pair similarity for different similarity intervals.
Intervals	0.0-0.1	0.1-0.2	0.2-0.3	0.3-0.4	0.4-0.5	0.5-0.6	0.6-0.7	0.7-0.8	0.8-0.9	0.9-1.0
Probability	0.0137	0.1246	0.2179	0.1439	0.0629	0.0347	0.0255	0.0193	0.0212	0.9903
shallow layers. To investigate the reasons behind the performance gap in shallow clustering, we analyze a representative scenario (ViT-B, layer index  $= 2$ , num cluster  $= 20*20$ ) and calculate the feature reconstruction quality in subsequent layers. The results presented in Tab. 11 indicate that Expedite experiences increasingly severe reconstruction distortion during the forward pass, while AiluRus maintains reconstruction quality across layers. This elucidates the poor performance of Expedite when applied in shallow layers.
iii) Why AiluRus has lower reconstruction errors and maintains its advantage across layers? We delve into the advantages of AiluRus over Expedite in terms of the intrinsic mechanism. Both methods aim to reduce the number of tokens while preserving the relationships between the retained tokens and the reduced ones, and subsequently utilize the preserved relationships for recovery at the output layer. The quality of recovery determines the overall performance, and it depends on whether the preserved relationships can be maintained after passing through several Transformer blocks. Expedite and AiluRus differ in the nature of the relationship they aim to preserve.