SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Token match-merge scheme. As discussed in Observation 3, dissimilarities among lower IIC tokens should be considered, necessitating a hierarchical processing. Thus, we employ the token
match-merge scheme among boundary tokens to guide the match-merge process of similar boundary tokens, aiming to reduce the number of tokens while maintaining accuracy to the best possible extent. We measure the similarity between two tokens by computing the cosine similarity of their value vectors.  $v^{a}, v^{b} \in \mathbb{R}^{C}$ , present the value vectors of tokens  $a$  and  $b$ , and the cosine similarity  $\cos(\theta)$  is expressed as follows:
$$
\cos (\theta) = \frac {\sum_ {i = 1} ^ {C + C ^ {\prime}} \left(v _ {i} ^ {a} \times v _ {i} ^ {b}\right)}{\sqrt {\sum_ {i = 1} ^ {C + C ^ {\prime}} \left(v _ {i} ^ {a}\right) ^ {2}} \times \sqrt {\sum_ {i = 1} ^ {C + C ^ {\prime}} \left(v _ {i} ^ {b}\right) ^ {2}}} \tag {11}
$$
The token match-merge scheme is demonstrated in Figure 5. Step1. Divide tokens at odd positions in CASR into Group A and those at even positions into Group B. Step2. Find the most similar token in set B for each token in set A by calculating cosine similarity. Step3. Put similar tokens together to complete the match. Step4. Merge the similar tokens by weights to reduce the number of tokens.
3.4 Training Mehtod
During the training of MG-ViT, setting  $\epsilon = 1$  makes MGIS be executed for every input image. Like knowledge distillation [37], our goal is to make the outputs of SGIS more similar to MGIS. Thus ViT in SGIS can have a stronger ability in classification. The training loss about ground-truth label  $gt$  is:
$$
\operatorname {l o s s} = C E \left(D ^ {M}, g t\right) + K L \left(D ^ {S}, D ^ {M}\right), \tag {12}
$$
where  $CE(\cdot ,\cdot)$  and  $KL(\cdot ,\cdot)$  represent cross entropy loss and Kullback-Leibler divergence.
4 Hierarchical Multi-Granularity Vision Transformer
Figure 6: The overall framework of Hierarchical MG-ViT.
4.1 Key Issue of Extension
Almost all classic ViT compression works, such as Dynamic ViT [12], PS-ViT [26], EViT [13], only focus on classification. Why not extend their methods to hierarchical ViT to execute downstream tasks like detection and segmentation? We analyze two reasons. The first reason is structural limitation. For instance, plain ViT only provides a single-scale feature representation, lacks the ability to handle multi-scale variation, and lacks image-related priors. While there are a few works that still employ plain ViT for downstream tasks, such as ViTDet [38], ViT-Adapter [39], SegViT [40; 41], they achieve this by integrating well-designed modules, which could be considered as individual research contributions. The second reason is method challenges. Most classic ViT compressions, as mentioned above, heavily rely on pruning or merging, which can be challenging to adapt to downstream tasks. Although we also employ pruning and merging methods, they serve as auxiliary means to reduce computational costs. The core of this paper - multi-granularity strategy - can be applied in hierarchical ViT for downstream tasks.
The key to extending multi-granularity strategy to hierarchical ViT lies in addressing the non-uniform issue of multi-grained patch scales. Subsplitting head and middle patches into  $3 \times 3$  fine patches and  $2 \times 2$  medium patches results in patches with multiple scales interweaving in an image, making it difficult to seamlessly enclose patches in a conventional window when using Swin Transformer. Therefore, in this paper, we propose proxy tokens to unify scales.
4.2 Introducing Proxy Tokens for Unifying Scales
Similar to MG-ViT, Hierarchical MG-ViT also employs a two-stage framework. In SGIS, 2D image  $\aleph \in \mathbb{R}^{H\times W}$  is split into patches  $P\in \mathbb{R}_{s}^{\frac{H}{s}\times \frac{W}{s}\times s^2}$ , where  $(H,W)$  donates the resolution of image  $\aleph$ ,
and  $s$  presents the patch size. The token sequence generated by  $P$  embedding is:
$$
T _ {o r i} = \left[ t _ {o r i} ^ {1}, t _ {o r i} ^ {2}, \dots , t _ {o r i} ^ {\frac {H}{s} \times \frac {W}{s}} \right], \tag {13}
$$
where  $T_{ori} \in \mathbb{R}^{\frac{HW}{s^2} \times C_1}$ , ori is a simplified representation of "origin". IIC of  $P$  is identified in SGIS. In MGIS, we assume that  $P^i \in \mathbb{R}^{s \times s}$  is a patch that requires subsplitting, which will be subsplit into  $\ell \times \ell$  fine patches:
$$
P ^ {i} = \left[ p ^ {i, 1}, p ^ {i, 2}, \dots , p ^ {i, \ell^ {2}} \right], \tag {14}
$$
where  $\ell \times \ell$  is the manner of subsplitting ( $3 \times 3$  or  $2 \times 2$ ),  $p^{i,j}$  present the  $j$ -th fine or medium patch subsplit from the  $i$ -th head or middle patch if  $i > 0$ , and  $j > 0$ .
Then the finer token sequence  $T_{ori}^{i} \in \mathbb{R}^{\ell^{2} \times C_{1}}$  originated from the embedding of these finer patches is denoted as:
$$
T _ {o r i} ^ {i} = \left[ t _ {o r i} ^ {i, 1}, t _ {o r i} ^ {i, 2}, \dots , t _ {o r i} ^ {i, \ell^ {2}} \right]. \tag {15}
$$
We introduce a proxy token  $t_{pro}^{i} \in \mathbb{R}^{C_{1}}$  for this group of finer tokens to compose the new sequence  $T^{i} = [T_{ori}^{i}, t_{pro}^{i}]$ . The computing procedure of a Transformer layer for this group can be summarized as follows:
$$
T ^ {i ^ {\prime}} = S A \left(T ^ {i}\right) + T ^ {i}, \quad T ^ {i ^ {\prime}} = M L P \left(T ^ {i ^ {\prime}}\right) + T ^ {i ^ {\prime}}. \tag {16}
$$
To generalization, We provide an optional proxy token set  $T_{pro}$  and its mask vector  $m$  for each image, where  $T_{pro} \in \mathbb{R}^{\frac{HW}{s^2} \times C_1}$  and  $m \in \mathbb{R}^{\frac{HW}{s^2}}$ . Thus, the unified scale token sequence  $T'$  before feature reuse can be denoted as:
$$
T ^ {\prime} = (- m) \times T _ {o r i} + m \times T _ {p r o} ^ {\prime}. \tag {17}
$$
After feature reuse, the token sequence is input into the pyramid-structured hierarchical ViT, through PVT stages 2, 3, and 4, feature maps reshape to  $\frac{H}{2s} \times \frac{W}{2s} \times C_2$ ,  $\frac{H}{4s} \times \frac{W}{4s} \times C_3$  and  $\frac{H}{8s} \times \frac{W}{8s} \times C_4$ . In this way, various downstream tasks can be performed.
4.3 Architectural Details
We develop two variants of Hierarchical MG-ViT based on PVT-Small: Hierarchical MG-ViT-A and Hierarchical MG-ViT-R. Hierarchical MG-ViT-A adds a Transformer layer with proxy tokens between stage 1 and stage 2 of PVT-Small. Hierarchical MG-ViT-R maintains the same capacity as PVT-Small but replaces one of the original Transformer layers in stage 1 with a Transformer layer introducing proxy tokens. Please refer to the appendix for architectural details.
Performance upgrade. Since our proposed method is plug-and-play and does not significantly alter the structure of the hierarchical ViT, methods aimed at improving the performance of hierarchical ViT, such as using relative position biases and overlapping patch embedding, are all effective. However, we haven't optimize Hierarchical MG-ViT for the best performance, as our goal was to explore the potential of extending the multi-granularity strategy from compressing plain ViT to hierarchical ViT for various downstream tasks. In the future, we will delve deeper into performance enhancements for Hierarchical MG-ViT.
Plain ViT vs. hierarchical ViT. Which ViT structure is superior? Plain ViT inherently has limitations when it comes to handling dense tasks. In order to utilize a general ViT for a variety of tasks, many new ViTs are designed based on hierarchical structures. However, we do not consider hierarchical ViT to be superior; both plain ViT and hierarchical ViT have their strengths and weaknesses. As for detection tasks, a multitude of ViTs built upon the hierarchical structure continue to remain the most competitively poised model. However, models like ViTDet [38], which are based on plain ViT, also demonstrate that hierarchical ViTs such as Swin-T and PVT are not the only efficient ways to accomplish visual tasks. As for segmentation tasks, what surprised us is that SegViTv2 [41], built upon the Plain ViT, has achieved state-of-the-art results. As for classification tasks, a variety of plain ViT models, with MG-ViT as a representative, exhibit a big performance advantage over hierarchical ViT. Moreover, multi-granularity strategy boost performance more effectively in plain structures than in hierarchical structures. This is because hierarchical ViT involves token merging, which
significantly diminishes the advantages of multi-granularity in feature representation. Regarding the question of which ViT structure is superior, we believe that it necessitates deliberation in the context of varying tasks and distinct constraints.
5.1 Backbones, Datasets and Evaluation Metrics.
We built MG-ViT based on DeiT-S and LV-ViT-S and evaluated its performance on the benchmark ImageNet [18]. To quantitatively compare performance, we report the Top-1 accuracy (Acc.), the number of floating-point operations (FLOPs), and throughput (TP). Moreover, we developed Hierarchical MG-ViT based on PVT-Small and evaluated its performance in object detection and semantic segmentation on the benchmark datasets MS-COCO2017 [19] and ADE20K [20]. We reported metrics such as box average precision  $(AP^{b})$ , mean intersection over union (mIoU) and the number of parameters (#Params). All metrics are measured on a single NVIDIA RTX 3090 GPU.
Comparison to backbones. Table 2 presents the comparison results between MG-ViT and backbones with different values of threshold  $\epsilon$ . It can be observed that Without any performance degradation, the computational cost of MG-ViT is significantly reduced. Moreover, when all images execute MGIS, the performance of ViT is improved, which can be attributed to subsplitting patches with multiple different granularities.
Table 2: Comparison between MG-ViT and backbones
Model	ε	Acc.(%)	FLOPs(G)	TP(img./s)
DeiT-S	-	79.8	4.6	1341
MG-ViT	0.49	79.8(+0.0)	2.0(-56%)	2404(+1.79×)
MG-ViT	0.73	80.8(+1.0)	2.4(-48%)	2054(+1.53×)
MG-ViT	1	81.0(+1.2)	3.9(-15%)	1591(+1.11×)
LV-ViT-S	-	83.3	6.6	989
MG-ViT	0.60	83.3(+0.0)	3.5(-47%)	1749(+1.77×)
MG-ViT	0.73	83.7(+0.4)	3.8(-42%)	1683(+1.70×)
MG-ViT	1	83.8(+0.5)	5.6(-15%)	1233(+1.25×)
Comparison to baselines. In Figure 7, we compare MG-ViT with 3 baselines: DynamicViT, EViT, and DVT. These models are chosen as representatives of 3 type methods: token pruning, token merging, and dynamic design of token format. With the same setting, MG-ViT outperforms baselines.
Comparison to other ViT compressing methods. In order to demonstrate the effectiveness of our dynamic framework, MG-ViT, in compressing ViT, we present a comparison of MG-ViT with various ViT compressing methods in Table 3. We visualize their accuracy, FLOPs, and throughput in Appendix. The results show that our method for compressing ViT based on the multi-granularity strategy, MG-ViT, is highly competitive when using DeiT-S and LV-ViT-S as backbones.
Table 3: Comparison between MG-ViT and other ViT compressing methods.
LV-ViT-S	DeiT-S
Model	FLOPs(G)	Acc.(%)	TP(img/s)	Model	FLOPs(G)	Acc.(%)	TP(img/s)
Baseline[17]	6.6	83.3	989	Baseline[15]	4.6	79.8	1341
PS-ViT[26]	4.7	82.4	-	IA-RED2[29]	3.3	79.1	1597
eTPS[42]	3.8	82.5	1665	DVT[14]	2.4	79.3	1485
EViT[13]	4.7	83.0	1447	DynamicViT[12]	2.9	79.3	1774
DynamicViT[12]	4.6	83.0	1302	Evo-ViT[27]	2.9	79.4	1863
SPViT[32]	4.3	83.1	1518	ATS[43]	2.9	79.7	1531
SiT[30]	4.0	83.4	1280	STViT[33]	3.2	80.6	1928
CF-ViT[28]	4.0	83.5	1711	CF-ViT[28]	2.6	80.7	2096
MG-ViT	3.8	83.7	1683	MG-ViT	2.4	80.8	2054
Comparison to other efficient ViTs. In Figure 8, we compare MG-ViT and various efficient ViTs [44; 45; 46; 47; 48; 49; 50; 51; 52; 53; 25; 54; 55; 56; 57; 58; 59; 60; 61; 62; 63; 64]. It can be observed that MG-ViT is competitive even among different backbones and other ViT variants.
Figure 7: Comparison with baselines.
Figure 8: Comparison MG-ViT between and static efficient ViT methods.
Object detection and semantic segmentation. We adopt the same implementation details as PVT-Small and conduct preliminary experiments on Hierarchical MG-ViT's performance in object detection and semantic segmentation tasks to represent its capability for executing downstream tasks. As shown in Table 4 and Table 5, extending the multi-granularity strategy to the hierarchical structure is beneficial for enhancing ViT's performance and serves as a nice attempt to slim hierarchical ViT.
Table 4: Object detection with Mask R-CNN  $1\times$  schedule.
Method	#Params(M)	FLOPs(G)	APb	AP50b	AP75b
PVT-Small [16]	44.1	245	40.4	62.9	43.8
Hierarchical MG-ViT-A	46.5	258	40.9	63.2	44.2
Hierarchical MG-ViT-R	41.9	227	40.3	62.7	43.6
Table 5: semantic segmentation with semantic FPN head.
Method	#Params(M)	FLOPs(G)	mIoU
PVT-Small [16]	28.2	44.5	39.8
Hierarchical MG-ViT-A	29.3	46.4	40.1
Hierarchical MG-ViT-R	27.1	41.3	39.7
MG-ViT, the two-stage dynamic framework based on the strategy we proposed, achieves a well-balanced trade-off between performance and computational cost. Within this novel framework, we contribute four new methods. Firstly, we subsplit patches with multiple different granularities, resulting in generating multi-granularity tokens with various levels of information, which enhances the capacity of ViT in feature representation, leading to performance improvement. Secondly, we introduce a three-way decision mechanism to achieve soft slimming of ViT. Thirdly, we propose the token match-merge scheme to guide the match-merge of boundary tokens, reducing their quantity. The second and third methods are reflected in our proposed PPSM. Fourthly, we extend multi-granularity strategy to hierarchical ViT for various downstream tasks, demonstrating that our method can also be applied to hierarchical structures for compressing, and we will explore this further in the future.
This work is supported by the National Key Research and Development Program of China (No. 2022YFB3104700), the National Natural Science Foundation of China (No. 61976158, No. 62376198, No. 62006172, No. 62076182, No. 62163016, and No. 62106091), the Jiangxi “Double Thousand Plan”, the Jiangxi Provincial Natural Science Foundation (No. 20212ACB202001) and the Shandong Provincial Natural Science Foundation (No. ZR2021MF054). The authors would like to thank Jun Wang, Yanping Li, Qixian Zhang, Zhiheng Wu for inspirational and feasible suggestions.
[51] Zizheng Pan, Bohan Zhuang, Haoyu He, Jing Liu, and Jianfei Cai. Less is more: Pay less attention in vision transformers. In AAAI, 2022.
[52] Yanghao Li, Chao-Yuan Wu, Haoqi Fan, Karttikeya Mangalam, Bo Xiong, Jitendra Malik, and Christoph Feichtenhofer. Mvitv2: Improved multiscale vision transformers for classification and detection, 2022.
[53] Weihao Yu, Mi Luo, Pan Zhou, Chenyang Si, Yichen Zhou, Xinchao Wang, Jiashi Feng, and Shuicheng Yan. Metaformer is actually what you need for vision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10819-10829, 2022.
[54] Zhengsu Chen, Lingxi Xie, Jianwei Niu, Xuefeng Liu, Longhui Wei, and Qi Tian. Visformer: The vision-friendly transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 589-598, 2021.
[55] Pengchuan Zhang, Xiyang Dai, Jianwei Yang, Bin Xiao, Lu Yuan, Lei Zhang, and Jianfeng Gao. Multi-scale vision longformer: A new vision transformer for high-resolution image encoding. ICCV 2021, 2021.
[56] Kunchang Li, Yali Wang, Junhao Zhang, Peng Gao, Guanglu Song, Yu Liu, Hongsheng Li, and Yu Qiao. Uniformer: Unifying convolution and self-attention for visual recognition, 2022.
[57] Ali Hassani, Steven Walton, Jiachen Li, Shen Li, and Humphrey Shi. Neighborhood attention transformer. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[58] Xiangxiang Chu, Zhi Tian, Yuqing Wang, Bo Zhang, Haibing Ren, Xiaolin Wei, Huaxia Xia, and Chunhua Shen. Twins: Revisiting the design of spatial attention in vision transformers, 2021.
[59] Weifeng Lin, Ziheng Wu, Jiayu Chen, Jun Huang, and Lianwen Jin. Scale-aware modulation meet transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6015-6026, 2023.
[60] Ali Hatamizadeh, Hongxu Yin, Greg Heinrich, Jan Kautz, and Pavlo Molchanov. Global context vision transformers. In International Conference on Machine Learning, pages 12633-12646. PMLR, 2023.
[61] Jiayu Jiao, Yu-Ming Tang, Kun-Yu Lin, Yipeng Gao, Jinhua Ma, Yaowei Wang, and Wei-Shi Zheng. Dilateformer: Multi-scale dilated transformer for visual recognition. IEEE Transactions on Multimedia, 2023.
[62] Dongchen Han, Xuran Pan, Yizeng Han, Shiji Song, and Gao Huang. Flatten transformer: Vision transformer using focused linear attention. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5961-5971, 2023.
[63] Zhiliang Peng, Zonghao Guo, Wei Huang, Yaowei Wang, Lingxi Xie, Jianbin Jiao, Qi Tian, and Qixiang Ye. Conformer: Local features coupling global representations for recognition and detection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
[64] Xuran Pan, Tianzhu Ye, Zhuofan Xia, Shiji Song, and Gao Huang. Slide-transformer: Hierarchical vision transformer with local self-attention. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2082–2091, 2023.
7.1 Hierarchical MG-ViT Architectural Details
We present two variants of the Hierarchical MG-ViT designed based on PVT-Small, as shown in Table 6. The relevant hyper parameters are listed as follows:  $P_{i}$ : the patch size of Stage  $i$ ;  $C_i$ : the channel number of the output of Stage  $i$ ;  $R_{i}$ : the reduction ratio of the spatial-reduction attention in Stage  $i$ ;  $N_{i}$ : the head number of the spatial-reduction attention in Stage  $i$ ;  $E_{i}$ : the expansion ratio of the feed-forward layer in Stage  $i$ ;  $H_{i}$ : the head number of the multi-head attention in Stage  $i$ .
Table 6: Architectural details of Hierarchical MG-ViT and PVT-Small
Output Size	Layer Name	PVT-Small	Hierarchical MG-ViT-A	Hierarchical MG-ViT-R
Stage 1	H/4 × H/4	Patch Embedding	P1 = 4; C1 = 64
Transformer Encoder	R1 = 8N1 = 1E1 = 8	×3	R1 = 8N1 = 1E1 = 8	×3	R1 = 8N1 = 1E1 = 8	×2
Stage MG	H/4 × H/4	MG Transformer	-	HMG = 8EMG = 8	×1	HMG = 8EMG = 8	×1
Stage 2	H/8 × H/8	Patch Embedding	P2 = 2; C2 = 128
Transformer Encoder	R2 = 4N2 = 2E2 = 8	×3	R2 = 4N2 = 2E2 = 8	×3	R2 = 4N2 = 2E2 = 8	×3
Stage 3	H/16 × H/16	Patch Embedding	P3 = 2; C3 = 320
Transformer Encoder	R3 = 2N3 = 5E3 = 4	×6	R3 = 2N3 = 5E3 = 4	×6	R3 = 2N3 = 5E3 = 4	×6
Stage 4	H/32 × H/32	Patch Embedding	P4 = 2; C4 = 512
Transformer Encoder	R4 = 1N4 = 8E4 = 4	×3	R4 = 1N4 = 8E4 = 4	×3	R4 = 1N4 = 8E4 = 4	×3
7.2 Implementation Details
The resolution of input images in our experiments is  $224 \times 224$ . In SGIS, we split each image into  $7 \times 7$  patches.  $r_h$  and  $r_t$  are set to 0.1 and 0.4, respectively. Therefore, each image is subsplit into 161 patches in MGIS. For DeiT-S, we inserted a total of three PPSM modules in the 3rd, 7th, and 10th layers. Similarly, for LV-ViT-S, we inserted a total of four PPSM modules in the 4th, 7th, 9th, and 12th layers. For conducting the training process, we set the batch size to 256 and use AdamW optimizer to train all models for 300 epochs.
In terms of training strategies and optimization methods, our training closely follows the original methods proposed in the studies conducted by DeiT and LV-ViT, but do not employ any knowledge distillation algorithms in our experiments. Throughout the training phase, we observed the impact on convergence when subsplitting was applied solely to the head and middle patches. To enhance the speed of model convergence, we adopted subsplitting specifically targeting the head and middle patches for the initial 200 epochs. Subsequently, for the following 100 epochs, we applied subsplitting to the head and tail patches. The training was performed on a workstation equipped with 8 RTX 3090 GPUs.
7.3 Visualization Result
In Figure 9, we specifically demonstrate the images that can be well recognized by MG-ViT (LV-ViT-S) in SGIS and MGIS. As observed, the images on the left are "simple," usually occupying a significant portion of the image pixels, with typically complete and clear contours. While the images on the right are "complex", containing intricate scenes and non-prominent objects or only include a small part of the objects, requiring multi-granularity representations through additional tokens.
Tennis
Figure 9: Example of images well recognized in SGIS and MGIS. To provide clarity, we use unfilled, semi-white, and semi-black boxes to represent the head, middle, and tail patches in MGIS, respectively.
Figure 10 visualizes the token slimming comparison between token merging (MG-ViT) and token pruning (DynamicViT). The result shows, the information loss caused by token pruning may result in incorrect slimming decisions, leading to a decrease in classification accuracy. However, the adoption of token merging in PPSM alleviates this situation.
Figure 10: Visual comparison between token merging and pruning. The boxes with the same color represent a merged token.
Figure 11: Comparison between MG-ViT and existing efficient ViT methods. The size of  $\bullet$  represents the computational cost required for each image (FLOPs). It can be observed that MG-ViT achieves a good balance in terms of accuracy, FLOPs, and throughput, demonstrating its overall competitiveness.
Table 8: Ablation experiment on global class attention.
Scheme	Acc.(%)
SGIS	MGIS
Random division	75.0	78.9
Random CASR	75.3	79.8
Last CASR	75.5	80.5
Ours	75.6	80.8
Table 9: Ablation experiment on Feature Reuse module.
Scheme	Acc.(%)
SGIS	MGIS
w/o Feature Reuse	75.0	79.9
Ours + Class Token	75.4	80.1
w/o Zero-Padding	75.5	80.7
Concat→add	75.4	80.7
Ours	75.6	80.8
7.4 More Experiments and Ablation Analysis
Necessity of global class attention.
We designed two experiments based on backbone DeiT-S to investigate the necessity of global class attention.
Table 7: Accuracy with different values of  $\lambda$  .
λ	0	0.5	0.9	0.95	0.98	0.99	0.999
Acc.(%)	80.3	80.5	80.7	80.7	80.8	80.7	80.7
Table 8 shows the gain of the proposed global attention class on model performance, compared to random division, which involves randomly dividing patches into three groups, random CASR, which divides patches according to CASR in a random layer of ViT, and last CASR, which divides patches based on CASR in the last layer. As we can see, the global attention class achieves the best performance improvement. Table 7 compares the contribution of the global attention class to model performance with different values of  $\lambda$ . When  $\lambda = 0$ , it represents the last CASR scheme. We set the default value of  $\lambda$  as 0.98.
Applicability of Feature Reuse. We introduce the Feature Reuse module from DVT[14] to enhance the feature representation of patches, and also explore its applicability in our model. Specially, we do not reuse the class tokens and apply zero padding to the tail tokens (i.e., not reusing the information from tail tokens). Table9 demonstrates the effective integration of the Feature Reuse module into our model, and indicates the effectiveness of our improvements.
Effect of token match-merge scheme. We conduct ablation experiments to investigate the effort of our proposed token match-merge scheme and establish three control schemes for comparison: token pruning, token clustering (K-means), and merging tokens into one. Especially, by employing our token match-merge scheme on boundary tokens, we achieved the best trade-off between accuracy and speed, as shown in Table 10. This result also supports the rationality of merging all negative tokens into one, because we need to process the tokens with low IIC in the most computationally
Table 10: Ablation experiment on the token match-marge scheme.
Scheme	Acc.(%)	SPD(img./s)
Token pruning	79.5	188
Token clustering	79.8	173
Merging tokens into 1	80.4	183
Match-marge (ours)	80.8	180
Table 11: Comparison among various merging approaches.
Approach	Acc.(%)
Keep one	80.3
Max pool	80.4
Avg pool	80.6
weighted avg(ours)	80.8