MG-ViT: A Multi-Granularity Method for Compact and Efficient Vision Transformers
Authors: Yu Zhang
Yepeng Liu
Duoqian Miao
Qi Zhang
Yiwei Shi
Liang Hu
Venue: NeurIPS 2023 poster (2023)
Keywords: Efficient AI
Vision Transformer
Image Classification
Multi-Granularity
Three-Way Decisions.
TL;DR: This paper proposes a two-stage multi-granularity method, MG-ViT, to balance ViT's remarkable performance and huge computational cost.

Abstract: Vision Transformer (ViT) faces obstacles in wide application due to its huge computational cost. Almost all existing studies on compressing ViT adopt the manner of splitting an image with a single granularity, with very few exploration of splitting an image with multi-granularity. As we know, important information often randomly concentrate in few regions of an image, necessitating multi-granularity attention allocation to an image. Enlightened by this, we introduce the multi-granularity strategy to compress ViT, which is simple but effective. We propose a two-stage multi-granularity framework, MG-ViT, to balance ViT’s performance and computational cost. In single-granularity inference stage, an input image is split into a small number of patches for simple inference. If necessary, multi-granularity inference stage will be instigated, where the important patches are further subsplit into multi-finer-grained patches for subsequent inference. Moreover, prior studies on compression only for classification, while we extend the multi-granularity strategy to hierarchical ViT for downstream tasks such as detection and segmentation. Extensive experiments Prove the effectiveness of the multi-granularity strategy. For instance, on ImageNet, without any loss of performance, MG-ViT reduces 47\% FLOPs of LV-ViT-S and 56\% FLOPs of DeiT-S.

MG-ViT: A Multi-Granularity Method for Compact and Efficient Vision Transformers
Yu Zhang' Yepeng Liu?
Duoqian Miao Qi Zhang'
Yiwei Shi Liang Hu'
Tongji University
2 University of Florida
University of Bristol
Abstract
Vision Transformer (ViT) faces obstacles in wide application due to its huge computational cost. Almost all existing studies compressing ViT adopt the manner of splitting an image with a single granularity, with very few exploration of splitting an image with multi-granularity. As we know, important information often randomly concentrate in few regions of an image, necessitating multi-granularity attention allocation to an image. Enlightened by this, we introduce the multi- granularity strategy to compress VIT. which is simple but effective. We propose a two-stage multi-grane gualliigaanllrrtty framework, MG-ViT to balance ViT's performance and computational cost. In single- granul-gatllrrtty inference stage, an input image is split into a small number of patches for simple inference. If necessary, multi- granularity inference stage will be instigated, where the important patches are further subsplit into multi-finer-graneed patches for subsequent inference. Moreover, prior studies on compression only for classification, while we extend the multi- granularity strategy to hierarchical VIT for downstream tasks such as detection and segmentation. Extensive experiments Prove the effectiveness of the multi- granularity strategy. For instance, on ImageNet, without any loss of performance, MG-ViT reduces 47% FLOPs of LV-ViT-S and 56% FLOPs of DeiT-S.
Introduction
Transformer [1 has gained tremendous achievements in computer vision. Despite the excellent performance of Convolutional Neural Networks (CNN) in computer vision tasks 22 5). due to VIT's superior ability capture global information and long-range interactions, ViT 6 outperforms CNN various tasks 7 cost of its huge computational overhead. Therefore,
8 9 10: [1]. However, the impressive performance of ViT comes at the researches focus VIT compression for greater efficiency. Since the computational cost of ViT increases quadratically with the number of tokens, minimizing the number of tokens is crucial in compressing ViT.
Notable investigations have adopted various methods to compress VIT. Dynamic ViT [12 reduces token number by pruning redundant tokens, EVTT [13] merges redundant tokens into one token. DVT [14] utilizes the particular and dynamic patch splitting manner (e.g., 444, 7x7, etc.) for each image, based on its complexity, rather than following the official manner of 14x 14, Although DVT innovatively considers the differences in complexity between images, still uses single granularity to split one image. As we all know, images often contain lot of redundancy, with important semantic information randomly concentrated in a few regions. Therefore, complexity and semantic density diversities across various regions of the image: different regions of an image should be assigned attention with multiple granularities. We initiate our exploration by observing the foundational task, classification, and have intriguing observations:
imperative to consider the
*Corresponding Author
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Observation 1: Most images can be correctly recognized by splitting them into a small number of patches, as has been analogously observed in research [14]. Training DeiT- [15] with the
Table I: Accuracy and FLOPs of DeiT-S on ImageNet different splitting manner.
patch splitting manner can achieve 63,4% accuracy at computational cost of 0.45G FLOPS. Although splitting an image to x 7 patches and 14 x 14
Manner
14 x 14
7 x 7
4 x
Accuracy FLOPs
79.8%
73.2%
63.4%
4.60G
1.10G
0.45G
patches can enhance the accuracy by 9.8% and 16.4%, the computation cost will also increase by 2.4x and 10.2x. How to substantially improve the accuracy with low growth rate of FLOPs is subject worth contemplating.
The classification accuracy is mainly affected by the several critical tokens, and subsplitting patches into finer patches is more beneficial to classification. We split each image into 14 14 patches and rank tokens from high to low based on their class attention scores. The 196 tokens (patches) are equally divided into head, middle, and tail groups. and respectively input into DeîT-S to measure the accuracy of each group (Fig [(a). Then, subsplitting the patches the head
Observation 2:
group expands the quantity to 4x and 9x, their accuracy is measured (Fig [[b]]. It is illustrated that head tokens ensure the base of classification accuracy, and finer tokens lead to higher accuracy.
80
80
80
75
70
70
70
gi ICtskers
60
66
Hightr ECTokers 1 tokon (200%)
go tchera talaro
50
basekito
60
baadina head tolkers
head tollens
56
madile sobera
Tell Tokene
50
mores
lizerr de
55
45
45
2
5 6 Layer
7
9 10
11
1 2
7
10 11 12
Layer
Loyer
(a)
(b)
(c)
Figure 1: The classification accuracy of the tokens from different groups in each layer of DeiT-S. (c) 4 input groups: 1#: Only higher IIC tokens (top50%) 2#: Merging lower IIC tokens into one token and together with higher IIC tokens. 3#: According to CASR, merging every 50% of lower IIC tokens into one token (2 in total) and together with higher IIC tokens, 4#: According to CASR, merging every 25% of lower IIC tokens into one token (4 in total) and together with higher IIC tokens.
Observation 3: All information matters, and proper handling of tokens with low important informa- tion content (IIC) contributes to avoiding partial loss of accuracy. We treat the bottom 50% tokens as lower IIC tokens based on class attention score rank (CASR), and create input groups for all tokens. The results in Fig [[c] suggest that retaining lower IIC tokens can maintain accuracy. This is because lower IIC tokens contain complementary information to higher IIC tokens, which helps bridge the accuracy gap with the baseline. The dissimilarities among lower IIC tokens should be considered, necessitating hierarchical processing. Performance improvement is from complementary informa- tion, certain lower IIC tokens make the higher contribution. If they are merged with other lower IIC tokens, the gain of performance improvement may be diluted.
Thus far, we can identify two sound reasons for utilizing the multi-grane granularity strategy to compress ViT. Firstly, as aforementioned. due the distinctiveness of important information distribution in images, it is imperative to assign multi-granul observations, we can consider multi gualiiirannllrrtt as an intermediate state between full fine- ane-ggrined and full coarse-grained. Full fine-gr fine-grained stage represents stronger performance, while full coarse-grained stage represents less computation. Therefore, multi quati-granularity can naturally be used to balance performance and computational cost.
attention to an image. Secondly, guided by our
gramulattyy frame- work for compressing ViT, which automatically splits an image into multi-granularid patches, as shown in Figure 2 MG-ViT consists of two parts: Sragulaarnn Inference Stage (SGIS) and Multi-Granule Granularity Inference Stage (MGIS), related to Observation 1. In SGIS, a small number of to- kens are used for classification inference. Should a prediction confident enough ensue, the inference terminates immediately; otherwise, MGIS will be
We propose MG-VIT:
Simpal dod
Single Indewane Stage
Complex S
Muri Crann-irndyrrtl vteennc Stage
53.5%
Safgerete
Predicton
66.3%
Figure 2: Example for MG-ViT.

---

triggered. In MGIS, We put forward three innovative methods. The first is gualti-gaanulritty patch subsplitting, related to Observation 2. All patches are divided into head, middle and tail groups based on CASR in SGIS, head and middle patches are subsplit, resulting in tokens with multiple information granularity that are input into ViT. The second is the three-way decision slimming, which can streamline ViT softly. Simply put, based on CASR, tokens are divided into three groups: positive, boundary, and negative. Boundary and negative tokens are respectively merged by different rules to reduce their numbers. The third is the token match-merge scheme for boundary tokens, related to Observation 3. The second and third methods are packaged as Plug-and- Slimming Module (PPSM) against the huge increase in computational cost caused by patch subsplitting. After PPSM, all tokens are fed into the next layer of ViT to continue inference.
Due to that ViT only outputs a single-scale feature representation and lacks the capability to handle multi-scale variations, possesses significant deficiencies in downstream tasks. In order to adapt multi-granularigg strategy to various downstream tasks, we extend the MG-ViT framework to hi- erarchical ViT. Unlike ViT, hierarchical ViT aggregates tokens within rectangular areas. However, patch subsplitting results in patches (tokens) with multiple scales interleaved in the image, making challenging to seamlessly enclose them within regular rectangular areas. To address this issue, we introduce proxy token of the same scale for each head or middle patch to represent itself. Additionally, each proxy token engages in attention calculations with its corresponding fine tokens or medium tokens to represent the subsplitting of the head or middle patch it stands for. Following the aggregation methodology of PVT [6],, Hierarchical MG- VIT can perform downstream tasks.
MG-VIT is dynamic and versatile framework that can be applied to most ViT models. We select LV- 17 and DeiT [13 assess the performance of MG-ViT on ImageNet [8. The experiments demonstrate that MG-ViT can balance performance and computation cost greatly, thereby signifi- cantly improving the efficiency of VIT. Morcover, while ensuring performance, MG-ViT reduces 47% FLOPs of LV-VIT and 56% FLOPs of DeiT. We also conduct simple experiments in object detection and semantic segmentation on the MS-COCO [9] and ADE20K (0) datasets, respectively. The results show that Hierarchical MG-VIT effectively reduces computational overhead without sigmificant performance loss, demonstrating the feasibility of extending multi cultti-ranularity strategy to the Hierarchical ViT structure.
2 Related Work
Vision Transformer. ViT (6] achieves great breakthrough in the image classification task by redesigning the Transformer structure, completely igniting researchers' passion for exploring the ViT
further. Later on, VIT is modified to hierarchical structure for various downstream tasks (21 8 22 23
VIT and Hierarchical VIT. VIT is a series of ViT models that adhere to the vanilla ViT design prin- ciples,such as DeiT [1] and LV-ViT[[7]. They only output a single-scale feature representation and lack the capability to handle multi-scale variations, making them typically suitable for classification tasks. Hierarchical ViT is series of ViT models that incorporate a hierarchical structure to aggregate tokens layer by layer, enabling them to better handle multi-scale information and well-suited for are representative architectures
various downstream tasks. Swin-Trant Transformer 24 and PVT [16] 25 of hicrarchical VIT.
VIT compression. ViT and hierarchical ViT can both enhance efficiency, Hicrarchical ViT primarily achieves efficiency by improving interaction rules among tokens, whereas ViT mainly relies on compression. ViT compression primarily aims to optimize ViT structures in a flexible and automated manner, developing a dynamic and efficient ViT where each image undergoes an individual com- putational process based on its unique features, all while minimizing computational costs without sacrificing performance. There are many classic VIT compression works, such as DynamicViT [2]
PS-ViT 26]. Evo-ViT 277. and so on 13 4 28
32 K3
3
Multi-Cranularity Vision Transformer
3.1 Overview
Figure 3 Illlustrates the overall framework of MG-VIT. For each image, SGIS must be executed. In SGIS, the image is split into small number of patches. These 2D patches are embedded into ID
3

---

Classfication
ludgement
Result
Image
0BS6
VIT
dlss
Head Faich
Malrrain
Add
Eotch
Middia Fuida
Mediun Tokara
Sandiars Marge
Patch
SubSpiltting
Marge Into
Proposed Method
Multigrandderty subspittin
Theee-way decision sliming
Taken match-merge scherme
Figure 3: The overall framework of VG.
tokens and fed into VIT for straightforward inference. The obtained class prediction will be evaluated against predetermined criteria (threshold). If the class prediction is acceptable, the inference can be terminated immediately, and the class prediction can be output as the final classification result. If the class prediction is unacceptable, MGIS will be executed.
In MGIS, according the IIC of each patch (referring to CASR), all patches proportionally divided into three groups: head, middle and tail. Head and middle patches are further subsplit. As head patches with high IIC are the most crucial for classification, they require subsplitting with fine granularity, and eventually are subsplit into fine patches with fine-grained information. Middle patches with middle IIC are require subsplitting with medium granularity, being subsplit into medium patches with medium- grained information. Tail patches, maintain their coarse granularity and serve coarse patches. All patches undergo patch embedding and feature reuse to generate multi-granularity tokens which input into ViT for inference, prediction, and output. Considering that patch subsplitting leads to huge increase in computational cost, Plug-and-Plle Slimming Module (PPSM) is introduced in certain layers of ViT reduce computational cost. The three-way decision slimming lies here. According to CASR, we divide all tokens into three groups: positive, boundary and negative, aiming realize soft slimming of ViT. Positive tokens remain unchanged. Among boundary tokens, similar tokens are matched and merged based on our proposed token match-marge scheme (more details in Section 3.3.2), resulting in fewer new tokens. Negative tokens are merged into one token by weights. In this way, the number of tokens is reduced, accomplishing ViT soft slimming.
crucial for classification but less than head patches. Whence, they
3.2 Single-Gra Inference Stage (SGIS)
In SGIS, the standard VIT is employed. Formally, for input image it is first split into patches P = [p=,p?,-. .? where N is the number of patches. In patch embedding, patches are mapped into C-dimension token embeddings via linear projection. Additionally, learnable token embedding x", referred to as a class token, is appended to the sequence of token embeddings to represent image for class prediction. After position embedding Epoz. the input token sequence for VIT in SGIS is:
X
[x x
N Eppo
where x' E R.C, present token embedding of the i-th patch if O, and Epos E (NN-1)xC
ViT consists of layers, each layer comprises a eelf-attention (SA) module and ford-forward network (FFN). In SA of the 1-th layer, the self atff-ittenton Attention(()) K,, Vi) is computed as follows:
QiKT VC
A = softmax(
[a], .f.....aa af
Attention Q1, K2, Vi) = A.V..
(2)
where Qi, K,, Vi E R(N+1)xC are query, key and value matrices of 1-th layer respectively and obtained through linear projection for X,-1- Ai is attention map of 1-th layer, and the first row of attention map at represent class attention in Z-th layer.
Ulteriorly, the processes of SA and FFN in the (-th layer follows:
X SA(X)-1) X
X₁ FFN(X(1)) X'-1-
(3)

---

After consecutive computation of L layers, class token zy from the L-th layer is input into the classifier C to obtain class prediction distribution D
= [d dg
(4)
where S indicates SGIS, .M denotes the class number, up to this point, the class prediction of image is the largest entry of DS i.e., argmaz,d..
Threshold for judgment. the value of maz,d, for input image large enough, it indicates that the class prediction in SGIS is acceptable, and the classification result can be generated. Therefore, the inference can be terminated immediately, saving computational cost. In order to use maz,d to determine whether to terminate the inference, we introduce a threshold e: if mor,d > the inference is terminated immediately, and the class prediction output as the classification result. mar,d < E, The image is transitioned to MGIS for consecutive inference.
3.3 MuanllGrauuyarity Inference Stage (MGIS)
3.3.1 Generation of Multi-Granularity Tokens
To bridge the ViTs of two stages, we perform multi multi-granularity patch subsplitting and feature reuse to generate tokens with multi-grainer information as the input for the VIT in MGIS.
Multi-granularity patch subsplitting, Since maz,dÿ of image N is less than further inference is required. As mentioned above, in MGIS, image N needs to be subsplit with multiple different granularities, and features of patches in SGIS need be reused. These operations require to be instructed by IIC of patches (tokens) in SGIS, as patches with different IIC will be divided into different groups. Thus, the key now lies in how identify the IIC of patches.
According to Equation class token y can be represented by class attention a E A as:
9" KT VC
yd = softmax
VV
(5)
we all known, class token yd represents the entire image N as input to produce the class prediction in classifier C. As V = [t0, , 1,,, ..., J. where v² is the value of the i-th token if >> The class token yd is obtained by integrating the value of each token using a the corresponding weights. In other words, a determines the amount of information from each token that enters the class token vd for classification. Therefore,
can class attention a to measure the contribution of each token to the class prediction, thence, we identify IIC according to CASR. However, if we only use CASR originating from the class attention of certain layer divide patches, patches of each group will be more random and unfixed, lacking overall consistency. Therefore, the flow of information across layers must also be considered. Thus, we come up with the global class attention instead of only using class attention of certain layer to identify IIC of patches. The global class attention à; in the 1-th block as following:
à = A- - (1-
where A = 0.98. In Fig ((a) we can observe that the classification accuracy of all groups exhibits intense fluctuation in shallow layers. evident that class attention is unstable in shallow layers. Therefore, the global class attention calculation starts from the 3-rd layer. We use CASR rooted from global class attention in the last layer å identify IIC of patches.
According to â all patches are divided into three groups: head, middle, and tail. The numbers of head, tail and middle patches, denoted by No. Nr, and Nm are given by:
N₂ NN TA + 0.5J, N₂ [N r, 0.5J,
Nm. N N -N,
(7)
where and represent the number rate of head and tail patches respectively.
Head and middle patches need be subsplit. Since head patches have high IIC, each is subsplit into 3 x 3 fine patches. Middle patches with middle IIC, each is subsplit into 2 x 2 medium patches. Each tail patch is considered as a coarse patch and remains unchanged. After patch subsplitting, the input token sequence is denoted as follows:
X [T y..., 5 Eposr
(8)
5

---

where K represents the total number of patches after patch subsplitting.
Feature reuse. Patch subsplitring disrupts the integrity of head and middle patches, resulting in lack of correlation among fine or medium patches which are severally subsplit from the same head middle patch. To address this issue, we incorporate Feature Reuse module from DVT [4] to inject the original information of head or middle patches into new fine or medium patches. It is advantageous enhance feature representation of each patch and strengthen the interconnectivity among them.
Figure demonstrates the Feature Reuse module. Tokens (except class token) output by the ViT of SGIS are taken as input and divided based on their IIC. As head and middle patches need to be subsplit in MGIS, we individually upsample head and middle tokens, so that the number of tokens obtained by upsampling is the same the number of new patches obtained by subsplitring. Different from the module in DVT [14], considering that tail tokens remain unchanged in MGIS, we utilize zero padding to handle them. Finally, all tokens flattened and sorted according to their position in image N. The token sequence X X output by Feature Reuse is obtained as follows:
US. Uisampllnn
ZP:
Zero Padlding
FRR.S. Flatten & Sort
X -FR(z,...., +
(9) Figure Feature Reuse module.
Specially, X E NxC" Due to concatenation expanding the dimension of multi granularatyl tokens to 2x, resulting in 4x increase in computational cost (detailed analysis in Appendix), we set MLP in the module realize dimension reduction: MLP(R = R The value of C' is small, thereby increasing the dimension of multi-granularity tokens from C to C + C², but still much smaller than 2C, which helps save computational cost.
The generated multi- galniiigranlrrrtty token sequence X IR(N4)x(C*+CC) which is input to VIT in MGIS for inference, slimming and output, is denoted as:
X = Concaa(X. X xF [=" x2 ..... N
(10)
3.3.2 Plug-and-Play Slimming Module (PPSM)
Most existing ViT slimming methods, such as Dynamic VIT. PS-VIT, adopt token pruning, undermin- ing the information integrity of the image and is not beneficial for classification. Therefore, select token merging, which diminishes the number of tokens while preserving information.
Three-way decision slimming. Most existing VIT slimming methods belong to hard slimming, which divides tokens into non-redundant and redundant groups based on certain criterion and reduces the number of redundant tokens to streamline VIT. Nevertheless, tokens in the middle part of CASR are arduous to distinguish as non-redundant or redundant during the inference. Mistakenly identifying non non-redundant tokens redundant and performing slimming will lead to degradation of ViT's performance. Considering that three-way decision model establishes a buffer zone, i.c., boundary, between positive and negative, it can aid in more appropriately handling tokens situated in the middle position of CASR. Therefore, we employ three-way decision mechanism to realize soft slimming for VIT.
Three way decision 34 is a decision mechanism in rough set [35] 36 that partitions set into positive, negative, and boundary domains, with elements each domain subject to positive, negative, and boundary rules, respectively. By setting the proportion of elements in positive and nega- tive domains to Tpos and Tneg. all tokens inputted into PPSM are divided into positive, negative and boundary domains. Positive tokens adhere to the positive rule: main- tain unchanged. Boundary tokens adhere to boundary rule, which refers to the token match-merge scheme pro- posed. Negative tokens adhere to the negative rule: all are merged into one token by weights.
Step1
Step2
CASR
Step3
Step4
Figure 5: The process of token match-merge.
Token match-merge seheme. As discussed in Observation 3, dissimilarities among lower IIC tokens should be considered, necessitating hierarchical processing. Thus, employ the token

---

match-merge scheme among boundary tokens to guide the match-merge process of similar boundary tokens, aiming to reduce the number of tokens while maintaining accuracy the best possible extent. We measure the similarity between two tokens by computing the cosine similarity of their value vectors. v", v E RC, present the value vectors of tokens a and b, and the cosine similarity cos expressed as follows:
Ee+e' (ug o
cos c0) = ECC (ug)2 x V (ut)2
(11)
The token match-merge scheme is demonstrated in Figure 5 Step1. Divide tokens at odd positions in CASR into Group and those at even positions into Group B. Step2. Find the most similar token in set B for each token in A by calculating cosine similarity. Step3. Put similar tokens together complete the match. Step4. Merge the similar tokens by weights to reduce the number of tokens.
3.4 Training Mehtod
During the training of MG-VIT, setting makes MGIS be executed for every input image. Like knowledge distillation (37), our goal to make the outputs of SGIS more similar to MGIS. Thus ViT in SGIS can have stronger ability in classification. The training loss about ground-truth label gt is:
loss CE(DM .g() + KL(D*, DDM,
(12)
where CE(:,)) and K K(.,,) represent cross entropy loss and Bellback-Leiblee divergence.
4 Hierachical Multi-Granularity Vision Transformer
ViT (PV Stage
Z oBas Ttd
& abes Utr
35es ltd
SA+MIP
Our Proposed Method
Proxy token
Introducing procy tokens to unify token scales
FR Feature Rouse
Figure 6: The overall framework Hierarchical MG-VIT.
4.1
Key Issue of Extension
Almost all classic VIT compression works, such Dynamic VIT [12), PS-VIT (6).. EViT 13 only focus on classification. Why not extend their methods to hierachical ViT to execute downstream tasks like detection and segmentation? We analyze two reasons. The first reason is structural limitation. For instance, plain VIT only provides single-scale feature representation, lacks the ability to handle multi-scale variation, and lacks image-related priors. While there are works that still employ plain VIT for downstream tasks, such as ViTDet [38]. ViT-Adapter [39,, SegViT ViT 40) 41. they achieve this by integrating well-designed modules, which could be considered as individual research contributions. The second reason is method challenges. Most classic ViT compressions, as mentioned above, heavily rely on pruning or merging, which can be challenging to adapt to downstream tasks. Although we also employ pruning and merging methods, they serve as auxiliary means to reduce computational costs. The core of this paper tiatiirrrnullrrtyy strategy can be applied hicrarchical ViT for downstream tasks.
The key extending multi multi-granularinn strategy to hierarchical VIT lies addressing the non-uniform multi-grained grained patch scales. Subsplitting head and middle patches into 3 x 3 fine patches and 2 medium patches results in patches with multiple scales interweaving an image, making it difficult to seamlessly enclose patches in a conventional window when using Swin Transformer. Therefore, in this paper, we propose proxy tokens to unify scales.
issue of multi
4.2 Introducing Proxy Tokens for Unifying Scales
Similar to MG-ViT, Hierarchical MG-ViT also employs two-stage framework. In SGIS, 2D image N R xx is split into patches P R
x where (H, W) donates the resolution of image N,

---

and presents the patch size. The token sequence generated by P embedding is:
Tori = [forts for ori xC1 ori a simplified representation "origin". IIC of identified in SGIS.
(13)
where Tori E
In MGIS, we assume that P E Rax* is patch that requires subsplitting, which will be subsplit into E fine patches:
Pi [p" ,p' p..
(14)
where l is the manner of subsplitting x subsplit from the i-th head or middle patch if > 0, and j > O.
2), p present the j-th fine or medium patch
Then the finer token sequence To E xCCx originated from the embedding of these finer patches denoted as:
To = [oors ors:
(15)
We introduce a proxy token "pro E R for this group of finer tokens compose the new sequence T [T ppro The computing procedure Transformer layer for this group can be summarized as follows:
T (S(TT)+
T (
(16)
To generalization, We provide optional proxy token set Tpro and its mask vector m for each image, mu
where Tro E R uw xC and m E IR Thus, the unified scale token sequence T' before feature reuse can be denoted as:
T' = (-m) x Tori + m x T pro
(17)
After feature reuse, the token sequence is input into the pyramid-structured hierarchical VIT, through PVT stages 2, 3, and 4, feature maps reshape to H. x C2.
2x 25
4s
H
4s
W
x C3 and
8s
x
x C4. In 88
this way, various downstream tasks can be performed.
4.3 Architectural Details
We develop two variants of Hierarchical MG-VIT based on PVT-Small: Hierarchical MG-ViT-A and Hierarchical MG-ViT-R. Hierarchical MG-ViT-A adds a Transformer layer with proxy tokens between stage and stage PVT-Small. Hierarchical MG-ViT VYT-R maintains the same capacity as PVT-Small but replaces one of the original Transformer layers in stage with Transformer layer introducing proxy tokens. Please refer to the appendix for architectural details.
4.4 Discussions
Performance upgrade. Since our proposed method is plug-and-pla and does not significantly alter the structure of the hierarchical ViT, methods aimed improving the performance of hierarchical VIT. such as using relative position biases and overlapping patch embedding, are all effective. However, we haven't 't optimize Hierarchical MG-ViT for the best performance, as our goal was to explore the potential of extending the multi- granularity strategy from compressing plain ViT to hierarchical VIT for various downstream tasks. the future, we will delve deeper into performance enhancements for Hierarchical MG-ViT.
Plain VIT vs. hierarchical VIT. Which ViT structure is superior? Plain ViT inherently has limitations when it comes to handling dense tasks. In order to utilize a general VIT for a variety of tasks, many new ViTs are designed based on hierarchical structures. However, we do not consider hierarchical ViT to be superior; both plain ViT and hierarchical ViT have their strengths and weaknesses. As for detection tasks, multitude of ViTs built upon the hierarchical structure continue remain the most competitively poised model. However, models like ViTDet (38, which are based on plain VIT, also demonstrate that hierarchical ViTs such Swin-T and PVT are not the only efficient ways to accomplish visual tasks. As for segmentation tasks, what surprised us is that Seg ViTv2 [41, built upon the Plain ViT, has achieved state-of-thes the-art results. for classification tasks, variety of plain ViT models, with MG-ViT representative, exhibit big performance advantage over hierarchical VIT. Moreover, multi-grannn gealiiigaaullrrtty strategy boost performance more effectively in plain structures than in hierarchical structures. This is because hierarchical ViT involves token merging, which

---

can be significantly diminishes the advantages of multi-grante granularity in feature representation. Regarding the question of which ViT structure is superior, believe that necessitates deliberation in the context of varying tasks and distinct constraints.
5 Experiments
5.1 Backbones, Datasets and Evaluation Metries.
We built MG-ViT based on DeïT-S and LV-ViT-S and evaluated its performance on the benchmark ImageNet [8] To quantitatively compare performance, we report the Top-1 accuracy (Acc.), the number of floating-point operations (FI .O)), and throughput (TP). Moreover, we developed Hicrarchical MG-VIT based on PVT-Small and evaluated its performance in object detection and semantic segmentation
the benchmark datasets MS- MSS-CCC22017 [19] and ADE20K [20]. We reported metrics such as box average precision (AP"), mean intersection over union (mloU) and the number of parameters (#Params). All metrics measured single NVIDIA RTX 3090 GPU.
5.2 Main Results
Comparison to backbones. Table 2 presents the comparison results between MG- MG-VIT and backbones with different values of threshold e. It can be observed that Without any performance degradation, the computational cost of MG-ViT is significantly reduced. Moreover, when all images execute MGIS, the performance of VIT is improved, which be attributed to subsplitting patches with multiple different granularities.
Table 2: Comparison between MG-ViT and backbones
Model
Acc((S)
FLOPs(G)
TP(img//)
DeiT-S
79.8
4.6
1341
MG-VT 0.49
79.8(+0.0)
2.0(-56%)
2404(+1.79) x
MG-VIT
0.73
80.8(+100)
2.4(-88%)
2054(+1.53x 1591(+1.11 x)
MG-VIT
81.0(+1.2)
3.9(-15%)
LV-VIT-S
83.3
6.6
989
MG-VIT
0.60
83.3(+0.0)
3.5(-47%)
1749(+1.77x)
MG-VIT
0.73
83.7(+0.4)
3.8(-42%)
1683(+1.70x)
MG-VIT
83.8(+0.5)
5.6(-15%)
1233(+1.25x)
Comparison baselines. In Figure 7 we compare MG-ViT with baselines: Dynamic VIT, EViT, and DVT. These models are chosen representatives of 3 type methods: token pruning, token merging, and dynamic design of token format. With the same setting, MG-ViT outperforms baselines.
Comparison to other VIT compressing methods. In order to demonstrate the effectiveness of our dynamic framework, MG-Vit VIT, compressing ViT, we present a comparison of MG-ViT with various VIT compressing methods in T We visualize their accuracy, FLOPs, and throughput Appendix. The results show that our method for compressing ViT based on the multi-granularity strategy, MG-ViT, is highly competitive when using DeiT-S and LV-ViT-S as backbones.
Table 3: Comparison between MG-ViT and other VIT compressing methods
LV-VI-SS
DeiT-S
Model
FLOPs(G)
Acc.(%)
TP(igg/s)
Model
FLOPs(G)
Acc.(%)
TP(img/s)
Buseline
6.6
Baseline
1341
PS-VIT
4.7
82,4
IA-RED* 29
79.1
1597
eTPS 42
3.8
82.5
1665
DVT
2.4
79.3
1485
EVIT 13
4.7
83,0
1447
Dynamic VrT
12
2.9
79.3
1774
Dynamie ViT
4.6
83,0
1302
Eve- Evo-ViT ViT 27
2.9
79.4
1863
SPVIT 32
4.3
83,1
1518
ATS
2.9
79.7
1531
SiT 30)
4.0
83,4
1280
ST 33
3.2
80.6
1928
CF-ViT
4.0
2096
MG-VIT
3.8
1683
MG-VIT
2054
Comparison to other efficient ViTs. In Figure 8
we compare MG-ViT and various efficient
ViTs 44 45 46
50
51
52
53
25
56
58 59
63 64
observed that MG- VIT competitive even among different backbones and other VIT variants.

---

VIT VIT
OUTTS
Flatas Cleiedd @
Acouracy
Conformen--03
NATAT
GEVITT
PiTi--33 plwird
Wide-PViii 47
FLaten-Prrn--2
Twitks
SWE-T VIT-XT
Accuracy(8)
UnVe-
Autsforen
Couti
12F-MI--14
Thi
CreatiT-115
FLOPs]()
ConktT-2
PVT-M
LITYT
81.7
LocaliiTS
T27-
Craf-Lia
$3.4
Poofformen
CLT-S
83.1
80
PVT-A
82.8
200
Siike PVTia at
825
FLOPUG)
25
3.5
FLOPs(G)
4.5
5.5
6.5
Figure 7: Comparison with baselines.
Figure 8: Comparison MG-ViT VIT between and static efficient VIT methods.
Object detection and semantic segmentation. We adopt the same implementation details as PVT-Small and conduct preliminary experiments Hierarchical MG-Vin ViT's performance in object detection and semantic segmentation tasks to represent its capability for executing downstream tasks. As shown in Table T extending the multi-granullrity strategy to the hierarchical structure is beneficial for enhancing ViT's performance and serves as a nice attempt to slim hierarchical ViT.
Table Object detection with Mask R R-CNN 1x schedule.
Method
#Params(M)
FLOPs(G)
Apb
AP
AP4
PVT-Small 16
44.1
245
40.4
62.9
43.8
Hierarchical MG-VIT-A
46.5
258
40.9
63.2
44.2
Hierarchical MG-ViT-R
41.9
227
40.3
62.7
43.6
Table 5: semantic segmentation with semantic FPN head.
Method
#Params(M)
FLOPs(G)
mIoU
PYT-Small 16
28.2
44.5
39.8
Hierarchical VIT-
29.3
46.4
40.1
Hierarchical MG-VIT-R
27.1
41.3
39.7
Conclusion
MG-ViT, the two-stage dynamic framework based on the strategy we proposed, achieves well- balanced trade-off between performance and computational cost. Within this novel framework, contribute four new methods. Firstly, subsplit patches with multiple different granularities, resulting in generating multi-granularity tokens with various levels of information, which enhances the capacity of VIT in feature representation, leading to performance improvement. Secondly, we introduce a three-way decision mechanism to achieve soft slimming of VIT. Thirdly, we propose the token match-merge scheme guide the match-merge of boundary tokens, reducing their quantity. The second and third methods are reflected in our proposed PPSM. Fourthly, we extend multi-granularity strategy hierarchical ViT for various downstream tasks, demonstrating that our method can also be applied to hierarchical structures for compressing, and we will explore this further in the future.
Acknowledgement
This work is supported by the National Key Research and Development Program of China (No. 2022YFB3104700 the National Natural Science Foundation of China (No. 61976158, No. 62376198, No. 62006172, No. 62076182, No. 62163016. and No. 62106091), the Jiangxi "Double Thousand Plan", the Jiangxi Provincial Natural Science Foundation (No. 20212ACB202001) and the Shandong Provincial Natural Science Foundation (No. PR2021MFO5A The authors would like to thank Jun Wang, Yanping Li, Qixian Zhang, Zhiheng Wu for inspirational and feasible suggestions.
10

---

References
[1] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Informa- tion Processing Systems, pages 1998-6008, 2017.
[2] Yadi Wang, Hongyun Zhang, Duoqian Miao, and Witold Pedrycz. Multi-granularity re-ranking for visible-infrared person re-identification. CAAI Transactions on Intelligence Technology, 2023.
[3] Changwei Wang, Rongtao Xu, Ke Lv, Shibiao Xu, Weiliang Meng, Yuyang Zhang, Bin Fan, and Xiaopeng Zhang, Attention weighted local descriptors. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
[4] Changwei Wang, Rongtao Xu, Shibiao Xu, Weiliang Meng, Jun Xiao, and Xiaopeng Zhang. Ac- curate lung nodule segmentation with detailed representation transfer and soft mask supervision. IEEE Transactions on Neural Networks and Learning Systems, 2023.
[5] Changwei Wang, Rongtao Xu, Shibiao Xu, Weiliang Meng, Ruisheng Wang. Jiguang Zhang, and Xiaopeng Zhang. Towards accurate and efficient road extraction by leveraging the characteristics of road shapes. IEEE Transactions Geoscience and Remote Sensing, 2023.
[6] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
[7] Zhengzhong Tu, Hossein Talebi, Han Zhang, Feng Yang, Peyman Milanfar, Alan C. Bovik, and Yinxiao Li. Maxvit: Multi-axis vision transformer. In European Conference on Computer Vision, pages 479, 2022.
[8] Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yabiao Wang, Yanwei Fu, Jianfeng Feng, Tao Xiang, Philip H.S. Torr, and Li Zhang. Rethinking semantic segmentation from sequence-to-sequene perspective with transformers. Jun 2021.
[9] Rongtao Xu, Changwei Wang, Jiguang Zhang, Shibiao Xu, Weiliang Meng, and Xiaopeng Zhang. Rssformer: Foreground saliency enhancement for remote sensing land-cover segmenta- tion. IEEE Transactions on Image Processing, 12:1052-1064 2023.
[10] Dong Chen, Duoqian Miao, and Xue Rong Zhao. Hyneter: Hybrid network transformer for object detection. CoRR, abs/2302.099655, 2023.
[11] Changwei Wang, Rongtao Xu, Shibiao Xu, Weiliang Meng, and Xiaopeng Zhang. Da-net: Dual branch transformer and adaptive strip upsampling for retinal vessels segmentation. In International Conference on Medical Image Computing and Computer-Assisted Intervention. pages 228-538. Springer, 2022.
[12] Yongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu, Jie Zhou, and Cho-Jui Hsieh. Dynamievit: Efficient vision transformers with dynamic token sparsification. Advances in neural information processing systems, 34: 13937-1997999 2021.
[13] Youwei Liang, Chongjian Ge, Zhan Tong, Yibing Song, Jue Wang, and Pengtao Xie. Not all patches are what you need: Expediting vision transformers via token reorganizations. 2022.
[14] Yulin Wang, Rui Huang, Shiji Song. Zeyi Huang, and Gao Huang. Not all images are worth 16x16 words: Dynamic vision transformers with adaptive sequence length. 2021.
[15] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa. Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers distillation through attention, 2021.
[16] Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In Proceedings of the IEFF/CVF international conference on computer vision, pages 568-578,40 2021.

---

[17] Zi-Hang Jiang, Qibin Hou, Li Yuan, Daquan Zhou, Yujun Shi, Xiaojie Jin, Anran Wang, and Jiashi Feng. All tokens matter: Token labeling for training better vision transformers. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances Neural Information Processing Systems, volume 34, pages 18590-18602. Curran Associates, Inc., 2021.
[18] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE Conference Computer Vision and Pattern Recognition, Mar 2010.
[19] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Purt V 13, pages 740-755. Springer, 2014.
[20] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 633-641, 2017.
[21] Yuxin Fang, Bencheng Liao, Xinggang Wang. Jiemin Fang. Jiyang Qi, Rui Wu, Jianwei Niu, and Wenyu Liu. You only look at one sequence: Rethinking transformer in vision through object detection. Advances in Neural Information Processing Systems, 34:26183-26199 2021.
[22] Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Segformer: Simple and efficient design for semantic segmentation with transformers. Advances in Neural Information Processing Systems, 34:12077-12009 2021.
[23] Ali Hatamizadeh, Greg Heinrich, Hongxu Yin, Andrew Tao, Jose M Alvarez, Jan Kautz, and Pavlo Molchanov. Fastervit: Fast vision transformers with hierarchical attention. arXiv preprint arXiv:2306.06 1899, 2023.
[24] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. arXiv preprint arXiv:2103. /4030, 2021.
[25] Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pvtv2: Improved baselines with pyramid vision transformer. CoRR, 11221006.1779,, 2021.
[26) Yehui Tang, Kai Han, Yunhe Wang, Chang Xu, Jianyuan Guo, Chao Xu, and Dacheng Tao. Patch slimming for efficient vision transformers. In IEEE/VV Conference on Computer Vision and Pattern Recognition, pages 12155-12164, 2022.
[27] Yifan Xu, Zhijie Zhang, Mengdan Zhang, Kekai Sheng. Ke Li, Weiming Dong, Liqing Zhang, Changsheng Xu, and Xing Sun. Evo-vit: Slow-fast token evolution for dynamic vision trans- former. In AAAI Conference on Artificial Intelligence, pages 2964-2999 2972, 2022.
[28] Mengzhao Chen, Mingbao Lin, Ke Li, Yunhang Shen, Yongjian Wu, Fei Chao, and Rongrong Ji. Cf-vit: A general coarse-to to-fine method for vision transformer, 2022.
[29] Bowen Pan, Rameswar Panda, Yifan Jiang, Zhangyang Wang, Rogério Feris, and Aude Oliva. la-redS interpretability-awaal redundancy reduction for vision transformers. In Advances in Neural Information Processing Systems, pages
2021.
[30] Zhuofan Zong, Kunchang Li, Guanglu Song, Yali Wang, Yu Qiao, Biao Leng, and Yu Liu. Self-slimmed vision transformer, 2022.
[31] Lingchen Meng, Hengduo Li, Bor-Chun Chen, Shiyi Lan, Zuxuan Wu, Yu-Gang Jiang, and Ser- Nam Lim. Adavit: Adaptive vision transformers for efficient image recognition. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 12309-12318, 2022.
12

---

[32] Zhenglun Kong, Peiyan Dong, Xiaolong Ma, Xin Meng. Wei Niu. Mengshu Sun, Xuan Shen, Geng Yuan, Bin Ren, Hao Tang, Minghai Qin, and Yanzhi Wang. Spvit: Enabling faster vision transformers via latency-awan soft token pruning. In European Conference Computer Vision, pages 620-640, 2022.
[33] Shuning Chang, Pichao Wang, Ming Lin, Fan Wang, David Junhao Zhang, Rong Jin, and Mike Zheng Shou. Making vision transformers efficient from A token sparsification view. 2023.
[34] Yiyu Yao. T'reeeWay Decision: An Interpretation of Rules in Rough Set Theory, page 642-649.
Jun 2009.
[35] ZDZISLAW PAWLAK. Rough set theory and its applications to data analysis. Cybernetics and Systems, 29(7):661 1666661188,, Jul 2002.
[36] Zdzistaw Pawlak. Rough set theory and its applications, Jan 2002.
[37] GeoffreyE. Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in neural network, Mar 2015.
[38] Yanghao Li, Hanzi Mao, Ross Girshick, and Kaiming He. Exploring plain vision transformer backbones for object detection. In European Conference on Computer Vision, pages 280-296. Springer, 2022.
[39] Zhe Chen, Yuchen Duan, Wenhai Wang, Junjun He, Tong Lu, Jifeng Dai, and Yu Qiao. Vision transformer adapter for dense predictions. arXiv preprint 2xXiv:2205.08334, 2022.
[40] Bowen Zhang, Zhi Tian, Quan Tang, Xiangxiang Chu, Xiaolin Wei, Chunhua Shen, et al. Segvit: Semantic segmentation with plain vision transformers. Advances in Neural Information Processing Systems, 35:4971 -4822, 2022.
[41] Bowen Zhang, Liyang Liu, Minh Hieu Phan, Zhi Tian, Chunhua Shen, and Yifan Liu. Segvitv2: Exploring efficient and continual semantic segmentation with plain vision transformers. arXiv preprint arXiv: 06289, 2023.
[42] Siyuan Wei, Tianzhu Ye, Shen Zhang, Yao Tang, and Jiajun Liang. Joint token pruning and squeezing towards more aggressive compression of vision transformers, 2023.
[43] Mohsen Fayyaz, Soroush Abbasi Koohpayegani, Farnoush Rezaei Jafari, Sunando Sengupta, Hamid Reza Vaezi Joze, Eric Sommerlade, Hamed Pirsiavash, and Juergen Gall. Adaptive token sampling for efficient vision transformers, 2022.
[44] Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition transformers with Auto-Correlation for long-term series forecasting. In Advances in Neural Information Processing Systems, 2021.
[45] Peng Gao, Jiasen Lu, Hongsheng Li, Roozbeh Mottaghi, and Aniruddha Kembhavi. Container: Context aggregation network. arXiv preprint arXiv:2106. 10222000014400 2021.
[46] Pengfei Gu, Yejia Zhang, Chaoli Wang, and Danny 7. Chen. Convformer: Combining cnn and transformer for medical image segmentation, 2022.
[47] Stéphane d' Ascoli, Hugo Touvron, Matthew L. Leavitt, Ari Morcos, Giulio Biroli, and Levent Sagun. Convit: Improving vision transformers with soft convolutional inductive biases. CoRR, 0t//2103.10699 2021.
[48] Mingyu Ding, Bin Xiao, Noel Codella, Ping Luo, Jingdong Wang, and Lu Yuan. Davit: Dual attention vision transformers. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviy, Israel, October 23-27, 2022, Proceedings, Purt XXIV, pages 74-92. Springer, 2022.
[49] Boyu Chen, Peixia Li, Chuming Li, Baopu Li, Lei Bai, Chen Lin, Ming Sun, Junjie Yan, and Wanli Ouyang. Glit: Neural architecture search for global and local image transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 12-21, 2021.
[50] Zizheng Pan, Jianfei Cai, and Bohan Zhuang, Fast vision transformers with hilo attention, 2023.
13

---

[51] Zizheng Pan, Bohan Zhuang, Haoyu He. Jing Liu, and Jianfei Cai. Less is more: Pay less attention in vision transformers. In AAAI, 2022.
[52] Yanghao Li, Chao-Yuan Wu, Haoqi Fan, Karttikeya Mangalam, Bo Xiong, Jitendra Malik, and Christoph Feichtenhofer. Mvitv2: Improved multiscale vision transformers for classification and detection, 2022.
[53] Weihao Yu, Mi Luo, Pan Zhou, Chenyang Si, Yichen Zhou, Xinchao Wang, Jiashi Feng, and Shuicheng Yan. Metaformer is actually what you need for vision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10819-10829, 2022.
[54] Zhengsu Chen, Lingxi Xie, Jianwei Niu, Xuefeng Liu, Longhui Wei, and Qi Tian. Visformer: The uision-friedlyy transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 589-598, 2021.
[55] Pengchuan Zhang, Xiyang Dai, Jianwei Yang, Bin Xiao, Lu Yuan, Lei Zhang, and Jianfeng Gao. Multi-scale vision longformer: A new vision transformer for high-resolution image encoding. ICCV 2021, 2021.
[56] Kunchang Li, Yali Wang, Junhao Zhang, Peng Gao, Guanglu Song. Yu Liu, Hongsheng Li, and Yu Qiao. Uniformer: Unifying convolution and self-attention for visual recognition, 2022.
[57] Ali Hassani, Steven Walton, Jiachen Li, Shen Li, and Humphrey Shi. Neighborhood attention transformer. In JEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[58] Xiangxiang Chu, Zhi Tian, Yuqing Wang, Bo Zhang, Haibing Ren. Xiaolin Wei, Huaxia Xia, and Chunhua Shen. Twins: Revisiting the design of spatial attention in vision transformers, 2021.
[59] Weifeng Lin, Ziheng Wu, Jiayu Chen, Jun Huang, and Lianwen Jin. Scale-aware modulation meet transformer. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 6015-6026, 2023.
[60] Ali Hatamizadeh, Hongxu Yin, Greg Heinrich, Jan Kautz, and Pavlo Molchanov. Global context vision transformers. In International Conference on Machine Learning, pages 12633-12646.
PMLR, 2023.
[61] Jiayu Jiao, Yu-Ming Tang, Kun-Yu Yu Lin, Yipeng Gao, Jinhua Ma, Yaowei Wang, and Wei-Shi Zheng. Dilateformer: Multi-scale dilated transformer for visual recognition. IEEE Transactions on Multimedia, 2023.
[62] Dongchen Han, Xuran Pan, Yizeng Han, Shiji Song, and Gao Huang. Flatten transformer: Vision transformer using focused linear attention. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 5961-5971, 2023.
[63] Zhiliang Peng, Zonghao Guo, Wei Huang, Yaowei Wang, Lingxi Xie, Jianbin Jiao, Qi Tian, and Qixiang Ye. Conformer: Local features coupling global representations for recognition and detection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
[64] Xuran Pan, Tianzhu Ye, Zhuofan Xia, Shiji Song, and Gao Huang. Slide-transformer: Hierar- chical vision transformer with local self-attention. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2082-2091, 2023.
14

---

7 Appendix
7.1 Hierarchical MG-VIT Architectural Details
We present two variants of the Hierarchical MG-ViT designed based on PVT-Small, as shown in Table The relevant hyper parameters are listed follows: P; the patch size of Stage i; Cj: the channel number of the output of Stage i; R3: the reduction ratio of the spatial-reduction attention in Stage N₃: the head number of the spatial-reduction attention Stage E;: the expansion ratio of the feed-forwate layer in Stage i; Hę: the head number of the multi-head attention in Stage i.
Table 6: Architectural details of Hierarchical MG- ViT and PVT-Small
Hierarchical ViT-iT-A
Hierarchica MG- VIT-R
Output Size
Layer Name
PVT-Small
Patch Embedding
4:0
Stage 1
M
R1 8
R1 8
Ri 8
Transformer Encoder
N1 = x
N1 1
x
N1 = 1
2
E1
E1 8
E1 8
Stage MG
H 4
x
Transformer
MG
HMG 8
FMG X
HMG
FMG
X
Patch
P3 C2 128
Stage 2
Embedding
Ra 1
R₂ = 1
Rs = 1
Transformer
N2 2 3
N2 2
x 3
N2 2
3
Encoder
Es
E2 8
bi 8
Patch
P3 2; C3 320
Stage 3
Embedding
16
R3 2
R3 2
R3 2
Transformer Encoder
N₂ 5 6
N3 5 6
N 5 6
Es
E3
Eg 4
Patch Embedding
P4 2; C4 = 512
Stage 4
R4
R4
R4 = 1
Transformer
N4 8 x
N4 8 x
N₂ 8
Encoder
E4
E4
7.2 Implementation Details
The resolution of input images in experiments is 224 x 224. In SGIS, split each image into 7 x patches. Th and F, are set to 0.1 and 0.4, respectively. Therefore. each image is subsplit into 161 patches in MGIS. For DeiT-S, we inserted a total of three PPSM modules in the 3rd, 7th, and 10th layers. Similarly, for LV-ViT-S, we inserted a total of four PPSM modules in the 4th, 7th, 9th, and 12th layers. For conducting the training process, optimizer to train all models for 300 epochs.
the batch size to 256 and use AdamW
In terms of training strategies and optimization methods, training closely follows the original methods proposed in the studies conducted by DeiT and LV-ViT, but do not employ any knowledge distillation algorithms in our experiments. Throughout the training phase, observed the impact on convergence when subsplitting was applied solely to the head and middle patches. To enhance the speed of model convergence,
adopted subsplitting specifically targeting the head and middle patches for the initial 200 epochs. Subsequently, for the following 100 epochs, we applied subsplitting to the head and tail patches. The training performed on a workstation equipped with 8 RTX 3090 GPUss
15

---

7.3 Visualization Result
In Figure we specifically demonstrate the images that can be well recognized by MG- ViT (LV- ViT-S) in SGIS and MGIS. As observed, the images the left are "simple," usually occupying sigmificant portion of the image pixels, with typically complete and clear contours. While the images on the right are "complex", containing intricate scenes and non-prominent objects or only include small part the objects, requiring multi granularrtlr representations through additional tokens.
Gablet
Figure Example images recognized SGIS MGIS. To provide clarity, we use unfilled, semi-white, and semi-black boxes to represent the head, middle, and tail patches in MGIS, respectively.
Figure 10 visualizes the token slimming comparison between token merging (MG-VIT) and token pruning (Dynamic ViT). The result shows, the information loss caused by token pruning may result in incorrect slimming decisions, leading to decrease in classification accuracy. However, the adoption of token merging in PPSM alleviates this situation.
input
Snowplow
Arctic fox
Castle
Gorill
Chrysanthenum
DynamicViT
Bulldozer
Wolf
Cathedral
Monkey
Cole flowers
MG-VIT
Snowplow
Arctic fox
Castle
Garilla
Chrysanthenum
Figure 10: Visual comparison between token merging and pruning. The boxes with the same color represent merged token.
16

---

D
LV-VIT-S
MT
MCNIT
CFVIT
STMT
ST
ATS
DVT
Model
Model
Throughput
DynanicVT
DVT
EVIT
DynamicHT
IMRES
PIVIT
70
79.2
79.4
19.6
79.0
an.2
60.4
80.6
80.0
82.2
62-4
62-8
82.6
83.2
82.4
62.6
61.
Accuracyrg)
Accuracy(8)
Figure 11: Comparison between MG-Vi VIT and existing efficient VIT methods. The size of represents the computational cost required for each image (FLOPs).It can be observed that MG-VIT VIT achieves good balance in terms of accuracy. FLOPs, and throughput, demonstrating overall competitiveness.
Table 8: Ablation experiment global class attention.
Table 9: Ablation experiment Feature
Reuse module.
Acc.(%
Acc.(%
Scheme
Scheme
SGIS
MGIS
SGIS
MGIS
Random division
75.0
78.9
Feature Reuse
75.0
79.9
Random CASR
75.3
79.8
Ours + Class Token
75.4
80.1
Last CASR
75.5
80.5
w/o Zero-Padding
75.5
80.7
Ours
75.6
80.8
Concat-vadd
75.4
80.7
Ours
75.6
80.8
7.4 More Experiments and Ablation Analysis
Necessity global class attention.
Table Accuracy with different values of A.
We designed two experiments based on backbone DeiT-S to investigate the necessity of global class attention.
0.5
0.9
0.95
0.98
0.99
0.999
Acc.(%) 80.3
80.5
80.7
80.7
80.8
80.7
80.7
Table 8 shows the gain of the proposed global attention class on model performance, compared to random division, which involves randomly dividing patches into three groups, random CASR. which divides patches according to CASR in a random layer of ViT, and last CASR, which divides patches based CASR in the last layer. As we can see, the global attention class achieves the best performance improvement. Table [ compares the contribution of the global attention class to model performance with different values of X. When À 0, it represents the last CASR scheme. We set the default value of À as 0.98.
Applicability of Feature Reuse. We introduce the Feature Reuse module from DVT to enhance the feature representation of patches, and also explore its applicability in our model. Specially, we do not reuse the class tokens and apply zero padding to the tail tokens (i.c., not reusing the information from tail tokens). Table" demonstaesss the effective integration the Feature Reuse module into our model, and indicates the effectiveness of our improvements.
Effect of token match-merge scheme. We conduct ablation experiments investigate the effort of our proposed token match- scheme and establish three control schemes for comparison: token pruning, token clustering (K-means), and merging tokens into one. Especially, by employing our token match-merge scheme on boundary tokens, we achieved the best trade-off between accuracy and speed, as shown Table TO This result also supports the rationality of merging all negative tokens into one, because need to process the tokens with low IIC in the most computationally
17

---

Table 10: Ablation experiment the
Table 11: Comparison among various merging approaches.
token match-marge scheme.
Scheme
Acc.(%)
SPD(img./s)
Approach
Acc.
Token pruning
79.5
188
Keep
80.3
Token clustering
79.8
173
Max pool
80.4
Merging tokens into
80.4
183
Avg pool
80.6
Match-marge (ours)
80.8
180
weighted avg(ours)
80.8
economical manner. Regarding the approach of generating new tokens by merging, the results in TI demonstrate that the weighted average method performs best maintaining accuracy.
Token merging us. token pruning. Table 12 displays the performance of token pruning and token merging under two types of slimming, namely the hard slimming type (dividing tokens into two groups) and the soft slimming type (dividing tokens into three groups). It is observed that utilizing three-way decision mechanism for token merging (i.e., our proposed PPSM) achieves the best performance. This is attributed to assigning tokens located in the middle part of CASR to the boundary domain within three-way decision model, where the tokens adhere boundary rules, resembling "delayed processing".
Table 12: Comparison between hard and soft slimming.
Table 13: Experiments fine-tuned and evaluated resolution 384.
Type
Method
Acc.(%)
Model
Acc.(%)
FLOPPs(G)
TP(img
Hard
Pruning Merging
79.8
LV-VIT-S
84.4
22.2
303
80.3
MG-ViT 0.66
84.4(+0.0)
10.7(-52%)
515(+1.70x)
Soft
Pruning Merging 80.8
80.2
MG-VIT 384
85.5(+0.7)
16.9(-21%)
345(+1.14x)
Effectiveness of PPSM. We integrate the two methods, three-way decision slimming and token match-merge ViT (PPSM) with existing methods of hard slimming and soft slimming, as well as the performance between MG-Vi- ViT with and without PPSM. The results demonstrate that our proposed PPSM module effectively streamlines the model while maintaining performance.
match-ne merge scheme, into our proposed PPSM module. Table [ the performance of MG-
Table 14: Comparison between MG-ViT and the existing soft and hard slimming methods for ViT
Type
Method
Model
Acc.(%)
FLOPPs())
Origin
MG-ViT PPSM)
80.8
3.7
Baseline
DeiT
79.8
PS-VIT
79.8(-0.0)
3.8(-1.2)
Hard slimming
Pruning
DynamieViT
79.3(-0.5)
2.9(-1.7)
IA RA-RED
79.1(-0.7)
3.3( (.3))
Merging
EVIT
79.5(-0.3)
3.0(-1.6)
Pruning
SiT
2.3( 2.3)
Soft slimming
Evo-Vi
79.4( 0.4)
2.9( 117)
Merging
MG-VIT
80.8(+1.00
2.4(- -2.2)
Experiment on images with higher resolution. We conduct experiments on higher resolution images, as shown Table T3] demonstrating the continued effectiveness of our proposed multi- multi-granularing strategy on higher resolution images.
18

---

7.5 Computational Complexity Analysis
In this discussion, we disregard the patch embedding block. The computational complexity of transformer layer responsible for processing tokens can be calculated as follows:
C ((Loken))))) = 4NC + 2N2C
(18)
(token) = 8NC².
Based on provided equations, can be observed that increasing the dimension C' leads to a quadratic growth in computational complexity. Therefore, if the dimension C' is expanded to 2x, the computa- tional cost will approximately increase by 4x, as discussed in the feature reuse part.
In the PPSM of MG-ViT, we introduce the three-way decision slimming to reduce the number of tokens. Specifically, based ratios Tpos and I'neg all tokens are divided into positive, negative, and boundary tokens. Positive tokens remain unchanged. Boundary tokens are merged to the half of their original quantity following the guidance of the token match-merge scheme. Negative tokens are merged into single token, and their quantity can be neglected. Therefore, the computational complexity in the transformer layer following the first PPSM can described as follows:
N = poos_ N + ppos Tneg N 2 ((PPSM')) 12N' +2
(19)
We employ DeiT-S as the backbone and insert PPSM before the self-attention (SA) layers in the 4th, 7th, and 10th layers. The positive, boundary, and negative tokens are divided according ratio of 5:4:1. The computational complexity of DeiT-S and our MG-ViT are:
((DeiT) = 144NC² + C, ((STVIT) = 91.2NC + 111.N2C.
(20)
7.6 Discussion of Ratio
Th and rt. Each head patch is subsplit into 9 fine patches, while each middle patch is subsplit into 4 medium patches. The majority of important information concentrated in few head patches, partially distributed among middle patches, and only small portion exists in tail patches. Conse- quently, we mandate that the count of middle patches exceeds half the total number. Additionally, as our aim is to develop efficient ViT framework, we strive to minimize the total number of patches obtained after subsplitting compared to the 196 tokens (the official splitting method 14x 14). Considering these factors, we set Ta < 0.2. Furthermore, based on the results presented in Fig.1(a) (a) and Fig. ICC), we set 0.3 < тл < 0.5. We conducted experiments on the origin MG-ViT (without PPSM) and selected several competitive control group results to showcase in table 15
Table 15: Accuracy and FLOPs with different values of T% and rt-
Th Va
0.05%,0.3
(..4
0.1
0.1
0.15%,0.35
(%).
80.7
80.5
81.0
81.0
81.1
FLOPs(G)
3.6
3.5
4.1
3.9
4.2
Tpos and Tneg- According to the discussion in Section 6.3, we exclude the match- of boundary tokens and only consider the merging of negative tokens. When I'ng 0.2, the number of tokens in the last layer of MG-ViT decreases to 51.2% of the original count. Therefore, the quantity of negative tokens should be relatively small, and we set Ineg < 0.2. On the other hand, if Tpos = 0.5, the minimum number of tokens in the last layer of MG-VIT should be 12.5% of the original count, which serves as the lowest limit for normal operation of MG-VIT. As a result, we set 0.5 I Tpos < 0.7. Our experimental findings, presented in Table I6 include several competitive control group results.
Taking all aspects into consideration, we set Th 0.1, Ft 0.4, Tpos 0.5 and Fneg 0.1.
7.7 Three-Wis Way Decision
Three-way decision decision mechanism in rough set theory. Let U be the set of all tokens, and given a pair of thresholds Tpos and Tneg- Let e(x) denote the rank of token based on CASR, and
19

---

Table 16: Accuracy and FLOPs with different values of ppos and Tneg-
Tpos Vineg
0.5
0.5
0.6
0.6\0.1
0.7%
Acc.(%)
80.8
80.6
81.0
81.0
81.1
FLOPs(G)
2,4
2.3
2.9
2.8
3,2
represent the total number of tokens in the PPSM. We can describe the positive, negative, and boundary domains as follows:
Pos E e(x) S num Tpost
Neg EI e(x) Z num (1 - Tneeg }
(21)
Bdy (a num Tpos e(x) num (1 Tneg))
For tokens in the positive and negative domains, employ deterministic operations to either maintain them unchanged or merge them into one token. This because we have determined their redundancy status. For tokens in the boundary domain whose redundancy status has not been determined, employ undeterministic operation, token match-merge. Token match-merge strikes a balance between maintaining and merging, representing form of slow merge that only merges similar tokens. If the newly generated boundary tokens are subsequently divided into the positive and negative domains in the next PPSM, they will undergo deterministic operations.
20


Reviews:
[{"review_id": "50iGjkewwH", "reviewer": "Reviewer_qw7Z", "summary": "This paper introduced a new vision transformer model, denoted as MG-ViT, which consists of a two-stage approach to balance the trade-off between the computational cost and performance. Specifically, it proposes a two-stage approach, in which a single-granularity inference stage is used to determine if the image can be correctly classified. If not, then a second multi-granularity stage is employed. The model is trained and tested on ImageNet-1K dataset and reportedly achieve competitive performance. ", "questions": "1. The proposed model is only evaluated for classification. Can the same ideas be extended to down-stream tasks such as detection and segmentation ? \n\n2. Typical ImageNet images with resolution 224 x 224 are used for training and evaluation of the proposed model. How about higher resolution images ? \n\n3. Can the proposed model be used with hierarchical vision transformers such as Swin Transformers ? ", "limitations": "Yes", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-written and easy to follow. \n\n2. The idea of improving vision transformer efficiency is important and has merits. \n\n3. The authors present comprehensive ablations studies and analysis of their proposed model. ", "weaknesses": "1. The proposed effort lacks novelty as it essentially proposes a two-stage approach with different level of granularity to strike a balance between accuracy vs efficiency. In the proposed framework, there are several assumptions that may limit the applicability of this work. First, it is not clear how to determine the confidence judgement threshold in the first stage which can significantly impact the performance. Second, the head.middle and tail patchification scheme cannot be generalized to all use-cases and can indeed introduce undesired biases that can negatively impact the model's performance. \n\n2. One of the main arguments of this work in justifying its usefulness is the efficiency aspect. However, the authors only report the gains in terms of number of FLOPs which does not include the entire picture. In fact, the throughput comparison is only shown in Figure 10 of the supplementary materials and without any actual quantitative comparisons. Since the method is comprised of two stages, it is expected that the throughput is negatively impacted. \n\n3. The method is only evaluated for smaller model regime and it is not clear whether it scales to larger models (e.g. ViT-L, ViT-H, ViT-G). \n\n4. The experimental results are not convincing enough and don't validate the effectiveness of this work. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank you for taking the time to review our work. We address the questions and clarify the issues accordingly as described below.\n\n**Question 1**: Transferring the multi-granularity strategy into dense tasks might be considered feasible. However, this topic will be explored in our subsequent research. Since we focus on compressing ViT to improve efficiency in this work, rather than proposing a new general ViT structure, similar to classical papers[r1-r7] in ViT compression, which mainly evaluated classification tasks, we've adopted the same approach. we're also eager to observe the performance of the multi-granularity strategy in downstream tasks, making it exciting and interesting for future work.\n\n**Question 2**: Thanks for your suggestion. The average resolution of images in ImageNet is 469×387. Standard works[r1-r7] on ViT compression all have employed a resolution of 224×224 for training and evaluation. However, a few studies have also fine-tuned and evaluated at 384 resolution. As a result, we have appended the result at 384 resolution. Please refer to Table 15 in the PDF.\n\n**Question 3**: Our proposed model can be combined with hierarchical ViT such as Swin-T. For the detailed approach and analysis, please refer to the response of Weakness 1 from Reviewer V3HF. However, we find that combining with hierarchical ViT is unnecessary for classification. The merging in hierarchical ViT considerably diminishes the advantages of multi-granularity in feature representation. This work offers fresh insights, challenging the notion that hierarchical ViT is superior. The original non-hierarchical ViT coupling with the multi-granularity method can achieve better results, simple but work. In object detection, ViTDet[r8] proposed by Kaiming He also demonstrates that hierarchical structures like Swin-T are not the only efficient ways to accomplish visual tasks.\n\n**Weakness 1**: ① While various variants of ViT have emerged, our motivation differs from theirs. Inspired by 3 observations during classification, we combined the idea of multi-granularity with common techniques like multi-stage and token merging, accomplishing classification tasks while adeptly balancing the computational cost and performance. Simple but work. This exploration provided us with a new view of problem-solving. And we made the first attempt to integrate multi-granularity patches in the same transformer block. Therefore, we believe our work is novel, gratefully, which is also appreciated by Reviewer J3bo and V3HF. ② In Table 2, we show the impact of thresholds on accuracy, FLOPs, and throughput. Taking MG-ViT with a DeiT-S backbone as an example, when $\\epsilon=0.49$, MG-ViT reduces FLOPs by 56% and increases throughput by 1.79x without any accuracy loss. We regret the absence of a unified method for determining the threshold as a hyperparameter. However, we provide additional data (Table 14 in PDF) to assist in threshold determination based on hardware configuration and other specific requirements. ③ During the inference stage, the patchification scheme (in MGIS) is not applied to all validation use-cases; it is only used for \"complex\" cases. Figure 8 provides the visualization of both “complex” and “simple” cases. This is because if an image can be well recognized in SGIS, there is no need to execute MGIS for the head, middle, tail patch identification and subsplitting. This plays a pivotal role in the effective balance of performance and computation achieved by MG-ViT. Additionally, during the training stage, as claimed in line 253, \"During the training of MG-ViT, setting $\\epsilon=1$ makes MGIS be executed for every input image.\" all training use-cases (images) are employed in the training stage (especially in MGIS). Therefore, MG-ViT avoids introducing undesired biases during training, fairness can be ensured.\n\n**Weakness 2**: In Table 2, we quantitatively show the throughput (TP) of MG-ViT. It can be observed that our strategy enhances throughput, and the two-stage method does not negatively affect it throughput. Responding to your and Reviewer L7yG's feedback, we've incorporated throughput and parameter comparisons in Table 3. (Please refer to Table 16 in PDF)\n\n**Weakness 3**: Classical works [r1-r7] on ViT compression have chosen DeiT, LV-ViT, and T2T (later replaced by LV-ViT) as backbones to develop efficient ViTs. Therefore, we have supplemented the experimental evaluations of the remaining sizes of DeiT and LV-ViT. A total of 4 backbones are covered, with experiments completed for DeiT-Ti, DeiT-B, and LV-ViT-M (please refer to Table 13 in the PDF). Due to limitations in computational resources and time, we have temporarily not conducted experiments for LV-ViT-L; we will add its experimental results in the revised version. We greatly appreciate your suggestions.\n\n**Weakness 4**: We have supplemented some experimental results based on your feedback and that of other reviewers. We appreciate your suggestions.\n\n\n\n"}, {"review_id": "CRs6R5NyWb", "reviewer": "Reviewer_J3bo", "summary": "This paper presents MG-ViT, a way to save computational cost without affecting the performance of ViT-based methods. MG-ViT is a two-stage model, the first stage is a lightweight ViT for single-granularity inference, and the second stage network for multi-granularity inference.", "questions": "The MG-VIT proposed in this paper is novel enough, but the effectiveness of the proposed method needs to be validated under stronger baselines.", "limitations": "Refer to weakness", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1、A series of experiments are conducted to demonstrate the influence of tokens of different IIC and granularity on network performance, which provides a clear motivation for the proposed MG-ViT.\n\n2、The design of MGIS, including multi-granularity patch subsplitting, PPSM, and token match-merge scheme is natural and reasonable.\n\n3、The ablation experiments of the proposed components are sufficient.\n", "weaknesses": "1、In recent years, a series of ViT-based methods based on window attention (Such as Swin Transformer, and Slide-Transformer[1]) have been proposed, which have better classification performance and lower computational cost than the original ViT (DeiT). This paper can better prove the effectiveness of MG-ViT, if it can be compatible with the frameworks based on window attention especially the current sota methods (Slide-Transformer).\n\n2、This paper can better prove the effectiveness of MG-ViT, if it can be compatible with ViTs pretrained by massive data (JFT-3B or LAION) or self-supervised pre-trained (Such as MAE[2], DINOv2), which achieves better performance (more than 85% acc on ImageNet).\n\n\n[1] Pan X, Ye T, Xia Z, et al. Slide-Transformer: Hierarchical Vision Transformer with Local Self-Attention[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023: 2082-2091.\n\n[2] He K, Chen X, Xie S, et al. Masked autoencoders are scalable vision learners[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022: 16000-16009.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review, recognition, and suggestions. We address the questions and clarify the issues accordingly as described below.\n\n**Weakness 1**: ① It is evident that hierarchical ViT models based on window attention, such as Swin-T, surpass the original ViT in many tasks or performance aspects. We have also explored combining our multi-granularity strategy with Swin-T. For the detailed approach and analysis, please refer to the response of Weakness1 from Reviewer V3HF. However, we find that combining with Swin-T is unnecessary for classification. The merging in Swin-T considerably diminishes the advantages of multi-granularity in feature representation. ② slide-transformer [r22] proposes a novel local attention module, Slide Attention. We have appended experimental results comparing the performance of MG-ViT and Slide-Transformer on classification tasks (Please refer to Table 17 in PDF). In the future, we are willing to carefully consider your suggestion to make the multi-granularity strategy and Slide-Transformer compatible. Thank you!\n\n**Weakness 2**：Indeed, utilizing more massive datasets like JFT-3B for pretraining can improve the performance. Currently, models pre-trained using datasets like Laion, such as OpenCLIP and the aforementioned DINOv2, have achieved encouraging outcomes in the ImageNet classification task. There are certain aspects to learn from them, but we haven't yet specifically considered how to combine them. We will carefully contemplate this in the future. Article [r25] found that after employing appropriate MAE pre-training and fine-tuning on ImageNet, the performance of the original ViT-Tiny surpassed that of various lightweight CNNs and meticulously designed ViT variants. \nIn line with your suggestion, we will consider selecting appropriate MAE pre-training in our next work. Thank you!\n\n**Question 1**: Thank you for recognizing the novelty of our work. Following your and Reviewer cCy5's common suggestion, we have found some recent works published in 2023 as stronger baselines and appended relevant experiments (Please refer to Table 17 in the PDF). Furthermore, our work focuses on compressing ViT, and the methods we compare with in our paper, such as eTPS [r23] and STViT [r24], were also published in 2023. Hence, within the domain of ViT compression, our method currently achieves state-of-the-art outcomes.\n"}, {"review_id": "XYOH7l1O3g", "reviewer": "Reviewer_V3HF", "summary": "The proposed MG-ViT framework introduces a two-stage multi-granularity approach for Vision Transformers (ViTs), aiming to balance performance and computational cost. Initially, an image is divided into patches for basic classification. If the image isn't accurately recognized, the system initiates a more detailed patch analysis stage.\n\nTo reduce the computational cost, a three-way decision slimming method is proposed. A token match-merge scheme also aids in making the ViT more compact and efficient. Tests on ImageNet show that without performance loss, MG-ViT reduces the FLOPs of LV-ViT by 47% and DeiT by 56%, demonstrating significant computational cost reduction.", "questions": "Please see weakness.", "limitations": "Please see weakness.", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "\n-The underlying motivation for the study is lucid, and the proposed method displays novelty.\n\n-The manuscript is well-written and easy to comprehend.\n\n-The experimental results highlight an impressive balance between accuracy and speed offered by the proposed method", "weaknesses": "While the Multi-Granularity strategy is well-intentioned and produces promising classification results, its application to other vision downstream tasks such as segmentation presents significant challenges.\n\nAdditionally, there is a lack of results regarding window-based Vision Transformers like the Swin Transformer. This could be due to the complexity and difficulty in applying this strategy to such models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank you for taking the time to review our work. We address the questions and clarify the issues accordingly as described below.\n\n**Weakness1：combining our proposed framework and Swin-T.**\n\nWe have previously investigated the integration of our proposed framework with Swin-T and found it to be feasible. In a general sense, we have made the following modifications to the framework:\n\n1. Replaced the original ViT in MGIS with a window-based hierarchical ViT, namely Swin-T.\n2. Diverging from the 7×7 splitting scheme discussed in the paper for SGIS, we employed an 8×8 scheme. The patches resulting from the 8×8 splitting are referred to as coarse patches.\n3. In MGIS, contiguous 2×2 coarse patches constitute a window.\n4. Utilizing CSAR, we selected the important coarse patches and further subsplitted each of these important coarse patches into four 4 patches through a 2×2 subsplitting procedure.\n5. During patch merging in Swin-T, each coarse patch (token) was replicated 4× and treated as fine patches (tokens) to merge.\n\nWe evaluated the model performance on the ImageNet. It was observed that the LV-ViT-S-based framework (referred to as MG-ViT) proposed in the paper achieves a reduction of 47% in FLOPs without sacrificing performance. In contrast, the Swin Transformer-S-based framework only achieves a 24% reduction in FLOPs and also lags behind MG-ViT in terms of accuracy. We attribute this discrepancy to two main factors:\n\nFirstly, the interaction between coarse and fine-grained patches (tokens) can only occur at shallower layers. With the continuous merging of patches, the granularity differences between tokens diminish, significantly eroding the benefits of multi-granularity in feature representation. However, the proposed PPSM effectively mitigates this issue by allowing patches of varying granularity to interact over several transformer layers before merging.\n\nSecondly, this discrepancy can also be attributed to inherent characteristics of Swin-T itself. As has been lamented by some researchers, the windowed attention mechanism deviates from the original intent of the ViT design, transforming global interactions into local interactions. In contrast, employing the original ViT circumvents such an issue from arising.\n\nOur work in the paper offers fresh insights, challenging the notion that hierarchical ViT is superior. The original non-hierarchical ViT coupling with the multi-granularity method can achieve better results, simple but work. In object detection, ViTDet[r8] proposed by Kaiming He also demonstrates that hierarchical structures like Swin-T are not the only efficient ways to accomplish visual tasks.\n\n**Weakness2**：While incorporating a multi-granularity strategy into downstream tasks presents significant challenges, it is not insurmountable. As previously discussed, the integration of our proposed framework with Swin-T offers a promising foundation for implementing the multi-granularity strategy in downstream tasks. Moreover, we extend our gratitude to Reviewer cCy5 for furnishing a valuable reference 4, which effectively illustrates the applicability of token pruning from ViT to dense tasks like detection and segmentation. This contribution has provided valuable insights and bolstered our confidence. Consequently, we eagerly anticipate observing the performance of the multi-granularity strategy in diverse downstream tasks. However, this topic will be explored in our subsequent research. Since we focus on compressing ViT to improve efficiency in this work, rather than proposing a new general ViT structure, similar to classical papers[r1-r7] in ViT compression, which mainly evaluated classification tasks, we've adopted the same approach.\n"}, {"review_id": "d7JtKh6iAu", "reviewer": "Reviewer_cCy5", "summary": "This paper presents a multi-granularity method to develop efficient vision Transformers. A two-stage inference pipeline is designed to handle easy and difficult samples in different ways. The method is evaluated on ImageNet based on DeiT and LVViT.", "questions": "Please refer to my comments above.", "limitations": "The limitations of the proposed method have not been discussed.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 1, "contribution": 2, "strengths": "- The motivation presented in the Introduction looks convincing and insightful.\n\n- The method achieves good performance on ImageNet classification and can outperform previous methods.", "weaknesses": "- The writing of this paper can be further improved. There are quite a few grammar errors and typos. Many sentences are confusing or hard to understand. We list some of them: line 65, \"we propose MG-ViT 2\"; line 80, \"Simply put, based on CASR, tokens are divided ...\".\n\n- Although the method is well-motivated, the proposed method is not very new: there are several existing methods to use classification probability to estimate the confidence of a prediction and use it to adjust computation [r1]; the token merging and multi-scale inference framework have also been explored in previous work like DVT and [r2].\n\n- The experimental results are not very impressive and competitive in 2023 since recent efficient vision Transformers are usually compatible with more diverse architectures like Swin/ConvNeXt [r1,r3] and more complex tasks like detection and segmentation [r3, r4]. The claim in line 87 \"MG-ViT is a dynamic and versatile framework that can be applied to most ViT models\" is also not well supported since it seems non-trivial to apply the method to hierarchical Transformers. \n\n[r1] Token Merging: Your ViT But Faster, ICLR 2023.\n\n[r2] MIA-Former: Efficient and Robust Vision Transformers via Multi-Grained Input Adaptation, AAAI 2022.\n\n[r3] Dynamic Spatial Sparsification for Efficient Vision Transformers and Convolutional Neural Networks, TPAMI. \n\n[r4] Revisiting Token Pruning for Object Detection and Instance Segmentation, arXiv:2306.07050", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your careful and insightful review! I greatly benefited from your feedback, especially the reference 4 you provided. We address the questions and clarify the issues accordingly as described below.\n\n**Weakness 1**: Thanks very much for your careful review! We apologize for the confusion caused by our presentation during your review. For instance, your mention of line 65, \"we propose MG-ViT 2,\" where 2 refers to the reference of Figure 2 set in our LaTeX document. We will thoroughly review similar errors and further polish our presentation. A well-written paper will be presented in the revised version. Thank you once again!\n\n**Weakness 2**: ① We agree that the idea of using classification probability to estimate the confidence of a prediction and adjusting computation is not new. However, in our case, we chose to compress the original non-hierarchical ViT for classification tasks. This inherently includes the class token, as it integrates information from each token for classification. Consequently, employing classification probability (class attention score) is the sole method to achieve a balance between performance and computation. ② Our work is inspired by 3 observations during classification. We combined the idea of multi-granularity with common techniques such as multi-stage and token merging to achieve the classification task while skillfully balancing computational costs and performance. It's a simple but effective approach. This exploration has offered us a new perspective on problem-solving. ③ While DVT and MIA-Former have explored multi-scale inference framework, we beg to differ that our proposed multi-granularity strategy aligns with them. DVT and MIA-Former integrate multi-scale tokens between different transformer layers, whereas, we integrate multi-granularity patches in the same transformer layer, which is a new attempt. Therefore, we believe our work is novel.\n\n**Weakness 3**: ① We have found several works [r9-r22] published in top journals and conferences in 2023, including 2 recently submitted articles [r18, r21] on arXiv this month. These works have employed various backbones such as ViT, CNN, Swin-T, GNN and ConvNeXt. Upon comparison (Please refer to Table 17 in PDF) with their performance, we find that MG-ViT remains competitive. Furthermore, our work focuses on compressing ViT, and the methods we compare with in our paper, such as eTPS [r23] and STViT[r24], were also published in 2023. Hence, within the domain of ViT compression, our method currently achieves state-of-the-art outcomes. ② Since we focus on compressing ViT to improve efficiency in this work, rather than proposing a new general ViT structure, similar to classical papers [r1-r7] in ViT compression, which mainly evaluated classification tasks, we've adopted the same approach. However, we are also actively attempting to extend the multi-granularity strategy to downstream tasks such as detection and segmentation. This topic is also being explored in our subsequent research. Just like the reference 3 you provided, which is the subsequent exploration of DynamicViT. It must be acknowledged that this general idea still requires many details to be worked out. However, we sincerely appreciate your provision of Reference 4, which has given me an entirely new perspective. The work in Reference 4 has effectively demonstrated that token pruning from ViT can be applied to dense tasks such as detection and segmentation. I will delve further into this work and attempt to integrate it into my future research. Once again, sincere gratitude for your provision! ③ Our proposed model can be combined with hierarchical ViT such as Swin-T. For the detailed approach and analysis, please refer to the response of Weakness 1 from Reviewer V3HF. However, we find that combining with hierarchical ViT is unnecessary for classification. The merging in hierarchical ViT considerably diminishes the advantages of multi-granularity in feature representation."}, {"review_id": "jIWA9231QU", "reviewer": "Reviewer_L7yG", "summary": "This paper improves the inference efficiency of Vision Transformers by token pruning/merging. Specifically, the authors proposed a Multi-Granularity method, which splits an image into patches of different sizes and merges similar tokens. Extensive experiments and ablation studies on ImageNet have demonstrated the advantage of their method.", "questions": "1. I would like to know the throughput/model parameter comparison in Table 3.\n2. Unlike this paper which trains a model from scratch with 300 epochs, DynamicViT utilises pretrained weights to finetune their model. How the performance would be if the MG-ViT is also finetuned based on pretrained DeiTs.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Simple idea but works. The proposed method achieves better speed-accuracy trade-off than many existing methods.\n2. The experiments are comprehensive, especially for the ablation study.\n3. This paper is easy to follow. The overall presentation is of good quality.", "weaknesses": "1. Despite the great performance, the evaluation is only conducted on one scale, i.e., the small version of DeiT and LV-ViT. It is not clear if the proposed method can generalize to other scales, such as DeiT-Ti and DeiT-B.\n2. Lack throughput comparison in Table 3. FLOPs may not reflect the real speed on hardware. It is important to directly evaluate the inference speed when comparing with other related works.\n3. Due to token merging, the proposed method may not be general for downstream dense prediction tasks. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate your meticulous review and professional suggestions! We address the questions and clarify the issues accordingly as described below.\n\n**Weakness 1**: Your suggestion is correct. In order to demonstrate the generality of the method, we indeed need to evaluate it on backbones of other scales. Therefore, we have supplemented the experimental evaluations of the remaining sizes of DeiT and LV-ViT. A total of 4 backbones are covered, with experiments completed for DeiT-Ti, DeiT-B, and LV-ViT-M (please refer to Table 13 in the PDF). Due to limitations in computational resources and time, we have temporarily not conducted experiments for LV-ViT-L; we will add its experimental results in the revised version. Thanks!\n\n**Weakness 2 & Question 1**: We apologize for not fully presenting our experimental results. Based on your suggestion, we have added a comparison of throughput and parameters in Table 3. (Please refer to Table 16 in the PDF.) Thank you!\n\n**Weakness 3**: Your perspective is correct. The proposed framework is not suited for handling dense tasks such as detection and segmentation. Since we focus on compressing ViT to improve efficiency in this work, rather than proposing a new general ViT structure, similarly to classical papers[r1-r7] in ViT compression, which mainly evaluated classification tasks. Therefore many designs are inclined towards classification. For instance, our proposed PPSM, which further reduces the computational overhead of ViT, is based on classification task. In our subsequent research, we are actively exploring the extension of the multi-granularity strategy to dense downstream tasks. However, for such scenarios, PPSM may not be applicable. Fortunately, Reviewer cCy5 provided a valuable reference 4, which effectively demonstrates that token pruning from ViT can be applied to dense tasks such as detection and segmentation. This leads me to Continue infer whether merging could also be explored in dense tasks? Although we have some general ideas about extending multi-granularity strategy to dense downstream tasks, they are not fully matured, and there are numerous details that need to be addressed. I believe this will be a highly challenging yet meaningful work.\n\n**Question 2**: We greatly appreciate your attention to details, as you helped us discover a slip in the paper. Similar to DynamicViT, MG-ViT is also achieved through fine-tuning. Therefore, the results we presented are obtained after MG-ViT fine-tuning on pre-trained DeiT or LV-ViT. We will thoroughly find similar slips and further polish our presentation. A well-written paper will be presented in the revised version. Thank you once again!\n\n"}]

Author rebuttal:
We sincerely thank all reviewers and ACs for their time and efforts. We are encouraged that the reviewers approve our motivation or idea (qw7Z J3bo V3HF cCy5 L7yG), consider our method novel (J3bo V3HF), and recognize our presentation (qw7Z J3bo V3HF L7yG). Meanwhile, we deeply value reviewers’ precious suggestions and questions. Concerning common aspects, I want to address and clarify them here once again.

In this work, our focus is on compressing ViT rather than proposing a new, general ViT structure. Drawing inspiration from three phenomena observed during the classification process, we contemplate how to effectively integrate the multi-granularity strategy with common techniques to solve the problem. It is within the impetus of this motivation that the MG-ViT framework is conceived. Similar to classical papers in ViT compression, which mainly evaluated classification tasks, we have evaluated the performance of MG-ViT in classification and devised numerous ablation experiments to substantiate the rationale and efficacy of our compression approachSimple but work. 

Reference\
[r1] DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsification. NIPS2021.\
[r2] Patch Slimming for Efficient Vision Transformers. CVPR2022.\
[r3] Evo-ViT: Slow-Fast Token Evolution for Dynamic Vision Transformer. AAAI2022.\
[r4] Self-slimmed Vision Transformer. ECCV2022.\
[r5] AdaViT: Adaptive Vision Transformers for Efficient Image Recognition. CVPR2022.\
[r6] Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizations. ICLR2022.\
[r7] ATS: Adaptive Token Sampling For Efficient Vision Transformers. ECCV2022.\
[r8] Exploring Plain Vision Transformer Backbones for Object Detection. arXiv:2203.16527v2.\
[r9] MixMAE: Mixed and masked autoencoder for efficient pretraining of hierarchical vision transformers. CVPR2023.\
[r10] Convnext v2: Co-designing and scaling convnets with masked autoencoders. CVPR2023.\
[r11] Scale-Aware Modulation Meet Transformer. ICCV2023.\
[r12] Internimage: Exploring large-scale vision foundation models with deformable convolutions. CVPR2023.\
[r13] More convnets in the 2020s: Scaling up kernels beyond 51x51 using sparsity. ICLR2023.\
[r14] Global context vision transformers. ICML2023.\
[r15] What Limits the Performance of Local Self-attention? IJCV2023.\
[r16] DeepMAD: Mathematical Architecture Design for Deep Convolutional Neural Network. CVPR2023.\
[r17] Dilateformer: Multi-scale dilated transformer for visual recognition. IEEE TMM2023.\
[r18] PVG: Progressive Vision Graph for Vision Recognition. ACM MM2023 arXiv:2308.00574.\
[r19] BViT: Broad Attention-Based Vision Transformer. IEEE TNNLS2023.\
[r20] Conformer: Local features coupling global representations for recognition and detection. IEEE TPAMI2023.\
[r21] FLatten Transformer: Vision Transformer using Focused Linear Attention. ICCV2023 arXiv:2308.00442.\
[r22] Slide-Transformer: Hierarchical Vision Transformer with Local Self-Attention. CVPR2023.\
[r23] Joint Token Pruning and Squeezing Towards More Aggressive Compression of Vision Transformers. CVPR2023.\
[r24] Making Vision Transformers Efficient from A Token Sparsification View. CVPR2023.\
[r25] A Closer Look at Self-Supervised Lightweight Vision Transformers. ICML2023.


Meta-review: After discussion, this submission received 4 positive scores and 1 negative score. After reading the paper, the review comments and the rebuttal, the AC think the major concern is about the lack of experiments for downstream tasks, which is encouraged to added to the camera-ready version. The AC discussed with the SAC, and he agreed the decision.
Decision: Accept (poster)