Title: MG-ViT: A Multi-Granularity Method for Compact and Efficient Vision Transformers
Venue: NeurIPS 2023
Accepted: yes
Keywords: Efficient AI

Vision Transformer

Image Classification

Multi-Granularity

Three-Way Decisions.
Abstract: Vision Transformer (ViT) faces obstacles in wide application due to its huge computational cost. Almost all existing studies on compressing ViT adopt the manner of splitting an image with a single granularity, with very few exploration of splitting an image with multi-granularity. As we know, important information often randomly concentrate in few regions of an image, necessitating multi-granularity attention allocation to an image. Enlightened by this, we introduce the multi-granularity strategy to compress ViT, which is simple but effective. We propose a two-stage multi-granularity framework, MG-ViT, to balance ViT’s performance and computational cost. In single-granularity inference stage, an input image is split into a small number of patches for simple inference. If necessary, multi-granularity inference stage will be instigated, where the important patches are further subsplit into multi-finer-grained patches for subsequent inference. Moreover, prior studies on compression only for classification, while we extend the multi-granularity strategy to hierarchical ViT for downstream tasks such as detection and segmentation. Extensive experiments Prove the effectiveness of the multi-granularity strategy. For instance, on ImageNet, without any loss of performance, MG-ViT reduces 47\% FLOPs of LV-ViT-S and 56\% FLOPs of DeiT-S.

MG-ViT: A Multi-Granularity Method for Compact and Efficient Vision Transformers

Abstract

Vision Transformer (ViT) faces obstacles in wide application due to its huge computational cost. Almost all existing studies on compressing ViT adopt the manner of splitting an image with a single granularity, with very few exploration of splitting an image with multi-granularity. As we know, important information often randomly concentrate in few regions of an image, necessitating multi-granularity attention allocation to an image. Enlightened by this, we introduce the multigranularity strategy to compress ViT, which is simple but effective. We propose a two-stage multi-granularity framework, MG-ViT, to balance ViT's performance and computational cost. In single-granularity inference stage, an input image is split into a small number of patches for simple inference. If necessary, multigranularity inference stage will be instigated, where the important patches are further subsplit into multi-finer-grained patches for subsequent inference. Moreover, prior studies on compression only for classification, while we extend the multigranularity strategy to hierarchical ViT for downstream tasks such as detection and segmentation. Extensive experiments Prove the effectiveness of the multigranularity strategy. For instance, on ImageNet, without any loss of performance, MG-ViT reduces 47% FLOPs of LV-ViT-S and 56% FLOPs of DeiT-S.

1 Introduction

Transformer [1] has gained tremendous achievements in computer vision. Despite the excellent performance of Convolutional Neural Networks (CNN) in computer vision tasks [2; 3; 4; 5], due to ViT's superior ability to capture global information and long-range interactions, ViT [6] outperforms CNN in various tasks [7; 8; 9; 10; 11]. However, the impressive performance of ViT comes at the cost of its huge computational overhead. Therefore, more researches focus on ViT compression for greater efficiency. Since the computational cost of ViT increases quadratically with the number of tokens, minimizing the number of tokens is crucial in compressing ViT.

Notable investigations have adopted various methods to compress ViT. DynamicViT [12] reduces token number by pruning redundant tokens, EViT [13] merges redundant tokens into one token. DVT [14] utilizes the particular and dynamic patch splitting manner (e.g., 4 × 4, 7 × 7, etc.) for each image, based on its complexity, rather than following the official manner of 14 × 14. Although DVT innovatively considers the differences in complexity between images, it still uses single granularity to split one image. As we all know, images often contain a lot of redundancy, with important semantic information randomly concentrated in a few regions. Therefore, it is imperative to consider the complexity and semantic density diversities across various regions of the image: different regions of an image should be assigned attention with multiple granularities. We initiate our exploration by observing the foundational task, classification, and we have 3 intriguing observations:

Observation 1 : Most images can be correctly recognized by splitting them into a small number of patches, as has been analogously observed in research [14]. Training a DeiTS [15] with the 4 × 4 patch splitting manner can achieve 63.4% accuracy at a computational cost of 0.45G FLOPs. Although splitting an image to 7 × 7 patches and 14 × 14

patches can enhance the accuracy by 9.8% and 16.4%, the computation cost will also increase by 2.4 × and 10.2 × . How to substantially improve the accuracy with a low growth rate of FLOPs is a subject worth contemplating.

Observation 2 : The classification accuracy is mainly affected by the several critical tokens, and subsplitting patches into finer patches is more beneficial to classification. We split each image into 14 × 14 patches and rank tokens from high to low based on their class attention scores. The 196 tokens (patches) are equally divided into head, middle, and tail groups, and respectively input into DeiT-S to measure the accuracy of each group (Fig.1(a)). Then, subsplitting the patches in the head group expands the quantity to 4 × and 9 × , their accuracy is measured (Fig.1(b)). It is illustrated that head tokens ensure the base of classification accuracy, and finer tokens lead to higher accuracy.

80

(a)

(b)

(c)

80

80

75

75

70

70

70

Higher IIC tokens

baseline

Accuracy(%)

60

Accuracy(%)

65

Accuracy(%)

65

Higher IIC tokens + 1 token (100%)

Higher IIC tokens + 2 tokens (50%)

50

baseline

60

--baseline

60

Higher IIC tokens + 4 tokens (25%)

40

head tokens

middle tokens

55

50

head tokens

tokens 4×

55

30

Tail Tokens

45

tokens 9x

50

20

40

45

123456789101112

123456789101112

123456789101112

Layer

Layer

Layer

Observation 3 : All information matters, and proper handling of tokens with low important information content (IIC) contributes to avoiding partial loss of accuracy. We treat the bottom 50% tokens as lower IIC tokens based on class attention score rank (CASR), and create 4 input groups for all tokens. The results in Fig.1(c) suggest that retaining lower IIC tokens can maintain accuracy. This is because lower IIC tokens contain complementary information to higher IIC tokens, which helps bridge the accuracy gap with the baseline. The dissimilarities among lower IIC tokens should be considered, necessitating a hierarchical processing. Performance improvement is from complementary information, certain lower IIC tokens make the higher contribution. If they are merged with other lower IIC tokens, the gain of performance improvement may be diluted.

Thus far, we can identify two sound reasons for utilizing the multi-granularity strategy to compress ViT. Firstly, as aforementioned, due to the distinctiveness of important information distribution in images, it is imperative to assign multi-granular attention to an image. Secondly, guided by our observations, we can consider multi-granularity as an intermediate state between full fine-grained and full coarse-grained. Full fine-grained stage represents stronger performance, while full coarse-grained stage represents less computation. Therefore, multi-granularity can naturally be used to balance performance and computational cost.

We propose MG-ViT: a multi-granularity framework for compressing ViT, which automatically splits an image into multi-granularity patches, as shown in Figure 2. MG-ViT consists of two parts: Single-Granularity Inference Stage (SGIS) and Multi-Granularity Inference Stage (MGIS), related to Observation 1. In SGIS, a small number of tokens are used for classification inference. Should a prediction confident enough ensue, the inference terminates immediately; otherwise, MGIS will be

Simple

Single-Granularity Inference Stage

Multi-Granularity Inference Stage

93.5%

Judgement

acceptable

Classification

Prediction

unacceptable

%E'98

Complex

40.3%

PPSM

triggered. In MGIS, We put forward three innovative methods. The first is multi-granularity patch subsplitting, related to Observation 2. All patches are divided into head, middle and tail groups based on CASR in SGIS, head and middle patches are subsplit, resulting in tokens with multiple information granularity that are input into ViT. The second is the three-way decision slimming, which can streamline ViT softly. Simply put, based on CASR, tokens are divided into three groups: positive, boundary, and negative. Boundary and negative tokens are respectively merged by different rules to reduce their numbers. The third is the token match-merge scheme for boundary tokens, related to Observation 3. The second and third methods are packaged as Plug-and-Play Slimming Module (PPSM) against the huge increase in computational cost caused by patch subsplitting. After PPSM, all tokens are fed into the next layer of ViT to continue inference.

Due to that ViT only outputs a single-scale feature representation and lacks the capability to handle multi-scale variations, it possesses significant deficiencies in downstream tasks. In order to adapt multi-granularity strategy to various downstream tasks, we extend the MG-ViT framework to hierarchical ViT. Unlike ViT, hierarchical ViT aggregates tokens within rectangular areas. However, patch subsplitting results in patches (tokens) with multiple scales interleaved in the image, making it challenging to seamlessly enclose them within regular rectangular areas. To address this issue, we introduce a proxy token of the same scale for each head or middle patch to represent itself. Additionally, each proxy token engages in attention calculations with its corresponding fine tokens or medium tokens to represent the subsplitting of the head or middle patch it stands for. Following the aggregation methodology of PVT [16], Hierarchical MG-ViT can perform downstream tasks.

MG-ViT is a dynamic and versatile framework that can be applied to most ViT models. We select LV-ViT [17] and DeiT [15] to assess the performance of MG-ViT on ImageNet [18]. The experiments demonstrate that MG-ViT can balance performance and computation cost greatly, thereby significantly improving the efficiency of ViT. Moreover, while ensuring performance, MG-ViT reduces 47% FLOPs of LV-ViT and 56% FLOPs of DeiT. We also conduct simple experiments in object detection and semantic segmentation on the MS-COCO [19] and ADE20K [20] datasets, respectively. The results show that Hierarchical MG-ViT effectively reduces computational overhead without significant performance loss, demonstrating the feasibility of extending multi-granularity strategy to the Hierarchical ViT structure.

2 Related Work

Vision Transformer. ViT [6] achieves a great breakthrough in the image classification task by redesigning the Transformer structure, completely igniting researchers' passion for exploring the ViT further. Later on, ViT is modified to hierarchical structure for various downstream tasks [21; 8; 22; 23].

ViT and Hierarchical ViT. ViT is a series of ViT models that adhere to the vanilla ViT design principles,such as DeiT [15] and LV-ViT[17]. They only output a single-scale feature representation and lack the capability to handle multi-scale variations, making them typically suitable for classification tasks. Hierarchical ViT is a series of ViT models that incorporate a hierarchical structure to aggregate tokens layer by layer, enabling them to better handle multi-scale information and well-suited for various downstream tasks. Swin-Transformer [24] and PVT [16; 25] are representative architectures of hierarchical ViT.

ViT compression. ViT and hierarchical ViT can both enhance efficiency. Hierarchical ViT primarily achieves efficiency by improving interaction rules among tokens, whereas ViT mainly relies on compression. ViT compression primarily aims to optimize ViT structures in a flexible and automated manner, developing a dynamic and efficient ViT where each image undergoes an individual computational process based on its unique features, all while minimizing computational costs without sacrificing performance. There are many classic ViT compression works, such as DynamicViT [12], PS-ViT [26], Evo-ViT [27], and so on [13; 14; 28; 29; 30; 31; 32; 33].

3 Multi-Granularity Vision Transformer

3.1 Overview

Figure 3 illustrates the overall framework of MG-ViT. For each image, SGIS must be executed. In SGIS, the image is split into a small number of patches. These 2D patches are embedded into 1D

Judgement

acceptable

Classification

Result

Image

class

unacceptable

ViT

class

Head Patchs

PPSM

2-1

2-2

2-3

ClassAttetion soreRank (CASR

28

Positive Rules

2-5

2-6

26

Maintain

2-8

Add&Norm

Add&Norm

Splitting

Patch

ture

D

Middle Patchs

Boundary Rules

FFN

Reuse

SimilarsMergel

8-1

SubSplitting

Patch

一

CoarseTokens

82

32

Negative Rules

B-

8-2

Merge into 1

OurProposed Method

SGIS

MGIS

Concat

Multi-granularity subsplitting

Three-way decision slimming

Tokenmatch-merge scheme

tokens and fed into ViT for straightforward inference. The obtained class prediction will be evaluated against predetermined criteria (threshold). If the class prediction is acceptable, the inference can be terminated immediately, and the class prediction can be output as the final classification result. If the class prediction is unacceptable, MGIS will be executed.

In MGIS, according to the IIC of each patch (referring to CASR), all patches are proportionally divided into three groups: head, middle and tail. Head and middle patches are further subsplit. As head patches with high IIC are the most crucial for classification, they require subsplitting with fine granularity, and eventually are subsplit into fine patches with fine-grained information. Middle patches with middle IIC are more crucial for classification but less than head patches. Whence, they require subsplitting with medium granularity, being subsplit into medium patches with mediumgrained information. Tail patches, maintain their coarse granularity and serve as coarse patches. All patches undergo patch embedding and feature reuse to generate multi-granularity tokens which are input into ViT for inference, prediction, and output. Considering that patch subsplitting leads to a huge increase in computational cost, Plug-and-Play Slimming Module (PPSM) is introduced in certain layers of ViT to reduce computational cost. The three-way decision slimming lies here. According to CASR, we divide all tokens into three groups: positive, boundary and negative, aiming to realize soft slimming of ViT. Positive tokens remain unchanged. Among boundary tokens, similar tokens are matched and merged based on our proposed token match-marge scheme (more details in Section 3.3.2), resulting in fewer new tokens. Negative tokens are merged into one token by weights. In this way, the number of tokens is reduced, accomplishing ViT soft slimming.

3.2 Single-Granularity Inference Stage (SGIS)

In SGIS, the standard ViT is employed. Formally, for an input image ℵ , it is first split into patches P = [ p 1 , p 2 , ..., p N ] where N is the number of patches. In patch embedding, patches are mapped into C -dimension token embeddings via linear projection. Additionally, a learnable token embedding x 0 , referred to as a class token, is appended to the sequence of token embeddings to represent image ℵ for class prediction. After position embedding E pos , the input token sequence for ViT in SGIS is:

where x i ∈ R C , present a token embedding of the i -th patch if i > 0 , and E pos ∈ R ( N +1) × C .

A ViT consists of L layers, each layer comprises a self-attention (SA) module and a feed-forward network (FFN). In SA of the l -th layer, the self-attention Attention ( Q l , K l , V l ) is computed as follows:

where Q l , K l , V l ∈ R ( N +1) × C , are query, key and value matrices of l -th layer respectively and obtained through linear projection for X l -1 . A l is attention map of l -th layer, and the first row of attention map a 0 l represent class attention in l -th layer.

Ulteriorly, the processes of SA and FFN in the l -th layer are as follows:

After consecutive computation of L layers, class token x 0 L from the L -th layer is input into the classifier C to obtain class prediction distribution D S :

where S indicates SGIS, M denotes the class number, up to this point, the class prediction of image ℵ is the largest entry of D S , i.e., argmax j d S j .

Threshold for judgment. If the value of max j d S j for input image ℵ is large enough, it indicates that the class prediction in SGIS is acceptable, and the classification result can be generated. Therefore, the inference can be terminated immediately, saving computational cost. In order to use max j d S j to determine whether to terminate the inference, we introduce a threshold ϵ : if max j d S j > ϵ , the inference is terminated immediately, and the class prediction is output as the classification result. If max j d S j < ϵ , The image is transitioned to MGIS for consecutive inference.

3.3 Multi-Granularity Inference Stage (MGIS)

3.3.1 Generation of Multi-Granularity Tokens

To bridge the ViTs of two stages, we perform multi-granularity patch subsplitting and feature reuse to generate tokens with multi-grained information as the input for the ViT in MGIS.

Multi-granularity patch subsplitting. Since max j d S j of image ℵ is less than ϵ , further inference is required. As mentioned above, in MGIS, image ℵ needs to be subsplit with multiple different granularities, and features of patches in SGIS need to be reused. These operations require to be instructed by IIC of patches (tokens) in SGIS, as patches with different IIC will be divided into different groups. Thus, the key now lies in how to identify the IIC of patches.

According to Equation 2, class token y 0 can be represented by class attention a 0 ∈ A as:

as we all known, class token y 0 represents the entire image ℵ as input to produce the class prediction in classifier C . As V = [ v 0 , v 1 , ..., v N ] , where v i is the value of the i -th token if i > 0 . The class token y 0 is obtained by integrating the value of each token using a 0 as the corresponding weights. In other words, a 0 determines the amount of information from each token that enters the class token y 0 for classification. Therefore, we can use class attention a 0 to measure the contribution of each token to the class prediction, thence, we identify IIC according to CASR. However, if we only use CASR originating from the class attention of a certain layer to divide patches, patches of each group will be more random and unfixed, lacking overall consistency. Therefore, the flow of information across layers must also be considered. Thus, we come up with the global class attention instead of only using class attention of a certain layer to identify IIC of patches. The global class attention ˆ a l in the l -th block is as following:

where λ = 0 . 98 . In Fig.1(a), we can observe that the classification accuracy of all groups exhibits intense fluctuation in shallow layers. It is evident that class attention is unstable in shallow layers. Therefore, the global class attention calculation starts from the 3-rd layer. We use CASR rooted from global class attention in the last layer ˆ a L to identify IIC of patches.

According to ˆ a L , all patches are divided into three groups: head, middle, and tail. The numbers of head, tail and middle patches, denoted by N h , N t , and N m are given by:

where r h and r t represent the number rate of head and tail patches respectively.

Head and middle patches need to be subsplit. Since head patches have high IIC, each is subsplit into 3 × 3 fine patches. Middle patches with middle IIC, each is subsplit into 2 × 2 medium patches. Each tail patch is considered as a coarse patch and remains unchanged. After patch subsplitting, the input token sequence is denoted as follows:

where K represents the total number of patches after patch subsplitting.

Feature reuse. Patch subsplitting disrupts the integrity of head and middle patches, resulting in a lack of correlation among fine or medium patches which are severally subsplit from the same head or middle patch. To address this issue, we incorporate Feature Reuse module from DVT [14] to inject the original information of head or middle patches into new fine or medium patches. It is advantageous to enhance feature representation of each patch and strengthen the interconnectivity among them.

Figure 4 demonstrates the Feature Reuse module. Tokens (except class token) output by the ViT of SGIS are taken as input and divided based on their IIC. As head and middle patches need to be subsplit in MGIS, we individually upsample head and middle tokens, so that the number of new tokens obtained by upsampling is the same as the number of new patches obtained by subsplitting. Different from the module in DVT [14], considering that tail tokens remain unchanged in MGIS, we utilize zero padding to handle them. Finally, all tokens are flattened and sorted according to their position in image ℵ . The token sequence X FR output by Feature Reuse is obtained as follows:

Head

US

2-1

2-2

Middle

US

F&S

8

6

Tail

8-1

ZP

8-2

8

US:UnsamplingZP:ZeroPaddingF&S:Flatten &Sort

(9)

Specially, X FR ∈ R N × C ′ . Due to concatenation expanding the dimension of multi-granularity tokens to 2 × , resulting in a 4 × increase in computational cost (detailed analysis in Appendix), we set a MLP in the module to realize dimension reduction: MLP ( R C ⇒ R C ′ ) . The value of C ′ is small, thereby increasing the dimension of multi-granularity tokens from C to C + C ′ , but still much smaller than 2 C , which helps save computational cost.

The generated multi-granularity token sequence ˜ X ∈ R ( N +1) × ( C ′ + C ) , which is input to ViT in MGIS for inference, slimming and output, is denoted as:

3.3.2 Plug-and-Play Slimming Module (PPSM)

Most existing ViT slimming methods, such as DynamicViT, PS-ViT, adopt token pruning, undermining the information integrity of the image and is not beneficial for classification. Therefore, we select token merging, which diminishes the number of tokens while preserving information.

Three-way decision slimming. Most existing ViT slimming methods belong to hard slimming, which divides tokens into non-redundant and redundant groups based on a certain criterion and reduces the number of redundant tokens to streamline ViT. Nevertheless, tokens in the middle part of CASR are arduous to distinguish as non-redundant or redundant during the inference. Mistakenly identifying non-redundant tokens as redundant and performing slimming will lead to a degradation of ViT's performance. Considering that three-way decision model establishes a buffer zone, i.e., boundary, between positive and negative, it can aid in more appropriately handling tokens situated in the middle position of CASR. Therefore, we employ three-way decision mechanism to realize soft slimming for ViT.

Three-way decision [34] is a decision mechanism in rough set [35; 36] that partitions a set into positive, negative, and boundary domains, with elements in each domain subject to positive, negative, and boundary rules, respectively. By setting the proportion of elements in positive and negative domains to r pos and r neg , all tokens inputted into PPSM are divided into positive, negative and boundary domains. Positive tokens adhere to the positive rule: maintain unchanged. Boundary tokens adhere to boundary rule, which refers to the token match-merge scheme we proposed. Negative tokens adhere to the negative rule: all are merged into one token by weights.

Step1

A

CASR

B

Step3

Step4

Token match-merge scheme. As discussed in Observation 3, dissimilarities among lower IIC tokens should be considered, necessitating a hierarchical processing. Thus, we employ the token

match-merge scheme among boundary tokens to guide the match-merge process of similar boundary tokens, aiming to reduce the number of tokens while maintaining accuracy to the best possible extent. We measure the similarity between two tokens by computing the cosine similarity of their value vectors. v a , v b ∈ R C , present the value vectors of tokens a and b , and the cosine similarity cos ( θ ) is expressed as follows:

The token match-merge scheme is demonstrated in Figure 5. Step1. Divide tokens at odd positions in CASR into Group A and those at even positions into Group B. Step2. Find the most similar token in set B for each token in set A by calculating cosine similarity. Step3. Put similar tokens together to complete the match. Step4. Merge the similar tokens by weights to reduce the number of tokens.

3.4 Training Mehtod

During the training of MG-ViT, setting ϵ = 1 makes MGIS be executed for every input image. Like knowledge distillation [37], our goal is to make the outputs of SGIS more similar to MGIS. Thus ViT in SGIS can have a stronger ability in classification. The training loss about ground-truth label gt is:

where CE ( · , · ) and KL ( · , · ) represent cross entropy loss and Kullback-Leibler divergence.

4 Hierachical Multi-Granularity Vision Transformer

ViT（PVTStage1）

FR

PVTStage

PVTStage

PVT Stage 4

④

2-2

2-3

2-5

SA+MLP

2

-7

2-8

2-9

3-1

3-2

SA+MLP

2

W

OurProposedMethod

Proxytoken

Introducingproxytokenstounifytokenscales

FRFeatureReuse

4.1 Key Issue of Extension

Almost all classic ViT compression works, such as Dynamic ViT [12], PS-ViT [26], EViT [13], only focus on classification. Why not extend their methods to hierachical ViT to execute downstream tasks like detection and segmentation? We analyze two reasons. The first reason is structural limitation. For instance, plain ViT only provides a single-scale feature representation, lacks the ability to handle multi-scale variation, and lacks image-related priors. While there are a few works that still employ plain ViT for downstream tasks, such as ViTDet [38], ViT-Adapter [39], SegViT [40; 41], they achieve this by integrating well-designed modules, which could be considered as individual research contributions. The second reason is method challenges. Most classic ViT compressions, as mentioned above, heavily rely on pruning or merging, which can be challenging to adapt to downstream tasks. Although we also employ pruning and merging methods, they serve as auxiliary means to reduce computational costs. The core of this paper - multi-granularity strategy - can be applied in hierarchical ViT for downstream tasks.

The key to extending multi-granularity strategy to hierarchical ViT lies in addressing the non-uniform issue of multi-grained patch scales. Subsplitting head and middle patches into 3 × 3 fine patches and 2 × 2 medium patches results in patches with multiple scales interweaving in an image, making it difficult to seamlessly enclose patches in a conventional window when using Swin Transformer. Therefore, in this paper, we propose proxy tokens to unify scales.

4.2 Introducing Proxy Tokens for Unifying Scales

Similar to MG-ViT, Hierarchical MG-ViT also employs a two-stage framework. In SGIS, 2D image ℵ ∈ R H × W is split into patches P ∈ R H s × W s × s 2 , where ( H,W ) donates the resolution of image ℵ ,

and s presents the patch size. The token sequence generated by P embedding is:

where T ori ∈ R HW s 2 × C 1 , ori is a simplified representation of "origin". IIC of P is identified in SGIS.

In MGIS, we assume that P i ∈ R s × s is a patch that requires subsplitting, which will be subsplit into ℓ × ℓ fine patches:

where ℓ × ℓ is the manner of subsplitting ( 3 × 3 or 2 × 2 ), p i,j present the j -th fine or medium patch subsplit from the i -th head or middle patch if i > 0 , and j > 0 .

Then the finer token sequence T i ori ∈ R ℓ 2 × C 1 originated from the embedding of these finer patches is denoted as:

We introduce a proxy token t i pro ∈ R C 1 for this group of finer tokens to compose the new sequence T i = [ T i ori , t i pro ] . The computing procedure of a Transformer layer for this group can be summarized as follows:

To generalization, We provide an optional proxy token set T pro and its mask vector m for each image, where T pro ∈ R HW s 2 × C 1 and m ∈ R HW s 2 . Thus, the unified scale token sequence T ′ before feature reuse can be denoted as:

After feature reuse, the token sequence is input into the pyramid-structured hierarchical ViT, through PVT stages 2, 3, and 4, feature maps reshape to H 2 s × W 2 s × C 2 , H 4 s × W 4 s × C 3 and H 8 s × W 8 s × C 4 . In this way, various downstream tasks can be performed.

4.3 Architectural Details

We develop two variants of Hierarchical MG-ViT based on PVT-Small: Hierarchical MG-ViT-A and Hierarchical MG-ViT-R. Hierarchical MG-ViT-A adds a Transformer layer with proxy tokens between stage 1 and stage 2 of PVT-Small. Hierarchical MG-ViT-R maintains the same capacity as PVT-Small but replaces one of the original Transformer layers in stage 1 with a Transformer layer introducing proxy tokens. Please refer to the appendix for architectural details.

4.4 Discussions

Performance upgrade. Since our proposed method is plug-and-play and does not significantly alter the structure of the hierarchical ViT, methods aimed at improving the performance of hierarchical ViT, such as using relative position biases and overlapping patch embedding, are all effective. However, we haven't optimize Hierarchical MG-ViT for the best performance, as our goal was to explore the potential of extending the multi-granularity strategy from compressing plain ViT to hierarchical ViT for various downstream tasks. In the future, we will delve deeper into performance enhancements for Hierarchical MG-ViT.

Plain ViT vs. hierarchical ViT. Which ViT structure is superior? Plain ViT inherently has limitations when it comes to handling dense tasks. In order to utilize a general ViT for a variety of tasks, many new ViTs are designed based on hierarchical structures. However, we do not consider hierarchical ViT to be superior; both plain ViT and hierarchical ViT have their strengths and weaknesses. As for detection tasks, a multitude of ViTs built upon the hierarchical structure continue to remain the most competitively poised model. However, models like ViTDet [38], which are based on plain ViT, also demonstrate that hierarchical ViTs such as Swin-T and PVT are not the only efficient ways to accomplish visual tasks. As for segmentation tasks, what surprised us is that SegViTv2 [41], built upon the Plain ViT, has achieved state-of-the-art results. As for classification tasks, a variety of plain ViT models, with MG-ViT as a representative, exhibit a big performance advantage over hierarchical ViT. Moreover, multi-granularity strategy boost performance more effectively in plain structures than in hierarchical structures. This is because hierarchical ViT involves token merging, which

significantly diminishes the advantages of multi-granularity in feature representation. Regarding the question of which ViT structure is superior, we believe that it necessitates deliberation in the context of varying tasks and distinct constraints.

5 Experiments

5.1 Backbones, Datasets and Evaluation Metrics.

We built MG-ViT based on DeiT-S and LV-ViT-S and evaluated its performance on the benchmark ImageNet [18]. To quantitatively compare performance, we report the Top-1 accuracy (Acc.), the number of floating-point operations (FLOPs), and throughput (TP). Moreover, we developed Hierarchical MG-ViT based on PVT-Small and evaluated its performance in object detection and semantic segmentation on the benchmark datasets MS-COCO2017 [19] and ADE20K [20]. We reported metrics such as box average precision ( AP b ), mean intersection over union (mIoU) and the number of parameters (#Params). All metrics are measured on a single NVIDIA RTX 3090 GPU.

5.2 Main Results

Comparison to backbones. Table 2 presents the comparison results between MG-ViT and backbones with different values of threshold ϵ . It can be observed that Without any performance degradation, the computational cost of MG-ViT is significantly reduced. Moreover, when all images execute MGIS, the performance of ViT is improved, which can be attributed to subsplitting patches with multiple different granularities.

Comparison to baselines. In Figure 7, we compare MG-ViT with 3 baselines: DynamicViT, EViT, and DVT. These models are chosen as representatives of 3 type methods: token pruning, token merging, and dynamic design of token format. With the same setting, MG-ViT outperforms baselines.

Comparison to other ViT compressing methods. In order to demonstrate the effectiveness of our dynamic framework, MG-ViT, in compressing ViT, we present a comparison of MG-ViT with various ViT compressing methods in Table 3. We visualize their accuracy, FLOPs, and throughput in Appendix. The results show that our method for compressing ViT based on the multi-granularity strategy, MG-ViT, is highly competitive when using DeiT-S and LV-ViT-S as backbones.

Comparison to other efficient ViTs. In Figure 8, we compare MG-ViT and various efficient ViTs [44; 45; 46; 47; 48; 49; 50; 51; 52; 53; 25; 54; 55; 56; 57; 58; 59; 60; 61; 62; 63; 64]. It can be observed that MG-ViT is competitive even among different backbones and other ViT variants.

84

84

Slide-CSwin-S

82

SMT-S

MaxViT-T

MG-ViT (LV-ViT-S)

CMT-S

GC ViT-T2

LV-ViT-S CSwin-S

Accuracy(%)

80

ConvFormer-S18 ●NAT-T

GC ViT-T

Dilate-S

MViTv2-S

78

83

● UniFormer-S CSwin-T

PVTv2-B3

gSwin-S

MG-VIT

Slide-PVTv2-B2

DaViT-T

76

74

72

84

1

2

FLOPs(G)

3

DynamicVIT

EViT

DeIT-S

DVT

4

5

Accuracy(%)

82

81

MG-ViT (DeiT-S)

● SMT-T

GC ViT-XT

Container-Light

Dilate-T

gSwin-To

LITv2-S

FLatten-PVT-S

LIT-T

CoaT-Lite-S

PVTv2-B2

LIT-S

Wave-ViT-S

CeiT-S

Slide-Swin-T

CvT-13

Swin-T

○DAT-T

MViTv2-T

T2T-ViT-14

PoolFormer-S36

Visformer-S

ViL-S

AutoFormer-S

TNT-S

●ConViT-S

Conformer-Ti

CrossViT-15

Twins-PCPVT-B

PVT-M

83.7

LocalViT-S

83.4

PoolFormer-S24

GLiT-S

T2T-ViTt-14

CoaT-Lite-M

gSwin-VT

83.1

82.8

MG-ViT

DynamicViT

EVIT

DVT

80

Slide-PVTv2-B1

○PVT-S

DeiT-S

LV-ViT-S

82.5

79

●LV-ViT-T

3

4

FLOPs(G)

5

6

7

2

2.5

3

3.5

4

4.5

5

5.5

6

6.5

/

FLOPs(G)

Object detection and semantic segmentation. We adopt the same implementation details as PVT-Small and conduct preliminary experiments on Hierarchical MG-ViT's performance in object detection and semantic segmentation tasks to represent its capability for executing downstream tasks. As shown in Table 4 and Table 5, extending the multi-granularity strategy to the hierarchical structure is beneficial for enhancing ViT's performance and serves as a nice attempt to slim hierarchical ViT.

6 Conclusion

MG-ViT, the two-stage dynamic framework based on the strategy we proposed, achieves a wellbalanced trade-off between performance and computational cost. Within this novel framework, we contribute four new methods. Firstly, we subsplit patches with multiple different granularities, resulting in generating multi-granularity tokens with various levels of information, which enhances the capacity of ViT in feature representation, leading to performance improvement. Secondly, we introduce a three-way decision mechanism to achieve soft slimming of ViT. Thirdly, we propose the token match-merge scheme to guide the match-merge of boundary tokens, reducing their quantity. The second and third methods are reflected in our proposed PPSM. Fourthly, we extend multi-granularity strategy to hierarchical ViT for various downstream tasks, demonstrating that our method can also be applied to hierarchical structures for compressing, and we will explore this further in the future.

References

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems , pages 5998-6008, 2017.

Yadi Wang, Hongyun Zhang, Duoqian Miao, and Witold Pedrycz. Multi-granularity re-ranking for visible-infrared person re-identification. CAAI Transactions on Intelligence Technology , 2023.

Changwei Wang, Rongtao Xu, Ke Lv, Shibiao Xu, Weiliang Meng, Yuyang Zhang, Bin Fan, and Xiaopeng Zhang. Attention weighted local descriptors. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2023.

Changwei Wang, Rongtao Xu, Shibiao Xu, Weiliang Meng, Jun Xiao, and Xiaopeng Zhang. Accurate lung nodule segmentation with detailed representation transfer and soft mask supervision. IEEE Transactions on Neural Networks and Learning Systems , 2023.

Changwei Wang, Rongtao Xu, Shibiao Xu, Weiliang Meng, Ruisheng Wang, Jiguang Zhang, and Xiaopeng Zhang. Towards accurate and efficient road extraction by leveraging the characteristics of road shapes. IEEE Transactions on Geoscience and Remote Sensing , 2023.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations , 2021.

Zhengzhong Tu, Hossein Talebi, Han Zhang, Feng Yang, Peyman Milanfar, Alan C. Bovik, and Yinxiao Li. Maxvit: Multi-axis vision transformer. In European Conference on Computer Vision , pages 459-479, 2022.

Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yabiao Wang, Yanwei Fu, Jianfeng Feng, Tao Xiang, Philip H.S. Torr, and Li Zhang. Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers. Jun 2021.

Rongtao Xu, Changwei Wang, Jiguang Zhang, Shibiao Xu, Weiliang Meng, and Xiaopeng Zhang. Rssformer: Foreground saliency enhancement for remote sensing land-cover segmentation. IEEE Transactions on Image Processing , 32:1052-1064, 2023.

Dong Chen, Duoqian Miao, and Xue Rong Zhao. Hyneter: Hybrid network transformer for object detection. CoRR , abs/2302.09365, 2023.

Changwei Wang, Rongtao Xu, Shibiao Xu, Weiliang Meng, and Xiaopeng Zhang. Da-net: Dual branch transformer and adaptive strip upsampling for retinal vessels segmentation. In International Conference on Medical Image Computing and Computer-Assisted Intervention , pages 528-538. Springer, 2022.

Yongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu, Jie Zhou, and Cho-Jui Hsieh. Dynamicvit: Efficient vision transformers with dynamic token sparsification. Advances in neural information processing systems , 34:13937-13949, 2021.

Youwei Liang, Chongjian Ge, Zhan Tong, Yibing Song, Jue Wang, and Pengtao Xie. Not all patches are what you need: Expediting vision transformers via token reorganizations. 2022.

Yulin Wang, Rui Huang, Shiji Song, Zeyi Huang, and Gao Huang. Not all images are worth 16x16 words: Dynamic vision transformers with adaptive sequence length. 2021.

Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers distillation through attention, 2021.

Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In Proceedings of the IEEE/CVF international conference on computer vision , pages 568-578, 2021.

Zi-Hang Jiang, Qibin Hou, Li Yuan, Daquan Zhou, Yujun Shi, Xiaojie Jin, Anran Wang, and Jiashi Feng. All tokens matter: Token labeling for training better vision transformers. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems , volume 34, pages 18590-18602. Curran Associates, Inc., 2021.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A largescale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition , Mar 2010.

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13 , pages 740-755. Springer, 2014.

Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 633-641, 2017.

Yuxin Fang, Bencheng Liao, Xinggang Wang, Jiemin Fang, Jiyang Qi, Rui Wu, Jianwei Niu, and Wenyu Liu. You only look at one sequence: Rethinking transformer in vision through object detection. Advances in Neural Information Processing Systems , 34:26183-26197, 2021.

Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Segformer: Simple and efficient design for semantic segmentation with transformers. Advances in Neural Information Processing Systems , 34:12077-12090, 2021.

Ali Hatamizadeh, Greg Heinrich, Hongxu Yin, Andrew Tao, Jose M Alvarez, Jan Kautz, and Pavlo Molchanov. Fastervit: Fast vision transformers with hierarchical attention. arXiv preprint arXiv:2306.06189 , 2023.

Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. arXiv preprint arXiv:2103.14030 , 2021.

Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pvtv2: Improved baselines with pyramid vision transformer. CoRR , abs/2106.13797, 2021.

Yehui Tang, Kai Han, Yunhe Wang, Chang Xu, Jianyuan Guo, Chao Xu, and Dacheng Tao. Patch slimming for efficient vision transformers. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 12155-12164, 2022.

Yifan Xu, Zhijie Zhang, Mengdan Zhang, Kekai Sheng, Ke Li, Weiming Dong, Liqing Zhang, Changsheng Xu, and Xing Sun. Evo-vit: Slow-fast token evolution for dynamic vision transformer. In AAAI Conference on Artificial Intelligence , pages 2964-2972, 2022.

Mengzhao Chen, Mingbao Lin, Ke Li, Yunhang Shen, Yongjian Wu, Fei Chao, and Rongrong Ji. Cf-vit: A general coarse-to-fine method for vision transformer, 2022.

Bowen Pan, Rameswar Panda, Yifan Jiang, Zhangyang Wang, Rogério Feris, and Aude Oliva. Ia-red$ˆ2$: Interpretability-aware redundancy reduction for vision transformers. In Advances in Neural Information Processing Systems , pages 24898-24911, 2021.

Zhuofan Zong, Kunchang Li, Guanglu Song, Yali Wang, Yu Qiao, Biao Leng, and Yu Liu. Self-slimmed vision transformer, 2022.

Lingchen Meng, Hengduo Li, Bor-Chun Chen, Shiyi Lan, Zuxuan Wu, Yu-Gang Jiang, and SerNam Lim. Adavit: Adaptive vision transformers for efficient image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 12309-12318, 2022.

Zhenglun Kong, Peiyan Dong, Xiaolong Ma, Xin Meng, Wei Niu, Mengshu Sun, Xuan Shen, Geng Yuan, Bin Ren, Hao Tang, Minghai Qin, and Yanzhi Wang. Spvit: Enabling faster vision transformers via latency-aware soft token pruning. In European Conference on Computer Vision , pages 620-640, 2022.

Shuning Chang, Pichao Wang, Ming Lin, Fan Wang, David Junhao Zhang, Rong Jin, and Mike Zheng Shou. Making vision transformers efficient from A token sparsification view. 2023.

Yiyu Yao. Three-Way Decision: An Interpretation of Rules in Rough Set Theory , page 642-649. Jun 2009.

ZDZISLAW PAWLAK. Rough set theory and its applications to data analysis. Cybernetics and Systems , 29(7):661-688, Jul 2002.

Zdzisław Pawlak. Rough set theory and its applications, Jan 2002.

GeoffreyE. Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in a neural network, Mar 2015.

Yanghao Li, Hanzi Mao, Ross Girshick, and Kaiming He. Exploring plain vision transformer backbones for object detection. In European Conference on Computer Vision , pages 280-296. Springer, 2022.

Zhe Chen, Yuchen Duan, Wenhai Wang, Junjun He, Tong Lu, Jifeng Dai, and Yu Qiao. Vision transformer adapter for dense predictions. arXiv preprint arXiv:2205.08534 , 2022.

Bowen Zhang, Zhi Tian, Quan Tang, Xiangxiang Chu, Xiaolin Wei, Chunhua Shen, et al. Segvit: Semantic segmentation with plain vision transformers. Advances in Neural Information Processing Systems , 35:4971-4982, 2022.

Bowen Zhang, Liyang Liu, Minh Hieu Phan, Zhi Tian, Chunhua Shen, and Yifan Liu. Segvitv2: Exploring efficient and continual semantic segmentation with plain vision transformers. arXiv preprint arXiv:2306.06289 , 2023.

Siyuan Wei, Tianzhu Ye, Shen Zhang, Yao Tang, and Jiajun Liang. Joint token pruning and squeezing towards more aggressive compression of vision transformers, 2023.

Mohsen Fayyaz, Soroush Abbasi Koohpayegani, Farnoush Rezaei Jafari, Sunando Sengupta, Hamid Reza Vaezi Joze, Eric Sommerlade, Hamed Pirsiavash, and Juergen Gall. Adaptive token sampling for efficient vision transformers, 2022.

Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition transformers with Auto-Correlation for long-term series forecasting. In Advances in Neural Information Processing Systems , 2021.

Peng Gao, Jiasen Lu, Hongsheng Li, Roozbeh Mottaghi, and Aniruddha Kembhavi. Container: Context aggregation network. arXiv preprint arXiv:2106.01401 , 2021.

Pengfei Gu, Yejia Zhang, Chaoli Wang, and Danny Z. Chen. Convformer: Combining cnn and transformer for medical image segmentation, 2022.

Stéphane d'Ascoli, Hugo Touvron, Matthew L. Leavitt, Ari S. Morcos, Giulio Biroli, and Levent Sagun. Convit: Improving vision transformers with soft convolutional inductive biases. CoRR , abs/2103.10697, 2021.

Mingyu Ding, Bin Xiao, Noel Codella, Ping Luo, Jingdong Wang, and Lu Yuan. Davit: Dual attention vision transformers. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXIV , pages 74-92. Springer, 2022.

Boyu Chen, Peixia Li, Chuming Li, Baopu Li, Lei Bai, Chen Lin, Ming Sun, Junjie Yan, and Wanli Ouyang. Glit: Neural architecture search for global and local image transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 12-21, 2021.

Zizheng Pan, Jianfei Cai, and Bohan Zhuang. Fast vision transformers with hilo attention, 2023.

Zizheng Pan, Bohan Zhuang, Haoyu He, Jing Liu, and Jianfei Cai. Less is more: Pay less attention in vision transformers. In AAAI , 2022.

Yanghao Li, Chao-Yuan Wu, Haoqi Fan, Karttikeya Mangalam, Bo Xiong, Jitendra Malik, and Christoph Feichtenhofer. Mvitv2: Improved multiscale vision transformers for classification and detection, 2022.

Weihao Yu, Mi Luo, Pan Zhou, Chenyang Si, Yichen Zhou, Xinchao Wang, Jiashi Feng, and Shuicheng Yan. Metaformer is actually what you need for vision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10819-10829, 2022.

Zhengsu Chen, Lingxi Xie, Jianwei Niu, Xuefeng Liu, Longhui Wei, and Qi Tian. Visformer: The vision-friendly transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 589-598, 2021.

Pengchuan Zhang, Xiyang Dai, Jianwei Yang, Bin Xiao, Lu Yuan, Lei Zhang, and Jianfeng Gao. Multi-scale vision longformer: A new vision transformer for high-resolution image encoding. ICCV 2021 , 2021.

Kunchang Li, Yali Wang, Junhao Zhang, Peng Gao, Guanglu Song, Yu Liu, Hongsheng Li, and Yu Qiao. Uniformer: Unifying convolution and self-attention for visual recognition, 2022.

Ali Hassani, Steven Walton, Jiachen Li, Shen Li, and Humphrey Shi. Neighborhood attention transformer. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2023.

Xiangxiang Chu, Zhi Tian, Yuqing Wang, Bo Zhang, Haibing Ren, Xiaolin Wei, Huaxia Xia, and Chunhua Shen. Twins: Revisiting the design of spatial attention in vision transformers, 2021.

Weifeng Lin, Ziheng Wu, Jiayu Chen, Jun Huang, and Lianwen Jin. Scale-aware modulation meet transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 6015-6026, 2023.

Ali Hatamizadeh, Hongxu Yin, Greg Heinrich, Jan Kautz, and Pavlo Molchanov. Global context vision transformers. In International Conference on Machine Learning , pages 12633-12646. PMLR, 2023.

Jiayu Jiao, Yu-Ming Tang, Kun-Yu Lin, Yipeng Gao, Jinhua Ma, Yaowei Wang, and Wei-Shi Zheng. Dilateformer: Multi-scale dilated transformer for visual recognition. IEEE Transactions on Multimedia , 2023.

Dongchen Han, Xuran Pan, Yizeng Han, Shiji Song, and Gao Huang. Flatten transformer: Vision transformer using focused linear attention. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 5961-5971, 2023.

Zhiliang Peng, Zonghao Guo, Wei Huang, Yaowei Wang, Lingxi Xie, Jianbin Jiao, Qi Tian, and Qixiang Ye. Conformer: Local features coupling global representations for recognition and detection. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2023.

Xuran Pan, Tianzhu Ye, Zhuofan Xia, Shiji Song, and Gao Huang. Slide-transformer: Hierarchical vision transformer with local self-attention. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2082-2091, 2023.

7 Appendix

7.1 Hierarchical MG-ViT Architectural Details

We present two variants of the Hierarchical MG-ViT designed based on PVT-Small, as shown in Table 6. The relevant hyper parameters are listed as follows: P i : the patch size of Stage i ; C i : the channel number of the output of Stage i ; R i : the reduction ratio of the spatial-reduction attention in Stage i ; N i : the head number of the spatial-reduction attention in Stage i ; E i : the expansion ratio of the feed-forward layer in Stage i ; H i : the head number of the multi-head attention in Stage i .

Output Size

Layer Name

PVT-Small

Hierarchical

MG-ViT-A

Hierarchical

MG-ViT-R

Stage 1

H

4

×

H

4

Patch

Embedding

P

1

= 4;

C

1

= 64

Transformer

Encoder





R

1

= 8

N

1

= 1

E

1

= 8





×

3





R

1

= 8

N

1

= 1

E

1

= 8





×

3





R

1

= 8

N

1

= 1

E

1

= 8





×

2

Stage MG

H

4

×

H

4

MG

Transformer

-

[

H

MG

= 8

E

MG

= 8

]

×

1

[

H

MG

= 8

E

MG

= 8

]

×

1

Stage 2

H

8

×

H

8

Patch

Embedding

P

2

= 2;

C

2

= 128

Transformer

Encoder





R

2

= 4

N

2

= 2

E

2

= 8





×

3





R

2

= 4

N

2

= 2

E

2

= 8





×

3





R

2

= 4

N

2

= 2

E

2

= 8





×

3

Stage 3

H

16

×

H

16

Patch

Embedding

P

3

= 2;

C

3

= 320

Transformer

Encoder





R

3

= 2

N

3

= 5

E

3

= 4





×

6





R

3

= 2

N

3

= 5

E

3

= 4





×

6





R

3

= 2

N

3

= 5

E

3

= 4





×

6

Stage 4

H

32

×

H

32

Patch

Embedding

P

4

= 2;

C

4

= 512

Transformer

Encoder





R

4

= 1

N

4

= 8

E

4

= 4





×

3





R

4

= 1

N

4

= 8

E

4

= 4





×

3





R

4

= 1

N

4

= 8

E

4

= 4





×

3

7.2 Implementation Details

The resolution of input images in our experiments is 224 × 224 . In SGIS, we split each image into 7 × 7 patches. r h and r t are set to 0.1 and 0.4, respectively. Therefore, each image is subsplit into 161 patches in MGIS. For DeiT-S, we inserted a total of three PPSM modules in the 3rd, 7th, and 10th layers. Similarly, for LV-ViT-S, we inserted a total of four PPSM modules in the 4th, 7th, 9th, and 12th layers. For conducting the training process, we set the batch size to 256 and use AdamW optimizer to train all models for 300 epochs.

In terms of training strategies and optimization methods, our training closely follows the original methods proposed in the studies conducted by DeiT and LV-ViT, but do not employ any knowledge distillation algorithms in our experiments. Throughout the training phase, we observed the impact on convergence when subsplitting was applied solely to the head and middle patches. To enhance the speed of model convergence, we adopted subsplitting specifically targeting the head and middle patches for the initial 200 epochs. Subsequently, for the following 100 epochs, we applied subsplitting to the head and tail patches. The training was performed on a workstation equipped with 8 RTX 3090 GPUs.

7.3 Visualization Result

In Figure 9, we specifically demonstrate the images that can be well recognized by MG-ViT (LVViT-S) in SGIS and MGIS. As observed, the images on the left are "simple," usually occupying a significant portion of the image pixels, with typically complete and clear contours. While the images on the right are 'complex', containing intricate scenes and non-prominent objects or only include a small part of the objects, requiring multi-granularity representations through additional tokens.

Watch

Goblet

Tennis

Figure 10 visualizes the token slimming comparison between token merging (MG-ViT) and token pruning (DynamicViT). The result shows, the information loss caused by token pruning may result in incorrect slimming decisions, leading to a decrease in classification accuracy. However, the adoption of token merging in PPSM alleviates this situation.

input

Snowplow

Arcticfox

Castle

Gorilla

Chrysanthemum

DynamicViT

Cathedral

Monkey

Coleflowers

MG-ViT

Snowplow

Arcticfox

Castle

Gorilla

Chrysanthemum

DeiT-S

LV-ViT-S

High

MG-ViT

MG-ViT

CF-ViT

CF-ViT

STViT

SiT

ATS

DVT

Model

Model

Throughput

EViT

SPViT

Evo-ViT

DynamicViT

DVT

EViT

DynamicViT

eTPS

IA-RED2

PS-ViT

3

OW

Accuracy(%)

Accuracy(%)

7.4 More Experiments and Ablation Analysis

Necessity of global class attention.

We designed two experiments based on backbone DeiT-S to investigate the necessity of global class attention.

Table 8 shows the gain of the proposed global attention class on model performance, compared to random division, which involves randomly dividing patches into three groups, random CASR, which divides patches according to CASR in a random layer of ViT, and last CASR, which divides patches based on CASR in the last layer. As we can see, the global attention class achieves the best performance improvement. Table 7 compares the contribution of the global attention class to model performance with different values of λ . When λ = 0 , it represents the last CASR scheme. We set the default value of λ as 0.98.

Applicability of Feature Reuse. We introduce the Feature Reuse module from DVT[14] to enhance the feature representation of patches, and also explore its applicability in our model. Specially, we do not reuse the class tokens and apply zero padding to the tail tokens (i.e., not reusing the information from tail tokens). Table9 demonstrates the effective integration of the Feature Reuse module into our model, and indicates the effectiveness of our improvements.

Effect of token match-merge scheme. We conduct ablation experiments to investigate the effort of our proposed token match-merge scheme and establish three control schemes for comparison: token pruning, token clustering (K-means), and merging tokens into one. Especially, by employing our token match-merge scheme on boundary tokens, we achieved the best trade-off between accuracy and speed, as shown in Table 10. This result also supports the rationality of merging all negative tokens into one, because we need to process the tokens with low IIC in the most computationally

economical manner. Regarding the approach of generating new tokens by merging, the results in Table 11 demonstrate that the weighted average method performs best in maintaining accuracy.

Token merging vs. token pruning. Table 12 displays the performance of token pruning and token merging under two types of slimming, namely the hard slimming type (dividing tokens into two groups) and the soft slimming type (dividing tokens into three groups). It is observed that utilizing three-way decision mechanism for token merging (i.e., our proposed PPSM) achieves the best performance. This is attributed to assigning tokens located in the middle part of CASR to the boundary domain within three-way decision model, where the tokens adhere to boundary rules, resembling a "delayed processing".

Effectiveness of PPSM. We integrate the two methods, three-way decision slimming and token match-merge scheme, into our proposed PPSM module. Table 14 compares the performance of MGViT (PPSM) with existing methods of hard slimming and soft slimming, as well as the performance between MG-ViT with and without PPSM. The results demonstrate that our proposed PPSM module effectively streamlines the model while maintaining performance.

Experiment on images with higher resolution. Weconduct experiments on higher resolution images, as shown in Table 13, demonstrating the continued effectiveness of our proposed multi-granularity strategy on higher resolution images.

7.5 Computational Complexity Analysis

In this discussion, we disregard the patch embedding block. The computational complexity of a transformer layer responsible for processing tokens can be calculated as follows:

Based on provided equations, it can be observed that increasing the dimension C leads to a quadratic growth in computational complexity. Therefore, if the dimension C is expanded to 2 × , the computational cost will approximately increase by 4 × , as discussed in the feature reuse part.

In the PPSM of MG-ViT, we introduce the three-way decision slimming to reduce the number of tokens. Specifically, based ratios r pos and r neg , all tokens are divided into positive, negative, and boundary tokens. Positive tokens remain unchanged. Boundary tokens are merged to the half of their original quantity following the guidance of the token match-merge scheme. Negative tokens are merged into a single token, and their quantity can be neglected. Therefore, the computational complexity in the transformer layer following the first PPSM can be described as follows:

We employ DeiT-S as the backbone and insert PPSM before the self-attention (SA) layers in the 4th, 7th, and 10th layers. The positive, boundary, and negative tokens are divided according to a ratio of 5:4:1. The computational complexity of DeiT-S and our MG-ViT are:

7.6 Discussion of Ratio r

r h and r t . Each head patch is subsplit into 9 fine patches, while each middle patch is subsplit into 4 medium patches. The majority of important information is concentrated in a few head patches, partially distributed among middle patches, and only a small portion exists in tail patches. Consequently, we mandate that the count of middle patches exceeds half of the total number. Additionally, as our aim is to develop an efficient ViT framework, we strive to minimize the total number of patches obtained after subsplitting compared to the 196 tokens (the official splitting method 14 × 14). Considering these factors, we set r h < 0 . 2 . Furthermore, based on the results presented in Fig.1(a) and Fig.1(c), we set 0 . 3 < r h < 0 . 5 . We conducted experiments on the origin MG-ViT (without PPSM) and selected several competitive control group results to showcase in table 15.

r pos and r neg . According to the discussion in Section 6.3, we exclude the match-merge of boundary tokens and only consider the merging of negative tokens. When r neg = 0 . 2 , the number of tokens in the last layer of MG-ViT decreases to 51.2% of the original count. Therefore, the quantity of negative tokens should be relatively small, and we set r neg < 0 . 2 . On the other hand, if r pos = 0 . 5 , the minimum number of tokens in the last layer of MG-ViT should be 12.5% of the original count, which serves as the lowest limit for normal operation of MG-ViT. As a result, we set 0 . 5 ≤ r pos ≤ 0 . 7 . Our experimental findings, as presented in Table 16, include several competitive control group results.

Taking all aspects into consideration, we set r h = 0 . 1 , r t = 0 . 4 , r pos = 0 . 5 and r neg = 0 . 1 .

7.7 Three-Way Decision

Three-way decision is a decision mechanism in rough set theory. Let U be the set of all tokens, and given a pair of thresholds r pos and r neg . Let e ( x ) denote the rank of token x based on CASR, and

num represent the total number of tokens in the PPSM. We can describe the positive, negative, and boundary domains as follows:

For tokens in the positive and negative domains, we employ deterministic operations to either maintain them unchanged or merge them into one token. This is because we have determined their redundancy status. For tokens in the boundary domain whose redundancy status has not been determined, we employ undeterministic operation, token match-merge. Token match-merge strikes a balance between maintaining and merging, representing a form of slow merge that only merges similar tokens. If the newly generated boundary tokens are subsequently divided into the positive and negative domains in the next PPSM, they will undergo deterministic operations.

Peer reviews:
comments: SUMMARY:
This paper improves the inference efficiency of Vision Transformers by token pruning/merging. Specifically, the authors proposed a Multi-Granularity method, which splits an image into patches of different sizes and merges similar tokens. Extensive experiments and ablation studies on ImageNet have demonstrated the advantage of their method.

STRENGTHS:
1. Simple idea but works. The proposed method achieves better speed-accuracy trade-off than many existing methods.
2. The experiments are comprehensive, especially for the ablation study.
3. This paper is easy to follow. The overall presentation is of good quality.

WEAKNESSES:
1. Despite the great performance, the evaluation is only conducted on one scale, i.e., the small version of DeiT and LV-ViT. It is not clear if the proposed method can generalize to other scales, such as DeiT-Ti and DeiT-B.
2. Lack throughput comparison in Table 3. FLOPs may not reflect the real speed on hardware. It is important to directly evaluate the inference speed when comparing with other related works.
3. Due to token merging, the proposed method may not be general for downstream dense prediction tasks.

QUESTIONS:
1. I would like to know the throughput/model parameter comparison in Table 3.
2. Unlike this paper which trains a model from scratch with 300 epochs, DynamicViT utilises pretrained weights to finetune their model. How the performance would be if the MG-ViT is also finetuned based on pretrained DeiTs.

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-06-27

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a multi-granularity method to develop efficient vision Transformers. A two-stage inference pipeline is designed to handle easy and difficult samples in different ways. The method is evaluated on ImageNet based on DeiT and LVViT.\n\nSTRENGTHS:\n- The motivation presented in the Introduction looks convincing and insightful.\n\n- The method achieves good performance on ImageNet classification and can outperform previous methods.\n\nWEAKNESSES:\n- The writing of this paper can be further improved. There are quite a few grammar errors and typos. Many sentences are confusing or hard to understand. We list some of them: line 65, \"we propose MG-ViT 2\"; line 80, \"Simply put, based on CASR, tokens are divided ...\".\n\n- Although the method is well-motivated, the proposed method is not very new: there are several existing methods to use classification probability to estimate the confidence of a prediction and use it to adjust computation [r1]; the token merging and multi-scale inference framework have also been explored in previous work like DVT and [r2].\n\n- The experimental results are not very impressive and competitive in 2023 since recent efficient vision Transformers are usually compatible with more diverse architectures like Swin/ConvNeXt [r1,r3] and more complex tasks like detection and segmentation [r3, r4]. The claim in line 87 \"MG-ViT is a dynamic and versatile framework that can be applied to most ViT models\" is also not well supported since it seems non-trivial to apply the method to hierarchical Transformers. \n\n[r1] Token Merging: Your ViT But Faster, ICLR 2023.\n\n[r2] MIA-Former: Efficient and Robust Vision Transformers via Multi-Grained Input Adaptation, AAAI 2022.\n\n[r3] Dynamic Spatial Sparsification for Efficient Vision Transformers and Convolutional Neural Networks, TPAMI. \n\n[r4] Revisiting Token Pruning for Object Detection and Instance Segmentation, arXiv:2306.07050\n\nQUESTIONS:\nPlease refer to my comments above.\n\nLIMITATIONS:\nThe limitations of the proposed method have not been discussed.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

comments: SUMMARY:
The proposed MG-ViT framework introduces a two-stage multi-granularity approach for Vision Transformers (ViTs), aiming to balance performance and computational cost. Initially, an image is divided into patches for basic classification. If the image isn't accurately recognized, the system initiates a more detailed patch analysis stage.

To reduce the computational cost, a three-way decision slimming method is proposed. A token match-merge scheme also aids in making the ViT more compact and efficient. Tests on ImageNet show that without performance loss, MG-ViT reduces the FLOPs of LV-ViT by 47% and DeiT by 56%, demonstrating significant computational cost reduction.

STRENGTHS:
-The underlying motivation for the study is lucid, and the proposed method displays novelty.

-The manuscript is well-written and easy to comprehend.

-The experimental results highlight an impressive balance between accuracy and speed offered by the proposed method

WEAKNESSES:
While the Multi-Granularity strategy is well-intentioned and produces promising classification results, its application to other vision downstream tasks such as segmentation presents significant challenges.

Additionally, there is a lack of results regarding window-based Vision Transformers like the Swin Transformer. This could be due to the complexity and difficulty in applying this strategy to such models.

QUESTIONS:
Please see weakness.

LIMITATIONS:
Please see weakness.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-08

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents MG-ViT, a way to save computational cost without affecting the performance of ViT-based methods. MG-ViT is a two-stage model, the first stage is a lightweight ViT for single-granularity inference, and the second stage network for multi-granularity inference.\n\nSTRENGTHS:\n1、A series of experiments are conducted to demonstrate the influence of tokens of different IIC and granularity on network performance, which provides a clear motivation for the proposed MG-ViT.\n\n2、The design of MGIS, including multi-granularity patch subsplitting, PPSM, and token match-merge scheme is natural and reasonable.\n\n3、The ablation experiments of the proposed components are sufficient.\n\nWEAKNESSES:\n1、In recent years, a series of ViT-based methods based on window attention (Such as Swin Transformer, and Slide-Transformer[1]) have been proposed, which have better classification performance and lower computational cost than the original ViT (DeiT). This paper can better prove the effectiveness of MG-ViT, if it can be compatible with the frameworks based on window attention especially the current sota methods (Slide-Transformer).\n\n2、This paper can better prove the effectiveness of MG-ViT, if it can be compatible with ViTs pretrained by massive data (JFT-3B or LAION) or self-supervised pre-trained (Such as MAE[2], DINOv2), which achieves better performance (more than 85% acc on ImageNet).\n\n\n[1] Pan X, Ye T, Xia Z, et al. Slide-Transformer: Hierarchical Vision Transformer with Local Self-Attention[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023: 2082-2091.\n\n[2] He K, Chen X, Xie S, et al. Masked autoencoders are scalable vision learners[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022: 16000-16009.\n\nQUESTIONS:\nThe MG-VIT proposed in this paper is novel enough, but the effectiveness of the proposed method needs to be validated under stronger baselines.\n\nLIMITATIONS:\nRefer to weakness", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduced a new vision transformer model, denoted as MG-ViT, which consists of a two-stage approach to balance the trade-off between the computational cost and performance. Specifically, it proposes a two-stage approach, in which a single-granularity inference stage is used to determine if the image can be correctly classified. If not, then a second multi-granularity stage is employed. The model is trained and tested on ImageNet-1K dataset and reportedly achieve competitive performance.\n\nSTRENGTHS:\n1. The paper is well-written and easy to follow. \n\n2. The idea of improving vision transformer efficiency is important and has merits. \n\n3. The authors present comprehensive ablations studies and analysis of their proposed model.\n\nWEAKNESSES:\n1. The proposed effort lacks novelty as it essentially proposes a two-stage approach with different level of granularity to strike a balance between accuracy vs efficiency. In the proposed framework, there are several assumptions that may limit the applicability of this work. First, it is not clear how to determine the confidence judgement threshold in the first stage which can significantly impact the performance. Second, the head.middle and tail patchification scheme cannot be generalized to all use-cases and can indeed introduce undesired biases that can negatively impact the model's performance. \n\n2. One of the main arguments of this work in justifying its usefulness is the efficiency aspect. However, the authors only report the gains in terms of number of FLOPs which does not include the entire picture. In fact, the throughput comparison is only shown in Figure 10 of the supplementary materials and without any actual quantitative comparisons. Since the method is comprised of two stages, it is expected that the throughput is negatively impacted. \n\n3. The method is only evaluated for smaller model regime and it is not clear whether it scales to larger models (e.g. ViT-L, ViT-H, ViT-G). \n\n4. The experimental results are not convincing enough and don't validate the effectiveness of this work.\n\nQUESTIONS:\n1. The proposed model is only evaluated for classification. Can the same ideas be extended to down-stream tasks such as detection and segmentation ? \n\n2. Typical ImageNet images with resolution 224 x 224 are used for training and evaluation of the proposed model. How about higher resolution images ? \n\n3. Can the proposed model be used with hierarchical vision transformers such as Swin Transformers ?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-16"}