SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yu Zhang $^{1}$  Yepeng Liu $^{2}$  Duoqian Miao $^{1*}$  Qi Zhang $^{1}$  Yiwei Shi $^{3}$  Liang Hu $^{1}$ $^{1}$ Tongji University  $^{2}$ University of Florida  $^{3}$ University of Bristol
Vision Transformer (ViT) faces obstacles in wide application due to its huge computational cost. Almost all existing studies on compressing ViT adopt the manner of splitting an image with a single granularity, with very few exploration of splitting an image with multi-granularity. As we know, important information often randomly concentrate in few regions of an image, necessitating multi-granularity attention allocation to an image. Enlightened by this, we introduce the multi-granularity strategy to compress ViT, which is simple but effective. We propose a two-stage multi-granularity framework, MG-ViT, to balance ViT's performance and computational cost. In single-granularity inference stage, an input image is split into a small number of patches for simple inference. If necessary, multi-granularity inference stage will be instigated, where the important patches are further subsplit into multi-finer-grained patches for subsequent inference. Moreover, prior studies on compression only for classification, while we extend the multi-granularity strategy to hierarchical ViT for downstream tasks such as detection and segmentation. Extensive experiments Prove the effectiveness of the multi-granularity strategy. For instance, on ImageNet, without any loss of performance, MG-ViT reduces  $47\%$  FLOPs of LV-ViT-S and  $56\%$  FLOPs of DeiT-S.
Transformer [1] has gained tremendous achievements in computer vision. Despite the excellent performance of Convolutional Neural Networks (CNN) in computer vision tasks [2; 3; 4; 5], due to ViT's superior ability to capture global information and long-range interactions, ViT [6] outperforms CNN in various tasks [7; 8; 9; 10; 11]. However, the impressive performance of ViT comes at the cost of its huge computational overhead. Therefore, more researches focus on ViT compression for greater efficiency. Since the computational cost of ViT increases quadratically with the number of tokens, minimizing the number of tokens is crucial in compressing ViT.
Notable investigations have adopted various methods to compress ViT. DynamicViT [12] reduces token number by pruning redundant tokens, EViT [13] merges redundant tokens into one token. DVT [14] utilizes the particular and dynamic patch splitting manner (e.g.,  $4 \times 4$ ,  $7 \times 7$ , etc.) for each image, based on its complexity, rather than following the official manner of  $14 \times 14$ . Although DVT innovatively considers the differences in complexity between images, it still uses single granularity to split one image. As we all know, images often contain a lot of redundancy, with important semantic information randomly concentrated in a few regions. Therefore, it is imperative to consider the complexity and semantic density diversities across various regions of the image: different regions of an image should be assigned attention with multiple granularities. We initiate our exploration by observing the foundational task, classification, and we have 3 intriguing observations:
Observation 1: Most images can be correctly recognized by splitting them into a small number of patches, as has been analogously observed in research [14]. Training a DeiT-S [15] with the  $4 \times 4$  patch splitting manner can achieve  $63.4\%$  accuracy at a computational cost of 0.45G FLOPs. Although splitting an image to  $7 \times 7$  patches and  $14 \times 14$
Table 1: Accuracy and FLOPs of DeiT-S on ImageNet in different splitting manner.
Manner	14 × 14	7 × 7	4 × 4
Accuracy	79.8%	73.2%	63.4%
FLOPs	4.60G	1.10G	0.45G
patches can enhance the accuracy by  $9.8\%$  and  $16.4\%$ , the computation cost will also increase by  $2.4\times$  and  $10.2\times$ . How to substantially improve the accuracy with a low growth rate of FLOPs is a subject worth contemplating.
**Observation 2:** The classification accuracy is mainly affected by the several critical tokens, and subsplitting patches into finer patches is more beneficial to classification. We split each image into  $14 \times 14$  patches and rank tokens from high to low based on their class attention scores. The 196 tokens (patches) are equally divided into head, middle, and tail groups, and respectively input into DeiT-S to measure the accuracy of each group (Fig.1(a)). Then, subsplitting the patches in the head group expands the quantity to  $4 \times$  and  $9 \times$ , their accuracy is measured (Fig.1(b)). It is illustrated that head tokens ensure the base of classification accuracy, and finer tokens lead to higher accuracy.
(b)
Figure 1: The classification accuracy of the tokens from different groups in each layer of DeiT-S. (c) 4 input groups: 1#: Only higher IIC tokens (top  $50\%$ ). 2#: Merging lower IIC tokens into one token and together with higher IIC tokens. 3#: According to CASR, merging every  $50\%$  of lower IIC tokens into one token (2 in total) and together with higher IIC tokens, 4#: According to CASR, merging every  $25\%$  of lower IIC tokens into one token (4 in total) and together with higher IIC tokens.
**Observation 3:** All information matters, and proper handling of tokens with low important information content (IIC) contributes to avoiding partial loss of accuracy. We treat the bottom  $50\%$  tokens as lower IIC tokens based on class attention score rank (CASR), and create 4 input groups for all tokens. The results in Fig.1(c) suggest that retaining lower IIC tokens can maintain accuracy. This is because lower IIC tokens contain complementary information to higher IIC tokens, which helps bridge the accuracy gap with the baseline. The dissimilarities among lower IIC tokens should be considered, necessitating a hierarchical processing. Performance improvement is from complementary information, certain lower IIC tokens make the higher contribution. If they are merged with other lower IIC tokens, the gain of performance improvement may be diluted.
Thus far, we can identify two sound reasons for utilizing the multi-granularity strategy to compress ViT. Firstly, as aforementioned, due to the distinctiveness of important information distribution in images, it is imperative to assign multi-granular attention to an image. Secondly, guided by our observations, we can consider multi-granularity as an intermediate state between full fine-grained and full coarse-grained. Full fine-grained stage represents stronger performance, while full coarse-grained stage represents less computation. Therefore, multi-granularity can naturally be used to balance performance and computational cost.
We propose MG-ViT: a multi-granularity framework for compressing ViT, which automatically splits an image into multi-granularity patches, as shown in Figure 2. MG-ViT consists of two parts: Single-Granularity Inference Stage (SGIS) and Multi-Granularity Inference Stage (MGIS), related to Observation 1. In SGIS, a small number of tokens are used for classification inference. Should a prediction confident enough ensue, the inference terminates immediately; otherwise, MGIS will be
Figure 2: Example for MG-ViT.
triggered. In MGIS, We put forward three innovative methods. The first is multi-granularity patch subsplitting, related to Observation 2. All patches are divided into head, middle and tail groups based on CASR in SGIS, head and middle patches are subsplit, resulting in tokens with multiple information granularity that are input into ViT. The second is the three-way decision slimming, which can streamline ViT softly. Simply put, based on CASR, tokens are divided into three groups: positive, boundary, and negative. Boundary and negative tokens are respectively merged by different rules to reduce their numbers. The third is the token match-merge scheme for boundary tokens, related to Observation 3. The second and third methods are packaged as Plug-and-Play Slimming Module (PPSM) against the huge increase in computational cost caused by patch subsplitting. After PPSM, all tokens are fed into the next layer of ViT to continue inference.
Due to that ViT only outputs a single-scale feature representation and lacks the capability to handle multi-scale variations, it possesses significant deficiencies in downstream tasks. In order to adapt multi-granularity strategy to various downstream tasks, we extend the MG-ViT framework to hierarchical ViT. Unlike ViT, hierarchical ViT aggregates tokens within rectangular areas. However, patch subsplitting results in patches (tokens) with multiple scales interleaved in the image, making it challenging to seamlessly enclose them within regular rectangular areas. To address this issue, we introduce a proxy token of the same scale for each head or middle patch to represent itself. Additionally, each proxy token engages in attention calculations with its corresponding fine tokens or medium tokens to represent the subsplitting of the head or middle patch it stands for. Following the aggregation methodology of PVT [16], Hierarchical MG-ViT can perform downstream tasks.
MG-ViT is a dynamic and versatile framework that can be applied to most ViT models. We select LV-ViT [17] and DeiT [15] to assess the performance of MG-ViT on ImageNet [18]. The experiments demonstrate that MG-ViT can balance performance and computation cost greatly, thereby significantly improving the efficiency of ViT. Moreover, while ensuring performance, MG-ViT reduces  $47\%$  FLOPs of LV-ViT and  $56\%$  FLOPs of DeiT. We also conduct simple experiments in object detection and semantic segmentation on the MS-COCO [19] and ADE20K [20] datasets, respectively. The results show that Hierarchical MG-ViT effectively reduces computational overhead without significant performance loss, demonstrating the feasibility of extending multi-granularity strategy to the Hierarchical ViT structure.
Vision Transformer. ViT [6] achieves a great breakthrough in the image classification task by redesigning the Transformer structure, completely igniting researchers' passion for exploring the ViT further. Later on, ViT is modified to hierarchical structure for various downstream tasks [21; 8; 22; 23].
ViT and Hierarchical ViT. ViT is a series of ViT models that adhere to the vanilla ViT design principles, such as DeiT [15] and LV-ViT[17]. They only output a single-scale feature representation and lack the capability to handle multi-scale variations, making them typically suitable for classification tasks. Hierarchical ViT is a series of ViT models that incorporate a hierarchical structure to aggregate tokens layer by layer, enabling them to better handle multi-scale information and well-suited for various downstream tasks. Swin-Transformer [24] and PVT [16; 25] are representative architectures of hierarchical ViT.
ViT compression. ViT and hierarchical ViT can both enhance efficiency. Hierarchical ViT primarily achieves efficiency by improving interaction rules among tokens, whereas ViT mainly relies on compression. ViT compression primarily aims to optimize ViT structures in a flexible and automated manner, developing a dynamic and efficient ViT where each image undergoes an individual computational process based on its unique features, all while minimizing computational costs without sacrificing performance. There are many classic ViT compression works, such as DynamicViT [12], PS-ViT [26], Evo-ViT [27], and so on [13; 14; 28; 29; 30; 31; 32; 33].
3 Multi-Granularity Vision Transformer
Figure 3 illustrates the overall framework of MG-ViT. For each image, SGIS must be executed. In SGIS, the image is split into a small number of patches. These 2D patches are embedded into 1D
Figure 3: The overall framework of MG-ViT.
tokens and fed into ViT for straightforward inference. The obtained class prediction will be evaluated against predetermined criteria (threshold). If the class prediction is acceptable, the inference can be terminated immediately, and the class prediction can be output as the final classification result. If the class prediction is unacceptable, MGIS will be executed.
In MGIS, according to the IIC of each patch (referring to CASR), all patches are proportionally divided into three groups: head, middle and tail. Head and middle patches are further subsplit. As head patches with high IIC are the most crucial for classification, they require subsplitting with fine granularity, and eventually are subsplit into fine patches with fine-grained information. Middle patches with middle IIC are more crucial for classification but less than head patches. Whence, they require subsplitting with medium granularity, being subsplit into medium patches with medium-grained information. Tail patches, maintain their coarse granularity and serve as coarse patches. All patches undergo patch embedding and feature reuse to generate multi-granularity tokens which are input into ViT for inference, prediction, and output. Considering that patch subsplitting leads to a huge increase in computational cost, Plug-and-Play Slimming Module (PPSM) is introduced in certain layers of ViT to reduce computational cost. The three-way decision slimming lies here. According to CASR, we divide all tokens into three groups: positive, boundary and negative, aiming to realize soft slimming of ViT. Positive tokens remain unchanged. Among boundary tokens, similar tokens are matched and merged based on our proposed token match-marge scheme (more details in Section 3.3.2), resulting in fewer new tokens. Negative tokens are merged into one token by weights. In this way, the number of tokens is reduced, accomplishing ViT soft slimming.
3.2 Single-Granularity Inference Stage (SGIS)
In SGIS, the standard ViT is employed. Formally, for an input image  $\aleph$ , it is first split into patches  $P = [p^{1}, p^{2}, \ldots, p^{N}]$  where  $N$  is the number of patches. In patch embedding, patches are mapped into  $C$ -dimension token embeddings via linear projection. Additionally, a learnable token embedding  $x^{0}$ , referred to as a class token, is appended to the sequence of token embeddings to represent image  $\aleph$  for class prediction. After position embedding  $E_{pos}$ , the input token sequence for ViT in SGIS is:
$$
X = \left[ x ^ {0}, x ^ {1}, \dots , x ^ {N} \right] + E _ {\text {p o s}} \tag {1}
$$
where  $x^{i} \in \mathbb{R}^{C}$ , present a token embedding of the  $i$ -th patch if  $i > 0$ , and  $E_{pos} \in \mathbb{R}^{(N + 1) \times C}$ .
A ViT consists of  $L$  layers, each layer comprises a self-attention (SA) module and a feed-forward network (FFN). In SA of the  $l$ -th layer, the self-attention  $Attention(Q_{l},K_{l},V_{l})$  is computed as follows:
$$
\mathcal {A} _ {l} = \operatorname {s o f t m a x} \left(\frac {Q _ {l} K _ {l} ^ {T}}{\sqrt {C}}\right) = \left[ a _ {l} ^ {0}, a _ {l} ^ {1}, \dots , a _ {l} ^ {N} \right], \quad \text {A t t e n t i o n} \left(Q _ {l}, K _ {l}, V _ {l}\right) = \mathcal {A} _ {l} V _ {l}, \tag {2}
$$
where  $Q_{l}, K_{l}, V_{l} \in \mathbb{R}^{(N + 1)\times C}$ , are query, key and value matrices of  $l$ -th layer respectively and obtained through linear projection for  $X_{l - 1}$ .  $\mathcal{A}_l$  is attention map of  $l$ -th layer, and the first row of attention map  $a_l^0$  represent class attention in  $l$ -th layer.
Ulteriorly, the processes of SA and FFN in the  $l$ -th layer are as follows:
$$
X _ {l - 1} ^ {\prime} = S A \left(X _ {l - 1}\right) + X _ {l - 1}, \quad X _ {l} = F F N \left(X _ {l - 1} ^ {\prime}\right) + X _ {l - 1} ^ {\prime}. \tag {3}
$$
After consecutive computation of  $L$  layers, class token  $x_{L}^{0}$  from the  $L$ -th layer is input into the classifier  $\mathfrak{C}$  to obtain class prediction distribution  $D^{S}$ :
$$
D ^ {S} = \mathfrak {C} (x _ {L} ^ {0}) = [ d _ {1} ^ {S}, d _ {2} ^ {S}, \dots , d _ {M} ^ {S} ], \tag {4}
$$
where  $S$  indicates SGIS,  $M$  denotes the class number, up to this point, the class prediction of image  $\aleph$  is the largest entry of  $D^{S}$ , i.e.,  $\arg\max_{j} d_{j}^{S}$ .
Threshold for judgment. If the value of  $\max_j d_j^S$  for input image  $\aleph$  is large enough, it indicates that the class prediction in SGIS is acceptable, and the classification result can be generated. Therefore, the inference can be terminated immediately, saving computational cost. In order to use  $\max_j d_j^S$  to determine whether to terminate the inference, we introduce a threshold  $\epsilon$ : if  $\max_j d_j^S > \epsilon$ , the inference is terminated immediately, and the class prediction is output as the classification result. If  $\max_j d_j^S < \epsilon$ , the image is transitioned to MGIS for consecutive inference.
3.3 Multi-Granularity Inference Stage (MGIS)
3.3.1 Generation of Multi-Granularity Tokens
To bridge the ViTs of two stages, we perform multi-granularity patch subsplitting and feature reuse to generate tokens with multi-grained information as the input for the ViT in MGIS.
Multi-granularity patch subsplitting. Since  $\max_j d_j^S$  of image  $\aleph$  is less than  $\epsilon$ , further inference is required. As mentioned above, in MGIS, image  $\aleph$  needs to be subsplit with multiple different granularities, and features of patches in SGIS need to be reused. These operations require to be instructed by IIC of patches (tokens) in SGIS, as patches with different IIC will be divided into different groups. Thus, the key now lies in how to identify the IIC of patches.
According to Equation 2, class token  $y^0$  can be represented by class attention  $a^0 \in \mathcal{A}$  as:
$$
y ^ {0} = \operatorname {s o f t m a x} \left(\frac {q ^ {0} K ^ {T}}{\sqrt {C}}\right) = a ^ {0} \cdot V, \tag {5}
$$
as we all know, class token  $y^0$  represents the entire image  $\aleph$  as input to produce the class prediction in classifier  $\mathfrak{C}$ . As  $V = [v^0, v^1, \dots, v^N]$ , where  $v^i$  is the value of the  $i$ -th token if  $i > 0$ . The class token  $y^0$  is obtained by integrating the value of each token using  $a^0$  as the corresponding weights. In other words,  $a^0$  determines the amount of information from each token that enters the class token  $y^0$  for classification. Therefore, we can use class attention  $a^0$  to measure the contribution of each token to the class prediction, thence, we identify IIC according to CASR. However, if we only use CASR originating from the class attention of a certain layer to divide patches, patches of each group will be more random and unfixed, lacking overall consistency. Therefore, the flow of information across layers must also be considered. Thus, we come up with the global class attention instead of only using class attention of a certain layer to identify IIC of patches. The global class attention  $\hat{a}_l$  in the  $l$ -th block is as following:
$$
\hat {a} _ {l} = \lambda \cdot \hat {a} _ {l - 1} + (1 - \lambda) \cdot a _ {l} \tag {6}
$$
where  $\lambda = 0.98$ . In Fig.1(a), we can observe that the classification accuracy of all groups exhibits intense fluctuation in shallow layers. It is evident that class attention is unstable in shallow layers. Therefore, the global class attention calculation starts from the 3-rd layer. We use CASR rooted from global class attention in the last layer  $\hat{a}_L$  to identify IIC of patches.
According to  $\hat{a}_L$ , all patches are divided into three groups: head, middle, and tail. The numbers of head, tail and middle patches, denoted by  $N_h$ ,  $N_t$ , and  $N_m$  are given by:
$$
N _ {h} = \left\lfloor N \cdot r _ {h} + 0. 5 \right\rfloor , \quad N _ {t} = \left\lfloor N \cdot r _ {t} + 0. 5 \right\rfloor , \quad N _ {m} = N - N _ {h} - N _ {t}, \tag {7}
$$
where  $r_h$  and  $r_t$  represent the number rate of head and tail patches respectively.
Head and middle patches need to be subsplit. Since head patches have high IIC, each is subsplit into  $3 \times 3$  fine patches. Middle patches with middle IIC, each is subsplit into  $2 \times 2$  medium patches. Each tail patch is considered as a coarse patch and remains unchanged. After patch subsplitting, the input token sequence is denoted as follows:
$$
\bar {X} = \left[ \bar {x} ^ {0}, \bar {x} ^ {1}, \dots , \bar {x} ^ {K} \right] + \bar {E} _ {p o s}, \tag {8}
$$
where  $K$  represents the total number of patches after patch subsplitting.
Feature reuse. Patch subsplitting disrupts the integrity of head and middle patches, resulting in a lack of correlation among fine or medium patches which are severally subsplit from the same head or middle patch. To address this issue, we incorporate Feature Reuse module from DVT [14] to inject the original information of head or middle patches into new fine or medium patches. It is advantageous to enhance feature representation of each patch and strengthen the interconnectivity among them.
Figure 4 demonstrates the Feature Reuse module. Tokens (except class token) output by the ViT of SGIS are taken as input and divided based on their IIC. As head and middle patches need to be subsplit in MGIS, we individually upsample head and middle tokens, so that the number of new tokens obtained by upsampling is the same as the number of new patches obtained by subsplitting. Different from the module in DVT [14], considering that tail tokens remain unchanged in MGIS, we utilize zero padding to handle them. Finally, all tokens are flattened and sorted according to their position in image  $\aleph$ . The token sequence  $X^{FR}$  output by Feature Reuse is obtained as follows:
$$
X ^ {F R} = F R \left(x _ {L} ^ {1}, \dots , x _ {L} ^ {N}\right).
$$
Figure 4: Feature Reuse module.
Specially,  $X^{FR} \in \mathbb{R}^{N \times C'}$ . Due to concatenation expanding the dimension of multi-granularity tokens to  $2 \times$ , resulting in a  $4 \times$  increase in computational cost (detailed analysis in Appendix), we set a MLP in the module to realize dimension reduction:  $MLP(\mathbb{R}^C \Rightarrow \mathbb{R}^{C'})$ . The value of  $C'$  is small, thereby increasing the dimension of multi-granularity tokens from  $C$  to  $C + C'$ , but still much smaller than  $2C$ , which helps save computational cost.
The generated multi-granularity token sequence  $\tilde{X} \in \mathbb{R}^{(N + 1) \times (C' + C)}$ , which is input to ViT in MGIS for inference, slimming and output, is denoted as:
$$
\tilde {X} = \operatorname {C o n c a t} \left(\bar {X}, X ^ {F R}\right) = \left[ \bar {x} ^ {0}, \tilde {x} ^ {1}, \dots , \tilde {x} ^ {N} \right] \tag {10}
$$
3.3.2 Plug-and-Play Slimming Module (PPSM)
Most existing ViT slimming methods, such as DynamicViT, PS-ViT, adopt token pruning, undermining the information integrity of the image and is not beneficial for classification. Therefore, we select token merging, which diminishes the number of tokens while preserving information.
Three-way decision slimming. Most existing ViT slimming methods belong to hard slimming, which divides tokens into non-redundant and redundant groups based on a certain criterion and reduces the number of redundant tokens to streamline ViT. Nevertheless, tokens in the middle part of CASR are arduous to distinguish as non-redundant or redundant during the inference. Mistakenly identifying non-redundant tokens as redundant and performing slimming will lead to a degradation of ViT's performance. Considering that three-way decision model establishes a buffer zone, i.e., boundary, between positive and negative, it can aid in more appropriately handling tokens situated in the middle position of CASR. Therefore, we employ three-way decision mechanism to realize soft slimming for ViT.
Three-way decision [34] is a decision mechanism in rough set [35; 36] that partitions a set into positive, negative, and boundary domains, with elements in each domain subject to positive, negative, and boundary rules, respectively. By setting the proportion of elements in positive and negative domains to  $r_{pos}$  and  $r_{neg}$ , all tokens inputted into PPSM are divided into positive, negative and boundary domains. Positive tokens adhere to the positive rule: maintain unchanged. Boundary tokens adhere to boundary rule, which refers to the token match-merge scheme we proposed. Negative tokens adhere to the negative rule: all are merged into one token by weights.
Figure 5: The process of token match-merge.