SlowGuess/NeurIPS_2023_OCR huggingface dataset:
economical manner. Regarding the approach of generating new tokens by merging, the results in Table 11 demonstrate that the weighted average method performs best in maintaining accuracy.
Token merging vs. token pruning. Table 12 displays the performance of token pruning and token merging under two types of slimming, namely the hard slimming type (dividing tokens into two groups) and the soft slimming type (dividing tokens into three groups). It is observed that utilizing three-way decision mechanism for token merging (i.e., our proposed PPSM) achieves the best performance. This is attributed to assigning tokens located in the middle part of CASR to the boundary domain within three-way decision model, where the tokens adhere to boundary rules, resembling a "delayed processing".
Table 12: Comparison between hard and soft slimming.
Type	Method	Acc.(%)
Hard	Pruning	79.8
Merging	80.3
Soft	Pruning	80.2
Merging	80.8
Table 13: Experiments fine-tuned and evaluated with a resolution of 384.
Model	ε	Acc.(%)	FLOPs(G)	TP(img./s)
LV-ViT-S ↑384	-	84.4	22.2	303
MG-ViT ↑384	0.66	84.4(+0.0)	10.7(-52%)	515(+1.70×)
MG-ViT ↑384	1	85.5(+0.7)	16.9(-24%)	345(+1.14×)
Effectiveness of PPSM. We integrate the two methods, three-way decision slimming and token match-merge scheme, into our proposed PPSM module. Table 14 compares the performance of MG-ViT (PPSM) with existing methods of hard slimming and soft slimming, as well as the performance between MG-ViT with and without PPSM. The results demonstrate that our proposed PPSM module effectively streamlines the model while maintaining performance.
Table 14: Comparison between MG-ViT and the existing soft and hard slimming methods for ViT
Type	Method	Model	Acc.(%)	FLOPs(G)
Origin	-	MG-ViT(w/o PPSM)	80.8	3.7
Baseline	-	DeiT	79.8	4.6
Hard slimming	Pruning	PS-ViT	79.8(-0.0)	3.8(-1.2)
DynamicViT	79.3(-0.5)	2.9(-1.7)
IA-RED2	79.1(-0.7)	3.3(-1.3)
Merging	EViT	79.5(-0.3)	3.0(-1.6)
Soft slimming	Pruning	SiT	79.8(-0.0)	2.3(-2.3)
Merging	Evo-ViT	79.4(-0.4)	2.9(-1.7)
MG-ViT	80.8(+1.0)	2.4(-2.2)
Experiment on images with higher resolution. We conduct experiments on higher resolution images, as shown in Table 13, demonstrating the continued effectiveness of our proposed multi-granularity strategy on higher resolution images.
7.5 Computational Complexity Analysis
In this discussion, we disregard the patch embedding block. The computational complexity of a transformer layer responsible for processing tokens can be calculated as follows:
$$
\Omega (S A (t o k e n s)) = 4 N C ^ {2} + 2 N ^ {2} C, \tag {18}
$$
$$
\Omega (F F N (t o k e n s)) = 8 N C ^ {2}.
$$
Based on provided equations, it can be observed that increasing the dimension  $C$  leads to a quadratic growth in computational complexity. Therefore, if the dimension  $C$  is expanded to  $2 \times$ , the computational cost will approximately increase by  $4 \times$ , as discussed in the feature reuse part.
In the PPSM of MG-ViT, we introduce the three-way decision slimming to reduce the number of tokens. Specifically, based ratios  $r_{pos}$  and  $r_{neg}$ , all tokens are divided into positive, negative, and boundary tokens. Positive tokens remain unchanged. Boundary tokens are merged to the half of their original quantity following the guidance of the token match-merge scheme. Negative tokens are merged into a single token, and their quantity can be neglected. Therefore, the computational complexity in the transformer layer following the first PPSM can be described as follows:
$$
N ^ {\prime} = r _ {p o s} N + \frac {1 - r _ {p o s} - r _ {n e g}}{2} N \tag {19}
$$
$$
\Omega (P P S M ^ {1 s t}) = 1 2 N ^ {\prime} C ^ {2} + 2 N ^ {\prime 2} C.
$$
We employ DeiT-S as the backbone and insert PPSM before the self-attention (SA) layers in the 4th, 7th, and 10th layers. The positive, boundary, and negative tokens are divided according to a ratio of 5:4:1. The computational complexity of DeiT-S and our MG-ViT are:
$$
\Omega (D e i T) = 1 4 4 N C ^ {2} + 2 4 N ^ {2} C, \tag {20}
$$
$$
\Omega (S T V i T) = 9 1. 2 N C ^ {2} + 1 1. 1 N ^ {2} C.
$$
7.6 Discussion of Ratio  $r$
$r_h$  and  $r_t$ . Each head patch is subsplit into 9 fine patches, while each middle patch is subsplit into 4 medium patches. The majority of important information is concentrated in a few head patches, partially distributed among middle patches, and only a small portion exists in tail patches. Consequently, we mandate that the count of middle patches exceeds half of the total number. Additionally, as our aim is to develop an efficient ViT framework, we strive to minimize the total number of patches obtained after subsplitting compared to the 196 tokens (the official splitting method  $14 \times 14$ ). Considering these factors, we set  $r_h < 0.2$ . Furthermore, based on the results presented in Fig.1(a) and Fig.1(c), we set  $0.3 < r_h < 0.5$ . We conducted experiments on the origin MG-ViT (without PPSM) and selected several competitive control group results to showcase in table 15.
Table 15: Accuracy and FLOPs with different values of  $r_h$  and  $r_t$ .
rh\rt	0.05\0.3	0.05\0.4	0.1\0.3	0.1\0.4	0.15\0.35
Acc.(%)	80.7	80.5	81.0	81.0	81.1
FLOPs(G)	3.6	3.5	4.1	3.9	4.2
$r_{pos}$  and  $r_{neg}$ . According to the discussion in Section 6.3, we exclude the match-merge of boundary tokens and only consider the merging of negative tokens. When  $r_{neg} = 0.2$ , the number of tokens in the last layer of MG-ViT decreases to  $51.2\%$  of the original count. Therefore, the quantity of negative tokens should be relatively small, and we set  $r_{neg} < 0.2$ . On the other hand, if  $r_{pos} = 0.5$ , the minimum number of tokens in the last layer of MG-ViT should be  $12.5\%$  of the original count, which serves as the lowest limit for normal operation of MG-ViT. As a result, we set  $0.5 \leq r_{pos} \leq 0.7$ . Our experimental findings, as presented in Table 16, include several competitive control group results.
Taking all aspects into consideration, we set  $r_h = 0.1$ ,  $r_t = 0.4$ ,  $r_{pos} = 0.5$  and  $r_{neg} = 0.1$ .
7.7 Three-Way Decision
Three-way decision is a decision mechanism in rough set theory. Let  $U$  be the set of all tokens, and given a pair of thresholds  $r_{pos}$  and  $r_{neg}$ . Let  $e(x)$  denote the rank of token  $x$  based on CASR, and
Table 16: Accuracy and FLOPs with different values of  $r_{pos}$  and  $r_{neg}$ .
rpos\neg neg	0.5\0.1	0.5\0.15	0.6\0.05	0.6\0.1	0.7\0.05
Acc.(%)	80.8	80.6	81.0	81.0	81.1
FLOPs(G)	2.4	2.3	2.9	2.8	3.2
num represent the total number of tokens in the PPSM. We can describe the positive, negative, and boundary domains as follows:
$$
P o s = \left\{x \in U \mid e (x) \leq n u m \cdot r _ {p o s} \right\},
$$
$$
N e g = \{x \in U \mid e (x) \geq n u m \cdot \left(1 - r _ {n e g}\right) \}, \tag {21}
$$
$$
B d y = \{x \in U \mid n u m \cdot r _ {p o s} <   e (x) <   n u m \cdot \left(1 - r _ {n e g}\right) \}.
$$
For tokens in the positive and negative domains, we employ deterministic operations to either maintain them unchanged or merge them into one token. This is because we have determined their redundancy status. For tokens in the boundary domain whose redundancy status has not been determined, we employ undeterministic operation, token match-merge. Token match-merge strikes a balance between maintaining and merging, representing a form of slow merge that only merges similar tokens. If the newly generated boundary tokens are subsequently divided into the positive and negative domains in the next PPSM, they will undergo deterministic operations.