SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Expedite employs K-means clustering to generate super-pixel features and aims to preserve the relationship between the super-pixel features and original tokens. Since the recovery for each token relies on several super-pixel features, the recorded relationships become complex, and the similarity between the removed token and corresponding super-pixel features is relatively low. In contrast, AiluRus selects representative tokens from the original ones and aggregates the remaining tokens into the nearest representative tokens, which allows AiluRus to recover tokens only relying on the corresponding aggregated token. Therefore, the relationship preserved by AiluRus is simpler, and the similarity between the removed token and the aggregated token is higher. To empirically verify this, we conducted a study on the similarity between the original tokens and the super-pixel features in Expedite, as well as the similarity between the aggregated tokens and the original tokens in AiluRus during the inference process. Specifically, we run Expedite and AiluRus with the same configuration and report the average similarity over the ADE20K validation set. The results confirm that the average similarity in Expedite is 0.6728, while in AiluRus, it is 0.9210, aligning with our expectations.
We further investigate the probability of token pairs in different similarity intervals to maintain their respective intervals at the output end. A higher probability indicates robust preservation of the relationship within that interval, while a lower probability suggests a higher propensity for distortion. Experiments are conducted on the entire validation set of ade20k, and the results are recorded in Tab. 12. The results demonstrate that token pairs with a similarity between 0.9 and 1.0 have a probability of over  $99\%$  of maintaining this similarity at the output layer. However, token pairs in other similarity intervals suffer from significant similarity distortion. This finding elucidates why Expedite experiences serious distortion while AiluRus consistently achieves low reconstruction errors across layers.
E Latency Analysis
As our motivation is to expedite vision transformers, it is necessary to ensure that the operations introduced by AiluRus are sufficiently efficient, avoiding significant latency that slows down the whole model. To investigate the latency brought by AiluRus, we decompose the running time of models into three components, i.e., time for transformer blocks, time for clustering, and time for recovering, and compare the elapsed time of Expedite and AiluRus. For fair comparisons, we run them under the same configuration over ViT-L and ViT-B respectively. For ViT-L, we start clustering at the 4-th layer and set the number of clusters as 576. For ViT-B, we start clustering at the 2-nd layer and set the number of clusters as 400. We report the average latency of each component on the ADE20K validation set in Tab. 13.
Table 13: The latency analysis. The percentage in parentheses indicates the percentage of latency occupied by the current operation.
Methods	Total (ms)	blocks (ms)	clustering (ms)	recovering (ms)	mIoU
Results on ViT-L
Expedite [15]	31.4	26.5 (84.4%)	3.9 (12.4%)	1.0 (3.2%)	47.19
AiluRus	26.6	25.3 (95.1%)	1.2 (4.5%)	0.09 (0.3%)	50.27
Results on ViT-B
Expedite [15]	11.1	8.4 (75.7%)	2.1 (18.9%)	0.6 (5.4%)	5.21
AiluRus	10.0	8.7 (87.0%)	1.2 (12.0%)	0.08 (0.8%)	47.35
The analysis results indicate that AiluRus exhibits significantly lower clustering and recovery costs compared to Expedite. As we aforementioned, Expedite replaces raw tokens with super-pixel features for forward passes. Consequently, Expedite relies on multiple iterations to achieve sufficiently accurate super-pixel features and necessitates the computation of reconstruction coefficients for recovering the original features. Thus, Expedite consumes more time in both clustering and reconstruction. In contrast, AiluRus directly operates tokens in the original feature space and thus could efficiently generate token assignments in a single iteration. The produced assignments can be directly used for subsequent reconstruction, thereby avoiding additional recovery costs.
F Prediction Visualizations
Figure 9: The prediction visualizations of the original Segmenter ViT-L model and the AiluRus integrated one.
We conducted a comparative analysis of the prediction results between AiluRus and the original model in order to elucidate the factors contributing to the performance improvement of AiluRus in Tab. 5. Specifically, we visualize the predictions generated by both AiluRus and the original models over randomly selected ADE20K [37] images. As illustrated in the figure, the predictions generated by AiluRus exhibit enhanced smoothness, thereby avoiding undesirable predictions such as voids or spikes. Consequently, in certain scenarios, AiluRus demonstrates superiority over the original model. This improved smoothness can be attributed to the application of the adaptive resolution strategy. By introducing a smoothness constraint in regions with low information density, such as backgrounds,
this strategy effectively incorporates a prior for generating smooth predictions in these areas. When retained an adequate number of tokens, this prior aligns well with the true information distribution of the images, leading to performance improvements.
G TCFormer for Dense Prediction.
Both TCFormer [32] and AiluRus employ DPC clustering. However, they have very different motivations and implementations. TCFormer is designed for human-centric tasks while AiluRus focuses on expediting dense prediction tasks. In this section, we show that it is untrivial to apply TCFormer for dense prediction tasks.
Table 14: The latency analysis for TCFormer and AiluRus.
Method	Total (ms)	blocks (ms)	extra (ms)	mIoU
TCFormer	33.8	8.6 (25%)	25.2 (75%)	0.14
AiluRus	10.0	8.7 (87%)	1.3 (13%)	47.35
Table 15: The performance comparison between AiluRus and ViT-CTM.
Method	Num clusters	FPS	mIoU
Baseline	1024	32.2	49.60
ViT-CTM	410 (1024 * 0.4)	29.1 (↓ 10%)	33.48 (↓ 16.12)
AiluRus	400	53.7 (↑ 67%)	49.04 (↓ 0.56)
Initially, we intend to integrate TCFormer into the mmseg framework for semantic segmentation. We carefully implement the TCFormer under the mmseg framework according to the officially released code. However, we find that TCFormer is an extremely computationally intensive backbone. Even with its lightest configuration, the FPS for inferring ADE20K images at a resolution of 512x512 under the Segmenter framework is only 2.03. In comparison, a typical ViT-Base model achieves an FPS of 32.2, while AiluRus can surpass 50 without sacrificing performance. Upon analyzing the TCFormer code, we discover that it heavily relies on sparse matrix multiplication. These operations may not have high FLOPs yet introducing significant latency. Therefore, it is nearly impossible and impractical to apply TCFormer for dense prediction tasks given its high computational complexity.
Furthermore, we attempt to integrate the core TCFormer design, the CTM module, into ViT for acceleration. This would allow us to compare the performance of TCFormer with AiluRus using the same backbone and initialization. However, we encountered the same issue of excessive complexity for the CTM module. In a similar setup, the time required for the CTM module to execute is even higher than the inference time of the model itself. The details are presented in Tab. 14:
Despite these issues, we proceeded to train the ViT model with the integrated CTM module in the Segmenter framework for comparison with AiluRus. We followed the instructions provided in the TCFoRer paper to set up the CTM module and trained ViT-CTM with the same optimization parameters and initialization as AiluRus. However, as demonstrated in Tab. 15, we find that even with only the integration of CTM into ViT, it raises a significant negative impact on training and causes very poor performance. Thus, it is untrivial to integrate the CTM module into ViTs for acceleration.
In conclusion, TCFormer is specifically designed for human-centric tasks, and adapting it to dense prediction tasks poses significant challenges. Additionally, TCFormer exhibits high computational complexity, rendering it unsuitable for accelerating ViTs. In contrast, AiluRus can be seamlessly integrated into well-trained models, providing immediate acceleration or expediting training without the need for additional hyper-parameter adjustments. These two methods differ significantly in terms of motivation, implementation, and application. AiluRus stands out for its flexibility and lightweight nature, enabling its deployment in various tasks that TCFormer is incapable of addressing.