Token-level Correlation-guided Compression for Efficient Multimodal Document Understanding

## Abstract

Abstract Cropping high-resolution document images into multiple sub-images is the most widely used approach for current Multimodal Large Language Models (MLLMs) to do document understanding. Most of current document understanding methods preserve all tokens within sub-images and treat them equally. This neglects their different informativeness and leads to a significant increase in the number of image tokens. To perform a more adaptive and efficient document understanding, we propose Token-level Correlation-guided Compression, a parameter-free and plug-and-play methodology to optimize token processing. Firstly, we propose an innovative approach for assessing the pattern repetitiveness based on the correlation between each patch token. This method identifies redundant tokens, allowing for the determination of the sub-image’s information density. Secondly, we present a token-level sampling method that efficiently captures the most informative tokens by delving into the correlation between the [CLS] token and patch tokens. By integrating these strategies, we develop a plug-and-play Token-level Correlation-guided Compressor module that can be seamlessly incorporated into MLLMs utilizing cropping techniques. This module not only enhances the processing speed during training and inference but also maintains comparable performance.
We conduct experiments with the SOTA document understanding model mPLUG-DocOwl1.5 and the effectiveness is demonstrated through extensive comparisons with other compression methods. The source codes can be found at https://github.com/JiuTian-VL/TokenCorrCompressor .

## 1 Introduction

Document understanding is a vital and complex task that combines computer vision with natural language processing. The challenge arises from dealing with high-resolution document images with diverse aspect ratios, and parsing sparse or dense text with varied formats such as graphic or table. Recently, the rapid development of Multimodal Large Language Models (MLLMs) has demonstrated significant capabilities in image comprehension and instruction following [, , , , , , ]. Some works further enhanced these models by incorporating high-resolution image processing and document parsing capabilities, leading to the development of sophisticated document understanding models [, , , , ].

Despite their impressive achievements, current MLLMs still struggle with efficient document understanding. As shown in Figure 1 (a), these models crop the original high-resolution image into multiple non-overlapping low-resolution sub-images [, , , , ]. A large number of visual tokens are encoded by a vision encoder from all sub-images, and then collectively fed into a Large Language Model (LLM). This paradigm makes it hard for MLLMs to scale up to higher resolution documents as a dramatically growing number of visual tokens have to be dealt with. This significantly hinders the scalability and deteriorates the efficiency of current document understanding MLLMs.

Figure: Figure 1: Comparison between (a) existing pipeline for cropping-based high-resolution processing methods and (b) proposed method. We can adaptively retain informative tokens, making models more efficient.
Refer to caption: x1.png

To efficiently process high-resolution images, it is commonly believed that tokens within sub-images have different degrees of informativeness [, , ], allowing for the compression of sub-images. Therefore, instead of simply feeding all tokens into MLLMs, we could further delve into each sub-image and select the most informative tokens, as shown in Figure 1 (b). This would significantly decrease the number of tokens and contribute to a more efficient document understanding model. Based on the above idea, two challenges arise: 1) how to determine the compression ratio for each sub-image, and 2) how to design a compression strategy for sampling informative tokens.

To address these challenges, it is essential to measure the informativeness of each token. In this paper, we try to leverage the correlation between tokens to reflect the relative degree of informativeness.
Specifically, we propose a Token-level Correlation-guided Compression method and explore the token-level correlations from two aspects, patch-patch and CLS-patch.
1) Using patch-patch correlation to determine the compression ratio. We observe that some tokens within a sub-image exhibit repeated patterns and can be regarded as relatively less informative. To identify such less informative tokens, we investigate the patch-patch correlation to quantify the degree of the pattern repetitiveness regarding each token,
and define the information density of a sub-image by the proportion of highly repetitive tokens.
This information density could be subsequently utilized as the cues to determine compression ratio for each sub-image. 2) Exploiting CLS-patch correlation to sample tokens. The [CLS] token could aggregate and summarize descriptive global information of an image when it exhibits a higher correlation with the informative patch tokens.
Consequently, we might detect and sample the most informative patch tokens based on the correlation between [CLS] and patch tokens. Based on this idea, to effectively sample the most informative tokens, we analyze the CLS-patch correlation and form a probability distribution to guide the sampling process.

Through the guidance of token-level correlation, we construct a plug-and-play high-resolution Token-level Correlation-guided Compressor module. It can be applied as plug-and-play to high-resolution MLLMs that use cropping methods, improving training and inference speed with slight or even no performance loss. We conduct experiments with mPLUG-DocOwl1.5 [], which is the state-of-the-art model in document understanding. Experimental results indicate that the proposed method can maintain comparable performance with DocOwl1.5 while achieving a maximum compression ratio of 11.5%.
Further extensive ablation experiments also verify the effectiveness of the method. Our contributions are summarized as follows:

- •
We conduct an in-depth study of patch-patch correlation to quantify the pattern repetitiveness of tokens and introduce the concept of information density of sub-image. Furthermore, we reveal the distribution of informative tokens with the guidance of CLS-patch correlation and propose a strategy to effectively sample informative tokens based on the distribution.
- •
We propose the Token-level Correlation-guided Adaptive Compression method. It aims to adaptively compress sub-images into different lengths, making document understanding more efficient.
- •
Comprehensive experiments demonstrate that the proposed approach can significantly compress image tokens while maintaining comparable performance. The proposed method achieve an average compression ratio of 66% across different datasets, with a maximum ratio of up to 11.5%, significantly enhancing the model’s efficiency.

## 2 Related Work

Figure: Figure 2: The illustration of the proposed method. (a) The overall architecture. The Token-level Correlation-guided Compressor is inserted between vision encoder and vision-to-text, which comprises two branches, (b) global information mining branch and (c) local information mining branch.
Refer to caption: x2.png

### 2.1 Document Understanding

To enable models to understand document images, a major challenge is the ability to process high-resolution images. There are currently two main methods for processing: one is to use heuristic cropping [, , , ], and another is to crop high-resolution image to a size that can be properly recognized by vision encoder. Pix2Struct [] first proposes a variable-resolution input representation for document understanding. While exhibiting promising ability in high-resolution perception, its language comprehension ability is much limited by the usage of a lightweight language decoder. And its vision encoder needs to be trained from scratch, unable to utilize the existing pre-trained model. To draw these issues, UReader [] further proposes a shape-adaptive cropping module to crop raw images into multiple non-overlapping sub-images with low-resolution that fit the size of pre-trained vision encoder, and conducts initial exploration on fine-tuning document understanding tasks based on MLLMs. Due to the strong high-resolution perception ability and language understanding ability demonstrated by UReader, the cropping-based high-resolution processing method has been widely adopted by subsequent works. For example, Monkey [] employs a sliding window technique to crop image and TextMonkey [] further introduces a shifted window attention mechanism to enable the interaction between different sub-images. mPLUG-Docowl1.5 [] utilizes the same shape-adaptive cropping module as UReader and strengthens the document understanding ability by unified structure learning, achieving state-of-the-art performance. Despite their robust capabilities for document understanding, these models remain significantly inefficient. We propose token-level correlation-guided compression to enhance the efficiency of document understanding in MLLMs.

### 2.2 Token Compression

Introducing high-resolution images into MLLM will significantly increase the number of visual tokens, necessitating methods to reduce the length of visual token sequences for efficient training and inference. Most existing works choose to use vision to text modules with compression capabilities, including using learnable queries along with cross attention mechanism [, , , , , ], convolutional layer with strides  [, ], or simply concatenate the adjacent tokens into a single new token via the channel dimension  [].
Although these methods demonstrate promising token compression ability, when it comes to cropping-based high-resolution MLLMs, they are still not efficient due to the different informativeness of different tokens.
Previous studies have explored the token compression for efficient transformer processing within the fields of natural language processing (NLP)  [, , , ] and computer vision (CV) [, , , , ] independently. However, to our knowledge, investigations into token compression in the multimodal domain remain relatively limited.
Concurrently, Shang et al. [] propose a PruMerge algorithm to adaptively select unpruned visual tokens based on their similarity to class tokens and spatial tokens. However, they did not consider token-level correlations, thereby limiting their adaptive compression capabilities.

## 3 Proposed Method

Figure: Figure 3: Visualization of token similarity. We select tokens corresponding to visually repetitive patches and visualize the similarity between the selected tokens and others. It can be observed that visually repetitive patches exhibit a high degree of similarity between their corresponding tokens.
Refer to caption: x3.png

The overall architecture of the proposed method is presented in Figure 2. Following previous works [, , ], the model begins by cropping high-resolution input image into multiple non-overlapping sub-images that fit the pre-training size of vision encoder. All sub-images along with the resized global image are fed into the vision encoder to get visual tokens. In previous methods, the visual token sequences are aligned with text information via a vision-to-text module. Then all of them would be concatenated with text tokens and collectively fed into LLM for processing, which is extremely inefficient in the face of high-resolution document images.
In this work, we introduce Token-level Correlation-guided Compressor module to adaptively compress visual tokens. The Token-level Correlation-guided Compressor first uses an information density calculation module to adaptively decide the compression ratio of each sub-image, which will be discussed in section 3.1. After that, this module utilizes a correlation-guided token sampling method to sample the most informative tokens, which will be discussed in section 3.2. The workflow of Token-level Correlation-guided Compressor module will be detailed in section 3.3.

### 3.1 Patch-patch Correlation-guided Information Density Calculation

Document images often contain large areas of whitespace and color blocks, which visually present repetitive patterns and can be considered relatively less informative and redundant for understanding the image. To determine an appropriate compression ratio for a sub-image, it is necessary to identify the redundant areas and reflect the proportion of unique regions within the image.

Since patch tokens typically contain local information within the image, we believe that the patch tokens corresponding to visually repetitive patches are highly correlated. This inspires us to explore patch-patch correlation for identifying redundant tokens.
Specifically, we utilize the keys of attention to represent each token, facilitating the calculation of similarity between pairwise token []. As shown in Figure 3, it can be observed that the tokens corresponding to visually repetitive patches in the image have many highly similar counterparts, which verifies our hypothesis. This finding enables the differentiation between redundant patch tokens from others.

Based on this finding, we design a method to adaptively calculate the proportion of non-redundant tokens in a sub-image, termed information density. Specifically, we calculate the cosine similarity between pairwise tokens based on the keys of attention in CLIP-ViT. For a given token, if the number of tokens with a similarity greater than the threshold exceeds an upper limit, it will be regarded as redundant. We calculate the proportion of redundant tokens in the sub-image to the total number of tokens as the information redundancy, noted as $r$, and $d=1-r$ represents the information density. Finally, we treat the informative density as the compression ratio of each sub-image. The patch-patch correlation-guided information density calculation is detailed in Algorithm 1.

### 3.2 CLS -patch Correlation-driven Informative Token Sampling

In CLIP-ViT [], the [CLS] token is introduced to aggregate information from each patch token to create a representation of the entire image.
There should be a higher correlation between the [CLS] token and the informative tokens.
As the aggregation process is achieved through the attention mechanism, the attention map can inherently indicate such correlation.
Therefore, to measure the informativeness of each token, we could interpret the attention scores as a quantification of the relative informativeness of patch tokens, with higher attention scores indicating that a patch token is relatively more informative.

Figure: Figure 4: Visualization of the attention maps between the [CLS] token and patch tokens across different layers of CLIP-ViT-L. (a) Original input image. (b) The attention maps from layers 1 to 12. (c) The attention maps from layers 13 to 24.
Refer to caption: x4.png

To unveil the underlying patterns of the CLS-patch correlation presented by the attention map, we conduct an in-depth study of the attention maps across different layers of CLIP-ViT. As shown in Figure 4, the low layer attention maps highlight the visually informative regions in the image, aligning with the consensus that patch tokens hold local information. Nevertheless, the attentions in deep layers exhibit several irregular outliers.
This phenomenon can be attributed to the model’s tendency to gradually integrate image information at multiple locations, facilitating the aggregation of global information into the [CLS] token. As suggested in [], the outlier can be considered to hold global information while containing minimal local information.
These distribution patterns of CLS-patch correlation in both deep and low layers inspire a potential approach to identify tokens that contain both global and local information. More visualization results can be found in Appendix C.1.

To effectively sample informative tokens from sub-images, we design methods that respectively sample tokens with the guidance of CLS-patch correlation from both deep and low layers.

Leveraging the CLS-patch correlation from deep layers, we adopt the Interquartile Range (IQR) method to distinguish outlier tokens [] for preserving informative tokens with global information. Specifically, IQR calculates the first quartile (Q1) and the third quartile (Q3) in a set of numbers. It considers values that are 1.5 times the IQR above Q3 as upper outliers and values that are 1.5 times the IQR below Q1 as lower outliers. As a typical outlier recognition method, IQR can adapt well to different data distributions and determine an adaptive number of outliers. We preserve tokens identified as upper outliers, thereby retaining informative tokens with global information.

Figure: Algorithm 1 Information Density Calculation

To further preserve informative tokens with local information, we quantify the relative informativeness of different tokens through the attention map from low layer. Since the attention map displays relative relationships, directly setting a threshold to determine informative tokens is not feasible.
Instead, we treat the attention score as the probability of sampling each token and perform sampling without replacement. This method ensures that more informative tokens are more likely to be selected.

### 3.3 Token-level Correlation-guided Compression

We propose a plug-and-play Token-level Correlation-guided Compressor module by integrating the information density calculation and informative tokens sampling. As shown in Figure 2, the module consists of two parallel branches, the global token mining branch and the local token mining branch.
The global token mining branch focuses on selecting and preserving tokens that contain global information. It utilizes the CLS-patch correlation in the deep layers of CLIP-ViT to adaptively retain an appropriate number of tokens through IQR.
Meanwhile, the local token mining branch aims to sample tokens that hold local information. To determine the sampling ratio adaptively, we calculate the information density of the sub-image with the guidance of patch-patch correlation and use it as the sampling ratio. To effectively sample the most informative tokens, the CLS-patch correlation in the low layers of CLIP-ViT is leveraged to form the sampling distribution. Consequently, tokens are sampled without replacement based on the token-level correlation-guided sampling ratio and distribution.
All sampled tokens are then concatenated as the preserved tokens.

To prevent the unintended discarding of important information, we adopted a token aggregation method as an alternative to directly discarding unsampled tokens [, ]. Specifically, this method utilizes keys of attention to calculate the similarity between tokens as a distance metric. Each sampled token is grouped with other tokens using k-nearest neighbors. Then the representations of each group are updated through a weighted sum, with attention scores serving as the weights. A similar approach has also been employed in a concurrent study []. Finally, the aggregated tokens are fed into the vision-to-text module and LLM as a compressed input. Specifically, to keep the overall information of the image, we do not compress the global image. The complete procedure of the Token-level Correlation-guided Compression is detailed in Algorithm 2.

Through the token-level correlation-guided compression, all sub-images are compressed into different lengths adaptively. The method maximally retains information while concurrently minimizing the length of the visual token sequence for all sub-images, thus significantly enhancing models’ efficiency.

Figure: Algorithm 2 Token-level Correlation-guided Compression

## 4 Experiments

We conduct experiments with mPLUG-DocOwl1.5 [].
For more implementation details, please refer to Appendix A.

### 4.1 Experimental Results

We evaluate the proposed method on 10 datasets based on previous experiments, including text-rich datasets like DocVQA [], InfoVQA [], DeepForm [], KLC [], table
datasets like WTQ [], TabFact [], chart datasets like chartQA [], natural datasets like TextVQA [], TextCaps [], and webpage
screenshots dataset VisualMRC []. As shown in table 1, we compare the results with the previous OCR-free method [, , , , , ] and also compare with some other token compression methods []. Our method performs better than many previous OCR-free methods.
Compared with the base model DocOwl1.5, in plug-and-play mode, the proposed method achieves a comparable performance, while both pruMerge and pruMerge+ [] can lead to significant performance degradation.
Our method outperforms many previous OCR-free approaches. In comparison to the base model DocOwl1.5, the proposed method achieves comparable performance with an average compression ratio of 66% in plug-and-play mode. Meanwhile, both pruMerge and pruMerge+ [] result in significant performance degradation. After finetuning 1 epoch, we can further narrow the performance gap with DocOwl.5.

**Table 1: Comparison with existing document understanding models. The scores marked with an underline represent the state-of-the-art (SOTA) performance. Although the proposed method exhibits a slight degradation compared to the SOTA, it achieves an average compression rate of 66% across various datasets, enhancing efficiency significantly. * The methods of PruMerge and PruMerge+ are both tested with DocOwl1.5 in plug-and-play mode.**
| Model | Doc<br>VQA | Info<br>VQA | Deep<br>Form | KLC | WTQ | Tab<br>Fact | Chart<br>QA | Text<br>VQA | Text<br>Caps | Visual<br>MRC |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Without compression |  |  |  |  |  |  |  |  |  |
| DocPeida [] | 47.1 | 15.2 | - | - | - | - | 46.9 | 60.2 | - | - |
| Monkey [] | 66.5 | 36.1 | 40.6 | 32.8 | 25.3 | - | - | 67.6 | 93.2 | - |
| DocOwl [] | 62.2 | 38.2 | 42.6 | 30.3 | 26.9 | 60.2 | 57.4 | 52.6 | 111.9 | 188.8 |
| UReader [] | 65.4 | 42.2 | 49.5 | 32.8 | 29.4 | 67.6 | 59.3 | 57.6 | 118.4 | 221.7 |
| CogAgent [] | 81.6 | 44.5 | - | - | - | - | 68.4 | 76.1 | - | - |
| TextMonkey [] | 71.5 | - | 61.6 | 37.8 | 30.6 | - | 65.5 | 68.0 | - | - |
| DocOwl1.5 [] | 81.6 | 50.4 | 68.8 | 37.9 | 39.8 | 80.4 | 70.5 | 68.8 | 132.0 | 239.5 |
|  | Compression without considering token-level correlation |  |  |  |  |  |  |  |  |  |
| *Prumerge [] | 53.6 | 29.6 | 9.3 | 28.3 | 23.7 | 71.4 | 55.8 | 60.0 | 120.7 | 125.9 |
| Prumerge+ [] | 55.0 | 33.2 | 26.7 | 30.8 | 24.6 | 70.9 | 58.3 | 62.3 | 124.8 | 185.0 |
|  | Adaptive Compression guided by token-level correlation |  |  |  |  |  |  |  |  |  |
| Ours (plug-and-play) | 72.6 | 49.6 | 63.2 | 34.6 | 35.2 | 75.2 | 64.0 | 68.0 | 132.1 | 254.3 |
| Ours (finetuning) | 78.3 | 50.2 | 65.7 | 35.9 | 38.6 | 79.3 | 68.9 | 66.6 | 125.9 | 243.7 |

We further investigate the token compression ratio of different adaptive compression methods on various datasets. Each cropped sub-image is treated as an individual sample, and we calculate the compression ratio for all sub-images. Here, the compression ratio is defined as the number of tokens after compression divided by the original number of tokens. A lower compression ratio indicates a more effective compression. As shown in Figure 5 and Figure 6, on different datasets, the compression ratios resulted from PruMerge and PruMerge+ algorithms remain within a relatively fixed interval, while the proposed method exhibits significantly different compression ratios on different datasets. This result demonstrates that our algorithm can adaptively recognize the information distribution patterns of different datasets and identify the most appropriate compression ratio.
More results of the compression ratio on different datasets can be found at Appendix B.2. As a summary, the proposed method achieves an average compression ratio of 66% across different datasets, with a maximum ratio of up to 11.5%, significantly enhancing the model’s efficiency.

### 4.2 Ablation Study

We perform a comprehensive ablation study to validate the effectiveness of CLS-patch correlation-guided token sampling, information density-based sampling ratio and global/local information mining. These experiments are conducted in plug-and-play mode for comparison.

Effectiveness of CLS-patch correlation-guided token sampling. For the sampling strategy in local information mining, we compare the CLS-patch correlation-guided sampling with uniform and random sampling under the same sampling ratio. As shown in Table 3, the token correlation-guided sampling significantly outperforms both uniform and random sampling.

Figure: Figure 5: Boxplot visualization of the compression ratio achieved by different compression methods across various datasets. The median numbers is presented adjacent to the boxes.
Refer to caption: x5.png

Figure: Figure 6: Histogram visualization of the compression ratio achieved by different compression methods across various datasets.
Refer to caption: x6.png

Figure: Figure 7: Comparisons between our adaptive compression ratio and fixed compression ratios. We set the sampling ratio at multiple fixed values and tested these on various datasets. Despite the higher average compression ratios resulted by fixed ratios, the evaluation scores are unable to surpass those of our adaptive compression method.
Refer to caption: x7.png

Effectiveness of information density calculation. We also conduct another set of experiments to verify the effectiveness of our adaptive sampling ratio. We set a group of fixed sampling ratios of $\frac{2}{3}$, $\frac{1}{2}$ and $\frac{1}{3}$ in local information mining for comparison. As shown in Figure 3, for a fixed sampling ratio setting, even though the fixed sampling ratio settings retain more tokens on average, its performance still cannot surpass our adaptive sampling ratio method.

Effectiveness of global and local information mining.
In Table 3, we conduct ablation experiments on two modules: global information mining and local information mining. As shown in table 3, simply removing any part here will result in performance degradation.

In order to validate that attention maps from low layers of CLIP-ViT can effectively guide the sampling of informative tokens, we conduct experiments to examine the selection of attention maps from different layers in local information mining. The results can be found in Appendix B.3.

### 4.3 Visualization

**Table 2: Comparison between different methods in local information mining branch.**
| Sampling Method | Deep<br>Form | Doc<br>VQA | Visual<br>MRC |
| --- | --- | --- | --- |
| Random | 55.8 | 69.1 | 245.4 |
| Uniform | 56.6 | 70.0 | 248.8 |
| Ours | 63.2 | 72.6 | 254.3 |

Figure: Figure 8: Visualization of redundant patch token. We visualize the redundant tokens identified during the information density calculation process, as represented by the unmasked areas. The calculated information density is highlighted at the bottom right of each sub-image. It can be observed that the identified redundant tokens are all concentrated in the parts that visually present repetitive patterns.
Refer to caption: x8.png

Figure: Figure 9: Visualization of selected token. The blue parts represent the tokens preserved by global information mining, while the orange parts represent the sampled tokens in local information mining.
Refer to caption: x9.png

To intuitively verify the effectiveness of the proposed method, we conduct several visualization experiments.
We first visualize the redundant tokens identified during information density calculation. As shown in Figure 8, the unmasked areas indicate the locations of identified redundant tokens, which are all concentrated in the visually repetitive regions. These results validate the effectiveness of our information density calculation method.
Simultaneously, the calculated information density accurately reflects the relative degree of informativeness of different sub-images, thereby demonstrating the effectiveness of our adaptive compression capability.
More visualization results of the information density calculation on different datasets can be found at Appendix C.2.

We also visualize the tokens sampling results achieved by the CLS-patch correlation-guided sampling method. Several samples with different distribution patterns are selected to verify the effectiveness of our method. In Figure 9, it can be observed that tokens sampled by local information mining are mainly concentrated in the area of informative areas, which verifies the effectiveness of our method. More results can be found at Appendix C.3.

## 5 Conclusion

In this paper, we present a token-level correlation-guided compression method to enhance document understanding efficiency in MLLMs. Experimental results show significant token sequence length reduction while maintaining performance comparability. The proposed method still has some limitations, including the necessity for fine-tuning the model to minimize the performance disparity with the base model. Additionally, it inhibits the capability for end-to-end learning. We hope to address these issues in future work.

## Appendix A Implementation Details

We conduct experiments with the mPLUG-DocOwl1.5 model [], which utilizes the CLIP-ViT/L-14 [] as the vision encoder and 7B LLaMA-2 model [] with the Modality Adaptive Module [] as the language decoder. We keep the adaptive cropping method in mPLUG-DocOwl1.5 with a fixed resolution of 448$\times$448. The similarity threshold $\alpha$ and upper limit $k$ in information density calculation is set to $0.7$ and $50$, respectively. We choose the last layer and the eighth layer to guide global and local information mining. Note that our method is a parameter-free token compression method. Thus we mainly conduct our experiment in the plug-and-play manner. We also further finetune mPLUG-Docowl.5 with our method using LoRA [] and DocDownstream dataset [] for 1 epoch. The initial learning rate for fine-tuning is set to 1e-4, with a batch size of 256. The vision-to-text module is frozen, and only the LoRA parameters are trained. We conduct experiments on a server equipped with four Nvidia A800 GPUs, each with 80GB of VRAM.

## Appendix B More Quantitative Experiments

### B.1 Statistical Significance Analysis

To mitigate the impact of randomness introduced by the sampling used in the proposed method, we conducted three repeated experiments across multiple datasets. As shown in Table 4, the random errors introduced by sampling are minimal. Even accounting for these errors, the method still demonstrates significant superiority over PruMerge and PruMerge+ [].

**Table 4: Replicate experiments for mitigating the impact of sampling.**
| Experiment | DeepForm | KLC | WTQ | ChartQA |
| --- | --- | --- | --- | --- |
| PruMerge | 9.33 | 28.25 | 23.67 | 55.84 |
| PruMerge+ | 26.72 | 30.84 | 24.55 | 58.28 |
| Ours | 62.80$\pm$0.29 | 34.54$\pm$0.02 | 35.32$\pm$0.17 | 64.38$\pm$0.39 |

### B.2 Comparison to the Compression Ratio

In this section, we present more results of the statistically adaptive compression ratio across additional datasets. As shown in Figure 10 and Figure 11, the PruMerge and PruMerge+ algorithms maintain relatively fixed compression ratios across various datasets. In contrast, our method adapts the compression rate based on the distribution pattern of the data, resulting in significant variations across different datasets.

### B.3 Ablation of Layer Selection

To demonstrate that attention maps from low layers of CLIP-ViT can effectively guide the sampling of informative tokens, we conduct experiments to evaluate the use of attention maps from different layers in local information mining. Specifically, for low layers, we select layers 4 and 8, and for deep layers, we select the 16th and 24th layers. As shown in Table 5, the performance of selecting attention maps from the 4th and 8th layers outperforms that of the 16th and 24th layers, demonstrating that low layer attention maps more accurately measure the distribution of informative tokens containing local information. Additionally, the superior performance of the 8th layer compared to the 4th suggests that shallower layers are not always better. This may be due to the inability of tokens in the very shallow layers to form descriptive representations. Moreover, the performance drops significantly when selecting the 24th layer compared to the 20th layer, further indicating that deep layers tend to describe the distribution of tokens with global information.

**Table 5: Comparisons of choosing different layers to guide local information mining**
| Layer | DeepForm | DocVQA | VisualMRC |
| --- | --- | --- | --- |
| 4 | 61.6 | 71.5 | 251.7 |
| 16 | 60.9 | 70.9 | 250.8 |
| 24 | 52.9 | 66.8 | 247.0 |
| 8 | 63.2 | 72.6 | 254.3 |

Figure: Figure 10: Boxplot visualization of the compression ratio achieved by different compression methods across additional datasets. The median numbers is presented adjacent to the boxes.
Refer to caption: x10.png

Figure: Figure 11: Histogram visualization of the compression ratio achieved by different compression methods across additional datasets.
Refer to caption: x11.png

## Appendix C More Visualization Results

### C.1 Attention Maps Across Different Layers of CLIP-ViT

We show more visualization results of attention maps across different datasets, including various types of tables, text-rich, and charts. As depicted in Figure 12, the attention maps generated by CLIP-ViT exhibit the same distribution patterns in various types of data, which further corroborates our findings.

Figure: Figure 12: More visualization results of attention maps in CLIP-ViT.
Refer to caption: x12.png

Figure: Figure 13: More visualization results of identified redundant tokens in additional datasets with various distributions.
Refer to caption: x13.png

### C.2 Redundant Patch Token Visualization

In this section, we will present more visualization results about redundant patch tokens. As shown in Figure 13, the information density calculated by our method is consistent with the visual perception.

### C.3 Selected Informative tokens Visualization

In Figure 14, there are more results regarding the informative tokens selected with the guidance of CLS-patch correlation.

Figure: Figure 14: More visualizations of CLS-patch correlation-based token selection across various datasets with different distributions.
Refer to caption: x14.png

## Appendix D Broader Impact and Potential Risk

Our approach employs readily available Multimodal Large Language Models (MLLMs), which means it shares some of their limitations, including the production of biased results. We recommend thoroughly evaluating its safety and fairness for the intended use before applying this method in practice.

## Peer Reviews

confidence: 4
contribution: 1
initial_rating: 3
presentation: 2
questions: Q1: Figure 8 and 9 suggests that the proposed method is primarily good at identifying the background pixels as uninformative. Would document background detection perform better as a baseline to detect and remove uninformative patches before feeding into MLLM? 
Q2: Can the proposed method be applied beyond document understanding? Is there an overarching theme?
soundness: 2
strengths: - Two simple and intuitive heuristics providing non-trivial speed improvements for MLLM models. 
- Multiple ablations are performed to justify the design decisions
summary: The paper proposes multiple heuristics to improve the speed of Multi-Modal Large Language Models for document understanding. This is achieved by discarding uninformative tokens before feeding into the MLLM. For this purpose, the paper also comes up with a heuristic definition of "information density". The proposed method speeds up one of the state of the art models (DocOW11.6) by 13% in average (in exchange of minor performance degradation).
weaknesses: - The paper proposes "incremental" (13%) efficiency improvements for the state of the art MLLM document understanding models in exchange for non-zero model accuracy regressions. 
- The method requires fine-tuning of target MLLM model for the best results. Otherwise, the trade-off between model accuracy loss and inference speed improvement is not justifiable. 
- Theoretical justification for the proposed "information density" method is absent. 
- Limitations of the proposed method needs to be discussed in depth.

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 3
questions: --  In Section A.2.3, the paper describes the results of Local Information Mining when selecting different layers. How does the performance of Global Information Mining vary with the selection of different layers?
soundness: 3
strengths: The paper introduces the idea of information density based on patch-patch correlations, defining the information density of sub-images as the proportion of non-redundant tokens. This is  then used to determine the token compression ratios adaptively. The proposed idea  has some technical values.
summary: This paper proposes a Token-level Correlation-guided Compression (TCC) method for dynamic visual token compression, aiming to improve the efficiency of Multimodal Large Language Models (MLLMs) in document understanding tasks. The method compute the information density through patch-patch correlations, leveraging token correlations to guide the compression process in document understanding tasks, aiming to reduce the number of visual tokens while maintaining model performance.  Experiments with mPLUG-DocOWL1.5 shows the effectiveness of the proposed method.
weaknesses: 1. The innovation of the proposed method is limited and not strong. Analyzing correlations or importance between visual tokens for compression of vision tokens has been explored in many recent researches. The simple idea of analyzing correlations between image patches and using [CLS] token correlations with patch tokens to sample the most informative tokens is straightforward and doesn't bring many new insights to the field.

2. In Table 1, the paper only compares with two token compression methods, which is not sufficiently convincing. There are several recent dynamic visual token compression methods, but the authors did not compare with these methods, such as FastV and more: 
   (1)  Chen, L. et al.,  An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models. arXiv preprint arXiv:2403.06764. 
   (2)  Lin, Z.; Lin, M.; Lin, L.; and Ji, R. 2024. Boosting  Multimodal Large Language Models with Visual Tokens Withdrawal for Rapid Inference arXiv preprint arXiv:2405.05803.
   (3). Zhang J. et al., Dockylin: A large multimodal model for visual document understanding with efficient visual slimming, arXiv preprint arXiv:2406.19101 
    ...

3. While the authors claim their method is plug-and-play and can be seamlessly integrated into existing MLLMs using the proposed techniques, they only validated it on mPLUG-DocOWL1.5. The lack of experiments on other document understanding multimodal models (especially some recent SoTA models) makes it unclear whether the method is universally effective,  or just works well for mPLUG-DocOWL1.5. The experiments are not  solid and convincing enough.
 
4. The compression effect is limited, achieving only 11.5% compression on mPLUG-DocOWL1.5. This compression ratio is not impressive compared to other recent multimodal model token compression methods.

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 2
questions: Does compressing tokens also offer advantages in terms of memory consumption? If so, the author should present this information. The proposed method does not seem to be designed specifically for documents, so why were experiments only conducted on document understaning benchmarks? Why not conduct experiments on more general multimodal understanding as well?
soundness: 3
strengths: The Token-level Correlation-guided Compression filters out the truly informative parts from the visual tokens through patch-patch correlation-guided information density calculation and cls-patch correlation-guided informative token sampling, accelerating the model's training and inference speed.
summary: This paper focus on the token compression for Multimodal Large Language Models. The authors propose Token-level Correlation-guided Compression. By calculating the information density of sub-images and efficiently capturing the most informative tokens through the correlation between the [CLS] token and patch tokens, the authors have developed a plug-and-play compression module to remove redundant tokens produced by existing image cropping methods, accelerating the model's training and inference speed.
weaknesses: 1.Although the authors' method compresses visual tokens and improves the speed of model training and inference, it results in a noticeable performance drop, and the speed improvement is not outstanding. 

2.The author claims that the proposed method is plug-and-play, but they only conducted experiments on the DocOwl-1.5 model, which is not very convincing. The proposed method needs to be validated on more models, especially on some higher-performance models such as InternVL2 and QWEN2VL.

3.Lacking comparison with some of the latest token compression methods, such as FastV, which has been accepted by ECCV.

Chen L, Zhao H, Liu T, et al. An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models[J]. arXiv preprint arXiv:2403.06764, 2024.

4.Lacking results on the commonly used benchmark OCRBench.

Liu Y, Li Z, Yang B, et al. On the hidden mystery of ocr in large multimodal models[J]. arXiv preprint arXiv:2305.07895, 2023.

5.In Line 139, the authors claim that "Despite their robust capabilities for document understanding, these models remain significantly inefficient." However, in the experimental tables, the author does not compare the efficiency with other methods. It would be better to include a comparison of efficiency with other methods in Table 1.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: Compared to PruMerge+, what are the advantages of the token compression method proposed in this paper?
soundness: 2
strengths: Topics are current and important, focusing on compressing redundant information to improve model inference efficiency. Experimental results show that the proposed method outperforms existing solutions in terms of token compression efficiency and performance retention.
summary: This paper focuses on document understanding. Most current methods ignore the different informativeness of the tokens in the sub-images, leading to a significant increase in the number of image tokens. Hence, the authors propose a plug-and-play Token-level Correlation-guided Compression module, which evaluates the repeatability and informativeness based on the correlation between image tokens. Experiments demonstrate the effectiveness of this module.
weaknesses: 1) The role of the prompt is overlooked. Given that token compression is based entirely on images, it remains unclear how to ensure that the information required for the prompt is retained. It would be better to compare CLS patch-based and prompt-based approaches to token compression guidance.
2) The effectiveness of the plug-and-play approach requires further validation. It would be beneficial to demonstrate results on more baselines.
3) The analysis of experiments needs to be enriched. For example, why the model fine-tuned on TextVQA, TextCaps and VisualMRC datasets is not as good as plug-and-play instead.
4) There is an absence of direct comparisons with other token compression methods. It would be better to evaluate the performance and efficiency of the proposed method alongside existing token compression methods under consistent baselines.
5) The effectiveness of the Global Info Token requires additional validation. As shown in Figure 9, the Global Info Token is not consistently positioned in information-rich areas. Expanding the results presented in Table 4 to include more datasets could provide a more comprehensive assessment.