LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Models

## Abstract

Abstract Large Multimodal Models (LMMs) have shown significant visual reasoning capabilities by connecting a visual encoder and a large language model.
LMMs typically take in a fixed and large amount of visual tokens, such as the penultimate layer features in the CLIP visual encoder, as the prefix content.
Recent LMMs incorporate more complex visual inputs, such as high-resolution images and videos, which further increases the number of visual tokens significantly.
However, due to the inherent design of the Transformer architecture, the computational costs of these models tend to increase quadratically with the number of input tokens.
To tackle this problem, we explore a token reduction mechanism that identifies significant spatial redundancy among visual tokens. In response, we propose P ruMerge , a novel adaptive visual token reduction strategy that significantly reduces the number of visual tokens without compromising the performance of LMMs.
Specifically, to metric the importance of each token, we exploit the sparsity observed in the visual encoder, characterized by the sparse distribution of attention scores between the class token and visual tokens.
This sparsity enables us to dynamically select the most crucial visual tokens to retain.
Subsequently, we cluster the selected (unpruned) tokens based on their key similarity and merge them with the unpruned tokens, effectively supplementing and enhancing their informational content.
Empirically, when applied to LLaVA-1.5 [ Liu et al. , 2023a ] , our approach can compress the visual tokens by 14 times on average, and achieve comparable performance across diverse visual question-answering and reasoning tasks.

## 1 Introduction

Large Language Models (LLMs) [OpenAI, , Team et al., , Jiang et al., , Touvron et al., ] have shown strong reasoning abilities. LLMs are usually high-capacity Transformers [Vaswani et al., ] pretrained with a large-scale text corpus. Large Multimodal Models (LMMs), inherit LLMs for text generation, while also leveraging a visual encoder such as CLIP-ViT [Radford et al., ] to embed image patches into visual tokens as the prefix visual context.

LMMs need substantial computation for inference. The LLM is the primary factor for the high computation cost, since the visual encoder is usually quite small relative to the LLM. For example, the commonly used CLIP visual encoder, ViT-L, only has 0.3B parameters, while the corresponding LLM such as LLaMA [Touvron et al., ] or Vicuna [Vicuna, ] can have 7B or 13B parameters.
As a result, reducing the LLM’s inference cost is the key to achieving low LMM inference cost.

Prior works [Chu et al., , , Yuan et al., ] mainly focus on replacing the LLM backbone with a smaller language model with less parameters, such as Phi-2 [Javaheripi et al., ].
However, such approaches sacrifice the reasoning abilities of LLMs, leading to a large performance gap on visual question-answering and reasoning tasks such as VQAv2 and MM-Bench [Chu et al., ]. A similar approach is to apply quantization for LLMs [Liu et al., , Yuan et al., ].

However, the cost of LLMs comes from not only its large number of parameters, but also the *length of the input context* due to the quadratic complexity of the Transformer’s attention operation. The context length in LMMs is especially important, where a fixed amount of visual tokens serves as the prefixed tokens. For example, in LLaVA-1.5, 576 visual tokens are appended, and in Video-LLaVA [Lin et al., ] that number is even higher, leading to high training and inference costs. Thus, an intriguing question is: Can we reduce the number of prefix visual tokens while maintaining comparable performance?

Figure: (a) Main idea of PurMerge.

In our study, we find that many visual tokens are redundant, similar to findings in previous related work [Bolya et al., , Liu et al., ], and most of the visual tokens can be pruned with little sacrifice in performance.
In particular, the similarity (i.e., attention scores in the visual encoder’s self-attention module) between the class token and spatial patches are sparse, indicating that only a small number of visual tokens are related to key visual information in the visual samples.
Motivated by this, we use this sparse similarity to adaptively select important visual tokens, as shown in Fig.1(b).
Specifically, we leverage the Interquartile Range (IQR) [Boukerche et al., ] scoring function in outlier detection to prune unimportant visual tokens. Moreover, we merge the visual tokens using $k$-nearest neighbor and update the selected important visual tokens via weighted averaging, which further enhances performance.
Finally, we design PruMerge+, which samples visual tokens spatial-uniformly to complement the unpruned tokens.
PruMerge+ not only minimizes performance degradation but also ensures substantial token reduction, maintaining a more comprehensive and representative selection of visual tokens.

Empirically, PruMerge can effectively and adaptively reduce the visual tokens in each image in LLaVA-1.5 [Liu et al., ], where with just 5.5% of visual tokens, which is around 32 tokens for an image on average, LLaVA-PruMerge can maintain comparable performance with that of retaining all 576 tokens across diverse benchmarks.
Furthermore, PruMerge showcases its versatility across various modalities, including video. By integrating PruMerge with Video-LLaVA during the inference phase alone—eliminating the need for additional training—we not only expedite processing within video-LLMs but also enhance their performance across multiple benchmarks.

## 2 Related Work

### 2.1 Efficient Large Multimodal Models (LMMs)

Large Language Models (LLMs) such as GPT-4 [OpenAI, ], LLaMA [Touvron et al., ], Mistral [Jiang et al., ], and Gemini [Team et al., ] have demonstrated strong question answering and reasoning capabilities over text. Large Multimodal Models (LMMs) [Liu et al., , Zhu et al., , Yin et al., , Zhang et al., ] extend these reasoning capabilities to images, where given an image and an associated question, a vision encoder and an LLM are leveraged to generate text responses in a chat format. More recent works extend whole-image understanding into region-level understanding [Cai et al., , Zhang et al., , Peng et al., , Chen et al., ], video understanding [Lin et al., , Zhang et al., ] and 3D scene understanding [Hong et al., ]. Such works typically feed the visual tokens directly into the LLM as prefix tokens, via either an MLP [Liu et al., ], Qformer [Dai et al., , Zhu et al., ], or resampler [Alayrac et al., ]. The number of visual tokens can be prohibitively long, especially when the images are high-resolution [Liu et al., , OpenAI, ]. In this paper, we reduce the number of visual tokens with a novel adaptive prune and merge procedure.

While LMMs have made significant advances, their large-scale training and deployment incur significant computational costs, requiring efficient parallel device implementations. Google’s Gemini [Team et al., ] is a pioneer in efficient LMMs, achieving state-of-the-art performance on multimodal benchmarks and introducing mobile-scale LMMs suitable for low-memory devices, although it is not open-source. Open-source alternatives like LLaVA-1.5 [Liu et al., ] employ advanced compression techniques such as 4/8 bit quantization [Dettmers et al., , Shang et al., ]. MobileVLM [Chu et al., ] and its improved version, MobileVLM-v2 [Chu et al., ], focus on compact architecture designs and training optimizations for mobile use. Similarly, TinyGPT-V [Yuan et al., ] uses the Phi-2 [Javaheripi et al., ] LLM backbone to exceed the performance of larger models. LLaVA-Phi [Zhu et al., ] and Vary-toy [Wei et al., ] introduce smaller backbones and enhanced vocabularies for better generalizability. TinyLLaVA [Zhou et al., ] explores the effects of architectural choices and training optimizations, achieving comparable performance to larger models. MoE-LLaVA [Lin et al., ] incorporates a Mixture of Experts to address model sparsity, enhancing efficiency and performance.

In most cases, LMM efficiency is enhanced by reducing the size of the backbone of the LMM, but no work has considered the efficiency of the LMM from the perspective of the number of visual tokens.

### 2.2 Token Reduction

The notorious quadratic complexity in Transformers [Vaswani et al., ] is a well-known problem, as it is one of the key bottlenecks in scaling the input sequence length. Sparse attention methods such as Linformer [Wang et al., ] and ReFormer [Kitaev et al., ] reduce the quadratic attention complexity by conducting attention operations within a certain region rather than the full context. Token merging [Bolya et al., ] utilizes full attention but gradually reduces the number of tokens in each transformer block by selecting the most representative tokens with bipartite matching.
One of the main causes of the inefficiency of recent LMMs is their use of a large number of prefix visual tokens that serve as a fixed budget for context [Liu et al., , Zhu et al., ].
In our study, we present a novel plug-and-play token reduction method based on visual token similarities, which achieves comparable performance with less than one tenth of the original tokens.

## 3 Method: Token Pru-Merging

In this section, we first review the basic implementation of large mutilmodal models (LMMs), with a particular focus on the visual encoder component (*i.e.*, Vision Transformer). We highlight the direct correlation between the number of visual tokens and the efficiency of LMMs (Sec. 3.1).
Next, we present a plug-and-play token reduction method specifically designed for LMMs, called token PruMerge.
Our method features two key components:
(1) Adaptive Important Token Selection (AITS) via Outlier Detection which adaptively determines the optimal number of visual tokens to retain based on the unique characteristics of the image (Sec. 3.2); and (2)
Token Supplement (TS) via Similar Key Clustering, which facilitates efficient processing without compromising the model’s performance by maintaining the integrity and richness of the visual information (Sec. 3.3).

### 3.1 Preliminaries

Figure: Figure 2: The conceptual idea of PruMerge. Our approach has 3 steps: (1) Sample important tokens according to the similarities between the class tokens and spatial visual tokens via an outlier detection algorithm (see Sec.3.2); (2) Cluster the visual tokens via $k$-nearest neighbor; and (3) Adjust the sampled visual tokens via weighted averaging for each cluster (see Sec.3.3). Here $m$ denotes the visual token compression ratio.

Vision Transformers (ViTs) [Dosovitskiy et al., ] are the most widely used vision encoder for LMMs, in which the input image is converted into a sequence of representative tokens by the ViT, and then fed into an LLM for understanding [Liu et al., , Zhu et al., , Hong et al., , Zhang et al., ].
An input image is divided into a grid of patches and each patch is projected into a token embedding by the ViT. In addition to the patch tokens, a class token (*i.e.*, [CLS] token) is computed to aggregate global image information for classification.
A ViT consists of a set of transformer blocks, which in turn consist of several essential components: a multi-head self-attention (MSA) layer, a feed-forward neural network (FFN), skip connections, and layer normalization [Ba et al., ]. These components work together to improve the model’s capability to understand visual data [Han et al., ].
In the self-attention layer, an input token is projected into three distinct vectors: query $\mathbf{q}$, key $\mathbf{k}$, and value $\mathbf{v}$, using three linear transformation matrices $\mathbf{W}_{q}$, $\mathbf{W}_{k}$, and $\mathbf{W}_{v}$.
These vectors, corresponding to different inputs, are assembled into matrices $\mathbf{Q}$, $\mathbf{K}$, and $\mathbf{V}$, respectively. The self-attention computes the relevance of each item to other items:

$$ $\mathbf{Y}=\text{Self-Attention}(\mathbf{Q},\mathbf{K},\mathbf{V})=\mathbf{A} \cdot\mathbf{V}$ (3.1) $$

where attention matrix $\mathbf{A}=\text{softmax}\left(\frac{\mathbf{Q}\cdot\mathbf{K}^{\mathbf{T}}}{
\sqrt{d_{k}}}\right)$ and $d_{k}$ is the dimension of $\mathbf{q}$ and $\mathbf{k}$.
In the last layer of the ViT, the [CLS] token is used for classification. Similarly, the attention between [CLS] token and other visual tokens is computed by the attention mechanism:

$$ $\mathbf{a}_{\texttt{cls}}=\text{softmax}\left(\frac{\mathbf{q}_{\texttt{cls}} \cdot\mathbf{K}^{\mathbf{T}}}{\sqrt{d_{k}}}\right).$ (3.2) $$

The MSA framework allows for simultaneous attention on multiple positions, offering diverse representation subspaces. This is achieved by employing distinct query, key, and value matrices for different heads, which project the input vectors into different representation subspaces.
After the self-attention layers is the feed-forward network (FFN), which consists of two linear transformation layers separated by a nonlinear activation function:

$$ $\text{FFN}(\mathbf{X})=\mathbf{W}_{2}\sigma(\mathbf{W}_{1}\mathbf{X})$ (3.3) $$

where $\mathbf{W}_{1}$ and $\mathbf{W}_{2}$ are the matrices of the linear transformation layers, and $\sigma$ denotes the nonlinear activation function. The general forward pass of ViT is illustrated in the left part of Figure 2.

Large Multimodal Models (LMMs). Following the forward pass through a Vision Transformer (ViT), a set of visual tokens is generated. These tokens are then processed by the input projector $\mathbf{\Theta}_{\mathbf{X}\rightarrow\mathbf{T}}$, which maps the encoded visual features from $\mathbf{F}_{X}$ into the text feature space $\mathbf{T}$.
The aligned features and the text prompts $\mathbf{P}_{T}$ are then fed into the LLM backbone [Zhang et al., ]. The overall architecture of an LMM is depicted in Figure 1.

It is important to note that the computational cost with these models increases quadratically with the number of input tokens to the LLM [Tay et al., ].
Mathematically, if there are $N$ tokens in the input, the self-attention mechanism computes a $N\times N$ matrix of attention scores, where each entry in this matrix represents the attention score between a pair of tokens.
However, there is an increasing demand for processing high-resolution images and videos, which increases the number of visual tokens, further exacerbating computation costs.
The reduction of visual tokens presents a promising approach to improving the efficiency of LMMs by reducing the escalating computational requirements.

### 3.2 Adaptive Important Token Selection via Outlier Detection

Figure: (a) Distribution of attention values.

The most straightforward solution to improve the efficiency of visual token utilization in LMMs is to prune redundant visual tokens [Liu et al., , Yin et al., , Liang et al., ].
To realize token pruning, we need to address a pivotal question: How do we determine the importance of each visual token?

As discussed in Sec. 3.1, LMMs typically leverage an extensive stack of visual tokens to represent the visual information.
On the other hand, self-/weakly-supervised learning paradigms, such as CLIP [Radford et al., ] simplify this complexity by representing an entire image with a single [cls] token, regarded as the most information-condensed token.
To balance those two extreme paradigms, we investigate the Key-Query attention between [cls] token and visual tokens, *i.e.*, $\mathbf{a}_{\texttt{cls}}$ in Equation 3.2.
Observing the distribution patterns of attention between the [cls] token and visual tokens unveils a sparse landscape, as depicted in Figure 3(a).
This sparse distribution underpins our methodology for identifying crucial visual tokens. By employing outlier detection algorithms, we aim to adaptively select visual tokens that best represent an image’s features effectively.

Interquartile Range (IQR) Method for outlier detection. To identify outliers within class attention values, we adopt the Interquartile Range (IQR) method [Boukerche et al., ], a statistical technique known for its robustness in outlier detection. The essence of the IQR method lies in its capability to establish a boundary or “fence” that delineates the normal range of data. This is achieved by calculating the IQR (the difference between the third quartile Q3 and the first quartile Q1) and subsequently defining the outer limits of the normal range as 1.5 times the IQR above Q3 and below Q1.
Specifically, the computation is as follows: the “lower fence” is set at 1.5 $\times$ IQR below Q1, and the “upper fence” is set at 1.5 $\times$ IQR above Q3. Any attention values residing outside these fences are classified as outliers.
In practice, only the “upper fence” is activated since the attention score is positive.
Through this method, we can adaptively identify and select the visual tokens for each image that exhibit outlier attention values, *i.e.*, those playing a significant role in representing the image within the LMM context.
Note that we use the class attention value from the penultimate layer for this calculation.

As shown in Figure 1(b), the sampled visual tokens demonstrate two behaviors: (1) The number of attentive tokens are proportional to the complexity of the image. Simpler images such as “Billboard among blue sky” owns fewer tokens while images with rich information such as a screen with dense texts own more tokens. (2) The sampled tokens are typically spatially aligned with important content. Such visualizations align with our visual token sampling design.
These trends are also observed at the benchmark level; in Table 6, the average token numbers on various benchmarks differ.

### 3.3 Token Supplement via Similar Key Clustering

Following the selection of informative visual tokens, we next optimize the utilization of the remaining tokens.
While pruned tokens may initially seem extraneous, they hold potential value for the perception capabilities of the LLM backbone.
This potential arises particularly in cases where an image contains large object parts that dominate the scene. In such scenarios, overly aggressive pruning could inadvertently diminish the model’s ability to represent the image comprehensively.

To address this, we devise a token merging method aimed at enhancing the representational capacity of the selected unpruned tokens. This method involves the strategic fusion of currently pruned tokens, as depicted in Figure 2.
To choose the pruned tokens to merge, we need a way to measure similarity between visual tokens.
Here we leverage the self-attention mechanism in ViTs.
Since the key vector of each patch token already contains information summarized in the self-attention module [Vaswani et al., ], the final layer’s key vector serves as the representation.
And then we use the dot product between keys to calculate which tokens have similar visual information [Bolya et al., ]:

$$ $\text{Sim}(\mathbf{y}_{i},\mathbf{y}_{j})=\mathbf{k}_{i}\cdot\mathbf{k}_{j}^{T},$ (3.4) $$

which yields $\mathbf{K}\mathbf{K}^{T}(i,j)$ for tokens $i,j$ in vectorized form for the set of all tokens ${1,2,\cdots,n}$, where $n$ is the number of input visual tokens.

With the similarities between visual tokens established, we simply find the $k$-nearest neighbors for each unpruned token, which act as the cluster centers.
The integration of pruned tokens into these clusters is guided by their respective class attentions $\mathbf{a}[i]$, enabling a refined representation of each unpruned token through a weighted sum.
This procedure is outlined in Algorithm 1.

### 3.4 P ruMerge +: Bridging the Efficiency-Performance Gap

While PruMerge achieves a remarkable reduction in the number of visual tokens—over tenfold compared to the original setup—the process is not without drawbacks. Specifically, the compression technique, though efficient, introduces a marginal performance discrepancy between the original LLaVA model and its PruMerge-optimized counterpart, LLaVA-PruMerge. To address this, we introduce PruMerge+, a refined version that strikes an optimal balance between token reduction and model performance.

PruMerge+ enhances our original method by maintaining the ability to significantly reduce visual token count—by an average of fourfold—with minimal performance degradation. This improvement is detailed in Algorithm 1, building upon the token selection strategies outlined in Section 3.2.
A new aspect of PruMerge+ lies in its enhanced token selection process. Beyond merely focusing on the previously identified important tokens, PruMerge+ extends its reach to encompass additional visual tokens from areas initially deemed less critical. This is achieved through a spatially uniform sampling of visual tokens, guided by a predetermined ratio informed by the distribution of outlier tokens. This methodology ensures a more comprehensive and representative selection of visual tokens, as depicted in Figure 3(b), thereby minimizing performance losses while still achieving substantial token reduction.

Figure: Algorithm 1 Token PruMerge and PruMerge+ algorithms for reducing the number of visual tokens.

### 3.5 Discussion

Distinction from Existing Token Reduction Methods.
Token reduction methods [Liu et al., , Yin et al., , Liang et al., ] have been proposed for accelerating ViT computation speed. Block by block, tokens are gradually reduced in number, correspondingly reducing the computation cost in the internal ViT.
By contrast, our approach is not specifically designed to increase the efficiency of a ViT.
It instead aims to improve the overall efficiency of a large multimodal model, where ViT is only one relatively light cost component as shown in Fig. 1.
There are two benefits of this design.
First, while ViTs typically rely on a single class token to represent the input image, which enables them to maintain performance despite a reduction in intermediate tokens, LMMs usually require a large stack of visual tokens. This ensures a comprehensive representation of the visual content, preserving the model’s ability to capture nuanced details.
Thus, using previous token merging methods to obtain one refined class token as representation of visual input is not consistent with the literature of large multimodal models.
Second, considering that the bulk of computational demand within LMMs is attributed to the LLM component rather than the ViT, our approach focuses not only on the reduction of tokens but also on maximizing the informational content of the pruned visual tokens. This strategy addresses the computational challenges inherent in LMMs with minimal compromise in the quality of the visual representation.

## 4 Experiments

We first show the empirical performance of our approach when applied to LLaVA-1.5 in Sec 4.1. We then analyze the efficiency improvement by using our PruMerge on LMM in Sec 4.2. To show the generalization ablity, we conduct a series of experiments in Sec. 4.3. Finally, we demonstrate the effectiveness of each component in our model in Sec 4.4.

**Table 1: Comparison with large multimodal models on six benchmarks. Our proposed approach PruMerge and PruMerge+ can adaptively reduce visual tokens, which use only (respectively) 5.5% and 25.0% visual tokens on average (on 6 tasks) and achieves competitive performance compared to the original LLaVA-1.5.**
| Method | LLM | Res. | PT | IT | VQA^v2 | SQA^I | VQA^T | POPE | MME | MMB |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| BLIP-2 | Vicuna-13B | 224 | 129M | - | 41.0 | 61 | 42.5 | 85.3 | 1293.8 | - |
| InstructBLIP | Vicuna-7B | 224 | 129M | 1.2M | - | 60.5 | 50.1 | - | - | 36 |
| InstructBLIP | Vicuna-13B | 224 | 129M | 1.2M | - | 63.1 | 50.7 | 78.9 | 1212.8 | - |
| Shikra | Vicuna-13B | 224 | 600K | 5.5M | 77.4 | - | - | - | - | 58.8 |
| IDEFICS-9B | LLaMA-7B | 224 | 353M | 1M | 50.9 | - | 25.9 | - | - | 48.2 |
| IDEFICS-80B | LLaMA-65B | 224 | 353M | 1M | 60.0 | - | 30.9 | - | - | 54.5 |
| Qwen-VL | Qwen-7B | 448 | 1.4B | 50M | 78.8 | 67.1 | 63.8 | - | - | 38.2 |
| Qwen-VL-Chat | Qwen-7B | 448 | 1.4B | 50M | 78.2 | 68.2 | 61.5 | - | 1487.5 | 60.6 |
| LLaVA-1.5 | Vicuna-7B | 336 | 558K | 665K | 78.5 | 66.8 | 58.2 | 85.9 | 1510.7 | 64.3 |
| LLaVA-1.5 + PruMerge | Vicuna-7B | 336 | 558K | 665K | 72.0 | 68.5 | 56.0 | 76.3 | 1350.3 | 60.9 |
| LLaVA-1.5 + PruMerge+ | Vicuna-7B | 336 | 558K | 665K | 76.8 | 68.3 | 57.1 | 84.0 | 1462.4 | 64.9 |
| LLaVA-1.5 | Vicuna-13B | 336 | 558K | 665K | 80.0 | 71.6 | 61.3 | 85.9 | 1531.3 | 67.7 |
| LLaVA-1.5 + PruMerge | Vicuna-13B | 336 | 558K | 665K | 72.8 | 71.0 | 58.4 | 78.5 | 1428.2 | 62.3 |
| LLaVA-1.5 + PruMerge+ | Vicuna-13B | 336 | 558K | 665K | 77.8 | 71.0 | 58.6 | 84.4 | 1485.5 | 65.7 |

### 4.1 Main Results

We apply our method to LLaVA-1.5 [Liu et al., ], a recent state-of-the-art LMM. We further finetune LLaVA-1.5 using LoRA [Hu et al., ] for 1 epoch using the LLaVA-1.5 instruction fine-tuning data [Liu et al., ] with our reduced visual tokens.

We evaluate on diverse visual question-answering and reasoning benchmarks including VQAv2 [Goyal et al., ], ScienceQA [Lu et al., ], TextVQA [Singh et al., ], POPE hallucination bench [Li et al., ], MME [Fu et al., ], and MMBench [Liu et al., ].
As shown in Table 1, our approach achieves comparable performance with LLaVA-1.5 despite using only a small fraction of the visual tokens, and performing better than previous works such as BLIP2 [Li et al., ] and InstructBLIP [Dai et al., ]. Specifically, in POPE and ScienceQA, our approach even shows better performance than LLaVA-1.5.
Note that due to the adaptive nature of PruMerge (see Sec. 3.2), the token numbers for various tasks are different (see 4.4.1), and thus we use the average number on numbers of 6 tasks (*i.e.*, 32) for simplicity.

### 4.2 Efficiency Analysis

**Table 2: Computation Cost Analysis. The development device is Tesla V100 GPU, and time estimated by the roofline model represents the theoretical performance that the hardware can achieve.**
| Method | LLM | Quantization | FLOPs | Prefill | Total | Storing |
| --- | --- | --- | --- | --- | --- | --- |
| Backbone | (TB) | Time (ms) | Memory (GB) | Activation (GB) |  |  |
| LLaVA-1.5 | Vicuna-7B | FP16 | 9.3 | 88.6 | 23.3 | 4.60 |
| LLaVA-1.5 w/ PruMerge | Vicuna-7B | FP16 | 0.91 | 15.3 | 13.7 | 0.28 |
| LLaVA-1.5 | Vicuna-7B | INT4 | 2.3 | 151.6 | 5.9 | 1.20 |
| LLaVA-1.5 w/ PruMerge | Vicuna-7B | INT4 | 0.28 | 14.9 | 3.5 | 0.07 |
| LLaVA-1.5 | Vicuna-13B | FP16 | 18.2 | 170.5 | 41.6 | 7.30 |
| LLaVA-1.5 w/ PruMerge | Vicuna-13B | FP16 | 1.80 | 29.5 | 26.6 | 0.44 |
| LLaVA-1.5 | Vicuna-13B | INT4 | 4.6 | 294.9 | 10.5 | 1.80 |
| LLaVA-1.5 w/ PruMerge | Vicuna-13B | INT4 | 0.45 | 29.0 | 6.8 | 0.11 |

To elucidate the computational efficiency afforded by PruMerge, we utilize the roofline-based LLM-Viewer analysis developed in [Yuan et al., ]. Our investigation is grounded in a theoretical scenario tailored to highlight the impact of PruMerge on processing efficiency within LMMs.
Consider a typical scenario where an image of dimensions $336\times 336$ pixels is processed using a CLIP-ViT model, resulting in 576 visual tokens. Accompanying this image is a text prompt, assumed to contain 40 tokens for the sake of this analysis. Through the application of PruMerge, we achieve a dramatic reduction in the number of visual tokens, decreasing the original count by approximately 14.4 times in MME/TextVQA to match the token count of the text prompt ($576/14.4\approx 40$).
The implications of this reduction are significant, as demonstrated in Table 2, which outlines the computational cost associated with the LMM prefill process. Notably, PruMerge not only enhances the speed of the LLM prefill process by reducing the required floating-point operations (FLOPs) but also contributes to a reduction in computational memory demands.

It is important to emphasize that the benefits of PruMerge  extend beyond mere efficiency gains. Our token reduction strategy can complement other LLM acceleration techniques, such as quantization and factorization [Yuan et al., ]. This orthogonal relationship underscores the versatile potential of PruMerge to contribute to a broader spectrum of efficiency-enhancing strategies.

### 4.3 Generalization on Video-LLM

To assess the generalization capabilities of PruMerge and PruMerge+ across different modalities, we next extend our approach to Video-LLaVA [Lin et al., ].
Video-LLaVA is one of the most popular open-soruced Video-LLMs.
We seamlessly integrate both algorithms into Video-LLaVA without the need for additional training, enabling us to bypass re-training on video datasets during inference. The outcome of this integration is shown in Table 3.
Video-LLaVA samples 8 frames from a video clip and extracts $8\times 16\times 16=2048$ visual tokens using a visual encoder for LLM perception, which is 4 times of visual token than LLaAV-1.5 [Liu et al., ].
Our Algorthms PruMerge and PruMerge+ can adaptively select important 256 (12.5% on average) and 256 (25.0% on average) important visual tokens, respectively.
The results demonstrate that our algorithms not only reduce the number of visual tokens in Video-LLaVA but also is able to enhance its performance.
This finding is noteworthy as it suggests a significant redundancy in the visual tokens used by video-LLMs. Exploring ways to further capitalize on this redundancy could shape future research directions.

**Table 3: A comparison of different LVMs on video reasoning benchmarks. Like Video-LLaVA, ChatGPT-Assistant is employed to evaluate performance. The version of ChatGPT is “gpt-3.5-turbo”. Note that we directly add PruMerge and PruMerge+ to Video-LLaVA during inference (without training our own model).**
| Methods | LLM size | MSVD-QA | MSRVT-QA | ActivityNet-QA |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Accuracy | Score | Accuracy | Score | Accuracy | Score |  |  |
| FrozenBiLM | 1B | 32.2 | - | 16.8 | - | 24.7 | - |
| VideoChat | 7B | 56.3 | 2.8 | 45.0 | 2.5 | - | 2.2 |
| LLaMA-Adapter | 7B | 54.9 | 3.1 | 43.8 | 2.7 | 34.2 | 2.7 |
| Video-LLaMA | 7B | 51.6 | 2.5 | 29.6 | 1.8 | 12.4 | 1.1 |
| Video-ChatGPT | 7B | 64.9 | 3.3 | 49.3 | 2.8 | 35.2 | 2.7 |
| Video-LLaVA | 7B | 70.7 | 3.9 | 59.2 | 3.5 | 45.3 | 3.3 |
| Video-LLaVA + PruMerge | 7B | 71.1 | 3.9 | 58.4 | 3.5 | 48.3 | 3.4 |
| Video-LLaVA + PruMerge+ | 7B | 71.1 | 3.9 | 59.3 | 3.6 | 47.7 | 3.4 |

### 4.4 Ablation Study

#### 4.4.1 Token Sampling Strategy Analysis

Here we show how our approach performs better than the vanilla visual token sampling strategy, including sequential sampling and spatial sampling.

LLaVA-PruMerge: Our approach dynamically samples key visual tokens (see Sec. 3.2), which results in 40 visual tokens per image on average for TextVQA/MME, 35 tokens for POPE, and 16 tokens for SQA. The visualization is shown in Figure 4 (b).

Sequential sampling: We sample $N$ tokens in the flatted visual tokens. For example, the first 40 tokens are sampled for an apples-to-apples comparison, shown in Figure 4 (c).

Figure: Figure 4: Different token sampling strategies.

Spatial sampling: The sampled $N$ tokens are evenly distributed across the image, as shown in Figure 4 (d-h). We study diverse settings, including 6 $\times$ 6 (36 tokens), 5 $\times$ 8 (40 tokens), 8 $\times$ 5 (40 tokens), 5 $\times$ 7 (35 tokens), 7 $\times$ 5 (35 tokens), and 4 $\times$ 4 (16 tokens).

**Table 4: Token Sampling Strategy Analysis on Different Tasks.**
| Approach | #Visual Tokens | Performance |
| --- | --- | --- |
| Task: VQA^T |  |  |
| LLaVA-PruMerge | 40 | 54.00 |
| Sequential | 40 | 42.72 |
| Spatial | $5\times 8=40$ | 46.85 |
| $8\times 5=40$ | 47.42 |  |
| Task: MME |  |  |
| LLaVA-PruMerge | 40 | 1250.07 |
| Sequential | 40 | 703.60 |
| Spatial | $5\times 8=40$ | 1180.23 |
| $8\times 5=40$ | 1142.32 |  |
| Task: POPE |  |  |
| LLaVA-PruMerge | 35 | 76.2 |
| Sequential | 35 | 11.7 |
| Spatial | $5\times 7=35$ | 69.8 |
| $7\times 5=35$ | 71.1 |  |
| $6\times 6=36$ | 67.9 |  |
| Task: SQA^I |  |  |
| LLaVA-PruMerge | 16 | 68.07 |
| Sequential | 16 | 64.20 |
| Spatial | $4\times 4=16$ | 66.29 |

Note that all the experiments are done via a training-free manner. As shown in Table 6, our approach is consistently better than sequential sampling and spatial sampling across all downstream tasks, which demonstrates the effectiveness of the sampling mechanism of LLaVA-PruMerge. Importantly, we observe that LLaVA-PruMerge shows much better performance on TextVQA [Singh et al., ]. Such Optical Character Recognition (OCR) task requires detailed information about the text, which demonstrates that LLaVA-PruMerge extracts the key information in the images with enough details. This quantitative result aligns with the visualization of LLaVA-PruMerge attentive tokens in Figure 1(b), where more attentive tokens are distributed on the foreground text in the images.

#### 4.4.2 Effectiveness of Each Module in P ruMerge

Here, we study the effectiveness of each module in our design based on LLaVA-1.5. Note that we maintain the same amount of visual tokens (6.9%, 40 tokens) across all settings.
As shown in Table 6, after progressively adding the proposed modules, including Adaptive Important Token Selection (AITS) and Token Supplement (TS), the downstream performance can be further enhanced.

#### 4.4.3 Training Analysis: Training-free v.s. Fine-tuning

Finally, LLaVA-PruMerge can be conducted in either a training-free or fine-tuning manner. With fine-tuning, the large language model can adapt to the new structure of visual tokens, which could further enhance the performance on vision-language tasks. As shown in Table 6, with fine-tuning, our approach does bring better performance for diverse tasks, including ScienceQA [Lu et al., ], TextVQA [Singh et al., ], POPE [Li et al., ], and MME [Fu et al., ].

## 5 Conclusion

In this paper, we improve the efficiency of Large Multimodal Models (LMMs) from the perspective of reducing the quantity of visual tokens. By leveraging the spatial redundancy in visual tokens, we proposed a plug-and-play token reduction module that employs the similarity between the class token and spatial tokens as a key criterion for pruning and merging visual tokens.
Our approach, applied to LLaVA-1.5, demonstrated that by utilizing only 6.9% of visual tokens on average, the pruned tokens can maintain comparable performance across a wide range of visual question-answering and reasoning tasks. Notably, our work highlights the potential for significant computational savings without sacrificing the reasoning capabilities of LMMs. We hope our work inspires further exploration into the interplay between efficiency and performance in LMMs.

## Appendix A Appendix

### A.1 Limitations and Future Work

Our exploration of LLaVA-PruMerge has two primary limitations.
First, the visual token compression, while efficient, is not entirely lossless. This results in a marginal performance gap between the original model (LLaVA) and our optimized version LLaVA-PruMerge. We are committed to advancing our research towards achieving a fully lossless token compression algorithm, aiming to close these performance gaps.
Second, the scope of our validation efforts is somewhat constrained by the computational resources typically available in academic settings. This limitation has precluded a comprehensive assessment of PruMerge’s applicability to larger-scale models, such as those envisioned in the LLaVA-Next [Liu et al., ] framework with a Yi-34B backbone. Future investigations will seek to extend our methodology to these more expansive model architectures, exploring its potential for generalization and broader impact.

### A.2 Societal Impacts

In this study, we propose a technique that improves the efficiency of Large Multimodal Models (LMMs), making them more accessible. This approach helps to democratize LMMs by lowering deployment costs and hardware barriers, facilitating their use in edge computing. However, it does not mitigate the potential misuse of LMMs by malicious actors.

## Peer Reviews

confidence: 5
contribution: 2
initial_rating: 3
presentation: 3
questions: Refer to the "Weakness" part.
soundness: 2
strengths: 1. The proposed method provides a new solution towards multimodal LLM efficiency in the direction of input compression. The proposed method is able to provide computation reduction without significant finetuning.

2. Experiments show the proposed method is effective, which can maintains performance using only 5.5% of original tokens in LLaVA-1.5 across six benchmarks. It can also reduces Video-LLaVA's tokens from 2048 to 256 while preserving or improving performance.

3. The proposed method doesn't requiring model architecture change, makes it easier to deploy.

4.. The paper writing is easy to understand.
summary: This paper introduces PruMerge, a method to improve the efficiency of Large Multimodal Models by reducing their visual token count. Specifically, PruMerge address the computing efficiency with two steps: 1) it first uses an Interquartile Range method to identify visual tokens outliers based on attention scores, 2) and then applies k-nearest neighbor clustering to merge similar tokens. The authors also introduce PruMerge+, which adds spatially uniform sampling. Experiments on LLaVA-1.5 show that PruMerge achieves comparable performance across various visual reasoning tasks while using only 5.5% of the original tokens. The method is also showed to be effective with video LMMs, reducing Video-LLaVA's 2048 tokens to 256 tokens (12.5%) without compromising the performance.
weaknesses: 1. The experiments are too limited to show the generalization ability of the proposed approach. The method is primarily validated on LLaVA-1.5 and Video-LLaVA. Additional experiments on other LMMs like Flamingo, Qwen, etc. would better show the method's generalizability across different MLLM modeling architectures like different image encoder and different feature fusion mechanism.

2. The experiment section missed real world efficiency analysis. The results on Tesla V100 GPU (e.g. in Table 2) doesn't represent typical deployment for efficient LMMs on resource-constrained environments (e.g. on mobile device).  For example, the paper would be more solid if the authors could show the memory, FLOPs, and latency on mobile device like Macbook, or Phone (e.g. iPhone or Pixel).

3. The experiment missed important baselines. While the paper compares with ToMe, ATS, and EViT, it misses several crucial token reduction baselines such as Token Pruning[1], Token summarization[2], Progressive Token Pruning[3], and recent prompt compression methods. These baselines are the latest work that achieved better performance over the aforementioned ones in various tasks.

4. The paper is based on the assumption of sparsity in visual encoder (CLIP). However, the paper doesn't check whether this holds true for other visual encoders (i.e. different encoder architectures like CNN and ViT).

5. The requirement of attention scores for outlier selection makes this method incompatible with FlashAttention, potentially leading to higher memory usage and increased latency. The authors should provide comparisons with FlashAttention-enabled baselines to address deployment concerns. Specifically, peak memory consumption and latency measurements comparing PruMerge/PruMerge+ vs FlashAttention-enhanced baselines are needed to verify the effectiveness of the proposed method.


[1] https://arxiv.org/abs/2403.06764
[2] https://arxiv.org/abs/2410.14072
[3] https://arxiv.org/abs/2301.13741

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: As PruMerge is effective for LLaVA1.5, can it be adapted for MLLMs with AnyRes (which is a widely adopted strategy), e.g., LLaVA-Next?
soundness: 3
strengths: 1) The method effectively chooses the most informative tokens from the visual encoder without further finetuning, hence largely reducing the computation costs of MLLM. 

2) PruMerge outperforms former SOTA methods on token pruning for ViTs.
summary: In this paper, authors propose a training-free token reduction strategy, PruMerge, for efficient MLLM inference. The method focuses on sampling the most important visual tokens by outlier token detection and clustering. PruMerge manages to reduce the computational costs by approximately 10 times in FLOPs with comparable performance.
weaknesses: Instead of being an adaptive reduction method for MLLM, PruMerge seems more like a strategy for the transformer-based visual encoder. It makes no exploration of the design of the LLM, particularly, ignoring the interaction between visual tokens and text tokens in MMLM.  If so, authors may consider exploring PriMerge on more visual encoders, e.g., SigLIP widely used in MLLMs and etc.

Also, as authors claim that token relevance matters more for MLLM encoder, the paper may support it with more experimental exploration as well as visualizations.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: - Consistency of Results in Table 1 and Table 6: Are the results reported in Table 1 based on additional fine-tuning? Table 6 shows that fine-tuning has a positive effect on performance, so confirming whether Table 1 results include fine-tuning would clarify the impact of this step on the final results, ensuring data consistency.

- Comparison with Visual Token Pruning Methods like FastV [1] and VTW [2] : Although FastV and VTW conduct token pruning within specific layers of the model, their goals align closely with the proposed method in terms of reducing computational cost through pruning. Since these methods reportedly maintain model performance with minimal loss, a comparison with them would provide a clearer reference, especially in terms of balancing accuracy with computational efficiency.

- Applicability Beyond VQA to More General Scenarios: VQA tasks primarily require only the main content of an image, allowing aggressive pruning strategies without significant impact on performance. However, in multi-turn dialogues with large language models, preserving comprehensive image information is essential, as responses may require details beyond the main content. Could you demonstrate the method’s performance on image captioning tasks, where capturing detailed information is crucial? This would better showcase its efficacy in tasks that require a fuller representation of visual data.

[1] Chen, Liang, et al. "An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models." arXiv preprint arXiv:2403.06764 (2024).  
[2] Lin, Zhihang, et al. "Boosting Multimodal Large Language Models with Visual Tokens Withdrawal for Rapid Inference." arXiv preprint arXiv:2405.05803 (2024).
soundness: 3
strengths: - High Visual Token Pruning Rate: The proposed method achieves a significant reduction in computational cost for Multimodal Large Language Models (MLLMs) by pruning a high proportion of visual tokens. This makes the model more efficient in resource-constrained settings.
- Inference Speed and Memory Efficiency: By pruning visual tokens before they are fed into the LLM, the approach reduces both inference time and memory usage. This aspect is particularly advantageous for practical applications where memory efficiency is crucial.
summary: This paper introduces LLaVA-PruMerge, a method designed to optimize Large Multimodal Models (LMMs) by reducing the number of visual tokens without compromising performance. The central innovation, the "PruMerge" algorithm, adaptively prunes and merges visual tokens based on their relevance, addressing the computational inefficiencies typically associated with LMMs. By exploiting sparsity patterns in attention scores between class and visual tokens, PruMerge reduces token counts by up to 94% on average. Empirical results demonstrate that this method improves efficiency while maintaining performance across diverse visual question-answering and reasoning benchmarks, including VQAv2 and ScienceQA. The approach has been evaluated on both image and video models, highlighting its adaptability in optimizing LMMs across different modalities.
weaknesses: - Potential Loss of Auxiliary Visual Information: The aggressive pruning of visual tokens could lead to a significant loss of auxiliary information, which is not evident from experiments on the VQA dataset alone. This likely contributes to weaker performance on the POPE dataset. Given the request-agnostic nature of pruning method , it may only preserve the primary content, limiting its broader applicability. Testing the method on tasks requiring richer image information, like image captioning, would better showcase its robustness.

- Performance Degradation Despite High Pruning Rate: Although the method achieves a high pruning rate, it incurs substantial performance drops, approximately 8% with prumerge and 2% with prumerge+, which is considerable, especially when compared to other token pruning approaches like FastV [1] and VTW [2] , which maintain performance integrity. Given the extra fine-tuning required, this performance decrease could be prohibitive in practical applications.

- Limited Memory Savings in Long Text Generation: While the proposed method reduces memory usage during inference, the focus on visual token pruning alone may have minimal impact for long text generation tasks, where large numbers of language tokens drive memory consumption. The memory savings from visual token reduction are less impactful in these contexts. Furthermore, in short text generation tasks, where accuracy and speed are primary concerns, the proposed method does not excel.

[1] Chen, Liang, et al. "An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models." arXiv preprint arXiv:2403.06764 (2024).  
[2] Lin, Zhihang, et al. "Boosting Multimodal Large Language Models with Visual Tokens Withdrawal for Rapid Inference." arXiv preprint arXiv:2405.05803 (2024).

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 3
questions: 1.Could the authors clarify if they have tested this method on tasks that require high levels of visual detail, such as image captioning? If so, how did the method perform in comparison to the non-pruned baseline?

2.Have the authors explored whether the assumption of visual token redundancy holds across different image complexities, such as high-texture scenes or densely populated images? How does token pruning adapt to varying levels of image detail?

Suggestion:It would be beneficial for the authors to explicitly state whether the results are obtained from the fine-tuned model or the base model. Additionally, providing a comparison between both scenarios would enhance the transparency of the experiments and allow readers to better assess the method's true performance and its dependence on fine-tuning.
soundness: 2
strengths: 1.Enhanced Computational Efficiency: PruMerge reduces visual tokens, decreasing computational complexity and speeding up large multimodal models, making them more suitable for resource-constrained environments.

2.Adaptive Selection Mechanism: By leveraging sparse attention, the method retains essential visual information while reducing computation, allowing for flexibility based on the complexity of input images.
summary: The paper presents PruMerge, a method designed to improve the computational efficiency of large multimodal models (LMMs) by adaptively reducing visual tokens. Utilizing sparse attention distribution from the visual encoder, PruMerge selects important tokens, achieving comparable performance to full models with fewer tokens on LLaVA.
weaknesses: 1.A key limitation of this method is that, by trimming visual tokens, it risks losing essential image details. This makes it less suited for tasks that rely heavily on fine-grained visual information, like OCR, object detection, or fine-grained classification. Since these tasks require precise visual cues to perform well, reducing tokens could hurt performance. Another issue is that the paper doesn't test on these detail-sensitive tasks, leaving some doubt about the method's broader applicability.

2.A key limitation is that experiments were only conducted on LLaVA-based models, limiting insights into the method’s generalizability. Testing solely on one model type makes it unclear if the token-pruning benefits would hold across different multimodal architectures. Broader testing would provide a fuller picture of its effectiveness.

3.The experimental section lacks clarity regarding whether the reported results are based on the pre-fine-tuned model or the fine-tuned version. This ambiguity can lead to confusion about the actual effectiveness of the proposed method and its reliance on fine-tuning.