Fine-grained Token Allocation Via Operation Pruning for Efficient MLLMs

## Abstract

Abstract Prior Vision Language Model (VLM) token pruning reduces computation by eliminating attention and feed-forward operations for pruned tokens while maintaining all operations for critical tokens. However, this binary approach conflates token/operation redundancy - critical operations may be removed along with discarded tokens, while preserved tokens retain all potentially redundant operations. To surgically eliminate redundant operations while preserving critical ones, we propose G reedily S orted O peration P runing (GSOP), a data-driven method that directly prunes operations rather than tokens. GSOP first decomposes a VLM decoder’s computations into atomic operations along three dimensions: token groups, layer positions, and computation modules.
GSOP determines the pruning order of operations through greedy sorting:
GSOP iteratively selects the redundant operation that incurs minimal performance drop considering previously pruned operations.
Different computational budgets can be accommodated without re-searching by simply pruning operations according to this order until the desired budget is met.
GSOP enhances sorting efficiency through: a) leveraging historical operation rankings to avoid redundant evaluations; b) excluding the “free-to-prune” and “danger-to-prune” operations from sorting.
GSOP achieves compelling efficiency-performance tradeoffs, reducing computation by 70% with only 4% performance loss while maintaining up to 18% higher performance than state-of-the-art methods when transferred across diverse VLMs and tasks. Real GPU efficiency evaluations confirm its practical value. The code is in GSOP .

## 1 Introduction

Figure: (a) Token Pruning.
Refer to caption: x1.png

Vision-language models (VLMs) [35, 34, 25, 13, 43, 3, 4, 12, 52] face significant computational overhead primarily in their language model decoders [2, 45, 38, 7, 18, 1], which dramatically outweigh the lightweight visual encoders (e.g., in LLaVA-1.5, 7B/13B decoder parameters versus 0.3B encoder parameters), creating substantial computational demands when processing extensive token sequences derived from visual inputs. Recent token pruning works provide a viable acceleration pathway by pruning redundant visual tokens [27, 51, 11, 56]. At a finer granularity, computation reduction is mainly achieved by pruning the attention and feed-forward operations associated with the pruned tokens. However, these methods [27, 11, 56] treat tokens as atomic units, forcing preserved/pruned tokens to retain/discard all associated operations. This coarse token-level selection (shown in [Fig. 1(a)](https://arxiv.org/html/2507.02909v1#S1.F1.sf1)) hinders optimal precision: critical operations are lost with discarded tokens while preserved tokens retain some unnecessary ones.

To surgically eliminate redundant operations while preserving critical operations, we introduce Greedily Sorted Operation Pruning (GSOP), a novel method that targets direct operation-level pruning in a VLM decoder ([Fig. 1(b)](https://arxiv.org/html/2507.02909v1#S1.F1.sf2)). GSOP decomposes a VLM decoder computations into operations defined by the tuple ($group$, $layer$, $module$), where tokens are partitioned into groups, each layer is considered independently, and each token’s operations in a single decoder layer are divided into three modules: MHA-out (which uses selected tokens to update other tokens), MHA-in (which updates selected tokens’ features) and MLP (token-wise feature update). Under this decomposition, token pruning acts as a coarse-grained approach that partitions tokens into ”redundant” and ”critical” groups, eliminating all operations tied to the ”redundant” group. In contrast, GSOP enables direct fine-grained operation-level pruning, achieving superior performance-efficiency tradeoffs through more flexible computation reduction.

To determine the pruning policy, GSOP performs a one-time greedy sorting of all operations. At each step, GSOP measures performance on a small validation set of each remaining operation by temporarily pruning it alongside previously selected operations. The operation with smallest performance drop is appended to the end of our sorted operation sequence. This process iterates until all operations are included. The resulting sequence enables budget-aware pruning without re-searching: given any computational budget constraint, we can simply prune operations along the sequence until the specified budget requirement is met. To accelerate the sorting process, GSOP further employs (1) adaptive evaluation that dynamically skips operation evaluations by reusing historical results; (2) excluding the “free-to-prune” operations in deep layers and “danger-to-prune” operations in shallow layers from greedy sorting.

Extensive experiments validate GSOP as a powerful acceleration framework across diverse VLM architectures and tasks. When applied to LLaVA-1.5 [35] and optimized on just 500 GQA [19] samples, GSOP reduces computations by 70% with only 4% performance drop across GQA and five unseen benchmarks. GSOP consistently outperforms all baselines, preserving up to 4% more performance than leading baselines. Beyond its strong performance on the model it was optimized for, GSOP demonstrates remarkable cross-model transferability, with policies optimized for LLaVA-1.5 transferring effectively to LLaVA-Phi3 [39] and VILA [29] while still outperforming all baselines. Notably, when transferred to VILA-1.5-3B, GSOP maintains up to 18% higher performance than state-of-the-art token pruning approaches. Real GPU efficiency evaluations confirm its practical value. Our contributions are as follows:

- 1.
We fundamentally extend the VLM acceleration framework from token pruning to a more fine-grained manner, enabling precise and direct control over operations defined by ($group$, $layer$, $module$).
- 2.
We propose GSOP, an efficient methodology to realize this framework and enables budget-aware operation pruning without re-searching by performing a one-time greedy sorting of all operations based on redundancy.
- 3.
GSOP significantly advances the state-of-the-art in VLM acceleration, preserving up to 18% more performance than leading baselines, and demonstrating exceptional generalization across 6 tasks and 5 VLM models without requiring task/model-specific re-optimization.
- 4.
Our analysis reveals valuable insights into VLM computational redundancy across layers, modalities, tasks, and modules for future VLM optimization.

## 2 Related Work

Token Pruning [6, 8, 9, 54, 41, 11, 56, 30, 53, 55, 50, 49] reduces Vision-Language Models (VLMs) [35, 34, 25, 13, 43, 3, 4, 12, 52, 16, 59, 28, 33, 47, 10, 57, 1, 29] computation by grouping visual tokens into redundant and critical groups, and pruning the redundant tokens to eliminate their associated attention and feed-forward operations. Visual token merging is also explored [41, 50], while unmerging, as used in vision transformer [6, 5, 22, 20, 23] remains unexplored. Following FastV [11] using decoder attention for one-time pruning, recent works advance in two directions: (1) implementing progressive fine-grained token pruning across multiple layers [53, 49, 56]; and (2) developing better redundancy metrics for more precise token grouping, such as using visual encoder’s [cls] attention [55]. However, token pruning’s computational granularity remains confined to the token level, limiting precision and overlooking the fact that true computation savings stem from eliminating associated attention and feed-forward operations. In contrast, our GSOP proposes a more fundamental solution that directly prunes fine-grained operations rather than tokens.

KV caching
[21] accelerates VLM inference by reusing computed key-value pairs of processed tokens, dividing the inference process into a prefilling stage, where all tokens are fully processed to generate the first token, and subsequent decoding stages, where KV cache exempts certain GSOP components (e.g., MHA-in and MLP modules) from full-token processing. Our GSOP primarily optimizes the prefilling stage, which aligns with prior token pruning approaches [6, 8, 9, 54, 41, 11, 56, 30, 53, 55, 50, 49]. Recent studies [58] confirm that prefilling is compute-bound, while decoding is memory-bandwidth-bound, thereby validating our focus on prefilling efficiency.

Neural Architecture Search (NAS). While sharing some similarity with NAS [60, 32, 31, 48, 42, 40, 44], GSOP is the extension of token pruning, not NAS. The key distinction is that NAS modifies network structure, while token pruning and our GSOP only changes the inputs / outputs of network components without structural changes. GSOP can be viewed as fine-grained token pruning on local components. While GSOP considers network structures for computational subdivision, these structures remain fixed. Similarly, prior token pruning methods [53, 49, 56] also leverage network structures (layer positions) for finer granularity. GSOP’s optimization-based policy search is not exclusive to NAS and has been adopted in prior token pruning methods [53].

## 3 Greedily Sorted Operation Pruning

Figure: (a) Decompose VLM decoder computations into operations.
Refer to caption: x4.png

Our goal is to enhance VLM efficiency by eliminating redundant computations while minimizing impact on performance. Following token pruning approaches [41, 11, 53, 55, 50, 49], we target the parameter-dense decoder (20× larger than the visual encoder) during the compute-bound prefilling stage [58]. We achieve this by decomposing the decoder’s computations into operations and identifying the most redundant operations to prune. Formally, denoting each operation as $\mathbf{o}$ and the set of all operations as $\mathcal{O}$, a pruning policy $\mathcal{P}$ is defined as:

$$ $\mathcal{P}=\{\mathbf{o}_{1},\mathbf{o}_{2},\dots\},\quad\mathcal{P}\subseteq \mathcal{O}.$ (1) $$

Note that some policies may overlap.
Our objective is to find the best policy $\mathcal{P}^{*}$ with ideal efficiency-performance trade-offs. Given $\tau$ as the minimum threshold of computation reduction required, we define the optimization objective as:

$$ $\displaystyle\mathcal{P}^{*}=\underset{\mathcal{P}\subseteq\mathcal{O}}{\arg\max}$ $\displaystyle\quad\text{Performance}(\mathcal{P})$ (2) s.t. $\displaystyle\quad\text{Computation}(\mathcal{P})\geq\tau$ $$

In this section, we present how GSOP decomposes VLM decoder computations into operations and how to efficiently determine the operation pruning policy.

### 3.1 Operation Pruning Framework

In this section, we introduce the framework of operation pruning by defining the atomic operations and optimization objective, as well as analyzing framework properties.

#### 3.1.1 Operation Definition

As shown in [Fig. 2(a)](https://arxiv.org/html/2507.02909v1#S3.F2.sf1), We decompose VLM decoder computations into atomic operations along three dimensions: token groups, layer positions, and modules. Formally, we define each basic operation as:

$$ $\mathbf{o}=(g,l,m),\quad\mathbf{o}\in\mathcal{O}=\mathcal{G}\times\mathcal{L} \times\mathcal{M}$ (3) $$

*Each operation $(g,l,m)$ represents the computations performed on tokens in group $g$ at layer $l$ in module $m$.*
$\mathcal{O}$ is the set of all operations, $\mathcal{G}=\{g_{1},g_{2}\}$ is the set of visual token groups, where $g_{1}$ represents critical tokens and $g_{2}$ represents redundant tokens, $\mathcal{L}=\{1,\ldots,L\}$ represents the set of layer indices, and $\mathcal{M}=\{\textit{MHA-out},\textit{MHA-in},\textit{MLP}\}$ is the set of module types, where MHA-out updates other tokens (key/value transforms), MHA-in updates current token (query/output transforms) and MLP applies token-wise update. These 3 dimensions are detailed below.

Token Groups. GSOP constructs operations by grouping tokens to avoid the excessive number of operations that would result from token-by-token division, thus reducing the computational cost of pruning policy optimization. Meanwhile, prior token pruning works [11, 41, 56] essentially divide visual tokens into two groups based on redundancy and prune the more redundant group. Thus, we adopt the redundancy-based binary grouping part to organize visual tokens into $\{g_{1},g_{2}\}$, without pruning tokens. While GSOP can encompass system and text tokens, we focuses on visual tokens for fair comparison with prior work [41, 11, 53, 55, 50, 49], which only prunes visual tokens. Following [55], GSOP uses attention scores from the visual encoder’s [CLS] token to divide visual tokens into two groups: $g_{1}$ containing the top-$r\%$ and $g_{2}$ containing the bottom-$(100-r)\%$.

Layer Positions. We consider the full layer range $l\in\{1,\ldots,L\}$ and optimize each layer individually in our GSOP. In contrast, prior token pruning works [11, 41, 56] typically apply coarse-grained optimizations by treating consecutive layers as a single block.

Modules. As shown in [Fig. 2(a)](https://arxiv.org/html/2507.02909v1#S3.F2.sf1), unlike prior work that treat a token as the atomic computation carrier, we further decompose each token’s computations at a single layer into three different modules:

- •
MHA-out: Computations when a token updates others, including key/value transformations and their matrix multiplications with all queries.
- •
MHA-in: Computations when a token is being updated, including query transformation, matrix multiplications with all keys/values, and output transformation.
- •
MLP: Token-wise feature transformation that operates independently on each token.

In a LLaVA1.5-7B [35] decoder layer, the computational proportions measured in Theoretical FLOPS are 17.5% for MHA-out, 17.5% for MHA-in, and 65% for MLP.

#### 3.1.2 Framework Analysis

Comparison with Token Pruning. Token pruning represents a constrained variant of operation pruning that works at a coarser token granularity. For token group $g$ and layer $l$, token pruning requires all operations $(g,l,*)$ to be pruned simultaneously, and once token group $g$ is removed, it remains excluded from all subsequent layers, forcing blocks $(g,l,*),(g,l+1,*),...$ to be pruned collectively. In addition, this paper primarily evaluates operation pruning alone for clarity, though our approach can be combined with token pruning as demonstrated in supplementary.

Policy Space Size. While our fine-grained operation pruning paradigm enables more precise computation control, it exponentially expands the policy space. For a 32-layer decoder with $|\mathcal{G}|=2$ token groups and $|\mathcal{M}|=3$ module types, the total operation count becomes:

$$ $|\mathcal{O}|=|\mathcal{L}|\times|\mathcal{M}|\times|\mathcal{G}|=32\times 3 \times 2=192$ (4) $$

resulting in $2^{192}\approx 6\times 10^{57}$ possible policies. Such an enormous policy space necessitates an efficient optimization algorithm, as conventional approaches would incur prohibitive computational costs.

### 3.2 Pruning Policy Optimization

In this section, we introduce GSOP’s operation sorting framework based on greedy search for determining the optimal pruning policy, as shown in [Fig. 2(b)](https://arxiv.org/html/2507.02909v1#S3.F2.sf2), along with two acceleration strategies: adaptive re-evaluation and pre-sorting operation filtering to reduce the search cost.

#### 3.2.1 Greedy Operation Sorting

The key idea of our pruning policy search is to employ a greedy approach that sorts operations for pruning according to their redundancy level, from the most to the least redundant. We denote this sorted sequence of operations by $\mathcal{T}$ as a permutation of the $n$ operations in operation set $\mathcal{O}$:

$$ $\mathcal{T}\triangleq\langle\hat{\mathbf{o}}_{1},\hat{\mathbf{o}}_{2},\dots, \hat{\mathbf{o}}_{n}\rangle=\text{Sort}(\mathcal{O}).$ (5) $$

To construct $\mathcal{T}$, we greedily sort operations based on model performance impact when pruned. The process is detailed as follows:

- •
Start with initial sequence $\mathcal{T}_{1}$ and initial prunable operation set $\mathcal{O}_{1}$ ($\mathcal{T}_{1}=\emptyset$ and $\mathcal{O}_{1}=\mathcal{O}$ if not pre-defined).
- •
At step $i\ (1\leq i\leq n)$, compute redundancy score for each candidate operation $\mathbf{o}\in\mathcal{O}_{i}$ by evaluating the validation performance when $\mathbf{o}$ is temporarily pruned with all operations in $\mathcal{T}_{i}$:
$\text{Score}(\mathbf{o})=\text{Performance}\big{(}\mathcal{T}_{i}\cup\{\mathbf
{o}\}\big{)}.$
(6)
- •
Select the operation with the highest redundancy score (minimal performance drop when pruned) as $\hat{\mathbf{o}}_{i}$:
$\hat{\mathbf{o}}_{i}=\arg\max_{\mathbf{o}\in\mathcal{O}_{i}}\text{Score}(
\mathbf{o}).$
(7)
Append it to $\mathcal{T}_{i}$ and remove it from $\mathcal{O}_{i}$:
$\mathcal{T}_{i+1}=\mathcal{T}_{i}\cup\{\hat{\mathbf{o}}_{i}\},\quad\mathcal{O}
_{i+1}=\mathcal{O}_{i}\setminus\{\hat{\mathbf{o}}_{i}\}.$
(8)
- •
If all operations are in $\mathcal{T}_{i}$, output it as the final $\mathcal{T}$

#### 3.2.2 Budget-Aware Pruning Policy

For computation measurements in [Eq. 2](https://arxiv.org/html/2507.02909v1#S3.E2), we adopt Theoretical FLOPS (TFLOPS) as our metric, which not only remains invariant to hardware devices and implementation details but has also been widely used in prior works [41, 11, 53, 55, 50, 49], while GSOP can also enable optimization based on hardware-specific metrics.

Given a target computation reduction $\tau$ in TFLOPS and the sorted operation sequence $\mathcal{T}=\langle\hat{\mathbf{o}}_{1},\hat{\mathbf{o}}_{2},\dots,\hat{
\mathbf{o}}_{n}\rangle$, we derive the optimal pruning policy $\mathcal{P}^{*}=\{\hat{\mathbf{o}}_{1},\dots,\hat{\mathbf{o}}_{k^{*}}\}$ by truncating $\mathcal{T}$ at the minimal position $k^{*}$ where the accumulated TFLOPS reduction meets $\tau$:

$$ $\mathcal{P}^{*}=\mathcal{P}_{k^{*}}=\{\hat{\mathbf{o}}_{1}\dots,\hat{\mathbf{o }}_{k^{*}}\},\$ $$

$$ $k^{*}=\min\{k\in[1,n]\ \big{|}\ \text{TFLOPS}(\mathcal{P}_{k})\geq\tau\}.$ (9) $$

Figure: (a) Adaptive Operation Evaluation.
Refer to caption: x6.png

#### 3.2.3 Accelerate Operation Sorting

GSOP uses operation sorting based on greedy search to reduce the policy search complexity from $O(2^{|\mathcal{O}|})$ to $O(|\mathcal{O}|^{2})$. However, it’s still computationally expensive due to frequent operation evaluations and large operation set size. Thus, GSOP further employs adaptive operation evaluation to reduce operation evaluation frequency; pre-sorting operation filtering to exclude the “free-to-prune” operations in deep layers and “danger-to-prune” operations in shallow layers from greedy sorting.

Adaptive Operation Re-evaluation. While naively re-evaluating all candidate operations at every step is computationally expensive, we observe that the relative redundancy ranking of operations remains largely consistent across consecutive steps and the ranking discrepancy grows gradually as expanding $\mathcal{T}_{i}$. As shown in [Fig. 3(a)](https://arxiv.org/html/2507.02909v1#S3.F3.sf1), we leverage these properties by dynamically reusing historical evaluation results and triggering operation re-evaluation only when pruning $\mathcal{T}_{i}$ drops model performance below predefined thresholds. Specifically, we define $Z$ monotonically decreasing performance thresholds $\mu_{1},\mu_{2},\dots,\mu_{Z}$ where $\mu_{1}>\mu_{2}>\dots>\mu_{Z}$. At step $i$ with performance threshold $\mu=\mu_{z}$, we reuse the most recent evaluation results at step $j$ ($j\leq i$) to select $\hat{\mathbf{o}}_{i}$ by:

$$ $\displaystyle\hat{\mathbf{o}}_{i}$ $\displaystyle=\arg\max_{\mathbf{o}\in\mathcal{O}_{i}}\text{Score}(\mathbf{o})$ $\displaystyle=\arg\min_{\mathbf{o}\in\mathcal{O}_{i}}\text{Performance}\big{(} \mathcal{T}_{j}\cup\{\mathbf{o}\}\big{)}.$ (10) $$

Then we decide whether re-evaluate all operations by $\text{Performance}\big{(}\mathcal{T}_{i}\cup\{\hat{\mathbf{o}}_{i}\}\big{)}$:

- •
If $\text{Performance}\big{(}\mathcal{T}_{i}\cup\{\hat{\mathbf{o}}_{i}\}\big{)}
\geq\mu_{z}$, we continue the sorting process with [Eq. 8](https://arxiv.org/html/2507.02909v1#S3.E8).
- •
If $\text{Performance}\big{(}\mathcal{T}_{i}\cup\{\hat{\mathbf{o}}_{i}\}\big{)}<
\mu_{z}$, we evaluate all operations based on $\mathcal{T}_{i}$ and update the redundancy scores:
$\text{Score}(\mathbf{o})=\text{Performance}\big{(}\mathcal{T}_{i}\cup\{\mathbf
{o}\}\big{)},\forall\mathbf{o}\in\mathcal{O}_{i}.$
(11)
–
If $\forall\mathbf{o}\in\mathcal{O}_{i},\ \text{Score}(\mathbf{o})<\mu_{z}$, decrease the performance threshold to $\mu=\mu_{z+1}$.

The complexity of our adaptive re-evaluation strategy depends on the consistency period of operation rankings. While the worst-case complexity remains $O(|\mathcal{O}|^{2})$, we observe that rankings typically remain effective for tens of steps, substantially reducing the block evaluation frequency.

Pre-Sorting Operation Filtering. To further reduce sorting complexity, we identify two operation types: (1) “free-to-prune” operations in deeper layers that can be safely pruned without performance impact; and (2) “danger-to-prune” operations in earlier layers that cause sharp performance drops when removed. As shown in [Fig. 3(b)](https://arxiv.org/html/2507.02909v1#S3.F3.sf2) we exclude both categories from the sorting process by automatically pruning the former and preserving the latter.

- •
Free-to-prune Operations. To ensure that “free-to-prune” pruning operations really induce negligible performance impact when pruned together, we adopt a strict depth-wise pruning criterion: For each group-module pair $(g,m)\in G\times M$, we binary-search for the earliest layer $l^{*}_{g,m}$ where pruning all operations $\{(g,l,m)\mid l\geq l^{*}_{g,m}\}$ causes no performance degradation:
$\displaystyle\mathcal{P}^{ll}_{g,m}$
$\displaystyle=\{(g,l,m)\mid l\geq l^{*}_{g,m}\}$
$\displaystyle\text{s.t.}\quad\text{Performance}(\mathcal{P}^{ll}_{g,m})\geq
\text{Performance}(\emptyset)$
(12)
This guarantees safety even under cumulative pruning effects while the binary search ensures efficiency.
Then, we aggregate all resulting operations into a “free-to-prune” operation set $\mathcal{O}^{f}$ to initialize the operation sorting:
$\mathcal{T}_{1}=\mathcal{O}^{f}=\bigcup_{(g,m)\in\mathcal{G}\times\mathcal{M}}
\mathcal{P}^{ll}_{g,m}$
(13)
- •
Danger-to-prune Operations. For “danger-to-prune” operations, we focus solely on group $g_{1}$, which constitutes the critical token group. Given a layer threshold $l_{d}$, we designate all operations involving $g_{1}$ in layers up to $l_{d}$ as “danger-to-prune” operations and exclude both these and the “free-to-prune” operations from the sorting set:
$\mathcal{O}_{1}=\mathcal{O}\setminus\left(\mathcal{O}^{f}\cup\{(g_{1},l,m)\mid
l
\leq l_{d},m\in\mathcal{M}\}\right)$
(14)

### 3.3 Implementation Details

Token Group Ordering. With tokens classified into critical group $g_{1}$ and redundant group $g_{2}$, we enforce that whenever $(g_{1},l,m)$ is pruned, $(g_{2},l,m)$ is also pruned if still present. This ensures more redundant operations are always removed no later than their less redundant counterparts.

Theoretical FLOPS. We calculate the theoretical FLOPs (TFLOPs) for VLM decoder layers following the LLaMA architecture. For one layer with hidden size $h$, key/value feature dimension $d$, and MLP intermediate dimension $m$, the total TFLOPs can be expressed as:

$$ $\displaystyle\text{TFLOPs}=$ $\displaystyle\underbrace{2d((n_{\text{out}}+2n_{\text{in}})h+2n_{\text{out}}n_ {\text{in}}+n_{\text{out}}h)}_{\text{Multi-Head Attention}}$ (15) $\displaystyle+\underbrace{6n_{\text{mlp}}hm}_{\text{LLaMA-style MLP}}$ $$

where $n_{\text{in}}$, $n_{\text{out}}$, and $n_{\text{mlp}}$ denote the number of remaining tokens in MHA-in, MHA-out, and MLP modules respectively. For architectures with different designs (e.g., Phi-3), we adjust the calculations accordingly in our implementation. Detailed derivations can be found in the supplementary materials.

## 4 Experiments

**Table 1: Performance comparison on LLaVA-1.5 [35] under different computation budgets. We evaluate LLaVA-1.5-7B and LLaVA-1.5-13B across 6 benchmarks: GQA [19], SeedBench($\text{Seed}^{\text{I}}$) [24], MME($\text{MME}^{\text{P}}$) [17], MMBench(MMB) [36], OKVQA [37] and POPE [26]. We report accuracy (%) for GQA, $\text{Seed}^{\text{I}}$, and OKVQA, gpt score for MMB, perception score for $\text{MME}^{\text{P}}$, and F1-score ($\times$100) for POPE. Avg. represents the mean performance ratio relative to Vanilla across all benchmarks (higher is better). Bold and underlined values indicate best and second-best performances in each setting. GSOP consistently outperforms all baselines with policies only optimized on a subset of 500 GQA samples, exceeding the strongest competitors FasterVLM and VisionZip by up to 4.3% in absolute average performance ratio. Lower Bound indicates the minimum possible computation in our settings. Our comparison matches actual TFLOPs, while token ratio only translates to methods with the same visual token number across layers [55, 50, 41] for easier comparison.**
|  | LLaVA-1.5-7B | LLaVA-1.5-13B |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Method | GQA | Seed${}^{\text{I}}$ | MME${}^{\text{P}}$ | MMB | OKVQA | POPE | Avg. | GQA | Seed${}^{\text{I}}$ | MME${}^{\text{P}}$ | MMB | OKVQA | POPE | Avg. |
| Base Model | 61.97 | 66.19 | 1506 | 64.80 | 53.40 | 85.90 | 100% | 63.30 | 67.06 | 1522 | 68.70 | 58.20 | 85.90 | 100% |
| Retain 35% TFLOPs (= 20% Visual Tokens) |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| PruMerge [41] | 53.01 | 54.80 | 1241 | 58.51 | 43.26 | 66.77 | 83.3% | 53.71 | 57.52 | 1243 | 60.57 | 53.33 | 66.69 | 84.9% |
| FastV [11] | 50.63 | 52.14 | 1143 | 55.84 | 42.11 | 56.39 | 63.6% | 53.08 | 56.87 | 1295 | 59.88 | 47.42 | 63.71 | 82.8% |
| PDrop [49] | 52.06 | 51.99 | 1204 | 56.19 | 43.80 | 66.73 | 81.5% | 57.54 | 60.46 | 1460 | 65.81 | 55.27 | 81.10 | 82.8% |
| SparseVLM [56] | 54.60 | 60.95 | 1314 | 62.63 | 48.79 | 78.09 | 91.1% | 56.92 | 63.53 | 1437 | 66.75 | 56.12 | 77.79 | 93.9% |
| FitPrune [53] | 57.64 | 61.80 | 1431 | 61.51 | 49.77 | 76.91 | 93.2% | 57.64 | 64.15 | 1455 | 66.84 | 54.76 | 77.31 | 93.9% |
| VisionZip [50] | 57.58 | 61.12 | 1428 | 62.20 | 50.23 | 83.05 | 94.5% | 57.90 | 63.61 | 1442 | 66.41 | 56.55 | 81.83 | 95.0% |
| FasterVLM [55] | 57.43 | 61.48 | 1431 | 62.63 | 50.17 | 82.38 | 94.5% | 57.41 | 63.44 | 1436 | 67.35 | 56.24 | 81.65 | 94.9% |
| GSOP | 58.04 | 63.80 | 1511 | 63.23 | 50.85 | 84.11 | 96.9% | 60.83 | 66.11 | 1449 | 66.32 | 54.48 | 84.17 | 96.3% |
| Retain 30% TFLOPs (= 14% Visual Tokens) |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| PruMerge [41] | 51.67 | 53.71 | 1230 | 55.76 | 42.43 | 65.80 | 81.4% | 52.89 | 56.45 | 1271 | 60.14 | 52.25 | 65.31 | 84.1% |
| FastV [11] | 41.06 | 41.40 | 749 | 32.99 | 17.42 | 15.37 | 46.7% | 52.22 | 55.32 | 1278 | 59.62 | 46.22 | 60.28 | 80.9% |
| PDrop [49] | 46.86 | 48.40 | 977 | 51.03 | 37.27 | 46.63 | 69.4% | 54.87 | 56.14 | 1339 | 62.89 | 51.36 | 74.26 | 87.4% |
| SparseVLM [56] | 53.14 | 58.13 | 1232 | 59.88 | 46.73 | 75.31 | 87.2% | 54.96 | 61.52 | 1363 | 65.89 | 53.29 | 75.50 | 90.6% |
| FitPrune [53] | 52.30 | 54.94 | 1150 | 57.04 | 41.16 | 57.70 | 79.3% | 54.09 | 57.81 | 1241 | 61.00 | 48.34 | 61.30 | 82.7% |
| VisionZip [50] | 55.93 | 58.93 | 1381 | 60.91 | 49.25 | 79.23 | 91.6% | 57.20 | 61.41 | 1429 | 66.67 | 54.88 | 78.06 | 93.0% |
| FasterVLM [55] | 56.10 | 59.40 | 1379 | 61.17 | 48.96 | 79.15 | 91.7% | 56.78 | 62.00 | 1439 | 66.92 | 54.78 | 78.40 | 93.2% |
| GSOP | 58.28 | 63.18 | 1439 | 63.06 | 51.04 | 84.26 | 96.0% | 60.42 | 65.14 | 1423 | 65.98 | 55.88 | 84.54 | 96.1% |
| Retain 25% TFLOPs (= 8% Visual Tokens) |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| PruMerge [41] | 51.60 | 52.93 | 1232 | 54.12 | 41.90 | 65.28 | 76.0% | 52.38 | 55.94 | 1259 | 59.19 | 50.83 | 64.06 | 82.8% |
| FastV [11] | 39.78 | 38.12 | 699 | 26.37 | 16.05 | 20.36 | 43.8% | 43.48 | 49.27 | 918 | 48.80 | 34.35 | 26.01 | 60.5% |
| PDrop [49] | 40.87 | 40.86 | 744 | 30.50 | 21.94 | 15.85 | 47.3% | 48.20 | 48.96 | 1047 | 52.49 | 42.85 | 52.23 | 71.5% |
| SparseVLM [56] | 46.34 | 52.22 | 1040 | 54.30 | 38.51 | 52.25 | 73.2% | 50.68 | 56.54 | 1122 | 61.17 | 46.81 | 64.47 | 80.4% |
| FitPrune [53] | 46.41 | 47.73 | 979 | 47.25 | 33.92 | 44.93 | 66.8% | 49.49 | 51.28 | 1142 | 55.93 | 42.28 | 52.18 | 74.1% |
| VisionZip [50] | 53.56 | 55.49 | 1318 | 60.22 | 46.56 | 74.02 | 87.3% | 54.87 | 58.24 | 1354 | 63.40 | 51.94 | 71.78 | 87.9% |
| FasterVLM [55] | 53.73 | 56.21 | 1332 | 59.62 | 45.02 | 73.22 | 86.9% | 54.44 | 58.47 | 1341 | 64.18 | 51.14 | 71.48 | 87.6% |
| GSOP | 55.61 | 59.40 | 1360 | 61.68 | 48.32 | 76.46 | 90.8% | 57.51 | 63.15 | 1329 | 64.09 | 51.69 | 81.91 | 91.6% |
| Lower Bound: Retain 18.6% TFLOPs (= 0 Visual Token) |  |  |  |  |  |  |  |  |  |  |  |  |  |  |

In this section, we evaluate GSOP across different VLM architectures and various multimodal tasks. Comprehensive comparisons with state-of-the-art token pruning and token merging methods demonstrate the superior performance of GSOP. We further investigate the cross-task and cross-architecture generalization capability of the learned pruning policies. Additionally, we also report GSOP’s actual acceleration as well as optimization costs on GPU.

### 4.1 Experimental Setup

VLM Models.
We evaluate GSOP across 5 different models: LLaVA-1.5 (7B&13B) [35], LLaVA-Phi3 [39], VILA-1.5(3B) [29] and LLaVA-Next(7B) [34] (in Supp).

Evaluation Benchmarks.
We select 6 benchmarks for major evaluation and comparison: GQA [19], SeedBench [24], MME [17], MMBench [36], OKVQA [37] and POPE [26]. We use the image parts of SeedBench ($\text{Seed}^{\text{I}}$), and the perception subset of MME ($\text{MME}^{\text{P}}$).

GSOP Method Configurations.
For fair comparison with token pruning and merging baselines, we primarily focus on operations associated with image tokens in our main evaluation and comparison, while additional operations are examined in the ablation studies. Image tokens are divided into two groups based on the attention scores from the visual encoder’s [CLS] token, with alternative grouping techniques investigated in our ablations. We optimize GSOP on a validation set consisting of 500 VQA instances. More detailed algorithm configurations are provided in the supplementary.

GSOP Optimization and Transfer.
To avoid data leakage providing unfair advantages to GSOP and to mitigate concerns about search costs, instead of conducting searches on each dataset and model combination, we only perform optimization on a randomly sampled subset of 500 instances from GQA [19] using LLaVA-1.5 (7B/13B) [35]. We evaluate generalization capabilities in two dimensions: (1) dataset generalization - testing GQA-optimized policies on GQA itself and five other benchmarks; (2) architecture generalization - applying LLaVA-1.5-7B policies to different architectures including LLaVA-Phi3 [39], VILA-1.5(3B) [29] and LLaVA-Next(7B) [34] (in supplementary).

Baselines. We compare GSOP with state-of-the-art token pruning and merging methods. Pruning baselines include FastV [11] (ECCV 2024), FitPrune [53] (AAAI 2025), PyramidDrop (PDrop) [49] (CVPR 2025), and FasterVLM [55] (2024.12). Merging baselines include PruMerge [41] (2024.03) and VisionZip [50] (CVPR 2025).

- •
Optimization-based Baselines. For fair comparison with GSOP, we optimize both FitPrune’s layer-wise pruning ratios and FastV’s pruning layer/ratio using the same validation set (details in supplementary).

### 4.2 Results and Comparisons

[Tab. 1](https://arxiv.org/html/2507.02909v1#S4.T1) evaluates GSOP on LLaVA-1.5-7B and 13B [35] across six benchmarks under three TFLOPs constraints, with policies optimized on GQA [19] but evaluated broadly. Similarly, [Tabs. 2](https://arxiv.org/html/2507.02909v1#S4.T2) and [3](https://arxiv.org/html/2507.02909v1#S4.T3) shows GSOP performance on LLaVA-Phi3[39] and VILA-1.5-3B [29] under two TFLOPs constraints, using policies transferred from LLaVA-1.5-7B and GQA optimizations. Across all settings, GSOP outperforms state-of-the-art with superior efficiency-performance tradeoffs, demonstrating both precise operation pruning and strong generalization capabilities.

Results on LLaVA-1.5 & GQA. As shown in [Tab. 1](https://arxiv.org/html/2507.02909v1#S4.T1), on the LLaVA-1.5 [35] & GQA [19] used for optimization, GSOP consistently outperforms all other methods on this “In-Domain” setting by substantial margins. Even when compared to FastV [11] and FitPrune [53] baselines optimized on the identical GQA subset, GSOP achieves remarkable improvements, with relative gains of up to 42% over FastV and up to 20% over FitPrune. These results confirm GSOP’s inherent methodological superiority over other optimization-based approaches under identical conditions.

Cross-Task Generalization. As shown in [Tab. 1](https://arxiv.org/html/2507.02909v1#S4.T1), the policy optimized on GQA [19] transfers remarkably well to other datasets, maintaining superior average performance across diverse benchmarks. When applied to SeedBench [24] and POPE [26], GSOP continues to significantly outperform competing methods. Similarly, when evaluated on MMB [36], MME^P [17], and OKVQA [37], our approach demonstrates robust and consistent results. Most significantly, GSOP surpasses even the strongest token pruning and merging baselines—FasterVLM [55] and VisionZip [50]—by retaining up to 4.3% more of the base model’s performance, confirming its state-of-the-art status across the full spectrum of evaluated tasks. These results demonstrate GSOP’s robust cross-task generalization, confirming its advantages are not from data leakage while addressing search cost concerns—a single task-optimized policy effectively generalizes across multiple domains without additional optimization.

**Table 2: Performance comparison on LLaVA-Phi3 [39] under different computation budgets. Benchmarks and metrics settings are identical to [Tab. 1](https://arxiv.org/html/2507.02909v1#S4.T1). GSOP pruning policies were optimized on LLaVA-1.5-7B [35] using 500 GQA [19] samples and transferred to LLaVA-Phi3, yet GSOP consistently outperforms state-of-the-art token pruning baselines. Lower Bound indicates the minimum possible computation in our settings. Our comparison matches actual TFLOPs, while token ratio only translates to methods with the same visual token number across layers [55].**
| Method | GQA | Seed${}^{\text{I}}$ | MME${}^{\text{P}}$ | MMB | OKVQA | POPE | Avg |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Base Model | 61.65 | 67.67 | 1471 | 67.96 | 55.69 | 85.68 | 100% |
| Retain 35% TFLOPs (= 23% Visual Tokens) |  |  |  |  |  |  |  |
| FastV [11] | 55.14 | 60.09 | 1342 | 65.38 | 53.94 | 74.78 | 91.6% |
| FasterVLM [55] | 56.26 | 64.38 | 1417 | 65.98 | 54.57 | 79.80 | 95.2% |
| GSOP | 58.24 | 64.93 | 1468 | 67.01 | 54.73 | 83.19 | 97.4% |
| Retain 25% TFLOPs (= 11% Visual Tokens) |  |  |  |  |  |  |  |
| FastV [11] | 46.55 | 48.63 | 991 | 52.84 | 45.19 | 52.12 | 72.4% |
| FasterVLM [55] | 52.68 | 60.41 | 1356 | 63.66 | 52.17 | 72.78 | 89.9% |
| GSOP | 53.93 | 61.00 | 1384 | 65.64 | 51.35 | 78.54 | 92.0% |
| Lower Bound: Retain 15.8% TFLOPs (= 0 Visual Token) |  |  |  |  |  |  |  |

**Table 3: Performance comparison on VILA-1.5-3B [29] under different computation budgets. Benchmarks and metrics settings are identical to [Tab. 1](https://arxiv.org/html/2507.02909v1#S4.T1). GSOP pruning policies were optimized on LLaVA-1.5-7B [35] using 500 GQA [19] samples and transferred to VILA-1.5-3B, yet GSOP consistently outperforms state-of-the-art token pruning baselines, notably surpassing the strongest baseline FasterVLM [55] by retaining up to 18% more of the base model’s performance. Lower Bound indicates the minimum possible computation in our settings. Our comparison matches actual TFLOPs, while token ratio only translates to methods with the same visual token number across layers [55].**
| Method | GQA | Seed${}^{\text{I}}$ | MME${}^{\text{P}}$ | MMB | OKVQA | POPE | Avg |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Base Model | 61.44 | 67.89 | 1434 | 61.77 | 50.42 | 85.91 | 100% |
| Retain 55% TFLOPs (= 25% Visual Tokens) |  |  |  |  |  |  |  |
| FastV [11] | 51.13 | 57.45 | 1202 | 49.48 | 37.00 | 63.15 | 79.8% |
| FasterVLM [55] | 55.18 | 62.78 | 1366 | 56.96 | 42.52 | 80.18 | 91.2% |
| GSOP | 56.34 | 63.51 | 1400 | 59.02 | 40.57 | 80.91 | 92.2% |
| Retain 45% TFLOPs (= 8% Visual Tokens) |  |  |  |  |  |  |  |
| FastV [11] | 46.64 | 51.38 | 1031 | 38.49 | 27.86 | 49.84 | 66.5% |
| FasterVLM [55] | 47.09 | 53.25 | 1092 | 41.07 | 23.20 | 56.14 | 68.2% |
| GSOP | 53.20 | 62.35 | 1246 | 55.33 | 36.36 | 75.93 | 85.9% |
| Lower Bound: Retain 40.3% TFLOPs (= 0 Visual Token) |  |  |  |  |  |  |  |

Cross-Model Generalization. We transfer GSOP optimized on LLaVA-1.5-7B [19] and GQA to LLaVA-Phi3 [39] ([Tab. 2](https://arxiv.org/html/2507.02909v1#S4.T2)) and VILA-1.5-3B [29] ([Tab. 3](https://arxiv.org/html/2507.02909v1#S4.T3)). Since FasterVLM [55] is the strongest baseline in [Tab. 1](https://arxiv.org/html/2507.02909v1#S4.T1), we primarily compare against it and against FasterVLM with parameters optimized on each model’s GQA [19] subset. GSOP consistently outperforms these baselines across all models. Results on LLaVA-Phi3 ([Tab. 2](https://arxiv.org/html/2507.02909v1#S4.T2)) demonstrate GSOP’s generalization capability, as policies optimized on LLaVA-1.5’s Vicuna decoder successfully transfer to Phi3’s architecture. Comparing LLaVA-1.5-7B ([Tab. 1](https://arxiv.org/html/2507.02909v1#S4.T1)) and VILA-1.5-3B ([Tab. 3](https://arxiv.org/html/2507.02909v1#S4.T3)) results reveals interesting patterns: LLaVA-1.5-7B uses 576 visual tokens while VILA-1.5-3B uses only 196, making each token a coarser computation carrier in VILA and reducing token pruning’s precision. This explains why, at the TFLOPS ratio corresponding to 8% visual tokens, GSOP outperforms FasterVLM by 18% absolute average performance ratio across all benchmarks on VILA-1.5-3B, compared to only 4% on LLaVA-1.5-7B. We also transfer GSOP to LLaVA-Next [34] in Supp.

### 4.3 Efficiency on GPU Device

**Table 4: Our GSOP’s acceleration effects on LLaVA-1.5 [35] measured on a NVIDIA A40 GPU compared to GSOP. Blue values represent LLaVA-1.5-7B results, while green values represent LLaVA-1.5-13B results. Time and latency are calculated on 200 GQA [19] samples with KVCache enabled. Our comparison matches actual TFLOPs, while token ratio only translates to methods with the same visual token number across layers [55].**
| Method | Latency (ms) | Time (s) |  |
| --- | --- | --- | --- |
| Prefill | Sample |  |  |
| Retain 100% TFLOPs |  |  |  |
| Vanilla | 154/260 | 219/357 | 55/80 |
| Retain 30% TFLOPs (= 14% Visual Tokens) |  |  |  |
| VisionZip [55] | 72/108 | 137/200 | 37/49 |
| GSOP | 76/120 | 136/215 | 35/50 |

Table [4](https://arxiv.org/html/2507.02909v1#S4.T4) showcases the acceleration benefits of our GSOP approach on LLaVA-1.5-7B [35] when tested on an NVIDIA A40 GPU at 30% TFLOPS ratios. For evaluation, we benchmarked against VisionZip [50] using identical VLM implementation and measured decoder prefilling latency, per-sample processing latency, and total processing time across 200 GQA [19] samples. The results demonstrate that GSOP achieves comparable acceleration to VisionZip. The minimal differences stem from implementation details like indexing, which optimized implementations would minimize. Additionally, deployment with FlashAttention [15, 14] is discussed in the supplementary materials.

## 5 Ablation Study

In this section, we conduct ablation studies on optimizing GSOP directly on target tasks and target models. More ablations on visual tokens grouping methods, pruning more than visual token/operations and applying GSOP based on token pruning are presented in the supplementary materials.

**Table 5: Evaluating task-specific GSOP optimization on LLaVA-1.5-7B [35]. Benchmarks and metrics settings are identical to [Tab. 1](https://arxiv.org/html/2507.02909v1#S4.T1). While task-specific optimization GSOP-Direct yields better results, GSOP-Transfer transferred from GQA [19] remains competitive. Our comparison matches actual TFLOPs, while token ratio only translates to methods with the same visual token number across layers [55].**
| Method | GQA | Seed${}^{\text{I}}$ | MME${}^{\text{P}}$ | MMB | OKVQA | POPE | Avg |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Vanilla | 61.97 | 66.19 | 1506 | 64.80 | 53.40 | 85.90 | 100% |
| Retain 35% TFLOPs (= 20% Visual Tokens) |  |  |  |  |  |  |  |
| GSOP-Transfer | 58.04 | 63.80 | 1511 | 63.23 | 50.85 | 84.11 | 96.9% |
| GSOP-Direct | 58.04 | 64.20 | 1498 | 63.48 | 51.64 | 84.47 | 97.2% |
| Retain 25% TFLOPs (= 8% Visual Tokens) |  |  |  |  |  |  |  |
| GSOP-Transfer | 55.61 | 59.40 | 1360 | 61.68 | 48.32 | 76.46 | 90.8% |
| GSOP-Direct | 55.61 | 60.43 | 1341 | 62.14 | 50.17 | 75.64 | 91.3% |
| Lower Bound: Retain 18.6% TFLOPs (= 0 Visual Token) |  |  |  |  |  |  |  |

Task-specific Optimization  [Tab. 5](https://arxiv.org/html/2507.02909v1#S5.T5) presents task-specific optimization ablation results on LLaVA-1.5-7B [35], where we directly optimize GSOP on each target benchmark using 500 randomly sampled examples. As expected, GSOP directly optimized on target tasks performs better, while GSOP transferred from GQA [19] remains competitive. However, considering the search costs that scale proportionally with the number of target benchmarks and concerns about data leakage, the transferred GSOP appears to be a safer and more cost-effective choice.

**Table 6: Evaluating model-specific GSOP optimization on LLaVA-Phi3 [39] and VILA-1.5.3B [29]. Benchmarks and metrics settings are identical to [Tab. 1](https://arxiv.org/html/2507.02909v1#S4.T1). While model-specific optimization GSOP-Direct yields better results, GSOP-Transfer transferred from LLaVA-1.5-7B [35] remains competitive.**
| Model | LLaVA-Phi3 | VILA-3B |  |  |
| --- | --- | --- | --- | --- |
| TFLOPS | 35% | 25% | 55% | 45% |
| GSOP-Direct | 97.1% | 92.6% | 92.4% | 86.2% |
| GSOP-Transfer | 97.4% | 92.0% | 92.2% | 85.9% |

Model-specific Optimization [Tab. 6](https://arxiv.org/html/2507.02909v1#S5.T6) illustrates model-specific results revealing that at higher FLOPs ratios, the performance difference between transferred GSOP and directly optimized GSOP is negligible. Although at lower FLOPs ratios, directly optimized GSOP exhibits better performance, transferred GSOP remains competitive in performance while saving optimization costs.

## 6 Conclusion

In this work, we investigated computational redundancy in Vision-Language Models and proposed GSOP, a precise operation pruning framework that outperforms token-centric approaches. By decomposing computations into fine-grained operations, our greedy sorting algorithm efficiently eliminates redundancies while maintaining performance. GSOP preserves up to 18% more performance than the state-of-the-art in VLM acceleration across 6 tasks and 5 VLM models without requiring task/model-specific re-optimization, demonstrating remarkable generalization. On-device experiments verify GSOP’s practical use.

## References

- Abdin et al. [2024]
Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, et al.
Phi-3 technical report: A highly capable language model locally on your phone.
*arXiv preprint arXiv:2404.14219*, 2024.
- Achiam et al. [2023]
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al.
Gpt-4 technical report.
*arXiv:2303.08774*, 2023.
- Bai et al. [2023a]
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al.
Qwen technical report.
*arXiv:2309.16609*, 2023a.
- Bai et al. [2023b]
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou.
Qwen-VL: A frontier large vision-language model with versatile abilities.
*arXiv:2308.12966*, 2023b.
- Bolya and Hoffman [2023]
Daniel Bolya and Judy Hoffman.
Token merging for fast stable diffusion.
In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pages 4599–4603, 2023.
- Bolya et al. [2022]
Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman.
Token merging: Your vit but faster.
*arXiv preprint arXiv:2210.09461*, 2022.
- Brown et al. [2020]
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.
Language models are few-shot learners.
*Advances in neural information processing systems*, 2020.
- Cao et al. [2023]
Qingqing Cao, Bhargavi Paranjape, and Hannaneh Hajishirzi.
Pumer: Pruning and merging tokens for efficient vision language models.
*arXiv preprint arXiv:2305.17530*, 2023.
- Chen et al. [2024a]
Jieneng Chen, Luoxin Ye, Ju He, Zhao-Yang Wang, Daniel Khashabi, and Alan Yuille.
Llavolta: Efficient multi-modal models via stage-wise visual context compression.
*arXiv preprint arXiv:2406.20092*, 2024a.
- Chen et al. [2023a]
Lin Chen, Jisong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin.
Sharegpt4v: Improving large multi-modal models with better captions.
*arXiv:2311.12793*, 2023a.
- Chen et al. [2024b]
Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang.
An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models.
In *Proceedings of the European Conference on Computer Vision (ECCV)*, 2024b.
- Chen et al. [2023b]
Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai.
InternVL: Scaling up vision foundation models and aligning for generic visual-linguistic tasks.
*arXiv:2312.14238*, 2023b.
- Dai et al. [2023]
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi.
InstructBLIP: Towards general-purpose vision-language models with instruction tuning.
*Advances in Neural Information Processing Systems*, 2023.
- Dao [2024]
Tri Dao.
FlashAttention-2: Faster attention with better parallelism and work partitioning.
In *International Conference on Learning Representations (ICLR)*, 2024.
- Dao et al. [2022]
Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré.
Flashattention: Fast and memory-efficient exact attention with io-awareness.
*Advances in Neural Information Processing Systems*, 35:16344–16359, 2022.
- Du et al. [2021]
Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang.
GLM: General language model pretraining with autoregressive blank infilling.
*arXiv:2103.10360*, 2021.
- Fu et al. [2023]
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, et al.
MME: A comprehensive evaluation benchmark for multimodal large language models.
*arXiv:2306.13394*, 2023.
- Gunasekar et al. [2023]
Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, et al.
Textbooks are all you need.
*arXiv preprint arXiv:2306.11644*, 2023.
- Hudson and Manning [2019]
Drew A Hudson and Christopher D Manning.
GQA: A new dataset for real-world visual reasoning and compositional question answering.
*Conference on Computer Vision and Pattern Recognition (CVPR)*, 2019.
- Kim et al. [2024]
Minchul Kim, Shangqian Gao, Yen-Chang Hsu, Yilin Shen, and Hongxia Jin.
Token fusion: Bridging the gap between token pruning and token merging.
In *Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision*, pages 1383–1392, 2024.
- Kwon et al. [2023]
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica.
Efficient memory management for large language model serving with pagedattention.
In *Proceedings of the 29th Symposium on Operating Systems Principles*, pages 611–626, 2023.
- Lee and Hong [2024]
Dong Hoon Lee and Seunghoon Hong.
Learning to merge tokens via decoupled embedding for efficient vision transformers.
*arXiv preprint arXiv:2412.10569*, 2024.
- Lee et al. [2024]
Seon-Ho Lee, Jue Wang, Zhikang Zhang, David Fan, and Xinyu Li.
Video token merging for long-form video understanding.
*arXiv preprint arXiv:2410.23782*, 2024.
- Li et al. [2023a]
Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan.
Seed-bench: Benchmarking multimodal llms with generative comprehension.
*arXiv:2307.16125*, 2023a.
- Li et al. [2023b]
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi.
Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.
In *International conference on machine learning*, 2023b.
- Li et al. [2023c]
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen.
Evaluating object hallucination in large vision-language models.
*arXiv:2305.10355*, 2023c.
- Li et al. [2024]
Yanwei Li, Chengyao Wang, and Jiaya Jia.
LLaMA-VID: An image is worth 2 tokens in large language models.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, 2024.
- Lin et al. [2023a]
Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, and Li Yuan.
Video-llava: Learning united visual representation by alignment before projection.
*arXiv:2311.10122*, 2023a.
- Lin et al. [2023b]
Ji Lin, Hongxu Yin, Wei Ping, Yao Lu, Pavlo Molchanov, Andrew Tao, Huizi Mao, Jan Kautz, Mohammad Shoeybi, and Song Han.
Vila: On pre-training for visual language models, 2023b.
- Lin et al. [2024]
Zhihang Lin, Mingbao Lin, Luxi Lin, and Rongrong Ji.
Boosting multimodal large language models with visual tokens withdrawal for rapid inference.
*arXiv preprint arXiv:2405.05803*, 2024.
- Liu et al. [2021]
Aoming Liu, Zehao Huang, Zhiwu Huang, and Naiyan Wang.
Direct differentiable augmentation search.
In *Proceedings of the IEEE/CVF international conference on computer vision*, pages 12219–12228, 2021.
- Liu et al. [2018]
Hanxiao Liu, Karen Simonyan, and Yiming Yang.
Darts: Differentiable architecture search.
*arXiv preprint arXiv:1806.09055*, 2018.
- Liu et al. [2023a]
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee.
Improved baselines with visual instruction tuning.
*arXiv:2310.03744*, 2023a.
- Liu et al. [2024a]
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee.
Llava-next: Improved reasoning, ocr, and world knowledge, 2024a.
- Liu et al. [2024b]
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee.
Visual instruction tuning.
*Advances in neural information processing systems*, 2024b.
- Liu et al. [2023b]
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al.
MMBench: Is your multi-modal model an all-around player?
*arXiv:2307.06281*, 2023b.
- Marino et al. [2019]
Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi.
Ok-vqa: A visual question answering benchmark requiring external knowledge.
In *Proceedings of the IEEE/cvf conference on computer vision and pattern recognition*, pages 3195–3204, 2019.
- Peng et al. [2023]
Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao.
Instruction tuning with gpt-4.
*arXiv:2304.03277*, 2023.
- Rasheed et al. [2024]
Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fahad S. Khan.
Llava++: Extending visual capabilities with llama-3 and phi-3, 2024.
- Sarah et al. [2025]
Anthony Sarah, Sharath Nittur Sridhar, Maciej Szankin, and Sairam Sundaresan.
Llama-nas: Efficient neural architecture search for large language models.
In *European Conference on Computer Vision*, pages 67–74. Springer, 2025.
- Shang et al. [2024]
Yuzhang Shang, Mu Cai, Bingxin Xu, Yong Jae Lee, and Yan Yan.
Llava-prumerge: Adaptive token reduction for efficient large multimodal models.
*arXiv preprint arXiv:2403.15388*, 2024.
- Sukthanker et al. [2024]
Rhea Sanjay Sukthanker, Benedikt Staffler, Frank Hutter, and Aaron Klein.
Large language model compression with neural architecture search.
*arXiv preprint arXiv:2410.06479*, 2024.
- Team et al. [2023]
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al.
Gemini: a family of highly capable multimodal models.
*arXiv:2312.11805*, 2023.
- Teterwak et al. [2024]
Piotr Teterwak, Soren Nelson, Nikoli Dryden, Dina Bashkirova, Kate Saenko, and Bryan A Plummer.
Learning to compose superweights for neural parameter allocation search.
In *Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision*, pages 2751–2760, 2024.
- Touvron et al. [2023]
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al.
Llama: Open and efficient foundation language models.
*arXiv:2302.13971*, 2023.
- Wang et al. [2024]
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al.
Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.
*arXiv preprint arXiv:2409.12191*, 2024.
- Wang et al. [2023]
Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, et al.
Cogvlm: Visual expert for pretrained language models.
*arXiv:2311.03079*, 2023.
- Wu et al. [2021]
Yan Wu, Aoming Liu, Zhiwu Huang, Siwei Zhang, and Luc Van Gool.
Neural architecture search as sparse supernet.
In *Proceedings of the AAAI Conference on Artificial Intelligence*, pages 10379–10387, 2021.
- Xing et al. [2024]
Long Xing, Qidong Huang, Xiaoyi Dong, Jiajie Lu, Pan Zhang, Yuhang Zang, Yuhang Cao, Conghui He, Jiaqi Wang, Feng Wu, et al.
Pyramiddrop: Accelerating your large vision-language models via pyramid visual redundancy reduction.
*arXiv preprint arXiv:2410.17247*, 2024.
- Yang et al. [2024]
Senqiao Yang, Yukang Chen, Zhuotao Tian, Chengyao Wang, Jingyao Li, Bei Yu, and Jiaya Jia.
Visionzip: Longer is better but not necessary in vision language models.
*arXiv preprint arXiv:2412.04467*, 2024.
- Yao et al. [2024]
Linli Yao, Lei Li, Shuhuai Ren, Lean Wang, Yuanxin Liu, Xu Sun, and Lu Hou.
DeCo: Decoupling token compression from semantic abstraction in multimodal large language models.
*arXiv:2405.20985*, 2024.
- Ye et al. [2023]
Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al.
mplug-owl: Modularization empowers large language models with multimodality.
*arXiv:2304.14178*, 2023.
- Ye et al. [2024]
Weihao Ye, Qiong Wu, Wenhao Lin, and Yiyi Zhou.
Fit and prune: Fast and training-free visual token pruning for multi-modal large language models.
*arXiv preprint arXiv:2409.10197*, 2024.
- Zeng et al. [2024]
Wang Zeng, Sheng Jin, Lumin Xu, Wentao Liu, Chen Qian, Wanli Ouyang, Ping Luo, and Xiaogang Wang.
Tcformer: Visual recognition via token clustering transformer.
*IEEE Transactions on Pattern Analysis and Machine Intelligence*, 2024.
- Zhang et al. [2024a]
Qizhe Zhang, Aosong Cheng, Ming Lu, Zhiyong Zhuo, MinQi Wang, Jiajun Cao, Shaobo Guo, Qi She, and Shanghang Zhang.
[cls] attention is all you need for training-free visual token pruning: Make vlm inference faster.
*arXiv preprint arXiv:2412.01818*, 2024a.
- Zhang et al. [2024b]
Yuan Zhang, Chun-Kai Fan, Junpeng Ma, Wenzhao Zheng, Tao Huang, Kuan Cheng, Denis Gudovskiy, Tomoyuki Okuno, Yohei Nakata, Kurt Keutzer, et al.
Sparsevlm: Visual token sparsification for efficient vision-language model inference.
*arXiv preprint arXiv:2410.04417*, 2024b.
- Zhao et al. [2024]
Yue Zhao, Long Zhao, Xingyi Zhou, Jialin Wu, Chun-Te Chu, Hui Miao, Florian Schroff, Hartwig Adam, Ting Liu, Boqing Gong, et al.
Distilling vision-language models on millions of videos.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pages 13106–13116, 2024.
- Zhong et al. [2024]
Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, and Hao Zhang.
$\{$DistServe$\}$: Disaggregating prefill and decoding for goodput-optimized large language model serving.
In *18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24)*, pages 193–210, 2024.
- Zhu et al. [2023]
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny.
Minigpt-4: Enhancing vision-language understanding with advanced large language models.
*arXiv:2304.10592*, 2023.
- Zoph and Le [2016]
Barret Zoph and Quoc V Le.
Neural architecture search with reinforcement learning.
*arXiv preprint arXiv:1611.01578*, 2016.

## 7 Additional Method Details

In this section, we describe the theoretical FLOPs (TFLOPs) calculation and GSOP’s settings in details.

### 7.1 TFLOPs Calculation

We provide a detailed derivation of the theoretical FLOPs (TFLOPs) calculation for our model based on the LLaMA architecture. Following the notation in the main paper, we define:

- •
$n_{\text{out}}$: Number of tokens in MHA-out
- •
$n_{\text{in}}$: Number of tokens in MHA-in
- •
$n_{\text{mlp}}$: Number of tokens processed by the MLP
- •
$h$: Hidden feature size
- •
$d$: Key/value feature dimension
- •
$m$: MLP intermediate dimension

Multi-Head Attention (MHA). The MHA module consists of three main components:

- 1.
Linear Projections:
•
Queries: $Q=X_{\text{in}}W_{Q}$ requires $2n_{\text{in}}hd$ FLOPs
•
Keys: $K=X_{\text{out}}W_{K}$ requires $2n_{\text{out}}hd$ FLOPs
•
Values: $V=X_{\text{out}}W_{V}$ requires $2n_{\text{out}}hd$ FLOPs
- 2.
Attention Computation:
•
Computing $QK^{\top}$: $2n_{\text{out}}n_{\text{in}}d$ FLOPs
•
Computing $AV$: $2n_{\text{out}}n_{\text{in}}d$ FLOPs
- 3.
Output Projection:
•
Final linear layer: $2n_{\text{in}}dh$ FLOPs

LLaMA-style MLP. The MLP module in LLaMA [45] architecture consists of:

- •
First linear layer: $2n_{\text{mlp}}hm$ FLOPs
- •
Second linear layer: $2n_{\text{mlp}}mh$ FLOPs
- •
Third linear layer: $2n_{\text{mlp}}hm$ FLOPs

Total FLOPs. Combining all components, we arrive at the equation presented in the main paper:

$$ $\displaystyle\text{TFLOPs}=$ $\displaystyle\underbrace{2d((n_{\text{out}}+2n_{\text{in}})h+2n_{\text{out}}n_ {\text{in}}+n_{\text{out}}h)}_{\text{Multi-Head Attention}}$ (16) $\displaystyle+\underbrace{6n_{\text{mlp}}hm}_{\text{LLaMA-style MLP}}$ $$

For different decoder architectures (e.g., Phi-3 [1]), the specific dimensions and computational patterns may vary. In our implementation, we adjust the calculations accordingly based on the architecture-specific design choices.

### 7.2 GSOP Algorithm

Search Algorithm. In Algorithm [1](https://arxiv.org/html/2507.02909v1#alg1), we represent the main process of GSOP’s operation sorting, with pre-sorting operation filtering simplified as OperationFiltering.

Figure: Algorithm 2 Operation Pruning in Layer $l$

Implementing Operation Pruning. Pruning operation $(g,l,m)$ denotes that tokens in group $g$ skip computations for module $m$ at layer $l$. This can be implemented by selectively excluding tokens from specific module computations. As demonstrated in [Algorithm 2](https://arxiv.org/html/2507.02909v1#alg2), the computational savings manifest differently for each module type:

- •
Pruning $(g,l,\textit{MHA-out})$ reduces the number of tokens participating in key/value transformations and their subsequent attention calculations (lines 1, 2, and 4).
- •
Pruning $(g,l,\textit{MHA-in})$ decreases tokens involved in query transformation, attention computation, and output projection (lines 3, 4, and 5).
- •
Pruning $(g,l,\textit{MLP})$ excludes tokens from feed-forward processing (line 7).

### 7.3 Additional GSOP Configurations

Visual Token Grouping We primarily follow FasterVLM [55]’s approach to group visual tokens based on the [CLS] attention scores from the visual encoder layer that outputs visual features in each VLM. The specific grouping method is identical to what is described in FasterVLM, with the only difference being that we do not perform further token pruning.

Visual Token Ratios GSOP uses [CLS] attention scores to divide visual tokens into two groups: $g_{1}$ containing the top-$r\%$ and $g_{2}$ containing the bottom-$(100-r)\%$. In this paper, for experiments with LLaVA-1.5, LLaVA-Phi3, and VILA, we set $r=25$, meaning a $g_{1}$ to $g_{2}$ ratio of 1:3. However, for LLaVA-Next experiments, we chose $r=20$, resulting in a $g_{1}$ to $g_{2}$ ratio of 1:4. This adjusted ratio was still optimized on LLaVA-1.5 & GQA, then transferred to LLaVA-Next.

Performance Thresholds Regarding the monotonically decreasing performance thresholds $\mu_{1},\mu_{2},\dots,\mu_{Z}$ used in Sec. 3.23, in all experiments in the paper, we uniformly set $Z=15$, with $\mu_{Z}$ as 20% of the base model’s performance on the validation task. The values $\mu_{1},\mu_{2},\dots,\mu_{Z}$ form an arithmetic sequence, so $\mu_{z}=\frac{z}{Z}\times 0.2\times\text{base_performance}$.

Danger-to-Prune Setting For “danger-to-prune” operations, we focus solely on group $g_{1}$, which constitutes the critical token group. Given a layer threshold $l_{d}$, we designate all operations involving $g_{1}$ in layers up to $l_{d}$ as “danger-to-prune” operations and exclude both these and the “free-to-prune” operations from the sorting set:

$$ $\mathcal{O}_{1}=\mathcal{O}\setminus\left(\mathcal{O}^{f}\cup\{(g_{1},l,m)\mid l \leq l_{d},m\in\mathcal{M}\}\right)$ (17) $$

In our paper, we set $l_{d}=12$ for LLaVA-1.5, LLaVA-Phi3, and VILA, while setting $l_{d}=20$ for LLaVA-Next

Free-to-Prune Setting Regarding the Free-to-Prune operations binary search, we set the search layer range to the posterior 50% of model layers, specifically layers 16-31 for LLaVA-1.5-7B which has 32 layers in total.

### 7.4 Optimization-based Baselines

In this paper, we employ two optimization-based baselines. While FitPrune [53] is inherently optimization-based and we simply adopt it directly, FastV [11] is not naturally designed as an optimization-based method. However, FastV has two critical parameters: layer position and retained token number, with performance varying significantly across different parameter combinations.

To establish a fair comparison, we conducted a comprehensive grid search for FastV parameters using the same GQA subset. Specifically, we selected 5 different retained token numbers $(288,144,72,36,18)$ and 10 different layer positions $(1,2,3,4,5,7,9,11,13,15)$, resulting in 50 distinct combinations. We evaluated each parameter configuration on the identical 500-sample GQA subset used for GSOP evaluation and recorded the corresponding performance metrics.

For a given target computation reduction, we select the configuration with the optimal performance among all configurations that achieve or exceed the target reduction. This carefully tuned version serves as our optimization-based FastV baseline, ensuring a rigorous comparison with our proposed approach.

## 8 Additional Experiments

### 8.1 LLaVA-Next

**Table 7: Performance comparison on LLaVA-Next-7B [34] under different computation budgets. Benchmarks and metrics settings are identical to Tab.1 in main paper. GSOP pruning policies were optimized on LLaVA-1.5-7B [35] using 500 GQA [19] samples and transferred to LLaVA-Next-7B, yet GSOP still consistently outperforms state-of-the-art token pruning baselines.**
| Method | GQA | Seed${}^{\text{I}}$ | MME${}^{\text{P}}$ | MMB | OKVQA | POPE | Avg |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Base Model | 64.28 | 70.14 | 1520 | 67.90 | 46.42 | 86.32 | 100% |
| Retain 30% TFLOPs (= 20% Visual Tokens) |  |  |  |  |  |  |  |
| FastV [11] | 59.24 | 61.85 | 1327 | 64.37 | 43.22 | 80.54 | 91.5% |
| FasterVLM [55] | 60.94 | 66.97 | 1460 | 65.75 | 45.95 | 85.33 | 96.8% |
| GSOP | 62.63 | 67.60 | 1454 | 65.83 | 45.38 | 85.63 | 97.2% |
| Retain 20% TFLOPs (= 8% Visual Tokens) |  |  |  |  |  |  |  |
| FastV [11] | 54.10 | 56.61 | 1247 | 59.97 | 34.65 | 68.52 | 81.5% |
| FasterVLM [55] | 59.76 | 64.58 | 1410 | 64.78 | 45.06 | 82.49 | 94.3% |
| GSOP | 62.41 | 66.14 | 1426 | 64.26 | 44.14 | 84.29 | 95.4% |
| Lower Bound: Retain 5.0% TFLOPs (= 0 Visual Token) |  |  |  |  |  |  |  |

We also evaluated GSOP on LLaVA-Next-7B in [Tab. 7](https://arxiv.org/html/2507.02909v1#S8.T7). Following our transfer protocol, we applied GSOP policies optimized on LLaVA-1.5-7B to LLaVA-Next and evaluated across the same benchmarks. Results demonstrate that our transferred GSOP consistently outperforms all baselines on LLaVA-Next-7B, further validating its cross-model transferability.

### 8.2 Deploy GSOP with FLashAttention

**Table 8: Evaluation of GSOP’s acceleration and performance with FlashAttention [15, 14]. GSOP represents the original method, while GSOP-F indicates GSOP re-optimized with the constraint [Eq. 18](https://arxiv.org/html/2507.02909v1#S8.E18). The Avg. column shows the average performance ratio across benchmarks relative to the vanilla model (higher is better). Efficiency was evaluated on an NVIDIA A40 GPU with 200 VQA samples from GQA with KVCache enabled. Despite the minor performance impact from constraint [Eq. 18](https://arxiv.org/html/2507.02909v1#S8.E18), GSOP-F maintains superior performance over baselines at comparable TFLOPS ratios in Tab.1 in main paper**
| Method | Latency (ms) | Time (s) | Avg. |  |
| --- | --- | --- | --- | --- |
| Prefill | Sample |  |  |  |
| Retain 100% TFLOPs |  |  |  |  |
| Vanilla | 132 | 187 | 45 | 100% |
| Retain 30% TFLOPs (= 14% Visual Tokens) |  |  |  |  |
| GSOP | - | - | - | 96.0% |
| GSOP-F | 68 | 121 | 30 | 94.8% |

Note that we deploy GSOP based on the ”eager” attention implementation in the main paper. While GSOP and FlashAttention [15, 14] are conceptually compatible, practical integration requires addressing a conflict with FlashAttention’s causal mask implementation. To accommodate this, we introduce an additional constraint: for all token groups $g$ and layers $i$, MHA-out and MHA-in operations must be jointly pruned or retained:

$$ $\displaystyle\forall g\in\mathcal{G},i\in\mathcal{L}:$ $\displaystyle(g,i,\text{MHA-out})\in\mathcal{P}\iff(g,i,\text{MHA-in})\in \mathcal{P}$ (18) $$

where $\mathcal{P}$ denotes the pruning policy. Table [8](https://arxiv.org/html/2507.02909v1#S8.T8) shows the efficiency and performance of GSOP re-optimized with constraint [Eq. 18](https://arxiv.org/html/2507.02909v1#S8.E18) on LLaVA-1.5-7B [35], enabling FlashAttention integration (GSOP-F). GSOP-F achieves effective acceleration with FlashAttention, particularly in prefilling latency. Despite the minor performance impact from constraint [Eq. 18](https://arxiv.org/html/2507.02909v1#S8.E18), GSOP-F maintains superior performance over baselines at comparable TFLOPS ratios in Tab.1 in main paper.

### 8.3 Sorting Cost

Figure: Figure 4: Sorting Cost of GSOP with LLaVA-1.5-7B [35] on 500 GQA [19] samples. The cost curve starts at 60% TFLOPS, as preliminary binary search identified 40% ”free-to-prune” operations in under 1 GPU hour. The sudden spikes in cost indicate operation evaluation triggers. During early sorting, performance degrades gradually with evaluations occurring every tens of steps, demonstrating the efficiency of adaptive evaluation. As retained TFLOPS decrease (pruning more), performance drops accelerate, triggering frequent evaluations and causing search costs to rise sharply at last.
Refer to caption: extracted/6568245/figs/search_cost.png

[Fig. 4](https://arxiv.org/html/2507.02909v1#S8.F4) illustrates the search cost for greedy operation sorting evaluated on LLaVA-1.5-7B [35] with 500 GQA [19] samples on an A40 GPU. The step-like increases in search cost correspond to operation evaluation triggers. Although GSOP requires time to sort operations, this cost is one-time only, and the resulting policies demonstrate strong generalization across tasks and models. For example, according to Tab.4 in main paper and [Fig. 4](https://arxiv.org/html/2507.02909v1#S8.F4), with just 12 GPU hours invested, we can obtain a policy that reduces LLaVA-1.5-7B’s prefilling TFLOPs to 30% while permanently decreasing prefilling latency by at least half. Not to mention that the cost will be much lower with a better GPU, e.g. A100 which is 2.4x faster that A40.

### 8.4 More Benchmarks

[Tab. 9](https://arxiv.org/html/2507.02909v1#S8.T9) shows GSOP outperforming FastV and PDrop significantly and matching VisionZip on TextVQA and MMMU. Over all 8 benchmarks, GSOP achieves 70% TFLOP reduction with $<4\%$ performance drop and outperforms VisionZip by $3.2\%$. MMMU shows less sensitivity to visual pruning.

**Table 9: Performance comparison on LLaVA-1.5 at 30% TFLOPs. GSOP applies the same policy as Table 1. Avg. All is computed over all datasets (6 in the main paper, 2 in supp)**
| Benchmark | TextVQA | MMMU | Avg. All |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Model | 7B | 13B | 7B | 13B | 7B | 13B |
| Base Model | 58.23 | 61.19 | 36.34 | 36.41 | 100% | 100% |
| FastV [11] | 48.64 | 52.71 | 34.54 | 35.03 | 57.3% | 83.5% |
| PDrop [42] | 48.71 | 55.97 | 35.22 | 35.78 | 74.6% | 89.3% |
| VisionZip [43] | 55.85 | 58.22 | 36.67 | 35.56 | 93.3% | 93.9% |
| GSOP (ours) | 55.42 | 58.29 | 36.67 | 35.89 | 96.5% | 96.3% |

### 8.5 Transferring LLaVA Policy to Qwen2-VL-7B

While direct optimization with Qwen2-VL [46] certainly works, we show transferability by adapting the LLaVA-1.5-7B’s operation sorting without re-search in Table [10](https://arxiv.org/html/2507.02909v1#S8.T10). Key adaptations: random token grouping and excluding operations beyond layer 28 (due to Qwen2-VL’s 28 decoder layers and lack of [CLS] token). We also change to 0.35:0.65 grouping and exclude $(g_{1},*,\text{MHA-out})$ operations. FasterVLM and VisionZip are unavailable due to lack of [CLS] token, so we compare against FastV and random pruning baselines. Transferred GSOP reduces 55% TFLOP ($\approx$ keeping 10% visual tokens in every layer) with only 12.3% performance drop, significantly outperforming the baselines. Larger absolute drops occur due to Qwen2-VL’s token scarcity from inherent merging, increasing pruning sensitivity.

**Table 10: Evaluation on Qwen2-VL-7B [46] (max resolution: $336^{2}$) at 45% TFLOPs. GSOP is transferred from LLaVA-1.5-7B.**
| Method | GQA | Seed^I | MME^P | OKVQA | POPE | MMB | Avg. All |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Base Model | 59.80 | 70.61 | 1661 | 48.56 | 86.63 | 77.66 | 100.0% |
| Random | 50.06 | 57.91 | 1320 | 34.52 | 77.29 | 55.67 | 79.5% |
| FastV [11] | 44.64 | 50.31 | 1136 | 22.41 | 70.11 | 38.14 | 65.1% |
| GSOP | 53.64 | 63.24 | 1455 | 38.54 | 81.50 | 66.92 | 87.7% |

## 9 Additional Ablation Study

### 9.1 Visual Token Grouping Methods

**Table 11: Comparison of different visual token grouping methods at various TFLOPS levels. CLS refers to [CLS] attention grouping, Uniform refers to uniform grouping, and LLM refers to using LLM attention for grouping visual tokens. Values represent percentage of base model performance retained, evaluated on the same benchmarks as other experiments.**
| Grouping | TFLOPS Ratio |  |  |
| --- | --- | --- | --- |
| 35% | 30% | 25% |  |
| CLS | 96.9% | 96.0% | 90.8% |
| Uniform | 96.0% | 94.4% | 87.6% |
| LLM | 92.4% | 89.0% | 77.8% |

We compare three approaches for grouping visual tokens: [CLS] attention from the visual encoder (our primary method), LLM decoder layer attention to last token (commonly used in prior works [11, 53, 49]), and uniform selection. Table [11](https://arxiv.org/html/2507.02909v1#S9.T11) demonstrates that [CLS] attention grouping consistently outperforms alternatives, especially under tight computational budgets. Uniform grouping provides a viable alternative when fixed token indices are required. LLM layer attention performs worst, with performance gaps widening at lower TFLOPs ratios—likely because, as noted in FasterVLM, LLM attention concentrates toward sequence ends, preserving only partial visual information.

### 9.2 Pruning beyond Visual Tokens/Operations

**Table 12: Comparison of different token ranges at various TFLOPS ratio levels. The table shows how Visual, System, and Text tokens perform under different computational constraints.**
| Token Range | TFLOPS Ratio |  |  |
| --- | --- | --- | --- |
| 35% | 30% | 25% |  |
| Visual | 96.9% | 96.0% | 90.8% |
| Visual + System | 97.2% | 96.3% | 91.4% |
| Visual + System + Text | 86.5% | 77.3% | 54.2% |

While our previous evaluations limited GSOP to visual token operations for fair comparison with baselines, GSOP can extend to system and text token operations. Table [12](https://arxiv.org/html/2507.02909v1#S9.T12) shows results from LLaVA-1.5-7B with these extensions. Leveraging the one-way attention pattern in causal LMs—where system tokens update others but are not updated by other tokens—we cache and reuse system token features, allowing GSOP to improve performance across all computational budgets by dropping all system token MHA-in and MLP operations. However, pruning text token operations proves harmful, causing significant performance degradation, particularly at tight computational budgets.

### 9.3 GSOP + Token Pruning

**Table 13: Evaluating GSOP based on token pruning on LLaVA-1.5-7B. Benchmarks and metrics settings are identical to Tab.1 in main paper. GSOP pruning policies were optimized using 500 GQA [19] samples while applying FasterVLM to pruning 20%.**
| Method | GQA | Seed${}^{\text{I}}$ | MME${}^{\text{P}}$ | MMB | OKVQA | POPE | Avg |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Baseline: Retain 35% TFLOPs (= 20% Visual Tokens) |  |  |  |  |  |  |  |
| FasterVLM [55] | 57.43 | 61.48 | 1431 | 62.63 | 50.17 | 82.38 | 100% |
| Retain 30% TFLOPs (= 14% Visual Tokens) |  |  |  |  |  |  |  |
| FasterVLM [55] | 56.10 | 59.40 | 1379 | 61.17 | 48.96 | 79.15 | 97.0% |
| Baseline + GSOP | 56.85 | 60.73 | 1368 | 62.33 | 49.13 | 81.44 | 98.3% |
| Retain 25% TFLOPs (= 8% Visual Tokens) |  |  |  |  |  |  |  |
| FasterVLM [55] | 53.73 | 56.21 | 1332 | 59.62 | 45.02 | 73.22 | 92.0% |
| Baseline + GSOP | 54.63 | 57.93 | 1341 | 60.35 | 47.05 | 72.09 | 93.5% |
| Lower Bound: Retain 18.6% TFLOPs (= 0 Visual Token) |  |  |  |  |  |  |  |

Finally, GSOP can be applied on top of token pruning methods. In this approach, we first use token pruning to reduce tokens, then treat the remaining visual tokens as a single token group $g_{1}$, while keeping other GSOP settings unchanged. In Table [13](https://arxiv.org/html/2507.02909v1#S9.T13), we use FasterVLM to prune tokens in LLaVA-1.5-7B to retain 35% of TFLOPs as a baseline, then apply GSOP to further reduce TFLOPs to 30% and 25%. We compare this combined approach against using FasterVLM alone at equivalent computation budgets. Results show that FasterVLM + GSOP achieves better performance than FasterVLM alone at both computation budgets.

## 10 Result Visualization

In [Fig. 5](https://arxiv.org/html/2507.02909v1#S10.F5), we visualize pruning policies discovered by our GSOP for LLaVA-1.5-7B and LLaVA-1.5-13B retaining 30% TFLOPS.

Figure: (a) GSOP policy for LLaVA-1.5-7B retaining 30% retained TFLOPs
Refer to caption: extracted/6568245/figs/supps/lv157.png

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 4
presentation: 3
questions: The work would benefit from a deeper analysis of why DOP works and under what conditions it fails.

There can be an analysis of failure cases, gradient sensitivity, or interplay between MHA vs. MLP pruning depth.
soundness: 2
strengths: * Addresses an important and timely efficiency problem in multimodal LLMs.

* The method is simple, easy to integrate, and empirically effective across different architectures.

* Extensive experimental coverage with clear ablations and latency analysis.
summary: The paper proposes **Depth-wise Operation Pruning (DOP)**, an operation-level pruning strategy for MLLMs that claims finer control than token pruning. The method prunes redundant operations (defined as module–token computations) using two heuristics: (1) depth-wise layer pruning and (2) additive divergence approximation for efficient search. Experiments on six MLLMs across 13 benchmarks show large FLOPs savings with minimal performance loss.
weaknesses: * **Incremental novelty:** The idea of pruning at finer granularity is not fundamentally new; the contribution is mostly heuristic refinements (depth ordering + additive scoring).

* **Limited theoretical insight:** The additive approximation is empirical, with no formal justification or guarantees; the correlation analysis is weak evidence for correctness.

* **Heavy reliance on validation heuristics:** The optimization objective (KL divergence on a small validation set) may not correlate well with downstream task performance; no robustness or sensitivity analysis is provided.

 * **Selective scope:** Despite framing as general “operation pruning,” the paper only applies it to visual tokens; this weakens the generality claim.

* **Unclear comparison fairness:** Some baselines (e.g., progressive pruning) are not re-tuned under identical FLOPs constraints. It’s unclear whether hyperparameters were optimized equally.

 * **Overemphasis on numbers:** The main results show marginal performance differences compared to strong baselines like CDPruner; the claimed 7\% gain seems dataset-specific.

 * **Ablations lack depth:** No analysis of failure cases, gradient sensitivity, or interplay between MHA vs. MLP pruning depth.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: 1.	Please see the weakness above.
2.	The evaluation benchmarks mainly focus on short-answer VQA tasks. How might DOP perform on multimodal tasks that require either short- or long-form generation, such as detailed image analysis or story generation? Could aggressively pruning visual operations in deeper layers prematurely remove the visual grounding necessary to maintain factual consistency throughout a longer narrative?
soundness: 3
strengths: 1. Through depth-wise pruning constraints and additive approximation, the complex optimization problem becomes highly efficient to solve.
2. The experimental validation is strong. The method is tested on diverse MLLMs (LLaVA, Qwen, InternVL) and 13 benchmarks, with comparisons against recent baselines. Using fixed TFLOPs budgets ensures fair evaluation.
summary: This paper presents the Depth-wise Operation Pruning (DOP) framework, designed to improve the inference efficiency of Multimodal Large Language Models (MLLMs). DOP breaks the computation process into small “operations” and focuses on reducing structural redundancy that traditional token-pruning methods often miss. By applying MHA/MLP depth decoupling and a depth-wise continuous pruning strategy, it turns a difficult policy search problem into a much simpler optimization task with only three parameters: $(\mathbf{D_A}, \mathbf{D_P}, \mathbf{n_v})$. Experiments on many MLLM models show that DOP delivers strong performance and clear efficiency gains.
weaknesses: 1.	The method relies on two strong simplifying assumptions—monotonic depth redundancy and additive independence across $\mathbf{D_A}$, $\mathbf{D_P}$, and $\mathbf{n_v}$—which may not always hold, as deeper layers are not necessarily more redundant for all tasks. This can limit the method’s ability to reach the global optimum and lead to suboptimal strategies in certain scenarios.
2.	The performance of DPO is highly sensitive to the validation set used for optimization, and strategies optimized on one task or data distribution may not generalize well to others.

---

confidence: 4
contribution: 3
initial_rating: 4
presentation: 3
questions: 1. Could the authors clarify the core methodological differences between DOP and recent works like GSOP and Skip-Vision, and please include experimental comparisons with these methods?

2. Why is DOP only applied to the LLM decoder and not to the computationally intensive visual encoder (e.g., ViT)?

3. Can the authors provide experimental results for DOP on video understanding benchmarks (e.g., using Qwen2.5-VL on VideoMME) and pure long-context text benchmarks (e.g., LongBench)?
soundness: 3
strengths: The paper tackles the critical problem of MLLM inference efficiency with a conceptually clear and technically sound approach. The shift from "token-level" to "operation-level" pruning is a well-motivated extension of the existing paradigm, offering finer-grained control with clear theoretical and practical value. The optimization strategy—combining depth-wise constraints with additive approximation—is cleverly designed to balance performance and efficiency, making DOP feasible for real-world deployment. The evaluation is comprehensive, covering 6 prominent MLLMs and 13 benchmarks, and includes real GPU latency measurements, lending strong credibility to the results.
summary: This paper proposes Depth-wise Operation Pruning (DOP), a fine-grained token allocation framework for accelerating Multimodal Large Language Models (MLLMs). DOP defines an "operation" as the computation of a specific module processing a group of tokens and enables different modules to process varying numbers of visual tokens, moving beyond uniform token reduction. It is a data-driven method that optimizes a pruning policy by minimizing the KL divergence from the original model's output distribution under a computational budget. To ensure efficiency, DOP employs depth-wise pruning constraints and an additive approximation to drastically reduce the policy search space and validation cost. Experiments show that DOP outperforms existing token pruning baselines across multiple MLLMs and benchmarks.
weaknesses: 1. The paper fails to adequately discuss or compare against recent works that also focus on "operation pruning," notably GSOP [1], Short-LVLM [2], and Skip-Vision [3]. These works similarly aim to accelerate models by pruning internal operations rather than just tokens, yet they are omitted from both the Related Work and experiments. This omission weakens the paper's novelty claim. The authors should clearly articulate the core methodological differences between DOP and GSOP (e.g., DOP's depth-wise policy vs. GSOP's greedy sorting) and include direct experimental comparisons.

2. The potential of DOP is not sufficiently explored, as follows: 

- (a) All experiments are confined to static image VQA tasks, with no evaluation on caption tasks like Nocaps and TextCaps or video understanding tasks like VideoMME or MLVU. Given the inherent redundancy in video data and the fact that recent works (e.g., VisPruner, CDPruner) have validated their methods on video tasks, demonstrating DOP's efficacy in video domain is crucial. 
- (b) DOP is only applied to the LLM decoder, ignoring the computationally expensive visual encoder (e.g., ViT), which is a major bottleneck for high-resolution image and video understanding. 
- (c) The paper does not investigate DOP's applicability to pure long-context text tasks (e.g., LongBench, RULER), limiting the demonstration of its generalizability.

3. For Qwen2.5-VL, the authors only report results using input resizing as the underlying token reduction method. In contrast, for LLaVA-NeXT and InternVL3, they evaluate DOP with multiple token compression strategies (e.g., VisPruner, CDPruner). To ensure fair and comprehensive assessment, the authors should at least include one experiment on Qwen2.5-VL combining DOP with a state-of-the-art token pruning method like CDPruner.

4. The paper does not provide any visualization of how DOP dynamically skips computations across different benchmarks or models. For instance, showing which layers or modules are pruned under DOP for Qwen2.5-VL on tasks like TextVQA versus MME would greatly enhance interpretability and help readers intuitively understand the adaptive nature of the proposed strategy.

5. The depth-wise pruning constraint is central to DOP’s efficiency, yet the paper justifies it only with the empirical observation that “deeper layers are typically more redundant in LLMs”, which is insufficient. The authors should provide stronger theoretical or empirical support for why pruning all operations beyond a fixed depth per module type is a valid and near-optimal assumption. As presented in Figure 2 (b), the DOP Rules appear as an unverified design choice; without ablation against more flexible strategies (e.g., non-consecutive layer pruning), this key simplification lacks credible justification.

[1] Beyond Token Pruning: Operation Pruning in Vision-Language Models. arXiv 2025.

[2] Skip-Vision: Efficient and Scalable Acceleration of Vision-Language Models via Adaptive Token Skipping. ICCV 2025.

[3] Short-LVLM: Compressing and Accelerating Large Vision-Language Models by Pruning Redundant Layers. ACM MM 2025.

---

confidence: 3
contribution: 2
initial_rating: 4
presentation: 3
questions: Although the authors empirically demonstrate a high correlation coefficient between the estimated divergence and the joint divergence when used as a ranking criterion, I remain interested in the error of the Additive Approximation. It would be very informative to see: how large is the gap between the strategies searched using the approximated divergence versus the true joint divergence (can be examined in smaller-scale scenarios, e.g., with smaller MLLM and less data)? Furthermore, what would be the corresponding impact on downstream task performance? I believe the paper would be strengthened by including such an ablation study.
soundness: 3
strengths: * The paper proposes several improvements addressing aspects such as strategy search complexity and data dependency. The resulting algorithm achieves low computational complexity and requires only a minimal calibration set.
* Searching different exiting point for {attn, mlp} is interesting.
* The paper provides comprehensive experiments, validating the algorithm's generalizability across multimodal large models with diverse architectures and scales.
summary: This paper investigates the token computation allocation problem in multimodal large models across three dimensions: tokens, model depth, and computational modules. It proposes DOP, a token pruning method (can be considered as token early exiting). To reduce the size of the strategy design space, this paper introduces an Additive Approximation approach, which effectively reduces the complexity of divergence evaluation. Furthermore, sparse sampling with interpolation is adopted to reduce the final strategy search complexity, enabling the algorithm to complete with only a small calibration set and limited search time. The experimental section is comprehensive, evaluating multimodal large models with different architectures and scales across multiple benchmarks and comparative methods, demonstrating the effectiveness of the proposed approach.
weaknesses: * Limited overall novelty. The core contribution--allocating computation for tokens along the depth dimension--is closely related to existing ideas of token-level early exit and dynamic computation allocation. Similarly, the decoupling of {attn, mlp} modules has also been explored in prior work on dynamic computation allocation. Here lists some works:

1. [ICLR 25] Routing Experts: Learning to Route Dynamic Experts in Multi-modal Large Language Models. It learns to dynamically skip some layer for <Q, A> pair tokens, achieves depth flexibility, which can also be applied to visual tokens.
2. [ACL 21] Accelerating BERT Inference for Sequence Labeling via Early-Exit. It learns token-level early-exit policy for inference acceleration. Its search strategy differs from that of this paper, but their core objectives are consistent.
3. [ICML 25] SkipGPT: Dynamic Layer Pruning Reinvented with Token Awareness and Module Decoupling. Token-level dynamic computation allocation with decoupled modules, i.e. attn/mlp.

* I think the paper appears to overestimate the complexity of the original search space (Sec. 3.1 - policy space size). In general, token pruning can be categorized as: (1) One-shot pruning: Redundant tokens are filtered out based on importance before being processed by the large model. (2) Early exit: The system decides during the forward pass at which layer a token should exit, after which it undergoes no further computation.(3) Dynamic computation allocation: Before each computational unit, a decision is made independently on whether to execute the current unit for the token. Unlike early exit, a token that "exiting" for one unit may still be executed in the next. In my view, the method in this paper falls into category (2), early exit, with a search space of $2 \times L \times N$ (finding an exit point for each token in both the attn and mlp modules across $L=32$ layers). In contrast, the search space for the  dynamic computation allocation in category (3) would be $2^{(2L)} \times N$.


* Need additional evidence. In Sec.3.2.1, the authors claim that: ‘we observe that for the same group-module type (g, m), a deeper operation (g, l2, m) remains generally more redundant than a shallower operation (g, l1, m) with $l2 > l1’$. However, no experimental evidence or cites are included. I think it is a crucial observation for the designing of method, i.e., if deeper operations are more redundant, then token early-exit (type 2) is sufficient, re-use token (type 3) in deeper layers is unnecessary.


Decision: Reject