Title: Why 1 + 1 < 1 in Visual Token Pruning: Beyond Naive Integration via Multi-Objective Balanced Covering
Venue: NeurIPS 2025
Accepted: yes
Keywords: Visual Token Compression

Multimodal LLMs

Hausdorff distance
Abstract: Existing visual token pruning methods target prompt alignment and visual preservation with static strategies, overlooking the varying relative importance of these objectives across tasks, which leads to inconsistent performance. To address this, we derive the first closed-form error bound for visual token pruning based on the Hausdorff distance, uniformly characterizing the contributions of both objectives. Moreover, leveraging $\epsilon$-covering theory, we reveal an intrinsic trade-off between these objectives and quantify their optimal attainment levels under a fixed budget. To practically handle this trade-off, we propose Multi-Objective Balanced Covering (MoB), which reformulates visual token pruning as a bi-objective covering problem. In this framework, the attainment trade-off reduces to budget allocation via greedy radius trading. MoB offers a provable performance bound and linear scalability with respect to the number of input visual tokens, enabling adaptation to challenging pruning scenarios. Extensive experiments show that MoB preserves 96.4\% of performance for LLaVA-1.5-7B using only 11.1\% of the original visual tokens and accelerates LLaVA-Next-7B by 1.3-1.5$\times$ with negligible performance loss. Additionally, evaluations on Qwen2-VL and Video-LLaVA confirm that MoB integrates seamlessly into advanced MLLMs and diverse vision-language tasks. The code will be made available soon.

Why 1 + 1 < 1 in Visual Token Pruning: Beyond Naïve Integration via Multi-Objective Balanced Covering

Abstract

Existing visual token pruning methods target prompt alignment and visual preservation with static strategies, overlooking the varying relative importance of these objectives across tasks, which leads to inconsistent performance. To address this, we derive the first closed-form error bound for visual token pruning based on the Hausdorff distance, uniformly characterizing the contributions of both objectives. Moreover, leveraging ϵ -covering theory, we reveal an intrinsic trade-off between these objectives and quantify their optimal attainment levels under a fixed budget. To practically handle this trade-off, we propose Multi-Objective Balanced Covering (MoB), which reformulates visual token pruning as a bi-objective covering problem. In this framework, the attainment trade-off reduces to budget allocation via greedy radius trading. MoB offers a provable performance bound and linear scalability with respect to the number of input visual tokens, enabling adaptation to challenging pruning scenarios. Extensive experiments show that MoB preserves 96.4% of performance for LLaVA-1.5-7B using only 11.1% of the original visual tokens and accelerates LLaVA-Next-7B by 1.3-1.5 × with negligible performance loss. Additionally, evaluations on Qwen2-VL and Video-LLaVA confirm that MoB integrates seamlessly into advanced MLLMs and diverse vision-language tasks. The code is available at https://github.com/YChenL/MoB .

1 Introduction

Multimodal large language models (MLLMs) have shown impressive performance across a variety of vision-language tasks, including visual understanding [30, 27, 20], visual question answering [40, 16, 37], and visual-language reasoning [9, 47, 45]. Since visual data exhibits much higher spatial redundancy than language, MLLMs are typically required to encode visual inputs as numerous tokens, resulting in substantial computational overhead.

To address this issue, visual token pruning methods are proposed to accelerate MLLMs by selecting representative subsets of visual tokens. Most pruning methods focus on two distinct objectives: Visual Preservation (VP) [6, 8, 59, 46], which retains tokens by minimizing redundancy or maximizing visual salience, and Prompt Alignment (PA) [58, 51, 48], which selects tokens most relevant to the prompt. Recently, several multi-objective approaches [31, 51, 42] have been proposed to integrate VP and PA through various complex strategies. Counterintuitively, these methods do not exhibit dominant superiority compared to single-objective approaches, as shown in Figure 1(a). This observation naturally raises a question: Does integrating different objectives offer fundamental advantages?

67

64

62

75

72

69

+2.93%

- 0.25%

+5.17%

- 0.81%

+1.40%

- 1.28%

+3.92%

- 0.11%

+3.42%

+0.51%

+2.47%

- 0.93%

+1.38%

- 0.52%

+2.10%

+4.76%

+4.89%

+3.26%

+1.28%

+0.86%

78

70

59

57

54

62

82

79

76

86

1713

1788

1862

1639

56

54

51

58

60

62

65

57

53

56

58

51

48

50

52

46

295

282

270

307

67

64

62

75

72

69

+3.69%

- 1.27%

+6.46%

- 1.62%

+1.86%

- 4.42%

+5.37%

- 0.91%

+2.58%

- 0.69%

+3.71%

+1.85%

+1.89%

- 1.03%

+1.14%

+5.12%

+4.68%

+0.67%

+1.58%

+0.14%

78

70

59

57

54

62

82

79

76

86

1713

1788

1862

1639

56

54

51

58

60

62

65

57

53

56

58

51

48

51

53

46

287

275

263

299

FastV

Vanilla

MustDrop

MoB

MoB

v.s.

Best

-of-three

MoB

v.s.

Vanilla

67

64

62

75

72

69

+7.90%

- 3.82%

+9.53%

- 4.68%

+2.44%

- 10.1%

+10.4%

- 3.01%

+4.81%

- 2.06%

+3.25%

-4.02%

+2.41%

- 6.20%

+1.73%

+4.03%

+9.17%

+0.43%

78

70

59

57

54

62

82

79

76

86

1713

1788

1862

1639

56

54

51

58

60

62

65

57

53

56

58

51

48

50

52

46

285

273

261

297

+3.37%

- 6.73%

Relative Performance Compared to MustDrop (VP + PA) (%)

ICML'25

FastV (VP)

ECCV'24

(a)

(c)

(b)

½ Results are

better

than

MustDrop (VP+PA)

½ Results are

worse

than

MustDrop (VP+PA)

Hausdorff Distance between Prompt and Visual Tokens

Inspired by this question, we formulate preservation using the Hausdorff distance between the original and pruned token sets and derive the first closed-form error bound for visual token pruning (Lemma 1). This bound depends on VP and PA, while it is also affected by a prompt-visual coupling, measured by the Hausdorff distance between prompt and visual tokens. Notably, we identify two patterns of this coupling across popular benchmarks, as presented in Figure 1(b): weak coupling with large distance ( e.g. , TextVQA, POPE) and strong coupling with small distance ( e.g. , MMB, VizWiz). Our further analysis reveals that the effectiveness of the pruning objectives varies under distinct coupling patterns (Lemma 2). However, existing multi-objective methods overlook this variation and integrate VP and PA via constant strategies, yielding inconsistent improvements over single-objective baselines.

To quantify the effect of prompt-visual coupling, we reexamine visual token pruning from a geometric covering perspective. In this view, the retained tokens can be thought of as the union of two disjoint covers for prompt and visual tokens, where each objective corresponds to a Hausdorff covering radius, and the prompt-visual coupling is represented by the inter-cover diameter. By analyzing the geometric relationship between the radii and the diameter, we reveal an intrinsic trade-off between the two objectives (Theorem 1), which identifies the optimal attainment level of each objective to achieve the performance ceiling under a fixed pruning budget and prompt-visual coupling.

For a practical solution to this trade-off, we propose Multi-objective Balanced Covering (MoB), a training-free visual token pruning method with provable performance guarantees and multilinear complexity (Theorem 2). MoB partitions the retained tokens into two disjoint subsets for PA and VP, employing greedy radius-trading strategies to reduce the trade-off in objective attainment to a budget allocation problem. This allows MoB to achieve the optimal balance under each coupling pattern by selecting appropriate subset sizes. As shown in Figure 1(c), MoB consistently outperforms both single-objective and multi-objective baselines by a clear margin at identical pruning rates. Besides, MoB accelerates LLaVA-Next-7B by 1 . 3 -1 . 5 × with negligible performance loss. Ablation studies further validate our theoretical analysis. Our key contributions are summarized as follows:

❶ To our knowledge, we present the first closed-form error bound for visual token pruning and its practical relaxation, characterizing the contributions of the two objectives to preservation quality.

❷ We quantify the trade-off between the objectives and identify their optimal attainment level under a fixed budget and prompt-visual coupling, offering valuable insights into visual token pruning.

❸ Wepropose Multi-objective Balanced Covering (MoB) for training-free visual token pruning, which reduces the trade-off of objective attainment to a budget allocation problem via two greedy radiustrading strategies, yielding both a provable performance guarantee and multilinear scalability.

❹ Extensive experiments across 14 public benchmarks demonstrate the superiority of MoB. For instance, it retains 96 . 4% and 97 . 9% performance for LLaVA-1.5-7B and Video-LLaVA-7B with an 88 . 9% reduction ratio, outperforming the second-best method by 2 . 7% and 1 . 6% , respectively. MoBcan also be readily incorporated into advanced MLLMs, such as LLaVA-Next and Qwen2-VL.

2 Background

2.1 Related Work

Multimodal Large Language Model (MLLM). MLLMs [30, 21, 60, 28] have achieved remarkable progress in vision-language reasoning, owing to their robust cross-modality modeling via attention mechanisms [43, 34]. However, the spatial redundancy inherent in visual signals typically leads to a large number of input tokens [25, 22, 29, 44], particularly in high-resolution images and multi-frame videos ( e.g. , 2048 tokens in Video-LLaVA [27]). This issue exacerbates the quadratic scaling problem of attention mechanisms, posing significant computational challenges. Moreover, to further enhance the visual capability by incorporating high-quality details, advanced MLLMs are now designed to support higher resolution images [24, 11, 10, 4], thereby necessitating the processing of even more visual tokens ( e.g. , 2880 tokens in LLaVA-NEXT [29]). In these scenarios, effectively selecting representative visual tokens becomes a critical requirement for the real-world application of MLLMs.

Visual Token Pruning. Due to the spatial redundancy, inputs to MLLMs contain numerous less informative visual tokens. Visual token pruning accelerates MLLMs by selectively retaining only the most critical tokens during inference. Existing methods typically focus on either visual preservation (VP) [6, 38, 8, 52, 57, 32, 46] or prompt alignment (PA) [58, 51, 48]. VP-driven methods, such as ToMe [6] and LLaVA-PruMerge [38], reduce redundancy by merging similar tokens, while FastV [8] and FasterVLM [57] select tokens based on visual salience. PA-driven approaches like SparseVLM [58] rely on cross-modal attention to identify prompt-relevant tokens. More recently, MustDrop [31] integrates VP and PA through a multi-stage pruning pipeline, reporting notable improvements. Despite these advances, existing methods largely overlook the varying relative importance of VP and PA across different scenarios. In this paper, we formally characterize the contribution of each objective under a fixed pruning budget, and propose an algorithm that balances these objectives per scenario, yielding consistent improvements across diverse pruning conditions.

2.2 Preliminaries

Pipeline of MLLM. MLLMs perform vision-language reasoning by jointly processing multimodal inputs in a shared representation space. Formally, given visual tokens V (1) extracted from the visual inputs and prompt tokens P (1) encoded from user prompts, the multimodal input is defined as

where N and L denote the numbers of visual and prompt tokens, respectively. We regard both V (1) and P (1) as compact sets on d -dimensional Euclidean space ( R d , ∥ · ∥ ) . The input X (1) is then fed into a language model F [1 ,I ] with I transformer block, and the final output is given by

In particular, each f ℓ follows the standard Transformer ( e.g. , multi-head self-attention [43], layer normalization [3, 50]). The intermediate feature for any layer ℓ ∈ { 2 , . . . , I } is defined as

with V ( ℓ ) and P ( ℓ ) representing the visual and prompt tokens after ℓ -1 layers, respectively.

Visual Token Pruning. To accelerate MLLMs with minimal performance loss, visual token pruning selectively removes less-informative visual tokens at chosen intermediate layers of the language model F [1 ,I ] . Specifically, for any chosen layer f ℓ , ℓ ∈ { 2 , . . . , I } , pruning algorithms first select a subset S ( ℓ ) ⊆ V ( ℓ ) of size K ( i.e. , pruning budget) and form the pruned input X ( ℓ ) s = S ( ℓ ) ⊔ P ( ℓ ) . The corresponding output before and after pruning are then defined as

Finally, the objective of visual token pruning is formulated as

Notation. For brevity we omit the layer index ( ℓ ) and simply write X = V ⊔ P and X s = S ⊔ P to denote the input and its pruned counterpart at an arbitrary layer f ℓ . We use F to denote any composition mapping of the full model F [1 ,I ] . Finally, we let ∥ · ∥ denote the Euclidean norm.

3 Methodology

3.1 Revisiting Visual Token Pruning: Insights into Prompt-Visual Coupling

As shown in Fig. 1(a), multi-objective pruning methods fail to achieve the expected improvements, and objective-specific methods exhibit inconsistent performance across benchmarks. These observations motivate us to reexamine the problem of visual token pruning. We begin by introducing Assumption 1, which quantifies pruning performance in terms of the preservation of the original token set.

Assumption 1 (Lipschitz Continuity w.r.t. the Hausdorff Distance) . Assume every partial composition F (from layer ℓ to I ) of the language model is Lipschitz continuous w.r.t. the Hausdorff distance with constant C ℓ ≥ 1 . Formally, for any intermediate token sets X , X s ⊂ R d ,

where d H is the Hausdorff distance induced by the Euclidean norm:

Subsequently, we measure the preservation of the original token set X using three pairwise distances among visual tokens V , retained tokens S , and prompt tokens P , thereby establishing a unified performance bound for various visual token pruning algorithms, as presented in Lemma 1.

Lemma 1 (An Error Bound for Visual Token Pruning) . Under Assump 1, given a token set with its pruned counterpart X = V ⊔ P , X s = S ⊔ P ⊆ R d , the pruning error bound is given by:

Remark. Here d H ( S , P ) and d H ( S , V ) describe the prompt alignment and visual preservation, while d H ( V , P ) is an inherent term that describes the prompt-visual coupling of input data.

/84

/85

/84

/86

/87

/86

/87

/88

/89

/i255

/88

/91

/92

/86

/89

/41

/39

/17 /18

/19

/20

/i255

/22

/23

/24

/25

/26

/27

/28

/29 /30

/14

/2 /7

/31

/9 /32

/33

/34

/35

/36

/37

/23

/28

/29

/i255

/22

/23

/24

/25

/26

/27

/28

/29 /30

/10

/38 /2

/14

/14

/32

/33

/34

/40 /39

/47 /47 /48

/35

Proof in Appendix E.1. By Lemma 1, in practical settings where |S| ≪ |V| , pruning performance is governed by a non-trivial interaction among visual preservation, prompt alignment, and prompt-visual coupling. However, existing multi-objective methods typically overlook the coupling term d H ( V , P ) and statically combine the two objectives across tasks, limiting their effectiveness. Our empirical evidence across popular benchmarks validates two distinct patterns of d H ( V , P ) , each favoring different pruning objectives, as shown in Figure 2. To further explicate the effect of prompt-visual coupling, we introduce Assumption 2 and propose a practical relaxed error bound in Lemma 3.

Assumption 2 (Prompt-Visual Coupling Bound) . We assume the input visual data and prompts are not entirely unrelated; hence, there exists a constant η > 0 for any intermediate token set X = V ⊔ P ⊆ R d such that d H ( V , P ) ≤ η , ensuring the reasonability of vision-language reasoning.

Lemma 2 (A Relaxed Error Bound under Practical Budgets) . Under Assumptions 1 and 2, let X = V ⊔ P , X s = S ⊔ P ⊆ R d with |S| = K ≪ N . Partition the retained token set S into two disjoint subsets: S = S p ⊔ S v , devoted to prompt alignment d H ( S p , P ) and visual preservation d H ( S v , V ) , respectively. Then, the pruning error bound reduces to

Proof in Appendix E.2. As Lemma 2 indicates, under weak coupling (large η ), most visual regions are distant from prompt tokens in the semantic space. Consequently, if S p misses the critical patches, d H ( S p , P ) dominates the pruning error, making the selection of S p i.e. , prompt alignment, more significant. Conversely, under strong coupling (small η ), d H ( S p , P ) tends to decrease in tandem with d H ( S v , V ) , reducing the marginal benefit of prompt alignment. To further guide pruning methods design, we next quantify this trade-off governed by η through an ϵ -covering argument.

3.2 Quantifying Prompt-Visual Trade-Off: A Geometric Covering Perspective

We first introduce some geometric metrics in Definition 1, recasting each objective term d H ( S p , P ) and d H ( S v , V ) as covering radii and the coupling term d H ( V , P ) as an inter-cover diameter. Next, we relate each recasted objective to its token budget |S p | , |S v | via covering regularity in Lemma 3. Finally, by loading the budget constraint and applying the triangle inequality between radii and diameter, we derive a quantitative trade-off jointly governed by K and η in Theorem 1.

Definition 1 ( ϵ -cover, Covering Number, and Covering Regularity) . Let ( R d , ∥ · ∥ ) be the d -dimensional Euclidean space and let X ⊆ R d be a compact set.

(a) ϵ -cover. if there exists a finite set C = { c 1 , . . . , c M } ⊂ R d , an ϵ -cover of X is given by

where C is the collection of covering centers, and ϵ is the covering radius.

(b) Covering number. The minimum cardinality of C is the covering number of X at radius ϵ :

/93

/94

/95

/84

/96

/97

/98

/95

(c) Covering regularity. We say that X satisfies d -dimensional covering regularity if there exist constants 0 < A ≤ B and ϵ 0 > 0 such that

Based on Definition 1(a) (b), S p , S v ⊆ V can be thought of as two collections of centers such that

where the radii are given by ϵ p := d H ( S p , P ) , ϵ v := d H ( S v , V ) , and the covering numbers satisfy N ( P , ϵ p ) ≤ |S p | , N ( V , ϵ v ) ≤ |S v | . Thereby, we derive a lower bound of the required budget, i.e. , |S p | , |S v | , to improve each objective, i.e. , ϵ p , ϵ v , based on d eff -dimensional covering regularity.

Lemma 3 (Covering Number Bounds) . Given P , V ⊂ R d with an effective dimension d eff . Suppose their δ -dilations V δ := ⋃ v ∈V B ( v, δ ) , P δ := ⋃ p ∈P B ( p, δ ) ( δ ≪ η ) satisfy d eff -dimensional covering regularity; thus, there exist constants b>a> 0 , b ′ >a ′ > 0 and ϵ 0 >δ such that

Remark. Previous work suggests that both visual and language embeddings concentrate on a low-dimensional manifold, so the effective covering dimension satisfies the typical relation d eff ≪ d .

Proof in Appendix E.3. Lemma 3 demonstrates that once the radius ( i.e. , the objective) falls below ϵ 0 , any further improvement of it demands a Θ( ϵ -d eff ) increase in the number of selected token.

By loading Lemma 3 into the budget constraint: |S p | + |S v | = K , and applying a two-step triangle inequality between the covering radii ϵ p , ϵ v and the inter-cover diameter η , we establish a K -η -bound in Theorem 1(b), which quantifies the trade-off governed by the budget and prompt-visual coupling.

Theorem 1 (Trade-off between Prompt Alignment and Visual Preservation) . Under Assumption 2 and the covering-regularity hypothesis of Lemma 3 with constants a, a ′ , d eff > 0 , there exist a radius-scaling factor z > 1 such that η/z > δ and K < N ( P , η/z ) + N ( V , η/z ) , for every pruning results S = ( S p ⊔ S v ) ⊆ V with budget K satisfying

Remark (Optimal Attainment Level) . D 1 K -2 /d eff is completely determined by the pruning budget, while D 2 η 2 quantifies the effect of prompt-visual coupling. The optimal attainment level per objective is given by ϵ ∗ = max { η/z, √ D 1 K -1 /d eff } . Any attempt to reduce one objective below ϵ ∗ forces the other above ϵ ∗ , thereby increasing the overall pruning error.

Remark (Effect of Budget and Coupling Strength) . As K decreases, z correspondingly shrinks, ultimately making D 2 η 2 dominate the bound; while as K increases, both of the terms reduce, thereby diminishing the trade-off and tightening the overall error bound.

Proof in Appendix E.4. Theorem 1 characterizes the optimal attainment level for each objective under a fixed pruning budget and prompt-visual coupling. However, it is actually very challenging to dynamically determine the attainment level per objective during the pruning process. To address this, we propose Multi-objective Balanced Covering, which leverages the monotonic relationship between covering radii and numbers to reduce the trade-off of attainment to a budget-allocation problem.

3.3 Multi-Objective Balanced Covering: From Trade-Off to Budget Allocation

Motivated by the insights in §3.2, Multi-objective Balanced Covering (MoB) recasts visual token pruning as bi-objective covering. Specifically, given a token set X = V ⊔ P ⊆ R d with a budget K , the retained token set S is defined as the union of a prompt center set S p and a visual center set S v :

MoB then selects the cover centers ( i.e. , retained tokens) by minimizing the overall maximum radius:

In practice, MoB solves this problem approximately by two sequential greedy covering procedures: selection of prompt center set S p with budget K p , and selection of visual center set S v with the remaining budget K -K p . By the covering number bounds given in Lemma 3, we have

where d eff is the effective dimension of V , P . Accordingly, by selecting the unique budget K p ( i.e. , fixing the remaining budget K -K p ) under each coupling pattern, MoB ensures ϵ p , ϵ v = Ω ( max { η/z, √ D 1 K -1 /d eff } ) , thus yielding provable performance guarantees across scenarios.

Normalization. For efficiency, MoB applies L2 normalization to each x ∈ X so that ∥ x ∥ = 1 . Hence, for any token pair x 1 , x 2 ∈ X , the Euclidean distance can be induced by their cosine similarity:

Selection of Prompt Center Set S p . Since all s p ∈ V lie outside P , a typical solution for minimizing the radius ϵ p is Nearest-Neighbor covering (NN covering) [15], which uniformly allocates the nearest s p ∈ V for each prompt token. However, the contribution of each prompt token is inequivalent, especially under weak prompt-visual coupling; thus, equal allocation risks missing the 'best-case tokens.' To remedy this, we introduce a k -fold NN covering procedure. Formally, let L = |P| and k > 1 be a hyperparameter; we first utilize a temporary budget of kL to form a candidate set.

thereby over-sampling the k nearest visual tokens for each prompt token. Subsequently, we refine the candidate set by selecting the final K p centers that maximize their worst-case alignment with P :

By concentrating the limited budget on those visual tokens most strongly aligned with the key prompt tokens, this strategy ensures a better preservation of the critical regions in the visual input. We determine the appropriate k by ablation to avoid the oversampling of a few salient prompt tokens.

Selection of Visual Center Set S v . Unlike the prompt center selection, each visual center s v lies in V . Thereby, we employ Farthest Point Sampling (FPS) [36] on the remaining tokens, i.e. , V \ S , to select the visual centers, which makes the visual centers S v well-spread over V , minimizing the covering radius ϵ v . Concretely, FPS operates by iteratively selecting the token farthest ( i.e. , the most different) from the current centers S , where the distance is given by

Subsequently, we initialize the visual centers with the empty set, i.e. , S (1) v := ∅ . We then successively add the farthest visual token to the current centers S ( i ) v ⊔ S p until it contains a total of K elements. Hence, the visual centers at the subsequent iteration, S ( i +1) v , is given by:

More details of the proposed MoB algorithm are provided in Appendix B.

Theorem 2 (Performance Guarantee) . Under Assump 1 and the covering-regularity of Lem 3 with consts a, a ′ , d eff > 0 and b>a, b ′ >a ′ , for any budget split ( K p , K -K p ) , covering fold k , and token set X = V ⊔ P ⊆ R d with |V| = N , |P| = L , d H ( V , P ) ≤ η , the following hold: (a) Performance bound: The Performance degradation caused by MoB is upper bounded by

(b) Multilinear complexity: The complexity of MoB is given by T MoB = O ( N ( L + K ) d ) .

Remark (Coupling Trade-off) . Under weak coupling (large α ), minimizing the bound requires a larger K p . Conversely, under strong coupling (small α ), the alignment term decays rapidly, favoring visual preservation (increasing K -K p ). Specially, under perfect coupling ( η = 0 ), the bound simplifies to ∥ ∆ y ∥ ≤ C ℓ β ( K -K p ) -1 /d eff , i.e. , MoB reduces to pure visual preservation.

Remark (Budget Scaling) . As the budget K increases, the preservation term β ( K -K p ) -1 /d eff decays, requiring a corresponding increase in K p (and thus a reduction in the alignment term) to re-balance the trade-off and further lower the overall error bound.

Remark (Scalability) . MoB exhibits a multilinear scalability w.r.t #visual tokens N , #prompt tokens L , and #retained tokens K ( K,L ≪ N ), making it easily adaptable to more challenging scenarios involving large token counts, e.g. , higher-resolution inputs or multi-frame video.

Table 2: Partial comparison of image understanding on the LLaVA-7B series. For MoB, we set K p ∈ { 64 , 48 , 32 } and k ∈ { 4 , 6 , 8 } , corresponding to token-reduction rates of { 88 . 9% , 77 . 8% , 66 . 7% } . For MoB with the η prior, we use K p ∈ { 3 K 8 , K 4 , K 4 } with k = 3 K p 40 for strong-coupling benchmarks and K p ∈ { K 2 , 7 K 16 , 5 K 12 } with k = K p 8 for weak-coupling benchmarks, corresponding to the same token-reduction rates; the pruning layer is fixed at ℓ = 2 . Blue and Orange denote the best and the second. See Appendix C.4 for the detailed setting, and see Appendix D.1 for the full results.

4 Experimental Results

Experiment Setting. We perform a comprehensive evaluation of the proposed MoB and several representative methods on two visual tasks: image understanding and visual understanding, together with an efficiency analysis. Our experiments employ four popular MLLMs and include a total of 14 widely recognized benchmarks. For further details regarding the benchmarks, models, baselines, and implement details please refer to Appendix C.

Image Understanding. Table 2 and Table 3 report the evaluation results across a variety of imageunderstanding tasks on LLaVA series and Qwen2VL, respectively. We highlight five key observations: (a) MoB consistently outperforms all base-

/0 /1

/0 /2 /3

/4 /3

/5 /i255

/7 /0 /8

/7 /0

lines on LLaVA-1.5-7B in most cases. This will be more pronounced when incorporating the η -prior, which highlights the inherent advantage of our approach; (b) single-objective baselines exhibit complementary strengths under different coupling patterns, whereas MoB consistently outperforms all baselines, demonstrating the benefit of balanced objectives; (c) the superiority of MoB becomes even more significant under aggressive token reduction. Specifically, the improvement of MoB over the best baseline in average scores increases from 1.8% at a 66.7% token reduction to 2.7% at an 88.8% reduction on LLaVA-1.5-7B; (d) MoB matches the performance of the vanilla LLaVA-1.5-7B with only 33.3% of visual tokens, which may be attributed to the mitigation of hallucinations caused by redundant tokens; and (e) MoB scales seamlessly to advanced models, preserving 95.2% performance on Qwen2-VL-7B using only 22.2% of visual tokens. These observations demonstrate the superiority of MoB in leveraging limited visual tokens while minimizing performance degradation.

Video Understanding. As presented in Table 4, MoB is general and can be readily extended to more challenging video scenarios without incurring additional cost. Specifically, MoB preserves 97.9% of average performance for Video-LLaVA7B using only 6.6% of the visual tokens, which sets new records in most VideoQA benchmarks, achieving 1.6% and 4.7% improvements over TwigVLM and VisionZip, respectively. These results validate the generalization ability of MoB.

Efficiency Analysis. We present a performance-latency trade-off measured on an NVIDIA A80080GB GPU in Figure 3. The results show that (a) MoB achieves a strong performance-latency trade-off, delivering a 1 . 3 -1 . 5 × speed-up for LLaVA-NEXT-7B with negligible performance loss; (b) due to ignoring the K -η trade-off, the multi-stage method MustDrop is outperformed by singleobjective methods FastV and SparseVLM on MME and POPE, and suffers significant performance drops as token budgets shrink ( i.e. , latency decreases). In contrast, MoB consistently maintains a robust trade-off across all benchmarks, surpassing all the baselines by a clear margin; (c) MoB does not rely on attention scores to identify important tokens, making it compatible with flash attention and more efficient than attention-based methods such as SparseVLM and FastV.

5 Ablation and Discussion

Impact of ⟨ K,η,K p , ⟩ . We study the impact of K , η , and K p on pruning performance across four benchmarks: GQA and TextVQA (weak coupling); VizWiz and MMB (strong coupling). As shown in Figure 4, the results can be interpreted by Theorem 1 and Theorem 2(a), respectively.

A. Theorem 1 Perspective: When K is large, e.g. , K = 192 , the trade-off is governed by D 1 K -2 /d eff , hence the trade-off intensity remains nearly identical across benchmarks. Conversely, When K is small, especially K = 64 , in weak-coupling benchmarks, the trade-off turns to be governed by D 2 η 2 ; thus, the trade-off intensity is obviously more pronounced in GQA and TextVQA than that in VizWiz and MMB. These observations exactly confirm the validity of Theorem 1.

B. Theorem 2(a) Perspective. (a) Under weak coupling, the alignment term α ( η, k, L )( K p ) -1 /d eff is amplified, which requires a larger K p to suppress the overall error. However, across benchmarks sharing the same coupling pattern, the optimal K p values exhibit only minor variation. (b) Increasing the total budget K pushes the optimal K p upward to rebalance the two bound terms. Since the prompt length L is fixed, adding more tokens yields diminishing returns for prompt alignment, which is reflected in the declining ratio K p /K . These validate the performance bound in Theorem 2(a).

/9

/10

/11

/12

/13

/14

/15

/11

/47

/35

/20

/i255

/37

/33

/38

/34

/39

/40

/33

/i255

/41

/38

/42

/43

/33

/i255

/44

/32 /33

/34

/35

/i255

/37

/33

/38

/34

/39

/40

/33

/i255

/41

/38

/42

/43

/33

/i255

/44

/32 /33

/34

/35

/i255

/37

/33

/38

/34

/39

/40

/33

/i255

/41

/38

/42

/43

/33

/i255

/44

/3 /4 /5

/6 /2

/8 /9

/21

/21

/12

/13

/14

/13

/22

/16

/3

/5

/6

/2

/8 /9

/8 /24

/12

/13

/14

/13

/25

/16

/17

/18

/17

/19

/20

/8 /9

/8 /28

/12

/13

/14

/13

/25

/16

/17

/18

/17

/19

/20

55

66

erforma

P

nance

59

58

57

56

55

54

erformance

Per

54

53

50

49

52

51

60

59

65

64

63

62

61

0

53

48

8

58

8

72

/48

/47

/49

/50

/49

/50

/51

/52

/17

/32 /33

/18

/17

/34

/19

/35

/20

/i255

/37

/33

/38

/34

/39

/40

/33

/i255

/41

/38

/42

/43

/33

/i255

/44

/17

/32 /33

/18

/17

/34

/23

/i255

/51

/54

/55

/52

/56

GQA

MME

TextVQA

OCR

MMB

SQA

VizWiz

0.07

0.13

0.25

0.50

Ratio of

k

/

Kp

Weak Coupling

Long Prompt

Short Prompt

TextVQA

MME

164s

565s

Dataset

Remarkably, the experimental results suggest that simply determining the optimal K p for each of the two coupling patterns suffices to guarantee effective generalization across all scenarios.

Impact of Covering Fold k . We chose the covering fold k by examining the normalized ratio k/K p across eight benchmarks and nine budget configurations. As shown in Figure 5, (a) weak-coupling benchmarks generally require a larger k to ensure critical region coverage, whereas strong-coupling settings suffice with a smaller k ; (b) benchmarks with longer prompts impose a lower cap on k to preserve sampling diversity and avoid redundant selection of salient tokens. Notably, weak-coupling benchmarks with long prompts ( e.g. , GQA, TextVQA) exhibit a narrowly clustered optimal k/K p range, reflecting their strict requirement to cover key tokens without excessive redundancy.

Impact of Pruning Layer. As shown in Figure 6, (a) models with visual token pruning consistently achieve a more favorable performance-efficiency trade-off than the vanilla model on both benchmarks. (b) Pruning in deeper layers provides more significant benefits for the weak-coupling TextVQA than strong-coupling MME. We attribute this to stronger cross-modal interactions in deeper MLLM layers, which facilitate identification of answer-relevant tokens under weak coupling, whereas pruning in shallow layers disrupts these interactions and incurs greater performance degradation.

6 Conclusion

In this paper, we present a comprehensive analysis of visual token pruning, deriving the first closedform error bound with a practical relaxation. Leveraging ϵ -covering theory, we quantify the intrinsic trade-off between the fundamental pruning objectives, i.e. , visual preservation and prompt alignment, and identify their optimal attainment levels under a fixed pruning budget. Building on these insights, we introduce MoB, a training-free algorithm for visual token pruning. Based on greedy radius trading, MoB ensures the near-optimal attainment per objective via budget allocation, offering a provable performance bound and multilinear scalability. Experimental results indicate that MoB matches the performance (100.6%) of LLaVA-1.5-7B with only 33.3% of visual tokens and can be seamlessly integrated into advanced MLLMs, such as LLaVA-Next-7B and Qwen2-VL-7B. Our work advances the understanding of visual token pruning and offers valuable insights for future MLLM compression.

Limitations. Our theoretical guarantees rely on assumption 1, which is generally satisfied in practice but may not hold for all MLLMs. Besides, MoB applies a preliminary search to select the proper K p , which potentially introduces extra tuning overhead in practical applications. Future work will focus on developing an adaptive K p selection mechanism driven by online estimation of the coupling η .

/57

/58

/59

/54

/60

/61

/62

/59

Acknowledgments

The work was performed at the Shanghai Key Laboratory of Multidimensional Information Processing, East China Normal University; the Institute of Natural Sciences and School of Mathematical Sciences, Shanghai Jiao Tong University; and the Chongqing Key Laboratory of Precision Optics, Chongqing Institute of East China Normal University, with joint support from the National Natural Science Foundation of China (62176091), the Natural Science Foundation of Chongqing (CSTB2024NSCQMSX0877), the Science and Technology Commission of Shanghai Municipality (21DZ2203100) and the Fundamental Research Funds for the Central Universities.

References

Yash Akhauri, Ahmed F AbouElhamayed, Yifei Gao, Chi-Chih Chang, Nilesh Jain, and Mohamed S Abdelfattah. Tokenbutler: Token importance is predictable. arXiv preprint arXiv:2503.07518 , 2025.

Kazi Hasan Ibn Arif, JinYi Yoon, Dimitrios S Nikolopoulos, Hans Vandierendonck, Deepu John, and Bo Ji. Hired: Attention-guided token dropping for efficient inference of high-resolution vision-language models in resource-constrained environments. arXiv preprint arXiv:2408.10945 , 2024.

Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450 , 2016.

Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923 , 2025.

Kaichen Zhang* Fanyi Pu* Xinrun Du Yuhao Dong Haotian Liu Yuanhan Zhang Ge Zhang Chunyuan Li Bo Li*, Peiyuan Zhang* and Ziwei Liu. Lmms-eval: Accelerating the development of large multimoal models, March 2024.

Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token merging: Your vit but faster. In The Eleventh International Conference on Learning Representations , 2022.

David Chen and William B Dolan. Collecting highly parallel data for paraphrase evaluation. In Proceedings of the 49th annual meeting of the association for computational linguistics: human language technologies , pages 190-200, 2011.

Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang. An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models. In European Conference on Computer Vision , pages 19-35. Springer, 2024.

Liangyu Chen, Bo Li, Sheng Shen, Jingkang Yang, Chunyuan Li, Kurt Keutzer, Trevor Darrell, and Ziwei Liu. Large language models are visual reasoning coordinators. Advances in Neural Information Processing Systems , 36:70115-70140, 2023.

Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271 , 2024.

Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 24185-24198, 2024.

Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 6904-6913, 2017.

Zhiyu Guo, Hidetaka Kamigaito, and Taro Watanabe. Attention score is not all you need for token importance indicator in kv cache reduction: Value also matters. arXiv preprint arXiv:2406.12335 , 2024.

Danna Gurari, Qing Li, Abigale J Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 3608-3617, 2018.

Dorit S Hochbaum and David B Shmoys. A best possible heuristic for the k-center problem. Mathematics of operations research , 10(2):180-184, 1985.

Wenbo Hu, Yifan Xu, Yi Li, Weiyue Li, Zeyuan Chen, and Zhuowen Tu. Bliva: A simple multimodal llm for better handling of text-rich visual questions. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 38, pages 2256-2264, 2024.

Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 6700-6709, 2019.

Yunseok Jang, Yale Song, Youngjae Yu, Youngjin Kim, and Gunhee Kim. Tgif-qa: Toward spatio-temporal reasoning in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 2758-2766, 2017.

Kasper Green Larsen and Jelani Nelson. The johnson-lindenstrauss lemma is optimal for linear dimensionality reduction. arXiv preprint arXiv:1411.2404 , 2014.

Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326 , 2024.

Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning , pages 19730-19742. PMLR, 2023.

Yanwei Li, Yuechen Zhang, Chengyao Wang, Zhisheng Zhong, Yixin Chen, Ruihang Chu, Shaoteng Liu, and Jiaya Jia. Mini-gemini: Mining the potential of multi-modality vision language models. arXiv preprint arXiv:2403.18814 , 2024.

Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. arXiv preprint arXiv:2305.10355 , 2023.

Zhang Li, Biao Yang, Qiang Liu, Zhiyin Ma, Shuo Zhang, Jingxu Yang, Yabo Sun, Yuliang Liu, and Xiang Bai. Monkey: Image resolution and text label are important things for large multi-modal models. In proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 26763-26773, 2024.

Youwei Liang, GE Chongjian, Zhan Tong, Yibing Song, Jue Wang, and Pengtao Xie. Evit: Expediting vision transformers via token reorganizations. In International Conference on Learning Representations , 2022.

Zijing Liang, Yanjie Xu, Yifan Hong, Penghui Shang, Qi Wang, Qiang Fu, and Ke Liu. A survey of multimodel large language models. In Proceedings of the 3rd International Conference on Computer, Artificial Intelligence and Control Engineering , pages 405-409, 2024.

Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection, 2024.

Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 26296-26306, 2024.

Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024.

Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023.

Ting Liu, Liangtao Shi, Richang Hong, Yue Hu, Quanjun Yin, and Linfeng Zhang. Multi-stage vision token dropping: Towards efficient multimodal large language model. arXiv preprint arXiv:2411.10803 , 2024.

Xuyang Liu, Ziming Wang, Yuhang Han, Yingyao Wang, Jiale Yuan, Jun Song, Bo Zheng, Linfeng Zhang, Siteng Huang, and Honggang Chen. Compression with global guidance: Towards training-free high-resolution mllms acceleration. arXiv preprint arXiv:2501.05179 , 2025.

Yuliang Liu, Zhang Li, Mingxin Huang, Biao Yang, Wenwen Yu, Chunyuan Li, Xu-Cheng Yin, Cheng-Lin Liu, Lianwen Jin, and Xiang Bai. Ocrbench: on the hidden mystery of ocr in large multimodal models. Science China Information Sciences , 67(12):220102, 2024.

Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. Advances in neural information processing systems , 32, 2019.

Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In The 36th Conference on Neural Information Processing Systems (NeurIPS) , 2022.

Carsten Moenning and Neil A Dodgson. Fast marching farthest point sampling. Technical report, University of Cambridge, Computer Laboratory, 2003.

Yingzhe Peng, Xinting Hu, Jiawei Peng, Xin Geng, Xu Yang, et al. Live: Learnable in-context vector for visual question answering. Advances in Neural Information Processing Systems , 37:9773-9800, 2024.

Yuzhang Shang, Mu Cai, Bingxin Xu, Yong Jae Lee, and Yan Yan. Llava-prumerge: Adaptive token reduction for efficient large multimodal models. arXiv preprint arXiv:2403.15388 , 2024.

Zhenwei Shao, Mingyang Wang, Zhou Yu, Wenwen Pan, Yan Yang, Tao Wei, Hongyuan Zhang, Ning Mao, Wei Chen, and Jun Yu. Growing a twig to accelerate large vision-language models. arXiv preprint arXiv:2503.14075 , 2025.

Zhenwei Shao, Zhou Yu, Meng Wang, and Jun Yu. Prompting large language models with answer heuristics for knowledge-based visual question answering. In Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition , pages 14974-14983, 2023.

Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 8317-8326, 2019.

Xudong Tan, Peng Ye, Chongjun Tu, Jianjian Cao, Yaoxin Yang, Lin Zhang, Dongzhan Zhou, and Tao Chen. Tokencarve: Information-preserving visual token compression in multimodal large language models. arXiv preprint arXiv:2503.10501 , 2025.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.

Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. arXiv preprint arXiv:2409.12191 , 2024.

Yiqi Wang, Wentao Chen, Xiaotian Han, Xudong Lin, Haiteng Zhao, Yongfei Liu, Bohan Zhai, Jianbo Yuan, Quanzeng You, and Hongxia Yang. Exploring the reasoning abilities of multimodal large language models (mllms): A comprehensive survey on emerging trends in multimodal reasoning. arXiv preprint arXiv:2401.06805 , 2024.

Zichen Wen, Yifeng Gao, Shaobo Wang, Junyuan Zhang, Qintong Zhang, Weijia Li, Conghui He, and Linfeng Zhang. Stop looking for important tokens in multimodal language models: Duplication matters more. arXiv preprint arXiv:2502.11494 , 2025.

Jiannan Wu, Muyan Zhong, Sen Xing, Zeqiang Lai, Zhaoyang Liu, Zhe Chen, Wenhai Wang, Xizhou Zhu, Lewei Lu, Tong Lu, et al. Visionllm v2: An end-to-end generalist multimodal large language model for hundreds of vision-language tasks. Advances in Neural Information Processing Systems , 37:69925-69975, 2024.

Long Xing, Qidong Huang, Xiaoyi Dong, Jiajie Lu, Pan Zhang, Yuhang Zang, Yuhang Cao, Conghui He, Jiaqi Wang, Feng Wu, et al. Pyramiddrop: Accelerating your large vision-language models via pyramid visual redundancy reduction. arXiv preprint arXiv:2410.17247 , 2024.

Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Yueting Zhuang. Video question answering via gradually refined attention over appearance and motion. In Proceedings of the 25th ACM international conference on Multimedia , pages 1645-1653, 2017.

Jingjing Xu, Xu Sun, Zhiyuan Zhang, Guangxiang Zhao, and Junyang Lin. Understanding and improving layer normalization. Advances in neural information processing systems , 32, 2019.

Senqiao Yang, Yukang Chen, Zhuotao Tian, Chengyao Wang, Jingyao Li, Bei Yu, and Jiaya Jia. Visionzip: Longer is better but not necessary in vision language models. arXiv preprint arXiv:2412.04467 , 2024.

Weihao Ye, Qiong Wu, Wenhao Lin, and Yiyi Zhou. Fit and prune: Fast and training-free visual token pruning for multi-modal large language models. arXiv preprint arXiv:2409.10197 , 2024.

Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang, and Dacheng Tao. Activitynet-qa: A dataset for understanding complex web videos via question answering. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 33, pages 9127-9134, 2019.

Yuanhan Zhang Bo Li Songyang Zhang Wangbo Zhao Yike Yuan Jiaqi Wang Conghui He Ziwei Liu Kai Chen Dahua Lin Yuan Liu, Haodong Duan. Mmbench: Is your multi-modal model an all-around player? arXiv:2307.06281 , 2023.

Zheng Zhan, Yushu Wu, Zhenglun Kong, Changdi Yang, Yifan Gong, Xuan Shen, Xue Lin, Pu Zhao, and Yanzhi Wang. Rethinking token reduction for state space models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , pages 16861697, 2024.

Kaichen Zhang, Bo Li, Peiyuan Zhang, Fanyi Pu, Joshua Adrian Cahyono, Kairui Hu, Shuai Liu, Yuanhan Zhang, Jingkang Yang, Chunyuan Li, and Ziwei Liu. Lmms-eval: Reality check on the evaluation of large multimodal models, 2024.

Qizhe Zhang, Aosong Cheng, Ming Lu, Zhiyong Zhuo, Minqi Wang, Jiajun Cao, Shaobo Guo, Qi She, and Shanghang Zhang. [cls] attention is all you need for training-free visual token pruning: Make vlm inference faster. arXiv preprint arXiv:2412.01818 , 2024.

Yuan Zhang, Chun-Kai Fan, Junpeng Ma, Wenzhao Zheng, Tao Huang, Kuan Cheng, Denis Gudovskiy, Tomoyuki Okuno, Yohei Nakata, Kurt Keutzer, et al. Sparsevlm: Visual token sparsification for efficient vision-language model inference. arXiv preprint arXiv:2410.04417 , 2024.

Yiwu Zhong, Zhuoming Liu, Yin Li, and Liwei Wang. Aim: Adaptive inference of multi-modal llms via token merging and pruning. arXiv preprint arXiv:2412.03248 , 2024.

Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592 , 2023.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: See Section 1 for the main claims; see Sections 3 to 5 and Appendices D and E for the detailed contributions and scope.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: See Section 6 for the discussion on the limitations.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: The assumptions are clearly stated in the statement of the theorems: any partial composition F of the language model satisfies a Lipschitz Continuity w.r.t. the Hausdorff Distance with constant C ℓ ≥ 1 , the δ -dilations V δ := ⋃ v ∈V B ( v, δ ) , P δ := ⋃ p ∈P B ( p, δ ) ( δ ≪ η ) satisfy d eff -dimensional covering regularity with constants a, a ′ , b, b ′ . See Theorems 1 and 2. The proof of Lemma 1 can be found in Appendix E.1, the proof of Lemma 2 can be found in Appendix E.2, the proof of Lemma 3 can be found in Appendix E.3, the proof of Theorems 1 and 2 are provided in Appendices E.4 and E.5, respectively.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: See Appendices B and C for the information needed to reproduce the main experimental results.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.

In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The code is provided in the supplemental material.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The experiment does not involve any training process. See Section 5 and appendix C for all the test details.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: The experimental results are accompanied by significance tests, and crossvalidation conducted using a publicly available third-party framework.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: See Appendix C for the computer resources needed to reproduce the experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: See Section 6 and appendix A for the discussion on the broader impacts.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The assets used in this paper are properly credited, and the license and terms are respected.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

16. Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.

Answer: [Yes]

Justification: MoB is an inference-time algorithm that prunes visual tokens in pre-trained multimodal LLMs-LLaVA-1.5-7B, LLaVA-Next-7B, Qwen2-VL-7B and Video-LLaVA7B. At Transformer layer ℓ = 2 , it removes redundant vision tokens using a bi-objective covering rule (see Section 3 and appendices B and C). The LLM weights remain frozen ; no additional data, gradient updates, or prompt engineering are used. Thus the LLMs serve as essential yet unmodified back-bones whose intermediate embeddings are the input to MoB.

Guidelines:

The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

Please refer to our LLM policy ( https://neurips.cc/Conferences/2025/LLM ) for what should or should not be described.

Appendix

In the appendix, we provide additional information as listed below:

§A provides the broader impacts of MoB

§B provides the algorithm details and pseudocode of MoB

§C provides the overview of the data, models, baselines and implementation details.

§D provides the additional experimental results.

§E provides the omitted technical details.

A Broader Impacts and Limitations

Theory Impacts. Beyond the visual setting, MoB's theoretical lens-balancing Visual Preservation (retaining sufficient context) and Prompt Alignment (isolating 'golden evidence')-naturally transfers to language domain. It makes the key challenging (context vs. evidence) in long-context LLM explicit and offers actionable guidance for token-level compression and scheduling under fixed context budgets. In practice, this perspective informs RAG (calibrating recall vs. precision) and summarization/LLM memory (trading coherence vs. conciseness).

Application Impacts. The proposed MoB yields substantial acceleration of MLLMs with negligible performance loss, thereby enabling high-resolution vision-language models to operate on resource-constrained platforms such as edge devices and mobile systems while supporting low-latency applications-including assistive technologies for the visually impaired, autonomous navigation, and AR/VR. Besides, MoB potentially benefits other redundancy-heavy domains ( e.g. , point clouds and multi-sensor fusion), guiding efficient token-level compression beyond vision.

Theory Limitations. The theoretical analysis (lemma 1 and theorem 1) and the performance guarantees (theorem 2) rely on assumption 1 (Lipschitz Continuity) and lemma 3 (Covering Regularity). In embedding spaces that violate metric properties or exhibit highly irregular token distributions, these conditions may fail to hold, and the provable performance bounds may no longer apply.

Application Limitations. Our deployment presently requires an a priori estimate of η to set the pruning hyperparameters K p and k . When η is misestimated for a new model, domain, or input distribution, the selected K p and k can deviate from their optimum, leading to suboptimal speed-accuracy trade-offs.

B Algorithm

Algorithm 1 Multi-Objective Balanced Covering (MoB)

Require: Visual token V ∈ R N × d , Prompt token P ∈ R L × d , Budget K p , K v , Covering fold k Ensure: Index list for select tokens S ∈ N Kp + Kv

1: Normalize all token embeddings to unit ℓ 2 norm: V ← V / ∥ V ∥ 2 , row , P ← P / ∥ P ∥ 2 , row

Step 1. Select Prompt Centers via Nearest-Neighbor Covering

2: Compute cosine-similarity matrix via PV ⊤ : M ← PV ⊤ ▷ M ∈ R L × N .

3: Retrieve k nearest token indices per prompt:

▷ C idx , C sim ∈ R L × k collects index and similarity of k closest centers per prompt token. # Deduplicate candidate indices

4: Flatten index and similarity arrays: C idx ← Flatten ( C idx ) , C sim ← Flatten ( C sim ) ▷ C idx ∈ N Lk , C sim ∈ R Lk

5: Remove duplicate indices, preserving associated similarities:

6: Identify topK p prompt centers by similarity: i p ← ArgTopK ( C ∗ sim , K p )

7: Form the prompt-center index list: S p ← C [ i p

∗ idx ] ▷ S p ∈ N K p

Step 2. Select Visual Centers via Farthest-Point Sampling

8: Initialize selected centers: S ← S

p # Initialize token-to-prompt minimum distances

9: Compute pairwise minimum distances between all tokens and selected prompt centers:

▷ Selected centers have zero distance in d ∈ R N .

# Farthest-Point Sampling

10: for t = 1 to K v do

11: Select the token farthest from current centers: i ∗ ← ArgMax ( d ) , S ← Concat ( S , i ∗ ) ▷ Selected tokens are excluded (distance = 0) from further sampling.

12: Compute cosine distances to the newly selected token: d ∆ ← 1 N -VV [ i ∗ ] ⊤

13: Update each token's minimum distance: d ← ElementwiseMin ( d , d ∆ ) ▷ Distance of newly selected token i ∗ set to zero in d .

14: end for

15: return S

Algorithm 2 Compute Prompt-Visual Coupling

Require: Visual embeddings V ∈ R nv × d , Prompt embeddings P ∈ R np × d Ensure: Hausdorff distance h ( V , P )

Step 1. Compute Pairwise Euclidean Distances

1: Compute distance matrix via cdist : D ← cdist( V , P , p = 2

Step 2. Directed Hausdorff Distances

2: Visual-to-prompt directed distance:

3: Prompt-to-visual directed distance:

Step 3. Final Hausdorff Distance

4: return max ( h v → p , h p → v )

) ▷ D ∈ R nv × np

C Experiment Details

C.1 Benchmarks

Our experiments evaluate the vision-language reasoning abilities of multimodal large language models using a comprehensive suite of widely recognized benchmarks. For image understanding tasks, we assess performance on ten public benchmarks: GQA, MMBench (MMB) and MMBench-CN (MMB CN ), MME, POPE, VizWiz, ScienceQA (SQA), VQA V2 , TextVQA (VQA T ), and OCRBench (OCR). For video understanding tasks, we conduct experiments on four popular benchmarks: TGIFQA (TGIF), MSVD-QA (MSVD), MSRVTT-QA (MSRV), and ActivityNet-QA (ActNet). The following section provides a concise overview of these benchmarks:

GQA [17] leverages scene graphs, questions, and images to evaluate visual scene understanding and reasoning. By incorporating detailed spatial relationships and object-level attributes, it poses significant challenges for models to perform accurate visual reasoning in complex environments.

MMBench [54] introduces a hierarchical evaluation framework where model capabilities are dissected into three levels. Level-1 focuses on basic perception and reasoning; Level-2 subdivides these abilities into six distinct sub-skills; and Level-3 further refines the evaluation into 20 specific dimensions. Its Chinese counterpart, MMBench-CN , adopts a similar structure.

MME [26] rigorously tests perceptual and cognitive abilities across 14 sub-tasks. By employing carefully crafted instruction-answer pairs and succinct instructions, MME minimizes data leakage and provides a robust, fair assessment of a model's multifaceted performance.

POPE [23] targets the evaluation of object hallucination by posing binary questions about object presence in images. It quantifies hallucination levels using metrics, e.g. , accuracy, recall, precision, and F1 score, offering a precise and focused measure of model reliability.

VizWiz [14] is a visual question answering benchmark derived from interactions with blind users. Comprising over 31 , 000 image-question pairs with 10 human-annotated answers per query, it encapsulates the challenges of low-quality image capture and conversational spoken queries, thereby emphasizing real-world visual understanding.

ScienceQA [35] spans multiple scientific domains by organizing questions into 26 topics, 127 categories, and 379 skills. This hierarchical categorization provides a diverse and rigorous testbed for evaluating multimodal understanding, multi-step reasoning, and interpretability across natural, language, and social sciences.

VQA V2 [12] challenges models with open-ended questions based on 265 , 016 images that depict a variety of real-world scenes. Each question is paired with 10 human-annotated answers, facilitating a thorough evaluation of a model's capacity to interpret and respond to diverse visual queries.

TextVQA [41] focuses on the integration of text within visual content. It evaluates a model's proficiency in reading and reasoning about textual information embedded in images, thereby requiring a balanced understanding of both visual and linguistic cues.

OCRBench [33] is a comprehensive benchmark for evaluating the OCR capabilities of multi-modal language models across five key tasks: text recognition, scene text-centric and document-oriented VQA, key information extraction, and handwritten mathematical expression recognition.

TGIF-QA [18] adapts the visual question answering task to the video domain by focusing on GIFs. With 165 K question-answer pairs, it incorporates tasks, e.g. , counting repetitions, identifying repeating actions, detecting state transitions, and frame-specific question answering, thereby demanding detailed spatio-temporal analysis.

MSVD-QA [49] builds upon the MSVD dataset by pairing 1 , 970 video clips with approximately 50 . 5 K QA pairs. Questions are categorized into five distinct types, e.g. , what, who, how, when, and where, making it a versatile tool for evaluating video understanding.

MSRVTT-QA [7] features 10 K video clips and 243 K QA pairs designed to test the integration of visual and temporal information. Its structure, which parallels that of MSVD-QA through the inclusion of five question types, further enriches the evaluation landscape for video-based tasks.

ActivityNet-QA [53] provides 58 K human-annotated question-answer pairs drawn from 5 . 8 K videos. Its focus on questions related to motion, spatial relationships, and temporal dynamics necessitates long-term spatio-temporal reasoning, thus serving as a benchmark for advanced video understanding.

C.2 Multi-modal Large Language Models

We evaluate MoB using various open-source multimodal large language models (MLLMs). For image understanding tasks, experiments are conducted on the LLaVA series, including LLaVA-1.5-7B and LLaVA-Next-7B, as well as the Qwen-VL series, such as Qwen2-VL-7B. Specifically, LLaVA-Next and Qwen2-VL are utilized to validate performance on high-resolution images, i.e. , those with a larger number of visual tokens. For video understanding tasks, we employ Video-LLaVA-7B as the baseline model, following the settings reported in its original paper to ensure a fair comparison.

LLaVA-1.5-7B [28] is a robust vision-language model built on the LLaV A framework. It processes images resized to 224 × 224 and tokenizes them into roughly 572 visual tokens using a patch-based vision encoder. This design balances fine-grained visual representation with computational efficiency, making it effective for diverse multimodal tasks.

LLaVA-Next-7B [29] extends the LLaVA-1.5 by incorporating refined training strategies and data curation. It supports higher-resolution inputs (up to 448 × 448 ), yielding up to 2880 visual tokens. These enhancements improve its visual reasoning capabilities and enable more precise alignment between visual content and language but also incur significantly increased computational cost.

Qwen2-VL-7B [44] augments the Qwen2 language model with visual input capabilities. This model leverages cross-modal pretraining to seamlessly merge vision and language, demonstrating strong performance in complex visual question answering and comprehensive scene understanding.

Video-LLaVA-7B [27] extends the LLaVA framework into the temporal domain by processing video inputs. It is designed to capture both spatial and temporal dynamics, enabling effective video comprehension and video-based question answering with coherent and context-aware responses.

C.3 Baselines

To validate the superiority of the proposed MoB, we construct a robust baseline that integrates a comprehensive set of representative existing methods, which encompass single-stage methods with both two distinct objectives and several multi-stage methods.

ToMe [6] employs a lightweight token-matching scheme to merge visually similar tokens across transformer layers, thereby reducing computation without additional training. Its simple yet effective design makes it well suited for real-time applications.

FastV [8] leverages attention maps in the early layers to identify and prune non-critical tokens, significantly reducing initial computational overhead. This focus on early-stage reduction allows the model to operate more efficiently while maintaining performance.

SparseVLM [58] ranks tokens based on cross-modal attention to assess image-prompt relevance and adopts adaptive sparsity ratios to retain key information. It further incorporates a token recycling mechanism to balance the trade-off between efficiency and accuracy.

HiRED [2] allocates token budgets across image partitions by using CLS token attention and then selects the most informative tokens within each partition. This spatially aware approach ensures balanced reduction while preserving contextual details.

LLaVA-PruMerge [38] combines pruning and merging strategies by dynamically removing less important tokens using sparse CLS-visual attention. It then clusters the retained tokens based on key similarity, ensuring that crucial visual features remain intact.

PyramidDrop [48] adopts a progressive token-dropping strategy across different model stages, resulting in a pyramid-like token structure. This method carefully balances the reduction of tokens with the preservation of performance as the processing advances.

MustDrop [31] integrates several token-reduction strategies including spatial merging, text-guided pruning, and output-aware cache policies. Its multi-faceted approach efficiently reduces token counts across various stages of the model.

VisionZip [51] first selects dominant tokens that capture the majority of an image's information and then merges the remaining tokens based on semantic similarity. This approach dramatically reduce token redundancy while accelerating inference and maintaining robust performance.

FasterVLM [57] evaluates token importance using CLS attention in the encoder and prunes tokens before they interact with the language model. This preemptive reduction streamlines the overall process and enhances model efficiency.

GlobalCom 2 [32] employs a hierarchical strategy by coordinating thumbnail tokens to allocate adaptive retention ratios for high-resolution crops. This approach successfully preserves local details while providing effective global context reduction.

DART [46] leverages token duplication to guide its pruning process instead of relying solely on attention scores. By selecting a small set of pivot tokens and retaining only those with minimal redundancy, DART achieves significant acceleration in a training-free manner.

TokenCarve [42] implements a two-stage, training-free compression framework that preserves critical visual information during aggressive token reduction. It first prunes low-information tokens using an information-preservation guided selection and then merges the remaining tokens based on similarity to minimize accuracy loss.

TwigVLM [39] accelerates large vision-language models by appending a lightweight twig block to an early layer of a frozen base VLM. It utilizes twig-guided token pruning coupled with selfspeculative decoding to boost generation speed while retaining high accuracy even under aggressive token reduction.

C.4 Implement Details

From Theorems 1 and 2, the balance between the visual preservation and prompt alignment, i.e. , the optimal budget K p applied for covering prompt tokens P , is jointly determined by the total budget K and the visual-prompt coupling η . To ensure fair comparison, we evaluate two settings.

(i) Without η prior. This setting deliberately avoids any benchmark-specific prior (w/o η prior). MoB adjusts K p solely as a function of K to balance the two objectives. Based on an ablation over ⟨ K,K p ⟩ , we set K p ∈{ 64 , 48 , 32 } and k ∈{ 4 , 6 , 8 } , corresponding to token-reduction rates of { 88 . 9% , 77 . 8% , 66 . 7% } .

(ii) With η prior. To verify the K -η -K p relationship formulated in Theorems 1 and 2, we introduce a coarse benchmark prior on η . Specifically, we do not meticulously search the optimal hyperparameters for MoB, i.e. , K p and the covering fold k , per benchmark. Instead, we partition benchmarks by their empirical η distribution into two groups (strong v.s. weak coupling) and employ the same configuration per group . From a joint ablation over ⟨ K,η,K p ⟩ , for image understanding we set

which again yield token-reduction rates of { 88 . 9% , 77 . 8% , 66 . 7% } .

As for video understanding, we set K p = 3 K 8 , k = 3 K p 40 for MSVD, MSRV, and ActNet, and K p = K 2 , k = K p 8 for TGIF. Unless otherwise stated, the pruning layer index is fixed to ℓ = 2 for both image and video tasks. The same configurations are applied across all MLLMs, and all baselines are run with their default settings.

To ensure reproducibility, we cross-validated our experimental results using the publicly available MLLMsevaluation tool lmms-eval (v0.3.0) [56, 5], with the random seed set to 1234 . All experiments were conducted on 4 × Nvidia A800-80GB GPUs paired with 2 × Intel Xeon ® Gold 6348 CPUs. The implementation was carried out in Python 3.10 using PyTorch 2.1.2 and CUDA 11.8.

D Additional Experimental Results

D.1 Quantitative Comparison

D.2 Visualization

/37

/0 /1

/2

/3 /4

/5

/i255

/8

/4

/9 /10

/11

/12 /13

/14

/15

/16

/17

/18

/15

/i255

/14

/20

/21

/10

/21

/i255

/22

/i255

/15

/23

/11

/24 /25

/11

/22

/10

/26

/i255

/27

/23

/i255

/14

/20

/21

/i255

/27

/12 /22

/28

/21

/29 /30

/9 /10

/11

/12 /13

/14

/15

/16

/17

/18

/15

/i255

/14

/20

/21

/10

/21

/i255

/22

/i255

/25

/11

/11

/31

/i255

/27

/23

/i255

/14

/20

/21

/i255

/27

/12 /22

/28

/21

/29 /30

/9 /10

/11

/12 /13

/14

/15

/16

/17

/18

/15

/i255

/14

/20

/21

/10

/21

/i255

/22

/i255

/25

/27

/10

/26

/i255

/27

/23

/i255

/14

/20

/21

/i255

/27

/12 /22

/28

/21

/29 /30

/9 /10

/11

/12 /13

/14

/15

/16

/17

/18

/15

/i255

/14

/20

/21

/10

/21

/i255

/22

/i255

/25

/27

/32

/33

/32

/34

/21

/i255

/27

/23

/i255

/14

/20

/21

/i255

/27

/12 /22

/28

/21

/29 /30

ELABORADOSTOTALMENTE

/38

/37

/39

/40

/39

/40

/41

/42

/i255

/41

/44

/45

/38

/46

/9 /10

/11

/12 /13

/14

/15

/16

/17

/35/20

/22

/14

/i255

/25

/10

/22

/23

/26

/i255

/11

/36

/i255

/24 /22

/14

/32

/20

/i255

/27

/15

/i255

/14

/20

/27

/15

/29 /30

/47

/48

/49

/50

/51

/52

/53

/49

/51

/52

/51

/53

/54

/53

/54

/55

/53

/i255

/55

/52

/57

/55

/58

Visual Data

/9 /10

/11

/12 /13

/16

/15

/14

/17

/18 /19

/21

/20

/19

/21

/i255

/i255

/24

/23

/25

/26

/14

/i255

/19

/26

/23

/i255

/15

/15

/11

/21

/20

/23

/14

/26

/27

/i255

/14

/26

/28

/14

/i255

/29 /20

/14

/19

/i255

/14

/19

/20

/15

/i255

/20

/12 /23

/30

/26

/i255

/13

/11

/15

/14

/26

/27

/i255

/11

/24

/i255

/14

/29 /20

/31

/26

/10 /32

/9 /10

/11

/12 /13

/14

/15

/16

/17

/18/19

/26

/10

/26

/i255

/20

/15

/i255

/20

/14

/33

/i255

/34

/35

/i255

/36

/19

/23

/24

/30

/19

/23

/20

/35

/i255

/37

/35

/i255

/38

/26

/29 /i255

/39

/11

/10

/40

/35

/i255

/41

/35

/i255

/18 /23

/15

/19

/20

/24

/30

/14

/11

/24

/35

/i255

/42

/35

/i255

/9

/23

/10

/20

/32

SLOW

/i255

/11

/43

/i255

/14

/19

/26

/12 /23

/20

/i255

/30

/26

/33

/34

/i255

/29 /11

/14

/i255

/35

/i255

/27

/11

/24

/44

/14

/15

/35

/i255

/45/32

/9 /10

/11

/12 /13

/14

/16

/15

/17

/18/19

/20

/21

/19

/i255

/20

/15

/i255

/26

/19

/14

/i255

/12 /23

/20

/24

/i255

/14

/11

/13

/20

/21

/35

/i255

/46

/19

/26

/i255

/12 /23

/24

/i255

/20

/15

/i255

/19

/11

/47

/27

/20

/24

/30

/i255

/14

/19

/26

/i255

/15

/20

/30

/24

/35

/i255

/9 /10

/11

/12 /13

/14

/15

/16

/17

/18/19

/20

/21

/19

/i255

/20

/15

/i255

/10

/20

/30

/19

/14

/33

/i255

/41

/13

/20

/21

/14

/26 29

/27

/i255

/20

/26

/i255

/27

/20

/15

/14

/i255

/12 /26

/24

/24

/10

/11

/20

/49

/26

/24

/43

/i255

/26

/i255

/11

/13

/i255

/14

/48

/14

/18/19

/23

/16

/17

/10

/26

/24

/50

/15

/i255

/13

/47

/23

/48

/27

/20

/47

/19

/i255

/41

/35

/37

/10

/26

/33

/i255

/14

/44

/20

/21

/i255

/13

/24

/i255

/14

/19

/26

/9 /10

/11

/12 /13

/14

/15

/30

/10

/11

/44

/24

/27

/35

/i255

/59

/60

/61

/62

/63

/64

/65

/61

MoB formulates visual token pruning as a bi-objective covering problem over ( V , P ) , which is expected to gather query-relevant, fine-grained evidence with S p while preserving global scene context with S v . The visualizations (Figures 7 and 8) qualitatively validate this design: tokens in S p concentrate in regions aligned with the text query and key visual evidence, whereas elements of S v spread more uniformly across the image to maintain the overall context. Together, this complementary allocation enables MoB to retain the most informative visual content for each image-query pair, accounting for its strong empirical performance.

D.3 Additional Ablation & Discussion

To assess MoB's sensitivity to the covering-fold parameter k -particularly under weak coupling with long prompts-we conduct a detailed ablation on k using GQA and TextVQA.

As Table 6 demonstrates, MoB is not overly sensitive to the choice of k , particularly within a clear optimal range. For instance, in both two benchmarks, performance only varies by approximately 0 . 3% for k values between [2 , 8] under ⟨ K = 64 , K p = 32 ⟩ setting.

There is also a principled, theoretical reason for this robustness, which stems from the relationship between the covering fold k , the budget K p , and the length L of prompt tokens P . From covering theory, every prompt token p ∈ P is covered by at least one visual token v ∈ V under the condition K p ≥ kL , thereby ensuring the performance guarantee of MoB. Therefore, as selected k satisfies k ≤ K p /L , the performance will remain stable.

Heuristic for estimating k . In practice, a robust range for k can be inferred from the prompt length L . Given the analysis above, we expect an adaptive, per-sample search for a fine-grained k to yield only limited gains, so we rely on this length-based heuristic instead.

D.4 Real-life Application

147

148

149

150

Hausdorff Distance

0.00

0.25

0.50

0.75

1.00

Density

MMB

147.0

148.5

150.0

151.5

Hausdorff Distance

0.00

0.15

0.30

0.45

0.60

Density

MMB_CN

147

148

149

150

151

Hausdorff Distance

0.00

0.25

0.50

0.75

1.00

Density

SQA

147

148

149

150

151

Hausdorff Distance

0.00

0.25

0.50

0.75

1.00

Density

VizWiz

148

149

150

151

152

Hausdorff Distance

0.0

0.3

0.6

0.9

1.2

Density

GQA

149.6

150.4

151.2

152.0

Hausdorff Distance

0.0

0.3

0.6

0.9

1.2

Density

MME

149.4

150.0

150.6

151.2

151.8

Hausdorff Distance

0.0

0.5

1.0

1.5

2.0

Density

149.6

150.4

151.2

152.0

Hausdorff Distance

0.0

0.3

0.6

0.9

1.2

Density

TextVQA

149

150

151

152

Hausdorff Distance

0.0

0.2

0.4

0.6

0.8

Density

OCR

Open-domain recipe. MoB is task-agnostic, which does not require pre-defined task labels and can operate online by classifying each sample's coupling pattern. For a given target model, we adopt a two-stage strategy:

Offline calibration. Analyze the empirical η distributions on a set of representative benchmarks (as shown in Figure 9) and set a robust threshold τ that separates weak vs. strong coupling.

Online classification and inference. For each incoming query, compute its Hausdorff distance using Algorithm 2 with tractable bilinear complexity O ( NLd ) . Classify the sample by comparing this value to τ , then apply the corresponding budget configuration ( e.g. , K p , k ) and run MoB + forward inference. In practice, this online cost is negligible relative to the pruned forward pass.

Computational Overhead. We provide a detailed cost breakdown for online computation of the Hausdorff distance using Algorithm 2 with complexity O ( NLd ) on LLaVA-1.5-7B and LLaVANext-7B, where N , L , and d denote the numbers of visual tokens, prompt tokens, and the feature dimension, respectively. As shown in Table 7, the measured cost (TFLOPs) of exact Hausdorff computation is orders of magnitude smaller than that of MoB itself and the model's forward pass, yielding a negligible overhead.

Concretely, computing d H ( e.g. , ∼ 1 . 2 × 10 -4 TFLOPs on LLaVA-Next) is insignificant relative to the pruned forward pass ( e.g. , ∼ 4 . 6 TFLOPs at K = 320 ) and, more importantly, to the savings from pruning ( ∼ 35 . 9 TFLOPs). Thus, exact online estimation is not a practical bottleneck; its cost is dwarfed by the efficiency gains of our method. Further acceleration is possible with standard techniques ( e.g. , heuristic support sampling or low-dimensional random projections), although it is unnecessary in our settings.

Heuristic Sampling: It computes the distance on smaller support sets of the tokens ( V ′ ⊂ V , P ′ ⊂ P ) , which can be constructed via random sampling [46] or more advanced heuristics such as Key-Norm selection [1, 13]. This reduces complexity to O ( N ′ L ′ d ) , where |V ′ | = N ′ , |P ′ | = L ′ .

Random Projections: For a more theoretically grounded approach, the Johnson-Lindenstrauss (JL) lemma [19] allows us to project embeddings to a much lower dimension ( d ′ ≪ d ) while preserving geometric structure, reducing complexity to O ( NLd ′ ) .

Potential Extensions. A natural extension is to maintain an online estimate of the coupling statistic η during inferencee.g. , a running summary of an approximate ˆ η computed from shallow-layer tokens. As more samples are processed, we expect the empirical distribution of ˆ η to become bimodal (consistent with the benchmark patterns in Figure 9), enabling a data-driven threshold to be derived on the fly that separates weak vs. strong coupling regimes. Using this live threshold, MoB could adapt K p (and k ) per sample or per mini-batch by selecting from a small budget pool or by scheduling K p as a function of ˆ η , with conservative warm-up and safeguards for distribution shift.

E Omitted Technical Details

E.1 Proof of Lemma 1

Restatement of Lemma 1 (An Error Bound for Visual Token Pruning) . Under Assumption 1, given any token set with its pruned counterpart X = V ⊔ P , X s = S ⊔ P ⊆ R d , the pruning error bound is given by:

Remark. Here d H ( S , P ) and d H ( S , V ) describe the prompt alignment and visual preservation, while d H ( V , P ) is an inherent term that describes the prompt-visual coupling of input data.

Proof. The intermediate input for any layer and its pruned counterpart are given by

By Equation (1), the Hausdorff distance is symmetric, i.e. ,

and induced by Euclidean distance.

Step 1. Bound the one-sided distances.

We analyze the distances by considering the membership of the points in the subsets.

Direction 1 ( X → X s ) For any x ∈ X :

Case (i): If x ∈ P , then since P ⊂ X s ,

Case (ii): If x ∈ V , then the candidate points in X s = S ⊔ P can be chosen either from S or P . Thus,

Taking the supremum over x ∈ V yields

By Equation (1), we derive the distance in direction 1:

Direction 2 ( X s →X ) For any y ∈ X s :

Case (i): If y ∈ P , then as P ⊂ X ,

Case (ii): If y ∈ S , the candidate points in X = V ⊔ P can be chosen from either V or P ; hence

Taking the supremum over y ∈ S yields

By Equation (1), we derive the distance in direction 2:

Step 2. Combine the bounds.

By Equation (1), combining the bounds in (E1-2) and (E1-3), we obtain

Based on (E1-1), we have

Loading the Assumption 1, we have the output discrepancy is bounded by

This completes the proof.

E.2 Proof of Lemma 2

Restatement of Lemma 2 (A Relaxed Error Bound under Practical Budgets) . Under Assumptions 1 and 2, let X = V ⊔ P , X s = S ⊔ P ⊆ R d with |S| = K ≪ N . Partition the retained token set S into two disjoint subsets: S = S p ⊔ S v , devoted to prompt alignment d H ( S p , P ) and visual preservation d H ( S v , V ) , respectively. Then, the pruning error bound reduces to

Proof. By Lemma 1, we obtain

Since min { a, b } ≤ max { a, b } , we have

For any p ∈ P , we have

Taking the supremum over p ∈ P yields

Using Assumption 2 ( d H ( V , P ) ≤ η ) and the triangle inequality for Hausdorff distance, we have

Hence,

Similarly, one can show that

Loading the maximum of (E2-2), (E2-3) and d H ( V , P ) into (E2-1), we obtain

Since d H ( S p , P ) ≥ 0 , d H ( S v , V ) ≥ 0 , η ≥ 0 , we have

Hence

This completes the proof.

Similarly, since S v ⊂ S ,

Thus, by Equation (1),

E.3 Proof of Lemma 3

Restatement of Lemma 3 ( d eff -regular lower bound on covering numbers) . Given P , V ⊂ R d with an effective dimension d eff . Suppose their δ -dilations V δ := ⋃ v ∈V B ( v, δ ) , P δ := ⋃ p ∈P B ( p, δ ) ( δ ≪ η ) satisfy d eff -dimensional covering regularity; thus, there exist constants b>a> 0 , b ′ >a ′ > 0 and ϵ 0 >δ such that

Remark Previous work suggests that both visual and language embeddings concentrate on a lowdimensional manifold, so the effective covering dimension satisfies the typical relation d eff ≪ d .

Proof. We prove the two-sided bound for P ; the argument for V is identical.

Notation.

N ( X,r ) : minimal number of closed balls of radius r covering X .

Step 1. Transfer trick for small ϵ .

Fix ϵ ∈ ( δ, ϵ 0 ] and define ϵ ′ = min { ϵ + δ, ϵ 0 } .

If ϵ ≤ ϵ 0 -δ (so ϵ ′ = ϵ + δ ), then any ϵ -cover { z i } i m =1 of P satisfies for each y ∈ P δ :

Hence

Note: For ϵ > ϵ 0 -δ , the above transfer argument is not applied.

Step 2. Lower bound on N ( P , ϵ ) .

Split into two cases:

Case I: ϵ ≤ ϵ 0 -δ . Since P δ satisfies d eff -dimensional covering regularity; loading the lower-bound for P δ at radius ϵ ′ = ϵ + δ , there exists a constant a δ ≥ 0 such that

Based on (E3-1), we obtain

Since ϵ > ϵ 0 -δ, we have

Hence

Since δ ≤ ϵ , it follows that ϵ + δ ≤ 2 ϵ ; thus, we have

Case II: ϵ > ϵ 0 -δ . Define ˜ a := ( ϵ 0 -δ ) d eff , such that

Since any nonempty set P has covering number at least one, the following holds

Therefore, set a := min { a δ 2 -d eff , ˜ a } > 0 , combining (E3-2) and (E3-3) yields

Similarly, V holds N ( V , ϵ ) ≥ a ′ ϵ -d eff , ∀ ϵ ∈ ( δ, ϵ 0 ] .

Step 3. Upper bound on N ( P , ϵ ) .

Since P δ satisfies d eff -dimensional covering regularity, there exists a constant b δ ≥ a δ ≥ 0 such that

Since P ⊆ P δ , we have N ( P , ϵ ) ≤ N ( P δ , ϵ ) ; thus, the following holds

Based on the monotonicity of covering numbers , for every radius ϵ ≥ δ , we have

Therefore, set b := max { b δ , N ( P , δ ) } , for all ϵ ∈ ( δ, ϵ 0 ] we have

Likewise for V , the following holds N ( V , ϵ ) ≤ b ′ ϵ -d eff , ∀ ϵ ∈ ( δ, ϵ 0 ] .

Step 4. Combine the bounds.

Based on (E3-4) and (E3-5), for all ϵ ∈ ( δ, ϵ 0 ] the following holds

This completes the proof.

E.4 Proof of Theorem 1

Restatement of Theorem 1 (Trade-off between Prompt Alignment and Visual Preservation) . Under Assumption 2 and the covering-regularity hypothesis of Lemma 3 with constants a, a ′ , d eff > 0 , there exist a radius-scaling factor z > 1 such that η/z > δ and K < N ( P , η/z ) + N ( V , η/z ) , for every pruning results S = ( S p ⊔ S v ) ⊆ V with budget K satisfying

Remark (Optimal Attainment Level) . The term D 1 K -2 /d eff is completely determined by the pruning budget, while D 2 η 2 quantifies the effect of prompt-visual coupling. Hence, the optimal attainment level per objective is given by ϵ ∗ = max { η/z, √ D 1 K -1 /d eff } . Any attempt to reduce one objective below ϵ ∗ forces the other above ϵ ∗ , thereby increasing the overall pruning error.

Remark (Effect of Budget and Coupling Strength) . As K decreases, z correspondingly shrinks ( D 2 growing as a power function), ultimately making D 2 η 2 dominate the bound; while as K increases, both of the terms reduce, thereby diminishing the trade-off and tightening the overall error bound.

Proof. We begin the proof by noting

Step 1. Quantify the impact of budget K .

By Lemma 3, for all ϵ p , ϵ v ∈ ( δ, ϵ 0 ] , we have

By AM-GM inequality, we have K p K v ≤ ( K 2 ) 2 ; thus, loading (E4-1) we have

Define D 1 := (4 aa ′ ) 1 /d eff > 0 , the K -bound is established by

Step 2. Quantify the impact of prompt-visual coupling η .

Based on the budget condition, the radius-scaling factor z holds

For contradiction, we suppose two covering radii is simultaneously small, such that ϵ p < η/z and ϵ v < η/z . Then, the monotonicity of covering numbers gives

Hence contradicting (E4-3). Therefore at least one of ϵ p , ϵ v is ≥ η/z . Consequently

Define D 2 := 1 z 2 > 0 , the η -bound is given by

Step 3. Combine the impacts.

By (E4-2) and (E4-4), we have

This completes the proof.

E.5 Proof of Theorem 2

Restatement of Theorem 2 (Performance Guarantee) . Under Assumption 1 and the coveringregularity of Lemma 3 with constants a, a ′ , d eff > 0 and b>a, b ′ >a ′ , for any budget split ( K p , K -K p ) , covering fold k , and token set X = V ⊔ P ⊆ R d with |V| = N , |P| = L , and d H ( V , P ) ≤ η , the following hold:

(a) Performance bound: The Performance degradation caused by MoB is upper bounded by

(b) Multilinear complexity: The complexity of MoB is given by T MoB = O ( N ( L + K ) d ) .

Remark (Coupling Trade-off) . Under weak coupling (large α ( η, k, L ) ), minimizing the bound requires a larger K p . Conversely, under strong coupling (small α ( η, k, L ) ), the alignment term decays rapidly, favoring visual preservation (increasing K -K p ). Specially, under perfect coupling ( η = 0 ), the bound simplifies to ∥ ∆ y ∥ ≤ C ℓ β ( K -K p ) -1 /d eff , i.e. , MoB reduces to pure visual preservation.

Remark (Budget Scaling) . As the total budget K increases, the preservation term β ( K -K p ) -1 /d eff decays, requiring a corresponding increase in K p (and thus a reduction in the alignment term) to rebalance the trade-off and further lower the overall error bound.

Remark (Scalability) . MoB exhibits a multilinear scalability with respect to visual tokens N , prompt tokens L , and retained tokens K (especially K,L ≪ N ), making it readily adaptable to more challenging scenarios, such as advanced MLLMs with higher-resolution inputs or multi-frame video.

Notation.

The intermediate input X is formulated as

Particularly, V , P are compact sets with d eff effective dimensions.

We define the pruned intermediate input as

The budget configuration is given by ⟨ K p , K v ⟩ , where K p + K v = K .

Proof. We separately proof the Performance Guarantee & Complexity in Part A & Part B

Part A: Performance Guarantee

Part A-1: Performance Guarantee of prompt alignment

Step A-1.1: Bound of the radius derived by k -fold NN-covering

Given any union set before K p -truncation

we define

By previous work [15], NN-covering achieves a 1 -approximation for the k -center problem with sufficient budget; i.e. , specifically for any p ∈ P we have

Thus,

Based on Assumption 2, since s ∈ S ′ p ⊆ V , the upper bound of the radius ϵ ′ p is given by

Step A-1.2: Impact of K p -truncation on the radius

Based on Lemma 3, we have

In particular:

and also

Combining the upper and lower bound for ϵ p and ϵ ′ p , respectively in terms of b, K p , K ′ p , we obtain

That is, truncating from K ′ p to K p centers increases the radius by at most the factor

Since kL ≥ K ′ p , loading into above, we have

By loading (E5-1) into the above, the performance guarantee of prompt alignment is given by

Part A-2: Performance Guarantee of Visual Preservation

By previous work [36], FPS achieves a 2 -approximation for the k -center problem:

where ϵ ⋆ ( K v ) is the optimal radius with K v centers. Based on Lemma 3, we have

thereby, the upper bound of optimal radius is given by

By loading the above into (E5-3), the performance guarantee of visual preservation is given by

Part A-3: Performance Guarantee of MoB

By substituting (E5-2) and (E5-4) into Lemma 2, the performance guarantee of the MoB is given by:

where α ( η, k, L ) = η ( b k L/a ) 1 /d eff , β = 2 b ′ 1 /d eff .

This completes the proof of Part A.

Part B: Complexity

Since k ≪ K p ≤ K ∼ L ≪ N , we restrict our complexity analysis to the leading-order terms.

Part B-1: Normalization

MoB do a L 2 normalization for each token x ∈ X ⊆ R d ; thus, the complexity is given by

Part B-2: Selection of Prompt Center

Firstly, MoB calculates the cosine similarity with each p ∈ P and v ∈ V via a matrix multiplication:

which leads a complexity of T step 1 -1 = O ( N Ld ) . Subsequent, MoB do a topk retrieval in the first dimension of M sim the select k most closed centers for each prompt token p ∈ P , which can be reduced to a partial sorting, thereby leading to a complexity of T step 1 -2 = O ( N L log k ) . Finally, MoB merge the selected result of each p ∈ P , and truncated the topK p ones with largest similarity, leading to a T step 1 -3 = O ( Lk log K p ) . Consequently, the total complexity T p -select of prompt center selection is given by:

Part B-3: Selection of Visual Center

Initially, MoB calculates the minimum distance (used in FPS) with each visual token v ∈ V\S p := V ′ and the selected prompt centers via a matrix multiplication together with an argmin operator:

thus, the complexity is given by

Subsequently, in K -K p iterations, MoB add the tokens with largest minimum distance with an argmax operator in d FPS , and update the d FPS with an inner production together with an N -K p -dimensional element-wise comparison; thus the complexity is given by

Consequently, the total complexity T v -select of visual center selection is given by:

Part B-4: Totally complexity

By (E5-5), (E5-6) and (E5-7), the totally complexity of MoB is given by

This completes the proof of Part B.

Combining the Part A & B, we complete the proof.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper provides a systematic analysis of visual token pruning in Multimodal Large Language Models (MLLMs). The authors identify that existing methods often combine the two objectives of Visual Preservation (VP) and Prompt Alignment (PA) in a static manner, neglecting their varying importance across different tasks, which results in inconsistent performance. To address this, the authors derive the first closed-form error bound for visual token pruning based on the Hausdorff distance, and employ ε-covering theory to quantify the intrinsic trade-off between VP and PA under a fixed pruning budget. Building upon this theoretical insight, the paper proposes a novel training-free method—Multi-Objective Balanced Covering (MoB)—which reformulates the pruning process as a bi-objective covering problem. MoB translates the trade-off into a budget allocation problem via greedy radius trading. Extensive experiments demonstrate that MoB outperforms existing methods across LLaVA, Video-LLaVA, and Qwen2-VL, maintaining high performance even under aggressive pruning.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\nTheoretical Contribution: The paper is the first to provide a closed-form error bound for visual token pruning, with rigorous theoretical analysis supported by a geometric interpretation using Hausdorff distance.\nNovel Methodology: The proposed MoB method addresses the bi-objective trade-off through a covering-based perspective, reformulating pruning as a budget allocation problem and offering provable performance guarantees.\nStrong Empirical Results: The method is evaluated across 14 benchmarks and three MLLMs, demonstrating superior performance. Notably, it retains 96.4% performance for LLaVA-1.5-7B with only 11.1% of tokens.\nPracticality: MoB is training-free and scales linearly with input size, making it applicable to high-resolution or multi-frame scenarios.\n\nWeaknesses:\nHyperparameter Sensitivity: MoB requires selecting the prompt center budget K_p, which may vary across tasks. Although the paper categorizes scenarios based on prompt-visual coupling strength (η), practical deployment may incur tuning overhead.\nAssumption Validity: The theoretical guarantees depend on the assumption of Lipschitz continuity of the model in the Hausdorff metric space. While reasonable in practice, this may not hold universally.\nLimited Discussion on Supervision Integration: While MoB is fully training-free, the paper does not explore the potential of incorporating supervised signals (e.g., task labels) to improve token selection relevance.\n\nQUESTIONS:\nAutomatic Estimation of 𝐾𝑝 : Currently, MoB requires manual selection of 𝐾𝑝. Could the authors consider estimating 𝐾𝑝 from prompt length or coupling strength 𝜂? Would meta-learning or lightweight heuristic predictors help automate this?\nComputational Cost of Hausdorff Distance: While theoretically elegant, the computation of Hausdorff distance in high-dimensional spaces can be expensive. Have the authors considered approximations or alternatives that maintain performance?\nPerformance with Fine-tuning: As MoB is training-free, have the authors tested its combination with lightweight fine-tuning post-pruning? Would such a setup improve performance under aggressive compression?\nApplicability to Language Token Pruning: While MoB focuses on visual tokens, could the same methodology be adapted to prune language tokens, especially for long-context LLMs?\n\nLIMITATIONS:\nThe authors explicitly discuss the reliance on theoretical assumptions and the potential cost of manually tuning Kp. They also suggest future work involving adaptive estimation of η. The discussion is honest and constructive.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-16"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the performance differences between single-objective and multi-objective approaches in visual token pruning, with the following main contributions:\n\nThe study finds that multi-objective methods do not significantly outperform single-objective methods in visual token pruning. By introducing the Hausdorff distance, the authors derive the first closed-form error bound, quantifying the contribution of the Visual Preservation (VP) and Prompt Alignment (PA) objectives.\n\nBased on the ε-covering theory, the authors reveal an inherent trade-off between VP and PA, showing how to balance these two objectives under different pruning budgets to optimize performance.\n\nThe authors propose a Multi-Objective Balanced Covering (MoB) method, reformulating visual token pruning as a bi-objective covering problem. MoB employs a greedy radius-trading strategy for budget allocation, offering provable performance bounds and linear scalability.\n\nExperimental results show that MoB maintains 96.4% of the original performance in LLaVA-1.5-7B using only 11.1% of the original visual tokens, and achieves 1.3–1.5× speedup in LLaVA-Next-7B with negligible performance loss. Moreover, evaluations on Qwen2-VL and Video-LLaVA confirm MoB’s seamless integration into advanced multimodal large language models and diverse vision-language tasks.\n\nIn summary, this paper presents a novel visual token pruning method through theoretical analysis and experimental validation, emphasizing the importance of balancing different objectives during the pruning process and demonstrating its effectiveness in practical applications.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. The paper proposes the Multi-Objective Balanced Covering (MoB) method, systematically addressing the objective trade-off problem in visual token pruning. It provides provable performance bounds and linear scalability.\n\n   By introducing the Hausdorff distance, the authors derive the first closed-form error bound, quantifying the contribution of Visual Preservation (VP) and Prompt Alignment (PA) objectives, laying a solid theoretical foundation for future research.\n\n   The paper is well-structured and logically rigorous, guiding readers step-by-step through complex concepts and methods. Especially in the introduction of the MoB method, the working principle and implementation steps are explained in detail.\n\n    By reformulating visual token pruning as a bi-objective covering problem, the paper offers a novel perspective and approach, demonstrating high innovativeness.\n\n   Experimental results show that MoB performs exceptionally well on multiple benchmarks, achieving significant token reduction while maintaining high performance, indicating its practical applicability.\n\nWeaknesses：\n\nAlthough a new method is proposed, the performance improvement of MoB is relatively modest in some experimental settings, which may fall short of expectations in certain scenarios.\n\nThe interpretation of some experimental results could be more in-depth. The paper does not fully explore the applicability and limitations of different pruning strategies.\n\nWhile the method shows improvements, compared to existing approaches such as CrossGET, LLaVA-PruMerge, TokenPacker, Turbo, and AIM, which also demonstrate similar effectiveness, the paper lacks comparison and discussion with these methods.\n\nQUESTIONS:\nSee weakness\n\nLIMITATIONS:\nSee weakness", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes MoB (Multi-objective Balanced covering), a novel pruning method for vision-language models that balances visual preservation (VP) and prompt alignment (PA) from a geometric perspective. The key idea is to leverage prompt-visual coupling (η) to statically allocate token budgets without requiring training. MoB achieves significant speedups and performance retention across diverse tasks, outperforming existing training-free and sparse pruning methods.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The method demonstrates strong empirical performance across a wide range of vision-language tasks, often outperforming previous training-free pruning approaches.\n﻿2. The experiments are relatively comprehensive, covering diverse datasets and settings (though lacking evaluation on larger models such as LLaVA-13B or QwenVL-32B).\n﻿3. The paper is well-written and clearly structured, making a technically dense topic accessible.\n﻿4. Importantly, the authors attempt to analyze the token pruning problem from a geometric and theoretical perspective, introducing prompt-visual coupling (η) and proving formal upper bounds on pruning error. This provides a novel and insightful angle to understand the pruning trade-off.\n﻿\nWeaknesses:\n﻿1. While theoretical analysis is a commendably ambitious direction, it is also risky—there is a concern that the theory may not be entirely complete. In particular, the paper assumes that all pruning-relevant information can be captured by VP and PA, which may overlook more complex token interactions, reasoning steps, or latent semantics.\n﻿2. The design and usage of η raise several questions. It is treated as a dataset-level prior, but not explicitly estimated during inference. Its dependency on model architecture, computational cost, and lack of deployable estimation strategy (especially in unseen, open-domain settings) limit the practicality of the approach.\n\nQUESTIONS:\n1. The proposed MoB method hinges on the prompt-visual coupling strength (η) to allocate token budgets between visual preservation (VP) and prompt alignment (PA). However, η is not directly observable in real-world settings. How can η be inferred when the task type is unknown, such as in open-domain or user-driven inference? Does MoB require pre-defined task labels or dataset-specific heuristics to function, and how does this constrain its applicability in deployment?\n2. While η plays a central role in MoB's theoretical guarantees and pruning strategy, it is computed via Hausdorff distance between prompt and visual tokens—an operation that can be computationally expensive and dataset-dependent. The authors assume η is known a priori based on dataset labels, but this introduces additional supervision and cost that methods like FastV or SparseVLM do not require. Can the authors justify this additional cost, and is it fair to compare MoB against zero-cost or training-free baselines under this assumption? Furthermore, could η be efficiently approximated via lightweight dataset sampling (e.g., estimating η from a few examples), and if so, how robust would such estimation be across domains?\n3. Given that η reflects the geometric distance between prompt and visual token embeddings, it may vary across models of different sizes and architectures (e.g., LLaVA-7B vs. QwenVL-32B). Does η need to be re-estimated or recalibrated when switching to a different model? How transferable is the chosen Kp configuration across models, and to what extent does MoB’s pruning effectiveness rely on model-specific representations?\n4. The MoB framework assumes that all pruning-relevant information can be captured through the dual objectives of visual preservation (VP) and prompt alignment (PA). However, is this decomposition theoretically complete? Could there exist tokens whose value arises from higher-order interactions, global reasoning, or latent semantics not directly reflected in either VP or PA distances.\n\nLIMITATIONS:\nyes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper revisits the problem of visual token pruning in Multimodal Large Language Models, which is essential for reducing computational overhead from high-resolution visual input. While prior methods often integrate the two main pruning objectives, visual preservation (VP) and prompt alignment (PA), using static or heuristic strategies, this work introduces a principled, theoretically grounded framework for their integration. The authors propose a closed-form error bound for pruning based on the Hausdorff distance (Lemma 1), which ties pruning performance to the balance between VP and PA. Also, a new characterization of the trade-off between these two objectives under varying prompt-visual coupling conditions (Theorem 1) is demonstrated. Finally, a novel algorithm, Multi-objective Balanced Covering (MoB) is proposed which translates the trade-off into a budget allocation problem between prompt and visual token retention and is implemented with efficient greedy heuristics.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n1 -  The paper derives an elegant, novel error bound for token pruning (Lemma 1 and 2), tying model output error to geometric distances in token embedding space. The application of covering theory and trade-off quantification (Theorem 1) is a significant conceptual leap over prior work.\n\n2 - The paper convincingly identifies that prior multi-objective token pruning methods do not adapt to variable prompt-visual coupling across tasks, explaining the failure of static integration schemes (e.g., MustDrop).\n\n3 - Extensive evaluations (Tables 1–3) across multiple MLLMs, datasets, and pruning budgets show consistent improvements, including outperforming strong baselines like FastV and MustDrop. MoB retains >95% of task performance with up to 88.9% token reduction.\n\n\n**Weaknesses**:\n\n1 - Despite having a solid technical background, the paper is extremely challenging to parse. The derivations, especially in Section 3.2 (covering regularity, trade-off derivations), are packed with notation and require significant effort to unpack. For a NeurIPS audience, clearer conceptual exposition with diagrams or more intuitive examples would greatly improve readability. The notation choices (e.g., reuse of \"X\", \"S\", \"K\", “η”) are overloaded and can be confusing. More thoughtful typographic separation or a notation table would help.\n\n\n2 - Assumption 1 (Lipschitz continuity of transformers w.r.t. Hausdorff distance) is plausible but unvalidated. While claimed to hold in practice, no empirical evidence (e.g., via probing or ablation) is shown. Similarly, Assumption 2 (bounded prompt-visual coupling η) might not hold in open-ended real-world prompts (e.g., in generative tasks). The robustness of MoB under such violations is not discussed.\n\n\n3- While the evaluation is extensive, it misses standard datasets like COCO, VQAv2 in full, or long-form visual reasoning tasks like ScienceQA. Evaluating on such benchmarks would further strengthen the case for generality. Some ablations, such as the role of pruning depth (Fig. 6), are shallow. For example, pruning at different layers beyond ℓ = 2 is not deeply explored, though it's likely to affect performance.\n\nQUESTIONS:\n1 - Could the authors provide more intuitive explanations or visual illustrations—particularly for the key theoretical insights in Section 3.2 (e.g., the trade-off in Theorem 1)? This would help clarify the core contributions for a broader NeurIPS audience.\n\n\n2 - Has a notation table or summary been considered to address the overloaded variables like \"X\", \"S\", \"K\", and “η”? This might significantly improve the clarity of mathematical sections.\n\n\n3 - What empirical evidence supports Assumption 1 (Lipschitz continuity of transformers with respect to Hausdorff distance)? Would it be possible to validate this assumption through ablation or probing experiments?\n\n4 - The experiments omit widely used benchmarks such as COCO and full VQAv2. Is there a reason for this exclusion, and could results on these datasets be included to better demonstrate generalizability?\n\n5 - How does pruning at deeper or multiple layers beyond ℓ = 2 impact performance and trade-off dynamics? Can additional ablations be provided to justify the pruning depth choice?\n\n6 - Given the dependency of MoB on the estimation of η, what method is used to compute or approximate η in practice? Is this estimation stable and efficient?\n\n7 - To what extent is MoB sensitive to the choice of covering fold parameter k, especially under weak coupling conditions with long prompts? Would an adaptive mechanism be feasible here?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}