Why 1 + 1 < 1 in Visual Token Pruning: Beyond Naive Integration via Multi-Objective Balanced Covering
Authors: Yangfu Li
Hongjian Zhan
Tianyi Chen
Qi Liu
Yu-Jie Xiong
Yue Lu
Venue: NeurIPS 2025 poster ()
Keywords: Visual Token Compression
Multimodal LLMs
Hausdorff distance
Primary area: deep_learning
TL;DR: This study reveals and quantifies key factors in visual token pruning from a geometric covering perspective, and proposes Multi-Objective Balanced Covering, which significantly accelerates MLLM reasoning with negligible performance loss.

Abstract: Existing visual token pruning methods target prompt alignment and visual preservation with static strategies, overlooking the varying relative importance of these objectives across tasks, which leads to inconsistent performance. To address this, we derive the first closed-form error bound for visual token pruning based on the Hausdorff distance, uniformly characterizing the contributions of both objectives. Moreover, leveraging $\epsilon$-covering theory, we reveal an intrinsic trade-off between these objectives and quantify their optimal attainment levels under a fixed budget. To practically handle this trade-off, we propose Multi-Objective Balanced Covering (MoB), which reformulates visual token pruning as a bi-objective covering problem. In this framework, the attainment trade-off reduces to budget allocation via greedy radius trading. MoB offers a provable performance bound and linear scalability with respect to the number of input visual tokens, enabling adaptation to challenging pruning scenarios. Extensive experiments show that MoB preserves 96.4\% of performance for LLaVA-1.5-7B using only 11.1\% of the original visual tokens and accelerates LLaVA-Next-7B by 1.3-1.5$\times$ with negligible performance loss. Additionally, evaluations on Qwen2-VL and Video-LLaVA confirm that MoB integrates seamlessly into advanced MLLMs and diverse vision-language tasks. The code will be made available soon.

Why +1<1 in Visual Token Pruning: Beyond Naïve Integration via Multi-Objective Balanced Covering
Yangfu Li
Hongjian Zhan 4. v. Tianyi Chen Liu & Yu-jie Xiong Yue Lu +
East China Normal University, Shanghai Jiao Tong University ? Chongqing Institute of East China Normal University
c Shanghai University of Engineering Science
{yeli_cee, qiliu)@stu. ecnu. edu. cn, hjzhan@cee ecnu edu en guleimurray@sjtu edu. cn, xiongosues ldng edu. cn, yluêcs ecnu edu. cn
Abstract
Existing visual token pruning methods target prompt alignment and visual preser- vation with static strategies, overlooking the varying relative importance of these objectives across tasks, which leads to inconsistent performance. To address this, we derive the first closed-form error bound for visual token pruning based on the Hausdorff distance, uniformly characterizing the contributions of both objectives. Moreover, leveraging e-covering theory, we reveal an intrinsic trade-off between these objectives and quantify their optimal attainment levels under a fixed budget. To practically handle this trade-off, we propose Multi-Objective Balanced Cov- ering (MoB), which reformulates visual token pruning bi-objective covering problem. In this framework, the attainment trade-off reduces to budget allocation via greedy radius trading. MoB offers provable performance bound and linear scalability with respect to the number of input visual tokens, enabling adaptation challenging pruning scenarios. Extensive experiments show that MoB preserves 96.4% of performance for LLaVA-1.55 using only 1 11.1 % of the original visual tokens and accelerates VLaVA-Next-7BB by 1.3-1.5x with negligible performance loss. Additionally, evaluations on Qwen2-VL, and Video-LLaVA confirm that MoB integrates seamlessly into advanced MLLMs and diverse vision-lan language tasks. The code is available at https:: //gtth///ttbttttg com/YChenL/MoB
1 Introduction
Multimodal large language models (MLLMs) have shown impressive performance across a variety of vision-language tasks, including visual understanding 330 27 20), visual question answering (40]
16 (7], and visual-language reasoning 9 47 45). Since visual data exhibits much higher spatial
redundancy than language, MILMs are typically required to encode visual inputs numerous tokens, resulting in substantial computational overhead.
To address this issue, visual token pruning methods are proposed to accelerate MLLMs by selecting representative subsets of visual tokens. Most pruning methods focus on two distinct objectives: Visual Preservation (VP) 66 & 59 46). which retains tokens by minimizing redundancy maximizing visual salience, and Prompt Alignment (PA) (58) 51
48 which selects tokens most relevant to the prompt. 42 have been proposed to integrate VP and PA through various complex strategies. Counterintuitively, these methods do not exhibit dominant superiority compared to single-objective approaches, shown in Figure [ This observation naturally raises question: Does integrating different objectives offer fundamental advantages?
Recently, several multi- oujei-oojecctve approaches 51
"Corresponiing Author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Strang Coupling
Weak Coupling
OGA
0.2
vidite
TedVOD
-5
-10
-15
-20
Fostl
-25
N. are Moree that Muzitrop (UPHAA)
SparooVEM (M4)
-20
347
143
340
150
151
352
147
143
349
(+)
Relative Performance Compared MustDrop (VP PA) (N]
(b)
Hausdorff Distance betwoen Prompt and Visual Tokens
Vartlla
+1.39 SUE
SOA 66-776
+1.5m SQA
GOA +EA6E
68-799 SQA
COA
+9.50%
FastV
Spansoln
44.0%%
CC1
7OTZ
11,0%%
1L.8%
7075
11.00%
13.37
DEE
7972
12,4N%
MustDrop
Moll
NNE
40.100
13.00%
15,2%
VVB-C
MYB-C
1445
1942
(0)
5.00%
77M
8.20%
MoB 1. Vonile
Figure 1: (a) Comparison of single- vs. bi-objective pruning methods on LLa at a 66.7% pruning rate; (b) distribution of the prompt-visure coupling, revealing two distinct patterns across various tasks; weak coupling (large distance) and strong coupling (small distance); (c) radar charts of LLaVA-1.5-7B with visual tokens reduced from 576 to 192, 128, and 64 (left-to-right), demonstrating the consistent improvements of MoB across 10 well-recognized benchmarks.
Inspired by this question, we formulate preservation using the Hausdorff distance between the original and pruned token sets and derive the first closed-form form error bound for visual token pruning (Lemma [ This bound depends on VP and PA, while it is also affected by a prompt-visual coupling, measured by the Hausdorff distance between prompt and visual tokens. Notably, we identify two patterns of this coupling across popular benchmarks, as presented in Figure [b): weak coupling with large distance (e.g., TeA,VQA, POPE) and strong coupling with small distance (e.g., MMB, VizWiz). Our further analysis reveals that the effectiveness of the pruning objectives varies under distinct coupling patterns (Lemma 2 However, existing multi-objective methods overlook this variation and integrate VP and PA via constant strategies, yielding inconsistent improvements over single-objective baselines.
To quantify the effect of prompt- visual coupling, we reexamine visual token pruning from a geometric covering perspective. In this view, the retained tokens can be thought of as the union of two disjoint covers for prompt and visual tokens, where each objective corresponds to a Hausdorff covering radius, and the prompt-visual coupling is represented by the inter-cover diameter. By analyzing the geometric relationship between the radii and the diameter, we reveal an intrinsic trade-off between the two objectives (Theorem IT. which identifies the optimal attainment level of each objective to achieve the performance ceiling under a fixed pruning budget and prompt-visual coupling.
For practical solution to this trade-off, we propose Multi objtti-obeective Balanced Covering (MoB), training-free ffre visual token pruming method with provable performance guarantees and multilinear complexity (Theorem 2 MoB partitions the retained tokens into two disjoint subsets for PA and VP, employing greedy radius-trading strategies to reduce the trade-off in objective attainment to a budget allocation problem. This allows MoB to achieve the optimal balance under each coupling pattern by selecting appropriate subset sizes. As shown in Figure [Tc, MoB consistently outperforms both single-objective and multi-objective basclines by clear margin at identical pruning rates. Besides, MoB accelerates LLLVN--e by 1.3-1.5x with negligible performance loss. Ablation studies further validate our theoretical analysis. Our key contributions are summarized as follows:
To our knowledge, we present the first closed-form error bound for visual token pruning and its practical relaxation, characterizing the contributions of the two objectives to preservation quality.
We quantify the trade-off between the objectives and identify their optimal attainment level under a fixed budget and prompt-visual coupling, offering valuable insights into visual token pruning.
We propose Multi-objective Balanced Covering (MoB) for training-free visual token pruning, which reduces the trade-off of objective attainment to a budget allocation problem via two greedy radius- trading strategies, yielding both a provable performance guarantee and multilinear scalability.
2

---

Extensive experiments across 14 public benchmarks demonstrate the superiority of MoB. For instance, retains 96.4% and 97.9% performance for LLaVA-1.5-78 and Video-LLaVA-7B with an 88.9% reduction ratio, outerforming the second-best method by 2.7% and 1.6%, respectively. MoB can also be readily incorporated into advanced MILLMs, such as VA-Nett and Qwen2-VI..
Background
2.1 Related Work
28 have achieved remarkable progress in vision-language reasoning, owing to their robust cross-modalig modeling via attention However, the spatial redundancy inherent in visual signals typically leads to a large number of imput tokens [25 22 29 44). particularly in high-resolution images and multi frame vidcos (e.g., 2048 tokens Video-LLa VA (27]). This issue exacerbates the quadratic scaling problem of attention mechanisms, posing significant computational challenges. Moreover, to further enhance the visual capability by incorporating high-quality details, advanced MLLMs are now designed to support higher resolution images [24 10 4. thereby necessitating the processing of even more visual tokens (e.g., 2880 tokens in LLa LLaVA-EEXT (29)). In these scenarios, effectively selecting representative visual tokens becomes critical requirement for the real-world application of MLLMs.
Multimodal Large Language Model (MLLM). MLLMS 30 21
mechanisms [43 34
Visual Token Pruning. Due to the spatial redundancy, inputs to MI.LMs contain numerous less informative visual tokens. Visual token pruning accelerates MILMss by selectively retaining only the most critical tokens during inference. Existing methods typically focus on either visual preservation (VP) 16 38 8 52, (57 [2, [46] or prompt alignment (PA) [58] [1 48). VP-driven methods, such as ToMe 6 and Va-VruMerrrr [38], reduce redundancy by merging similar tokens, while F 8 and FLM [57] select tokens based visual salience. PA-driven approaches like Sparse VLM 58) rely on cross-modal attention to identify prompt-relevant tokens. More recently, integrates VP and PA through multi-stage pruning pipeline, reporting notable improvements. Despite these advances, existing methods largely overlook the varying relative importance of VP and PA across different scenarios. In this paper, formally characterize the contribution of each objective under a fixed pruning budget, and propose an algorithm that balances these objectives per scenario, yielding consistent improvements across diverse pruning conditions.
MustDrop 3i
2.2 Preliminaries
Pipeline of MILM. MLLMS perform vision-language reasoning by jointly processing multimodal inputs in a shared representation space. Formally, given visual tokens y(u) extracted from the visual inputs and prompt tokens p(1) encoded from user prompts, the multimodal input defined as
(1) = (()) LI p(),, D(I) = (vf)
1), p(1) = (r()) . -P') C
where N and L denote the numbers of visual and prompt tokens, respectively. We regard both ((1) and ((1) as compact sets on d-dimensional Euclidean space (IRd, II). The input ( is then fed into language model F(I.1) with I transformer block, and the final output is given by
y = F (2()) where Fu.M = f1
f1,
In particular, each fe follows the standard Transformer (e.g., multi-head self attentttennon (43, layer normalization 3 50).
. The intermediate feature for any layer E E (2...., is defined as
x = F1e-1(((()) = vit u p(o)
F1,.-11 = fe- ....o fi.
with (0) and (()) representing the visual and prompt tokens after é- layers, respectively.
Visual Token Pruning. To accelerate MILMs with minimal performance loss, visual token pruning selectively removes less-informative visual tokens at chosen intermediate layers of the language model Fl..Ij- Specifically, for any chosen layer fe, & E (2.....
... 7), pruning algorithms first select subset (tt) C (tt) of size K (i.c, pruning budget) and form the pruned input x! s L p(O. The corresponding output before and after pruning are then defined as
y= { ya == =
where F fI f
3

---

Notation
Description
Notation
Description
Pruming layer index: Final layer index. Embedding dimension.
Transformer black layer f.
d
Visual tokens layer ie.. V R²
Prompt tokens layer Le. R
sto
Retained visual tokens layer f. Le.. 8 V.
X
All tokens at layer Le. R= P.P. Avisual tokens, IV|=N; #prompt tokens, -P Full model (layers Outputs with full tokens 2' pruned tokens X, Lipechitz, constant of Jie, I] w.t. dø- Prompt center/ / Visual center set, S=Sp Ll Sv. Covering radius P. (y P).
Retained tokens layer le.. X,= P.P.
L
Pruning budget, |S|=K.
111
1).
Jie,
Submodel from layer I.
M. 9b
du (AA, 8))
Hausdorff distance between sets A and
Visual Visual-prompt coupling bound: (V. Budgets for Sp and Sv. K= Kp Kv-
Sp. Se
Kp. Kv
Eg
by
Covering radius V. d (S,, V). Effective dimension of V. P.
N(K,e)
Covering number of X radius e.
a,
Covering-rumber lower/upper constants for Validity radius covering bounds. 8-dilation of V and P.
a',
Covering- lower/upper constants for V. Small dilation radius (6
to
Va. Pk
(c,
Ball {ar ||æ-<]] S c)
Radius scaling factor (> 1).
D1
Trade-off constant (4ma') )
Da
Trade-off constant 1/2².
Optimal radius (oax(1//, vD, 1/ Candidate before
k
proposed ncarest-ncighibr covering.
Sp
by Kp- Preservation constant 1//eff.. Asymptotic lower bound (at least the order of Le., F. > O, no f(m) 2 eg(n) for 2 no-
o(0
Alignment consrant ((kkT//a)) /a)
e())
Asymptotically equal (same order); Le. 3
(3--)
0, no ei (ng) S [(n) & eag(n) for no-
Table I: Summary of notation used in the theoretical framework.
Finally, the objective of visual token pruning is formulated as
8 gvg,,,
lyy Vsll2
Notation. For brevity we omit the layer index (f) and simply write X' VUP and X₂ = SUP to denote the input and its pruned counterpart at arbitrary layer fe- We F to denote any composition mapping of the full model Fi,,J- Finally, we let [| denote the Euclidean norm.
Methodology
3.1 Revisiting Visual Token Pruning: Insights into Prompt-Visun Coupling
As shown in Fig. Da), multi-objective pruning methods fail to achieve the expected improvements, and objective-specific methods exhibit inconsistent performance across benchmarks. These observations motivate us to reexamine the problem visual token pruning. We begin by introducing Assumption which quantifies pruning performance in terms of the preservation of the original token set.
Assumption 1 (Lipschitz Continuity w.r.t. the Hausdorff Distance). Assume every partial composition FF (from layer to I) of the language model is Lipschitz continuous w.r.. the Hausdorff distance with constant C2 2 1. Formally, for any intermediate token sets X X Rd.
|[((X) F(X)l S Ce (X,
where dy is the Hausdorff distance induced by the Euclidean norm:
dH(X, max suuree infa,ex, || xell. supr,ex, inf || x.
(1)
Subsequently, we measure the preservation of the original token set X' using three pairwise distances among visual tokens V. retained tokens S, and prompt tokens P. thereby establishing unified performance bound for various visual token pruning algorithms, as presented in Lemma]
Lemma 1 (An Error Bound for Visual Token Pruning). Under Assump | given a token set with its pruned counterpart X' = =UP, X = SUP C Rd, the pruning error bound is given by:
[[(())-F(X)] < Cs max { min (S,
(S, V), du (V, (V,P), min (dn(S, V), dH }
d
Remark. Here d (S, P) and (S V) describe the prompt alignment and visual preservation, while dy (V, P) an inherent term that describes the prompt-visue coupling of input data.

---

Weak Coupling (large dy)
Strong Coupling (smali dg)
Text
MMB
dR
dp
dæ
clit
Is there a frtsbee the image?
What word does one of the hot batloana feature?
"Paris
"Shonghal"
Which country hightighte?? [ I'Jomolca "Hald" ...!
Relevant patch
relevant
Best-case case
Worst case relevant
Distant from the best-case patch
Near best
Illustration of prompt-visual coupling with two distinct patterns: In tne-ggraned tasks (e.g. POPE), only a few patches are critical, so the worst-case patch lies far from best-case ones, resulting in large Hausdorff distance and making prompt alignment valuable. In coarse-grained tasks (e.g. MMB), many relevant patches contain the answer cues; thus, the worst-case patch remains close to best-case ones, yielding small Hausdorff distance and making visual preservation more efficient.
Figure 2:
Proof in Appendix E.1 By Lemma in practical settings where |S| < [V], pruning performance is governed by non-trivial interaction among visual preservation, prompt alignment, and prompt-visual coupling. However, existing multi-objective methods typically overlook the coupling term dæ (V, P) and statically combine the two objectives across tasks, limiting their effectivvnesss Our empirical evidence across popular benchmarks validates two distinct patterns of dæ (V, P), each favoring different pruning objectives, as shown in Figure To further explicate the effect of prompt-visual coupling. introduce Assumption and propose practical relaxed error bound in Lemma
Assumption 2 (Prompt- Visual Coupling Bound). We assume the input visual data and prompts not entirely unrelated; hence, there exists constant n 0 for any intermediate token set X VUP C gd such that du (V, P) S. T), ensuring the reasonability of vision- language reasoning.
Lemma (A Relaxed Error Bound under Practical Budgets). Under Assumptions 7 and 2 let X= = =VUP, X, = SUP C Rd with = K & N. Partition the retained token set into two disjoint subsets: S Sp Ll Su. devoted to prompt alignment du (S,,.. P) and visual preservation d (S., V), respectively. Then, the pruning error bound reduces to
[[((X) - F(X)) <
Ce max m (d P), du (S,, V)} + C.
As Lemma 2 Jidicates, under weak coupling (large 17), most visual regions distant from prompt tokens in the semantic space. Consequently, if Sp misses the critical patches, du (Sp. P) dominates the pruming error, making the selection of S, i.e., prompt alignment, more significant. Conversely, under strong coupling (small 1)). du (Sp, P) tends to decrease in tandem with du (S,, V), reducing the marginal benefit of prompt alignment. To further guide pruning methods design, next quantify this trade-off governed by 7) through e-covering argument.
Proof in Appendix
E.2
3.2 Quantifying Prompt-Visual Visual Trade-Off: A Geometric Covering Perspective
We first introduce some geometric metrics in Definition I recasting each objective term du (Sp, P) and (S., V) as covering radii and the coupling term dæ (V, P) as an inter-cover diameter. Next. we relate each recasted objective its token budget [Spl. S. via covering regularity Lemma Finally, by loading the budget constraint and applying the triangle inequality between radii and diameter, we derive a quantitative trade-off jointly governed by K and n in Theorem
Definition (c-cover, Covering Number, and Covering Regularity). Let (R3. |l |]) be the d- dimensional Euclidean space and let X C Rdd be a compact set.
(a) e-cover. if there exists finite set C (cI,..., C Rd, E-Cover of X is given by B(c,c) = E where C is the collection of covering centers, and is the covering radius. (b) Covering number. The minimum cardinality of is the covering number of X at radius e:
X C Ucec B(c,
cll S c),
(xx,)) = min minfMee M 3C Rd, |C| M, X S UsEc B(c,t)}
5

---

(c) Covering regularity. constants A S and Eo > 0 such that
We say that X' satisfies d-dimensional covering regularity if there exist
A < e)
Be
Vee E (0,
Based on Definition Ia (b), Sp: S, CV can be thought of as two collections of centers such that P c U B(sp)
(0) (p), U B(s(" .v),
where the radii are given by G == dy du (Sp. co 6= du V). and the covering numbers satisfy ((P, cp) < [Spl. NVV, 6v) I |Syl. Thereby, we derive a lower bound of the required budget, i.e. (Sp). [Sv], to improve each objective, i.e., Ep. Eys based on def- -de-dnnnniional covering regularity.
Lemma 3 (Covering Number Bounds), Given P.V c Rd with an effective dimension dett. Sup- pose their 8-dilations V UvEV B(u, )). P3 = U pEdp B(p, 8) (ò < 1)) satisfy dest dimensional covering regularity; thus, there exist constants b a> O, P > a' O and co > 5 such that
EEp det < <P,, b
a' d'Euuæ < NV(V S & E,
Ee. E (o],
Remark. Previous work suggests that both visual and language embeddings concentrate on now-dimensional manifold, so the effective covering dimension satisfies the typical relation dest <
Lemma 3 demonstrates that once the radius (i.e., the objective) falls below co. any further improvement of it demands a ((c-dat) increase in the number of selected token.
Proof in Appendix E.3
By loading Lemma 3 into the budget constraint: S | K, and applying a two-step triangle inequality between the covering radii ép. 6v and the inter-cover diameter 7), we establish a K --y--ound in Theorem [b), which quantifies the trade-off governed by the budget and prompt-visudl coupling.
Theorem 1 (Trade-off between Prompt Alignment and Visual Preservation). Under Assump- tion 2 and the covering reeularryyy hypothesis of Lemmal 3 with constants a, a', dest >> 0, there exist radius-scaling factor z >> 1 such that n/2 > 8 and K < <N(P n/2) + NNV, f/rr for (Sp Lv),) C V with budget K satisfying
every pruning results S
max (Di -- -2/dar D2n²) S dn(Sp; P) du (S,,)), V),
where D1 = (Aaa')) > 0. D2
Remark (Optimal Attainment Level). D1 -2//en completely determined by the pruning budget, while D2 n2 quantifies the effect of prompt-visual coupling. The optimal attainment level per objective is given by c" = max(n/z,, vD, K-1/dat Any attempt to reduce one objective below c* forces the other above c", thereby increasing the overall pruming error.
Remark (Effect of Budget and Coupling Strength). As K decreases, correspondingly shrinks, ultimately making D2 n dominate the bound; while K increases, both of the terms reduce, thereby diminishing the trade-off and tightening the overall error bound.
Theorem T characterizes the optimal attainment level for each objective under a fixed pruning budget and prompt-visual coupling. However, is actually very challenging to dynamically determine the attainment level per objective during the pruning process. To address this, we propose Multi oojeit-b Balanced Covering, which leverages the monotonic relationship between covering radii and numbers to reduce the trade-off of attainment to budget-allocatte problem.
Proof in Appendix E.4
3.3 Multi-Objective Balanced Covering: From Trade-Off to Budget Allocation
Motivated by the insights in $3.2 Multi-objective Balanced Covering (MoB) recasts visual token pruning as bi-objective covering. Specifically, given a token set = V 1. P C Rdd with a budget K, the retained token set S is defined as the union of prompt center set Sp and visual center set S,:
S = Sp L S C
Rd
where PCU K2 B(
(0) .p). V VCU
K-K, B(s ..).
MoB then selects the cover centers (i.e., retained tokens) by minimizing the overall maximum radius:
(S): St) = s,us, cV, Kkp |S, K
min
maxfcp(S,). (S.)))

---

In practice, MoB solves this problem approximately by two sequential greedy covering procedures: selection of prompt center set Sp with budget Kp. and selection of visual center set S, with the remaining budget K - Kp. By the covering number bounds given in Lemma we have
Kp -p ((p K- - ((
where dett is the effective dimension of V, P. Accordingly, by selecting the unique budget Kp (i.e., fixing the remaining budget K Kp) under each coupling pattern, MoB ensures Ep. Ey (mmax(n/2, vD, K K-1/daa)) thus yielding provable performance guarantees across scenarios.
Normalization. For efficiency, MoB applies L.2 normalization to each i E X' so that |||||| 1. Hence, for any token pair æ1, 22 E X, the Euclidean distance can be induced by their cosine similarity:
|| x2ll = / 11,
Selection of Prompt Center Set Sp. Since all 8p V lie outside P, a typical solution for minimizing the radius Ep is Nearest-Neighbor covering (NN covering) [[5]. which uniformly allocates the nearest 8p E for each prompt token. However, the contribution of each prompt token is incquivalent, especially under weak prompt-visual coupling; thus, equal allocation risks missing the "best-case tokens." " To remedy this, we introduce a k-fold NN covering procedure. Formally, let L = [P] and first utilize temporary budget of kL to form candidate set.
k > 1 be hyperparameter;
Sp- = UpEP arg to (s.sss..))) p), k)), |S 2 Kp:
thereby over-sampling the k nearest visual tokens for each prompt token. Subsequently, we refine the candidate set by selecting the final Kp centers that maximize their worst-case alignment with P:
Sp arg topk (maxpep cos(8, Kp)-
By concentrating the limited budget on those visual tokens most strongly aligned with the key prompt tokens, this strategy ensures a better preservation of the critical regions in the visual input. We determine the appropriate & by ablation avoid the oversampling of few salient prompt tokens.
Selection of Visual Center Set S... Unlike the prompt center selection, each visual center sy lies in V. Thereby, we employ Farthest Point Sampling (FPS) 33) the remaining tokens, i.e., V S, to select the visual centers, which makes the visual centers Sy well-spread over V, minimizing the covering radius cy. Concretely, FPS operates by iteratively selecting the token farthest (ie., the most different) from the current centers S, where the distance given by
distyrs (v. S) min cos(sv. s)), Vs, S.
Subsequently, we initialize the visual centers with the empty set, i.e., s(")) = Ø. We then successively add the farthest visual token to the current centers S( Ll S, until it contains total of K elements. Hence, the visual centers at the subsequent iteration, S(++1), is given by:
1(++1) S(()) Li arg max s.EV. (stous,) disters(sv, S L..,), for (1..., K Kpl-
More details of the proposed MoB algorithm are provided in Appendix B
Theorem 2 (Performance Guarantee). Under Assump 7 and the revurirg of Lem 3
with consts a, a', dett 0 and b>> a. V/ a', for any budget split (K,, h K Kp), covering fold k and token set X V P Rdd with [V|= N, |P] L, d (V, P) I 17, the following hold: (a) Performance bound: The Performance degradation caused by MoB is upper bounded by
F(MO))((M)(())) S C2 Cymax ann. k, L) (KCp) 3 B(K - (KKK)))* -1/dat +Ce
where al1, k. = 17 Lbb 1/her
B = (b')
(b) Multilinear complexity: The complexity of MoB is given by TMo8= O(N
Remark (Coupling Tradc-off). Under weak coupling (large a.), minimizing the bound requires larger Ky- Conversely, under strong coupling (small a), the alignment term decays rapidly, favoring visual preservation (increasing Kp). Specially, under perfect coupling (1 = 0), the bound simplifies to |Ayll < C. (K- K,)) -1/dnn i.e. MoB reduces pure visual preservation.
7

---

Remark (Budget Scaling). As the budget K increases, the preservation term B(K Kg)- decays, requiring a corresponding increase in Kp (and thus a reduction in the alignment term) to re-balance the trade-off and further lower the overall error bound.
Remark (Scalability). MoB exhibits a multilinear scalability w.I.t #visual tokens N, #prompt tokens L, and #retained tokens K (K, L N), making easily adaptable to more chailenging scenarios involving large token counts, c.g., higher--reoluttonn inputs or multi-frame video.
Proof in Appendix H..5
Strong Coupling
Weak Coupling
Method
Objectives
Avg.
MMB MMBeN SQA VizWiz GQA
MME POPE VQA"
VQA"?
OCR
LLAVA-1
who Pruming, N
576; Taken Reduction Rate 0.0%
Vanilla
64.7
58.1
69.5
50.0
61.9
1862
58.2
78.5
297
100%
LLaVA-1.5-7B
Pruning budget K
192. Token Reduction Rate 66,76% %
FastV (ECCV"22)
[]
61.2
57.0
67.3
50.8
52.7
1612
64.8
52.5
67.1
291
91. 1.22%
Spa (5)
62.5
75.6
292
3%
MusiDrop [1
62.3
69.2
58.2
76.0
289
24%
DART (EMNI.P22))
VP
63.6
57.0
69.8
51.2
60.0
1856
82.8
57.4
76.7
296
98.8%
MoB (wlo 1)-rior)
PA 63.8
VP
57.5
70.0
5244
61.2
1858
84.5
58.2
77.9
304
100.2% 2%
11-prior
64.1
57.8
70.1
52.5
61
1860
84.8
58.5
78.3
307
100.6%
15578
Praning budget
Token
Reduuction
77.8%
FastY (ECCV"22)
VP
56.1
56.4
60.2
$1.3
49.6
1490
59.6
50.6
61.8
285
86.4%
SparseV LM (ICML'25)
FA
60.0
51.1
67.1
$1.4
56.0
1696
80.5
34.9
73.8
280
93.8%
MusiDrop (24.11)
BA vP
61.1
55.2
68.5
$2.1
56.9
1745
78.7
$6.3
74.6
281
95 95.6%
DART (EMNNTP'25)
VP
63.2
57.5
69.1
51.7
58.7
1840
80.1 1
56.4
75.9
296
98 0%%
MoB (w/o prior)
63.2
57.3
69.3
52.8
60.7
1842
81.7
$7.5
77.2
299
99 2%
+7y-prior
63.5
57.5
69.6
$2.7
60.9
1845
82.1
57.8
77.5
299
99 .4%.
LLaVA-1.5-7B
Pruming budget K
64; Token
Reduction Rate
88,
FastV (EOCV"24 24)
VP
48.0
$2.7
$1.1
50.8
46.1
1256
48.0
47.8
55.0
245
77. 3%
SparseVLM (ICML'25)
56.2
46.1
62.2
50.1
52.7
1505
75.1
51.8
68.2
180
84.6%
MustDrop (24.1l)
60.0
53.1
63.4
$1.2
53.1
68.0
54.2
69.
267
90. 1%
DART (EMNLP'22
72.4
MoB (wll p-prior)
61.7
69.7
59.0
75.5
277
96. .%%
+ +7p-rrorr
62.1
54.5
69.8
52.1
59.0
1806
71.2
57.0
75.5
277
96.4%
LLNVA-NoNt-B
w/o Prining, N 2880; Token Reduction Rate 0.0%
Vanilla 29
67.4
60.6
70.1
57.6
64.2
1851
86.3
64.9
81.8
517
100%
LA-No-Nx
Pruning budget K = 320: Token Reduction Rate = -8,9 %
FastV (ECCCV22)
VP
61.6
$1.9
62.8
$1.1
55.9
1661
71.7
55.7
71.9
374
86.4%
M (ICM)/25)
PA
60.6
54.5
66.
52.0
56.1
1533
82.4
58.4
71.5
270
85.9%
MusiDrop 11
62.8
55.1
68.0
54.0
57.3
1641
82.1
59.9
73.7
382
90.4%.
FasterVLM (24.12) 63
61.6
53.5
66.5
52.6
56.9
1701
83.6
74.0
401
89 8%
DART (EMNLP'25)
65.3
61.7
79.1
406
MoB (with yp-rior)
65.8
68.7
62.6
80.1
418
95.4%
Table 2: Partial comparison of image understanding on the LLaVA-7B series. For MoB, we set K, (64, 48, 32} and & E {4, 8), corresponding to token-reduction rates of (88.9%, 77.8%, 66.7%) For MoB with the 77 prior, we use Kp E @ 4 }) with A
3Kg 40 for strong-coupling benchmarks to the same token-reduction rates; the pruning layer is fixed at l = 2. Blue and Orange denote the best and the second. See Appendix for the detailed setting, and Appendix D.I for the full results.
and K, E ( 7K 5K T6 T2 } with h: = Ke for ceak-coupling benchmarks, corresponding
8
Experimental Results
Experiment Setting. We perform a comprehen- sive evaluation of the proposed MoB and several representative methods on two visual tasks: image understanding and visual understanding, together with an efficiency analysis. Our experiments em- ploy four popular MLLMs and include a total of 14 widely recognized benchmarks. For further de- tails regarding the benchmarks, models, baselines, and implement details please refer to Appendix
Method
GQA MME POPE VQAT MMB SQA Avg.
QwenZ-VL VL-7B
w/o Proming: Taken Reduction Rate
0.0%
Vanilla (44
62.2
2317
86.1
82.1
80.5
84.7 100%
DWL72-VL--B
Token Reduction Rute
66.7
FastV DART MoB
58.0
2130
82.1
77.3
76.1
80.0 94.0%
60.2
2345
83.9
80.5
81.4
(with m)
61.8
2268
84.7
81.1
79.5
82.3
98.4%
Qwen2- VL-7B
Token Reduction Rate = 77.8%
FastV DART
36.7
2051
79.2
12.0
74.1
78.3 91.0%
C
58.5
2175
82.1
75.3
77.3
79.6 94.3%
MoB (with m))
59.4
2203
82.8
95.2%
Image Understanding. Table Table the evaluation results across a variety of image- understanding tasks on L. series and Qwen2- VI., respectively. We highlight five key observa- tions: (a) MoB consistently outperforms all base-
Qwen2-VL-PD
Token Reduction
88.9%
FastV
51.9
1962
16.1
60.3
70.1
75.8 84.4%
DART MoB (with
55.5
2052
77.9
61.8
72.0
77.6
87.4%
56.5
2094
78.5
62.7
72.8
78.4 88.6%
Table 3: Comparative experiments on image under- standing with Owen2-VL--B..
8

---

MMB
MME
POPE
VizWiz
1281
1850
65
86
Pertamarre
Purfomannc
1800
Porfommrno
84
Perforrmnnc
10 1
1.700
Spersc/UM
Outs
Madthop
80
MestDra
PaxiDhop
1005
970
035
000
863
83
470
440
430
380
350
320
1700
1600
1500
1400
130
2000
1850
1705
1550 1400 1250
Latency 16)
Latency (4)
Latency (0)
Latency IN
Figure 3: Performance-Latenge trade-off comparisons across four benchmarks LLaVA-Next-7B.
lines on LLaVA-I.5-7BB in most cases. This will be more pronounced when incorporating the 77-riior which highlights the inherent advantage of our approach; (b) singlc-objective baselines exhibit com- plementary strengths under different coupling patterns, whereas MoB consistently outperforms all baselines, demonstrating the benefit of balanced objectives; (c) the superiority of MoB becomes even more significant under aggressive token reduction. Specifically, the improvement of MoB over the best baseline in average scores increases from 1.8% at 66.7% token reduction to 2.7% at an 88.8% reduction on ALaVA-1.55-8 (d) MoB matches the performance of the vanilla LLaVA-1.5-77 with only 33.3% of visual tokens, which may be attributed to the mitigation of hallucinations caused by redundant tokens; and (e) MoB scales seamlessly to advanced models, preserving 95.2% performance on Qwen2-VI--7 using only 22.2% of visual tokens. These observations demonstrate the superiority of MoB in leveraging limited visual tokens while minimizing performance degradation.
Video Understanding, As presented in Table 4 MoB is general and can be readily extended more challenging video scenarios without incur- ring additional cost. Specifically, MoB preserves 97.9% of average performance for Video-LLaVA- 7B using only 6.6% of the visual tokens, which sets new records in most VideoQA benchmarks, achieving 1.6% and 4.7% improvements over TwigVLM and VisionZip, respectively. These re- sults validate the generalization ability of MoB.
Method
TGIF MSVD MSRV ActNet Avg.
Video Mideo-LLaVA.1 LLaVA-7B Vanilla
Tokon Reduction Rate
0.0%
47.1
69.8
56.7
43.1
100%
Video-LLaVA-7B
Token Reduction Rute
93.4%
FastV 24)
23.1
38.0
19.3
30.6
52.1%
SparseVLM (ICME:25)
44.7
68.2
31.0
42.6
86.5%
VisionZip (24 51
42.4
52.1
TwigVLM (ICCV22)
30
44.7
68.3
54.6
41.5
96.3%
MoB (with y-prior)
45.3
68.8
55.2
42.8
97.9%
Table 4:
Comparative experiments on video under-
standing with Video-I.LaVA-7B.
We present a performance-latente trade-off measured on an NVIDIA A800- 80GB GPU in Figure 3 The results show that (a) MoB achieves a strong performance- latency trade-off, delivering a 1.5- speed-up for (b) due ignoring the K-r trade-off, the multi-stage method MustDrop is outperformed by single- objective methods V and Sparse on MME and POPE, and suffers significant performance drops as token budgets shrink (i.e. latency decreases). In contrast, MoB consistently maintains a robust trade-off across all benchmarks, surpassing all the baselines by clear margin; (c) MoB does not rely on attention scores to identify important tokens, making it compatible with flash attention and more efficient than attention-based methods such as Sparse VLM and FastV.
Efficiency Analysis.
with negligible performance loss;
5 Ablation and Discussion
Impact of (K, 7. Kp:). We study the impact of K, 77 and Kp on pruning performance across four benchmarks: GQA and TextVQA (weak coupling); VizWiz and MMB (strong coupling). As shown in Figure 4 the results can be interpreted by Theorem and Theorem aa),, respectively.
A. Theorem Perspective: When K is large, e.g., K 192, the trade-off is governed by D, K -2/daa, hence the trade-off intensity remains nearly identical across benchmarks. Conversely, When K is small, especially K 64, weak-coupling benchmarks, the trade-off turns to be governed by D2n2 thus, the trade-off intensity is obviously more pronounced in GQA and TextVQA than that in VizWiz. and MMB. These observations exactly confirm the validity of Theorem [
Theorem a Perspective. (a) Under weak coupling, the alignment term a(1, k. (),,))-1/40
is amplified. which requires a larger Kp to suppress the overall error. However, across benchmarks sharing the same coupling pattern, the optimal K, values exhibit only minor variation. (b) Increasing the total budget K pushes the optimal Kp upward to rebalance the two bound terms. Since the prompt length I. is fixed, adding more tokens yields diminishing returns for prompt alignment, which reflected in the declining ratio Kp/ K. These validate the performance bound in Theorem 2 a).

---

GQA /weea coupling)
(ireak coupling)
VizWiz/Strong coupling)
MMB (strong coupling)
iest velotvee Slive /N/
d4
reca rehovv Josc M%
55
66
C4
82
00%
0.00%
54
MAAS melet ctree
mven relatve sloww (N)
of
0.075
Perforaance
0.0%%
0.00%
58
500
1.64
130.8
50
17.0%
60
37.9%
340
140.7
158.1
20.25
14..0 6
25%
340
Size Prompr Coverage Set
Size of Prompt Geverage Set
Stze Prompt Coverage Sot
Sloc of Prompt Coverage Sut
Figure 4: Comprehensive ablation on the budget configuration (Kps K) across four benchmarks with distinct prompt-visual coupling 7) on LLaVA-115-7B, where K (64, 128, 192); the mean relative slope (%) is given by ai 01
1 Mo+1 quantifying the trade-off intensity; the ratio Kę reflects K
the cost-effectiveness of prompt alignment, and the box plot presents the distribution of 17.
0.50
Weak Couplig
TextVQ4
Long Prompt
String Coup
Perferancce
1860
1646 -+
Deffomunee
5i05s
Short Prompt
3840
58.0
0 lovra
Rato
0.25
tysc
layentc
Inyerb
levet2
0.13
S.parir
Saperi:
0.07
GGA
MME
POPE
MHB
3800
Yuria
132
57.0
450
520
550
Dataset
Latency 66
Latenic 1s]
Figure 5: Ablation on the ratio of Kpp
Figure 6: Ablation on the pruning layer.
Remarkably, the experimental results suggest that simply determining the optimal Kp for each of the two coupling patterns suffices to guarantee effective generalization across all scenarios.
Impact of Covering Fold k. We chose the covering fold & by examining the normalized ratio k/ Kp across eight benchmarks and nine budget configurations. As shown in Figure (a) weak-coupling benchmarks generally require a larger l to ensure critical region coverage, whereas strong- crong-coupping settings suffice with a smaller k; (b) benchmarks with longer prompts impose a lower cap on & to preserve sampling diversity and avoid redundant selection of salient tokens. Notably, weak- weak-coupling benchmarks with long prompts (e.g., GQA, Te exhibit a narrowly clustered optimal k. Kp range, reflecting their strict requirement to cover key tokens without excessive redundancy.
Impact of Pruning Layer. As shown in Figure 6 (a) models with visual token pruning consistently achieve a more favorable (b) Pruning in deeper layers provides more significant benefits for the weak-coupling TextVQA than strong-coupling MME. We attribute this to stronger cross-modal interactions in deeper MLLM layers, which facilitate identification of answer-relevant tokens under weak coupling, whereas pruning in shallow layers disrupts these interactions and incurs greater performance degradation.
cesormncceefficienn. trade-off than the vanilla model on both benchmarks.
Conclusion
In this paper, we present comprehensive analysis of visual token pruning, deriving the first closed- form error bound with practical relaxation. Leveraging e-covering theory, we quantify the intrinsic trade-off between the fundamental pruning objectives, iee,, visual preservation and prompt alignment, and identify their optimal attainment levels under a fixed pruning budget. Building on these insights, we introduce MoB, a training-frr free algorithm for visual token pruning. Based on greedy radius trading, MoB ensures the near-optimal attainment per objective via budget allocation, offering provable performance bound and multilinear scalability. Experimental results indicate that MoB matches the performance (100.6%) of -.55- with only 33.3% of visual tokens and can be seamlessly integrated into advanced MLLMs, such as LLaVA-Next-7B and Qwe72-VL-7B. Our work advances the understanding of visual token pruning and offers valuable insights for future MLLM compression.
Limitations. Our theoretical guarantees rely on assumption which is generally satisfied in practice but may not hold for all MI.LMs. Besides, MoB applies a preliminary search to select the proper Kp- which potentially introduces extra tuning overhead in practical applications. Future work will focus on developing an adaptive Kp selection mechanism driven by online estimation of the coupling 1,
10

---

Acknowledgments
The work performed the Shanghai Key Laboratory of Multidimensional Information Processing, East China Normal University; the Institute of Natural Sciences and School of Mathematical Sciences, Shanghai Jiao Tong University; and the Chongqing Key Laboratory of Precision Optics, Chongqing Institute of East China Normal University, with joint support from the National Natural Science Foundation of China (62176091), the Natural Science Foundation of Chongqing (CSTB2024NSCQ- MSX0877), the Science and Technology Commission of Shanghai Municipality (2IDZ2203100) and the Fundamental Research Funds for the Central Universities.
References
[1] Yash Akhauri, Ahmed F AbouElhhamyed, Yifei Gao, Chi-Chih Chang, Nilesh Jain, and Mohamed S Abdelfattah. Tokenbutler: Token importance is predictable. arXiv preprint arXiv:2503.07518. 2025.
[2] Kazi Hasan Ibn Arif, Jin Yoon, Dimitrios S Nikolopoulos, Hans Vandierendonck, Deepu John, and Bo Ji. Hired: Attention guided token dropping for efficient inference of high-resolution vision-language models in resource-constrriine environments. arXiv preprint arXiv:2408. 10945, 2024.
[3] Jimmy Lei Ba. Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint 1607.:1607.0644 2016.
[4] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang. Peng Wang. Shijie Wang, Jun Tang, al. Qwen2. 5-vl technical report. arXiv preprint 2502:22002.1923: 2025.
[5] Kaichen Zhang* Fanyi Pu* Xinrun Du Yuhao Dong Haotian Liu Yuanhan Zhang Ge Zhang Chunyuan Li Bo Li*, Peiyuan Zhang* and Ziwei Liu. Lmms-eval: Accelerating the development of large multimoal models, March 2024.
[6] Daniel Bolya, Cheng- Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token merging: Your vit but faster. In The Eleventh International Conference on Learning Representations, 2022.
[7] David Chen and William B Dolan. Collecting highly parallel data for paraphrase evaluation. In Proceedings of the 49th annual meeting of the association for computational linguistics: human language technologies, pages 190-200, 2011.
[8] Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang. An image is worth 1/2 tokens after layer 2: Plug-and playaynd inference acceleration for large vision- languaee models. European Conference on Computer Vision, pages 19-35. Springer, 2024.
[9] Liangyu Chen, Bo Li, Sheng Shen, Jingkang Yang, Chunyuan Li, Kurt Keutzer, Trevor Darrell, and Ziwei Liu. Large language models are visual reasoning coordinators. Advances Neural Information Processing Systems, 36:70115-70144 2023.
[10] Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shen- glong Ye, Hao Tian, Zhaoyang Liu, al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05221,, 2024.
[11) Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic assual-liguuistee tasks. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages
2024.
[12] Yash Goyal, Tejas Khot, Douglas Sommers-SStay Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEFE conference on computer vision and puttern recognition, pages 1904-69133 2017.
11

---

[13] Zhiyu Guo, Hidetaka Kamigaito, and Taro Watanabe. Attention score is not all you need for token importance indicator in kv cache reduction: Value also matters. arXiv preprint ar0iv:2406. /2335, 2024.
[14] Danna Gurari, Qing Li, Abigale J Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In Proceedings of the IEEE conference computer vision and puttern recognition, pages 3608-3617, 2018.
[15] Dorit S Hochbaum and David B Shmoys. A best possible heuristic for the k-center problem. Mathematics of operations research, 10(2):180-185 184, 1985.
[16] Wenbo Hu, Yifan Xu, Yi Li, Weiyue Li, Zeyuan Chen, and Zhuowen Tu. Bliva: A simple multimodal Ilm for better handling of text-rich visual questions. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 2256-2264, 2024.
[17] Drew A Hudson and Christopher Manning. Gqa: new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6700-6709, 2019.
[18] Yunseok Jang, Yale Song, Youngjae Yu, Youngjin Kim, and Gunhee Kim. Tgif-qa: Toward spatio-temporal reasoning in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages
2017.
[19] Kasper Green Larsen and Jelani Nelson. The joniooonnnnndddnstaaass lemma optimal for linear dimensionality reduction. arXiv preprint arXiv: /11/1441114200 2014.
[20] Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv preprint 20X 2024.
[21] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: 2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19730-19742. PMLR, 2023.
[22] Yanwei Li, Yuechen Zhang, Chengyao Wang, Zhisheng Zhong, Yixin Chen, Ruihang Chu, Shaoteng Liu, and Jiaya Jia. Mini-gemini: Mining the potential of multi-modality vision language models. arXiv preprint arXiv:2403.18814 2024.
[23] Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. arXiv preprint arXiv:2305. 10355, 2023.
[24] Zhang Li, Biao Yang. Qiang Liu, Zhiyin Ma, Shuo Zhang, Jingxu Yang, Yabo Sun, Yuliang Liu, and Xiang Bai. Monkey: Image resolution and text label are important things for large multi-modal models. In proceedings of the IEEF/CVF conference computer vision and pattern recognition, pages 26763-26773, 2024.
[25] Youwei Liang, GE Chongjian, Zhan Tong, Yibing Song, Jue Wang, and Pengtao Xie. Evit: Expediting vision transformers via token reorganizations. In International Conference Learning Representations, 2022.
[26] Zijing Liang, Yanjie Xu, Yifan Hong, Penghui Shang, Qi Wang, Qiang Fu, and Ke Liu, A survey of multimodel large language models. In Proceedings of the 3rd International Conference on Computer, Artificial Intelligence and Control Engineering, pages 405-409, 2024.
[27] Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection, 2024.
[28] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 26296-26330 26306, 2024.
[29] Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024.
12

---

[30] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023,
[31] Ting Liu, Liangtao Shi, Richang Hong, Yue Hu, Quanjun Yin, and Linfeng Zhang. Multi-stage vision token dropping: Towards efficient multimodal large language model. arXiv preprint arXiv:2411.10003 2024.
[32] Xuyang Liu, Ziming Wang, Yuhang Han, Yingyao Wang, Jiale Yuan, Jun Song, Bo Zheng, Linfeng Zhang, Siteng Huang, and Honggang Chen. Compression with global guidance: Towards training free high-resolution milms acceleration. arXiv preprint arXiv:2501.05179, 2025.
[33] Yuliang Liu, Zhang Li, Mingxin Huang, Biao Yang, Wenwen Yu, Chunyuan Li, Xu-Cheng Yin, Cheng-L.n Liu, Lianwen Jin, and Xiang Bai. Ocrbench: on the hidden mystery of ocr in large multimodal models. Science China Information Sciences, 67(12))220102, 2024.
[34] Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task agno-sgo visiolinguistic representations for lisionaadd-lanuage tasks. Advances in neural information processing systems, 32, 2019.
[35] Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-We Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In The 36th Conference on Neural Information Process- ing Systems (NeurIPS), 2022.
[36] Carsten Moenning and Neil A Dodgson. Fast marching farthest point sampling. Technical report, University of Cambridge, Computer Laboratory, 2003.
[37] Yingzhe Peng, Xinting Hu, Jiawei Peng, Xin Geng, Xu Yang, et al. Live: Learnable in-context vector for visual question answering. Advances in Neural Information Processing Systems, 37:9773-9800, 2024.
[38] Yuzhang Shang, Mu Cai, Bingxin Xu, Yong Jae Lee, and Yan Yan. Llava-prumerge: Adaptive token reduction for efficient large multimodal models. arXiv preprint arXiv:2403.15.66 2024.
[39] Zhenwei Shao, Mingyang Wang, Zhou Yu, Wenwen Pan, Yan Yang, Tao Wei, Hongyuan Zhang, Ning Mao, Wei Chen, and Jun Yu. Growing twig to accelerate large vision-language models. arXiv preprint arXiv:2503. /4075, 2025.
[40] Zhenwei Shao, Zhou Yu, Meng Wang, and Jun Yu. Prompting large language models with answer heuristics for knowledge-based visual question answering. In Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, pages 14974-14983, 2023.
[41] Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8317-8326, 2019.
[42] Xudong Tan, Peng Ye, Chongjun Tu, Jianjian Cao, Yaoxin Yang, Lin Zhang. Dongzhan Zhou, and Tao Chen. Tokencarve: Information-peee peaattioo-reeeviinggng.. visual token compression in multimodal large language models. arXiv preprint arXiv:2503. 10501, 2025.
[43] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[44] Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge, et al. Qwen2-vl: Enhancing language model's perception of the world at any resolution. arXiv preprint arXiv: 12191, 2024.
[45] Yiqi Wang, Wentao Chen, Xiaotian Han, Xudong Lin, Haiteng Zhao, Yongfei Liu, Bohan Zhai, Jianbo Yuan, Quanzeng You, and Hongxia Yang. Exploring the reasoning abilities of multimodal large language models (mllms): comprehensive survey on emerging trends in multimodal reasoning. arXiv preprint arXiv:2401.06800, 2024.
13

---

[46] Zichen Wen, Yifeng Gao, Shaobo Wang, Junyuan Zhang, Qintong Zhang, Weijia Li, Conghui He, and Linfeng Zhang. Stop looking for important tokens multimodal language models: Duplication matters more. arXiv preprint arXiv:2502.1 1494, 2025.
[47] Jiannan Wu, Muyan Zhong, Sen Xing, Zeqiang Lai, Zhaoyang Liu, Zhe Chen, Wenhai Wang, Xizhou Zhu, Lewei Lu, Tong Lu, et al. Visionllm v2: An end-to-end generalist multimodal large language model for hundreds of vision- language tasks. Advances in Neural Information 2024.
vision-language
Processing Systems, 37:69925-69999
[48] Long Xing, Qidong Huang, Xiaoyi Dong, Jiajie Lu, Pan Zhang, Yuhang Zang, Yuhang Cao, Conghui He, Jiaqi Wang, Feng Wu, et al. Pyramiddrop: Accelerating your large vision-language models via pyramid visual redundancy reduction. arXiv preprini arXiv:2410. 2224441..777777 17247, 2024.
[49] Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Yueting Zhuang. Video question answering via gradually refined attention over appearance and motion. In Proceedings of the 25th ACM international conference on Multimedia, pages 1645-1653, 2017.
[50] Jingjing Xu, Xu Sun, Zhiyuan Zhang, Guangxiang Zhao, and Junyang Lin. Understanding and improving layer normalization. Advances in neural information processing systems, 32, 2019.
[51] Senqiao Yang, Yukang Chen, Zhuotao Tian, Chengyao Wang, Jingyao Li, Bei Yu, and Jiaya Jia. Visionzip: Longer is better but not necessary in vision language models. arXiv preprint arXiv:2412.0444 044667 2024.
[52] Weihao Ye, Qiong Wu, Wenhao Lin, and Yiyi Zhou. Fit and prune: Fast and training- visual token pruning for multi-modal large language models. arXiv preprint arXiv:2409.100 10197, 2024.
[53] Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang. and Dacheng Tao. Activitynet-qa: A dataset for understanding complex web videos via question answering. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 9127-9134, 2019.
[54] Yuanhan Zhang Bo Li Songyang Zhang Wangbo Zhao Yike Yuan Jiaqi Wang Conghui He Ziwei Liu Kai Chen Dahua Lin Yuan Liu, Haodong Duan. Mmbench: Is your multi molti- model an all-around player? arXiv:2307 200.20682.688 2023.
[55] Zheng Zhan, Yushu Wu, Zhenglun Kong, Changdi Yang, Yifan Gong, Xuan Shen, Xue Lin, Pu Zhao, and Yanzhi Wang. Rethinking token reduction for state space models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 1686- 1697, 2024.
[56] Kaichen Zhang, Bo Li, Peiyuan Zhang, Fanyi Pu, Joshua Adrian Cahyono, Kairui Hu, Shuai Liu, Yuanhan Zhang. Jingkang Yang. Chunyuan Li, and Ziwei Liu. Lmms-eval: Reality check on the evaluation of large multimodal models, 2024.
[57] Qizhe Zhang, Aosong Cheng, Ming Lu, Zhiyong Zhuo, Minqi Wang, Jiajun Cao, Shaobo Guo, Qi She, and Shanghang Zhang. [cls] attention is all you need for training-fres visual token pruning: Make vlm inference faster. arXiv preprint arXiv:2412.018.
2024.
[58] Yuan Zhang, Chun-Kai Fan, Junpeng Ma, Wenzhao Zheng, Tao Huang, Kuan Cheng, Denis Gudovskiy, Tomoyuki Okuno, Yohei Nakata, Kurt Keutzer, et al. Sparsevim: Visual token sparsification for efficient vision-language model inference. arXiv preprint arXiv:2410.04417,
2024.
[59] Yiwu Zhong, Zhuoming Liu, Yin Li, and Liwei Wang. Aim: Adaptive inference of multi-modal Ilms via token merging and pruning. arXiv preprint arXiv:2412.03248, 2024.
[60] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En- hancing lannuan understanding with advanced large language models. arXiv preprint arXiv:2304.10522 2023.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: See Section for the main claims; see Sections 1o and Appendices D and for the detailed contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: See Section for the discussion on the limitations.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
15

---

Justification:
The assumptions are clearly stated in the statement of the theorems: any partial composition F of the language model satisfies a Lipschitz Continuity w.r.f. the Hausdorff
Distance with constant Ce 2 1, the 5-dilations V == Uvey B(v, 5), Ps = pe Bp,, 6) (ó < 7)) satisfy det-dnimesio covering regularity with constants a, a', b,, D.. Seee Theo-
rems [ and 2 The proof of Lemma T can be found in Appendix F. the proof of Lemma
can be found in Appendix E.2 the proof Lemma can be found in Appendix E.3 the
proof of Theorems and are provided in Appendices E.4 and E.5 respectively.
E.d
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: See Appendices B and C for the information needed to reproduce the main experimental results.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c)
If the contribution is new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility.
16

---

In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
The code is provided in the supplemental material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. ec/
public/guides httii//ggiides/oooeuuuinssssssss onliiccyooyy) for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nips //iips.c/pput http:liiigguidde//oeeeeeeeeeeeee
wwwiensssooniiii ssionpolicy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended the paper) is recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: The experiment does not involve any training process. See Section and ap- pendix for all the test details.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: The experimental results are accompanied by significance tests, and cross- validation conducted using a publicly available third-party framework.
Guidelines:
The answer NA means that the paper does not include experiments.
17

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: See Appendix for the computer resources needed to reproduce the experi- ments.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make it into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: The research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: See Section and appendix A for the discussion on the broader impacts.
18

---

Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
The assets used in this paper are properly credited, and the license and terms are respected.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
19

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the datasets.paperssiii paperswi thcode. www.coo//aaaaa datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing nor research with human subjects
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
20

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [Yes]
Justification: MoB is an inference-time algorithm that prunes visual tokens in pre-trained multimodal
LLaVA-Next-75 Diven2-VL-7B and Video-LLaVA. 7B. At Transformer layer & = 2, removes redundant vision tokens using a bi-objective covering rule (see Section 3 and appendices and The LLM weights remain frozen; no additional data, gradient updates, or prompt engineering are used. Thus the LLMs serve as essential yet unmodified back-bones whose intermediate embeddings the input MoB.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non noa-standard components.
Please refer to our LLM policy hbtppp///eerrrpsccc/oooffrrrrrrp for what should or should not be described.
21

---

Appendix
In the appendix, we provide additional information as listed below:
provides the broader impacts of MoB
provides the algorithm details and pseudocode of MoB
provides the overview of the data, models, baselines and implementation details.
provides the additional experimental results.
provides the omitted technical details.
A Broader Impacts and Limitations
Theory Impacts. Beyond the visual setting, MoB's ' theoretical lenl-eellleee balancing Visual Preservation (retaining sufficient context) and Prompt Alighmennn (isolating "golden evidence")- naaurrlly transfers to language domain. It makes the key challenging (context vs. evidence) in long- cong-context LLM explicit and offers actionable guidance for token-level compression and scheduling under fixed context budgets. In practice, this perspective informs RAG (calibrating recall vs. precision) and Summarization/LL.M memory (trading coherence vs. conciseness).
Application Impacts. The proposed MoB yields substantial acceleration of MLLMs with neg- ligible performance loss, thereby enabling high-resolution vision-language models to operate on resource-constrained platforms such as edge devices and mobile systems while supporting low-latency applications- inplications-innludinn assistive technologies for the visually impaired, autonomous navigation, and AR/VR. Besides, MoB potentially benefits other redundancy-heat domains (e.g., point clouds and multi-sensor fusion), guiding efficient token-level compression beyond vision.
Theory Limitations. The theoretical analysis (lemma T and theorem D and the performance guar- antees (theorem 2 rely on assumption [ (Lipschitz Continuity) and lemma 3 (Covering Regularity). In embedding spaces that violate metrie properties exhibit highly irregular token distributions, these conditions may fail to hold, and the provable performance bounds may no longer apply.
Application Limitations. Our deployment presently requires priori estimate of rj to set the pruning hyperparameters K, and When 7j is misestimated for a new model, domain, or input distribution, the selected Kp and can deviate from their optimum, leading suboptimal speed-accural trade-offs.
22

---

B Algorithm
Algorithm I Multi-Objective Balanced Covering (MoB)
Require: Visual token V E Rx Prompt token P E -Rixa Budget Ky. Ky, Covering fold k Ensure: Index list for select tokens N
1: Normalize all token embeddings to unit l norm: V + V 2orw P P 2,row
Step 1. Select Prompt Centers via Nearest-Neighhor Covering
2: Compute cosine-sinee similarii matrix via P V M + PV
P M E RLXY
Retrieve nearest token indices per prompt:
Cidx ArgTopK(M, k, axis 1), Csin +- TopK(M, k, axis 1)
> Ciázi Cain E RL*k collects index and similarity of ls closest centers per prompt token.
Deduplicate candidate indices
4: Flatten index and similarity arrays: Ciax + Platten(Cias) C f- Flatten (C.in)
Ciax Nuk Csin E R
5: Remove duplicate indices, preserving associated similarities:
(Ciaz Chim) <- UniqueIndices Cata)
Kp S Ciia S Lk
6: Identify top-Kp prompt centers by similarity: ip ArgTopK(Cā..) Kp)
7: Form the prompt-center index list: Sp + Ci [pp
> Sp E N
Step 2. Select Visual Centers via Farthest-Poinl Sampling
8: Initialize selected centers: S + Sp
# Initialize coken-to-prompt minimum distances
9: Compute pairwise minimum distances between all tokens and selected prompt centers:
d + nxx V
d t Min(d, axis = 1)
Selected centers have zero distance in R.V.
# Farthest-Pood Sampling
for 1 to Kv do
11:
Select the token farthest from current centers: i* + (rgMax(d),
S +- Concat(S, i²)
Selected tokens are excluded (distance 0) from further sampling.
12:
Compute cosine distances to the newly selected token: da < 1,
1V [
Update each token's 's minimum distance: <- ElementwiseMin(d, d,)
D Distance of
newly selected token i" set to zero in cl.
14:
end for
15: return S
Algorithm 2 Compute Prompt- Visual Coupling
Visual embeddings V E xd, Prompt embeddings E Rd Ensure: Hausdorff distance h( a(V, P)
Require:
Step 1. Compute Pairwise Euclidean Distances
1:
Compute distance matrix via edist: D edist V, P, p
D Ra.xa
Step 2. Directed Hausdorff Distances
2: Visual-to-prompt directed distance:
dy + min axis = 2) h + max(dy-pp)
3: Prompt-to-visual directed distance:
dp +- min (D, axis = 1) hp f max(dp-vv)
Step 3. Final Hausdorff Distance
4: return max max(hy-pp. (hy-pg bp
23

---

C Experiment Details
C.1 Benchmarks
Our experiments evaluate the vision-language reasoning abilities of multimodal large language models using comprehensive suite of widely recognized benchmarks. For image understanding tasks, we assess performance on ten public benchmarks: GQA, MMBench (MMB) and MMBench-CN (MMBex). MME, POPE, VizWiz, ScienceQA (SQA), VQA V2, TextVQA (VQA"), and OCRBench (OCR). For video understanding tasks, we conduct experiments on four popular benchmarks: TGIF- QA (TGIF), MSVD-QA (MSVD), MSRVTT-QA (MSRV), and ActivityNet-QA (ActNet). The following section provides a concise overview of these benchmarks:
leverages scene graphs. questions, and images to evaluate visual scene understanding and reasoning. By incorporating detailed spatial relationships and object-level attributes, it poses significant challenges for models to perform accurate visual reasoning in complex environments.
GQA 17
MMBench [54] introduces a hierarchical evaluation framework where model capabilities are dissected into three levels. Level-1 focuses on basic perception and reasoning; Level-2 subdivides these abilities into six distinct sub-skills; and Level-3 further refines the evaluation into 20 specific dimensions. Its Chinese counterpart, MMBench-CN, adopts a similar structure.
rigorously tests perceptual and cognitive abilities across 14 sub-tasks. By employing carefully crafted instruction-answer pairs and succinct instructions, MME minimizes data leakage and provides robust, fair assessment of model's 's multifaceted performance.
MMEE 226]
targets the evaluation of object hallucination by posing binary questions about object presence in images. It quantifies hallucination levels using metries, e.g., accuracy, recall, precision, and Fl score, offering precise and focused measure of model reliability.
POPE 23
Wiz 14 is visual question answering benchmark derived from interactions with blind users. Comprising over 31, 000 image-question pairs with 10 human-annotated answers per query, encapsulates the challenges of low-quality image capture and conversational spoken queries, thereby emphasizing real-world visual understanding.
ScienceQA 33 spans multiple scientific domains by organizing questions into 26 topics, 127 categories, and 379 skills. This hierarchical categorization provides diverse and rigorous testbed for evaluating multimodal understanding, multi-step reasoning, and interpretability across natural, language, and social sciences.
VQA V2 12 challenges models with open-ended questions based on 265, 016 images that depict a variety of real-world scenes. Each question is paired with 10 human-annotated answers, facilitating thorough evaluation of a model's 's capacity to interpret and respond to diverse visual queries.
TextVQA [41) focuses the integration of text within visual content. It evaluates model's proficiency in reading and reasoning about textual information embedded in images, thereby requiring balanced understanding of both visual and linguistic cues.
OCRBench 333 is a comprehensive benchmark for evaluating the OCR capabilities of
language models across five key tasks: text recognition, scene text-centric and document-oriented VQA, key information extraction, and handwritten mathematical expression recognition.
TGIF-QA 18 adapts the visual question answering task to the video domain by focusing on GIFs. With 165K question-answer pairs, incorporates tasks, e,g., counting repetitions, identifying repeat- ing actions, detecting state transitions, and frame-specific question answering, thereby demanding detailed spatio-temporal analysis.
MSVD-QA 49 builds upon the MSVD dataset by pairing 1, 970 video clips with approximately 50.5K QA pairs. Questions categorized into five distinct types, e.g., what, who, how, when, and where, making it a versatile tool for evaluating video understanding.
MSRVTT-QA 7 features 10K video clips and 243K QA pairs designed to test the integration of visual and temporal information. Its structure, which parallels that of MSVD-QA through the inclusion of five question types, further enriches the evaluation landscape for video-based tasks.
24

---

ActivityNet-QA [53 provides 58K annottatd question- answer pairs drawn from 5.8K videos. Its focus on questions related to motion, spatial relationships, and temporal dynamics necessitates long-term spatio-temporal reasoning, thus serving as benchmark for advanced video understanding.
C.2 Multi-modal Large Language Models
We evaluate MoB using various open-source multimodal large language models (MLLMs), For image understanding tasks, experiments are conducted on the LLaVA series, including LLa and LLaVA-Next-7B, as well as the Qwen-Vi series, such as Qwen2-VL-7B. Specifically, LLaVA-NexI and Qwen2-VL are utilized to validate performance on high-resolution images, i.e., those with larger number of visual tokens. For video understanding tasks, we employ Video-LLaVA-P7 as the baseline model, following the settings reported in its original paper to ensure a fair comparison.
LLaVA-1.5-7B (8 a robust vision-language model built on the LLaVA framework. It processes images resized 224 x 224 and tokenizes them into roughly 572 visual tokens using base vision encoder. This design balances fine-grained visual representation with computational efficiency, making it effective for diverse multimodal tasks.
LLaVA-Next-7B [29] extends the LL. by incorporating refined training strategies and data curation. It supports higher-resolution inputs (up to 448 x 448), yielding up to 2880 visual tokens. These enhancements improve its visual reasoning capabilities and enable more precise alignment between visual content and language but also incur significantly increased computational cost.
Qwen2-VL-7B 44 augments the Qwen2 language model with visual input capabilities. This model leverages cross-modal pretraining to seamlessly merge vision and language, demonstrating strong performance in complex visual question answering and comprehensive scene understanding.
Video-LLa VA-7B [27] extends the LLaVA framework into the temporal domain by processing video inputs. It is designed capture both spatial and temporal dynamics, enabling effective video comprehension and video-based question answering with coherent and context-aware responses.
C.3 Baselines
To validate the superiority of the proposed MoB, we construct a robust baseline that integrates a comprehensive set of representative existing methods, which encompass single-stage methods with both two distinct objectives and several multi-stage methods.
ToMe O employs a lightweight metthnnt scheme to merge visually similar tokens across transformer layers, thereby reducing computation without additional training. Its simple yet effective design makes it well suited for real-time applications.
F g leverages attention maps in the early layers to identify and prune non-critical tokens, significantly reducing initial computational overhead. This focus on early-stage reduction allows the model operate more efficiently while maintaining performance.
Sparse LM (58) ranks tokens based on cross-modal attention to assess image- image-prompt relevance and adopts adaptive sparsity ratios retain key information. It further incorporates token recycling mechanism to balance the trade-off between efficiency and accuracy.
allocates token budgets across image partitions by using CLS token attention and then HiRED 2
selects the most informative tokens within each partition. This spatially aware approach ensures balanced reduction while preserving contextual details.
LLa VAPuueree 38 combines pruning and merging strategies by dynamically removing less important tokens using sparse CLS-visual attention. It then clusters the retained tokens based on key similarity, ensuring that crucial visual features remain intact.
PyramidDrop 48 adopts a progressive token-dropping strategy across different model stages, resulting in pyramid-like token structure. This method carefully balances the reduction of tokens with the preservation of performance as the processing advances.
integrates several token-reduction strategies including spatial merging, text-guided pruning, and output-aware cache policies. Its multi-faceted approach efficiently reduces token counts across various stages of the model.
MustDrop 31
25

---

VisionZip [5]]
first selects dominant tokens that capture the majority of an image's 's information and then merges the remaining tokens based on semantic similarity. This approach dramatically reduce token redundancy while accelerating inference and maintaining robust performance.
evaluates token importance using CLS attention in the encoder and prunes tokens before they interact with the language model. This preemptive reduction streamlines the overall process and enhances model efficiency.
Faster [57
GlobalCom' 32 employs a hierarchical strategy by coordinating thumbnail tokens to allocate adaptive retention ratios for high-resolution crops. This approach successfully preserves local details while providing effective global context reduction.
DART [46] leverages token duplication to guide its pruning process instead of relying solely on attention scores. By selecting a small set of pivot tokens and retaining only those with minimal redundancy, DART achieves significant acceleration in training-free manner.
CorenCarve 442 implements two-stage, training-free compression framework that preserves critical visual information during aggressive token reduction. It first prunes low-information tokens using an cntommation-peservalleee similarity to minimize accuracy loss.
guided selection and then merges the remaining tokens based on
TwigVLM [39] accelerates large vision- laiiuae models by appending lightweight twig block to an early layer of a frozen base VLM. It utilizes twig-guided token pruning coupled with self- speculative decoding boost generation speed while retaining high accuracy even under aggressive token reduction.
C.4 Implement Details
From Theorems T and 2 the balance between the visual preservation and prompt alignment, i.e., the optimal budget Kp applied for covering prompt tokens P, is jointly determined by the total budget K and the visual-prompt coupling 17. To ensure fair comparison, we evaluate two settings.
(i) Without 7) prior. This setting deliberately avoids any benchmark -spciiifc prior (w/o n) prior). MoB adjusts Kp solely as function of K to balance the two objectives. Based on an ablation over (K, Kp). we set Kp E (64, 48, 32] and & E (4, 6, 8), corresponding token-reduction rates of (88.9%, 77.8%, 66.7%
(ii) With 7) prior. To verify the K- relationship formulated in Theorems and we introduce coarse benchmark prior on Specifically, we do not meticulously search the optimal hyperparameters for MoB. i.e., Kp and the covering fold k. per benchmark. Instead, we partition benchmarks by their empirical 1) distribution into two groups (strong v.s. weak coupling) and employ the same configuration per group. From a joint ablation over (K, 17, Kp), for image understanding we set
strong coupling:
Kp
{
3K 8
K
11K
k
3Kg
1
24
weak coupling: Kp E {
K
2
7K
16
5K 1
12
k
=
Kp 8
which again yield token-reduction rates of (88.9%, 77.8%, 66.7% 66.7%), }.
As for video understanding, we set Kp 3K A 3Kg for MSVD, MSRV, and ActNet, and
K, K 3: k Ke for TGIF. Unless otherwise stated, the pruning layer index is fixed to = 2 for both image and video tasks. The same configurations are applied across all MLLMs, and all baselines are run with their default settings.
To ensure reproducibility, cross- validaad our experimental results using the publicly available MLLMs evaluation tool Immss-val (v0.3.0) [56] [] with the random seed set to 1234. All experiments were conducted on 4x Nvidia A800-80GB GPUs paired with 2x Intel Xeon Gold 6348 CPUs. The implementation was carried out in Python 3.10 using Py PyTccch 2.1.2 and CUDA 11.8.
26

---

D Additional Experimental Results
D.1 Quantitative Comparison
Strong Coupling
Weak Coupling
Method
Ohjectives
MMB MMBON SQA VizWiz GQA
MME POPE VQA" VQA
OCR
LLAVA-L5-78
w/o Pruning, N 576; Token Reduction Rate
0.0%
Vanilla 28
64.7
58.1
69.3
50.0
61.9
1862
85.9
58.2
78.3
297
100%
LLAVA-1.5-7B
Praning budget =
192; Token Reduction Rate
66.7%
ToMe (5C [6
VP
60.5
65.2
54.3
1563
12.4
52.1
6%.0
88.5%
FastV (ECCV"24) 8
61.2
57.0
67.3
50.8
52.7
1612
64.8
$2.5
67.1
291
91.2%
HIRED (5AAT225)
62.8
54.7
68.4
50.1
58.7
1737
82.8
47.4
74,9
190
91.5%
LLaVA-PruMerge (24.0)
X
VP
59.6
52.9
67.9
50.1
54.3
1632
71.3
54.3
70.6
253
90.8%
SpurseVLMM (ICML [88]
62.5
53.7
69.1
50.5
57.6
1721
83.6
56.1
75.6
292
96.3%
PyramidDrop (CVPRR25)
48
PA
63.3
56.8
68.8
51.1
57.1
1797
82.3
56.1
75.1
290
96.7%
FiCoCo-V (EMNLPP22)
VP
62.1
55.3
67.8
51.0
58.5
1732
82.5
$5.7
74.4
9
MustDrop 12) X
62.3
55.8
51.4
58.2
1787
82.6
56.5
76.0
289
97.2%
VisionZip (24.12)
[1
VP
63.0
68.9
59.3
1783
85,3
57.3
76.8
97.7%
DART (EMNLP"25)
46
VP
63.6
$7.0
69.8
51.2
60.0
1856
82.8
$7.4
76.7
296
98.8%
TokenCarve (25.03)
(
63.0
69.1
50.9
1830
84.9
58.4
78.0
99. 35g
TLigVLM (ICCV'25)
PA
64.0
68.8
61.2
1848
87.2
58.0
78.1
99, 59.
MoB (w/o n-prior)
PA
VP
63.8
57.5
70.0
52.4
61.2
1858
84.5
58.2
77.9
304
100.2%
+ -77-piorr
57.8
52.5
58.5
307
LLaVA-1.5-7B
Pruning budger K
128: Token
Reduction Rate
77.8%
ToMe (ICLR'23) 23)
VP
53.3
59.6
52.4
1343
62.8
49.1
63.0
80.4%
FasVV (2)
VP
56.1
56.4
60.2
51.3
49.6
1490
59.6
50.6
61.88
2855
86.4%
HiRED (5)AI
61.5
53.6
68.1
51.3
57.2
1710
79.8
46.
73.4
191
90. 2%
LLaVA-PruMegg (24.05)
VP
58.1
51.7
67.1
50.3
53.3
1554
67.2
54.3
68.88
248
88.8%
SparseVLM (5)
60.0
51.1
67.1
51.4
56.0
1696
80.5
54.9
73.8
280
93.8%
PyramsdDrop (CVPR'25)
61.6
1761
55.1
287
FiCoCo (EM)LIP'24)
VP
61.1
54.3
68.3
49.4
57.6
1711
82.2
35.6
73.1
94, 9%
MustDrop (24_
PA VP
61.1
55.2
68.5
52.1
56.9
1745
78.7
56.3
74.6
281
25.6%
VisionZip (24.12)
62.0
56.8
2%
DART (EM)LP"25)
VP
63.2
57.5
69.1
51.7
58.7
1840
80.1
56.4
75.9
296
98.0%
TokenCarve (25.03)
VP
62.7
68.9
51.0
1829
84.5
58.1
77.3
99.0%
Twig (ICCV'25)
63.5
57.8
MoB (w/o 7-prior)
PA VP
63.2
57.3
69.3
52.8
60.7
1842
81.7
57.5
77.2
299
99, 2%
>77-riorr
63.5
57,5
69.6
52.7
60.9
1845
82.1
57.8
77.5
299
99.4%
LLaVA-1.5:7B
Pruming hudget K
64;
Token
Reduction Rate
88.9%
(ICLRR'25
43.7
48.6
45.3
FastV (BCCV"24)
VP
48.0
$2.7
51.1
50.8
46.1
1256
48.0
47.8
55.0
245
77.
HiRED (AAA1'25)
60.2
51.4
68.2
50.2
54.6
1599
73.6
44.2
69.7
191
87.0%
PraWA.PruMerge (24.05)
55.3
1549
54.0
250
SparscVLM (ICML'25)
PA
56.2
46.
62.2
50.1
52.7
1505
75.1
51.8
68.2
180
84, 6%
PyramidDrop (CVPR'25)
58.8
50.5
68.66
50.7
41.9
1561
55.9
45.9
69.2
250
78. 1%
(EM)LPP'2))
60.3
53.0
68.1
49.8
1591
$3.6
MustDrop (24.11)
PA VP
60.0
53.1
63.4
51.2
53.1
1612
6%.0
$4.2
69.3
267
90.1
VisionZip (24.
60.1
55.1
1690
77.0
55.5
72.4
92.8%
DART (EMNLPP'55
60.6
53.2
69.8
51.6
55.9
1765
73.9
54.4
72.4
270
93.7%
TokenCarve (25.03)
PA VP
62.0
69.7
51.4
1754
79.9
$7.0
74.8
97. 97.0%
VLM (CC)
60.4
70.0
58.8
1760
62.7
55.8
75.6
96. 1
MoB (w/o p-proo))
PA VP
61.7
54,2
69.7
52.0
59.0
1806
77.2
$7.0
75.5
277
96.3%
+7-prior
62.1
54.5
69.8
52.1
59.0
1806
77.2
57.0
75.5
277
96.4%
IJaVA-Next-77
w/o Pruning, N
2880: Token Reduction Rote 0.0%
Vanilla 20
67.4
60.6
70.1
$7.6
64.2
1851
86.5
64.9
81.8
517
100%
LLVVA-Next-BB
Prining budget K
320: Token Reduction Rate
88.9%
FastV (ECCV'24)
61.6
$1.9
53.1
55.9
1661
55.7
374
86.4%
HiRED (AAAM125)
VP
64.2
55.9
66.7
54.2
59.3
1690
83.3
58.8
75.7
404
91.8%
P.aVA-PruMerrge (24.05
VV
61.3
55.3
66.4
54.0
53.6
1534
60.8
50.6
69.7
146
79.9%
SparsVLLM (ICML225))
60.6
$4.5
58.4
270
PyrmiddDrop (CVPR'2S)
PA
63.4
56.2
67.5
54.1
56.4
1663
77.6
54.4
73.5
250
86.8%
MustDrop (24.11)
62.8
55.1
68.0
$4.0
$7.3
1641
82.1
59.9
73.7
382
90.4%
VisionZip (24.12)
63.1
58.9
FasterVI.M (24.12) 621
VP
61.6
53.5
66.5
52.6
56.9
1701
83.6
56.5
74.0
401
89.8%
GlobalCom (25.01)
VP
61.8
53.4
54.6
57.1
57.2
76.7
375
DART (EMNLP'25
VP
65.3
58.2
68.4
56.1
61.7
1710
84.1
58.7
79.1
406
93.9%
TLigVLM (5C
65.0
68.7
62.2
1758
57.4
79.7
95.4%
(with vy-riorr)
65.8
58.9
68.7
62.6
1760
60.2
80.1
418
Table 5: Full results on image understanding with the LLaVA-7B Series. For MoB, we set Kp E (64, 48, 32} and & E (4, 6, 8), corresponding to token-reduction rates of (88.9%, 77 77.8%, 66.7%). For MoB with the 7) prior, we use Kp { : {} with k 3kg for strong-coupling benchmarks with & Kp for weak-coupling benchmarks, corresponding to the same token-reduction rates; the pruning layer is fixed at e = 2. B and O denote the best and the second.
40
and Kp : 16: : T2
7K 5K
27

---

D.2
Visualization
Visual Data
Sp
Sy
S
Prompts: "Is there snowboord in the image?"
Prompts: "Is there a book in the image?"
Prompts: there bird the image?" "
Prompts: "Is there bicycle image?
Prompts:
"Whot brand of watch this?" "
Figure 7: Visualization the selected prompt and visual centers under weak coupling.
28

---

Visual Data
Sp
Sy
S
Prompts: "Which can be the associated text with this image posted on twitter
Prompts: "Where is it? A. Shanghai. New York. Washington. D. Pari"
Prompts: "Which is the main topic of the image? A. two donuts.
SLOW
SLOW
SLOW
SLOW
Prompts: "Which is right? C. The man is holding the sign.
Prompts:
"What type of environment is depicted in the picture? B. Children's playground.
Figure 8: Visualization of the selected prompt and visual centers under strong coupling.
29

---

MoB formulates visual token pruning bi-objective covering problem over (V. P), which is expected gather query-relevant, fine-grained evidence with Sp while preserving global scene context with Sy- The visualizations (Figures a 8 qualitatively validate this design: tokens in Sp concentrate regions aligned with the text query and key visual evidence, whereas elements of Sy spread more uniformly across the image to maintain the overall context. Together, this complementary allocation enables MoB to retain the most informative visual content for each imagc-query pair, accounting for its strong empirical performance.
D.3
Additional Ablation & Discussion
GQA
Kp)
0
2
4
8
12
16
24
32
48
64
96
(64, 32)
58.3
58.8
59.0
58.7
58.2
57.4
(128, 64)
60.2
60.5
60.7
60.6
60.0
59.5
(192, 96)
60.6
61.1
61.2
60.9
60.7
60.5
TextVQA
kp)
0
2
4
8
12
16
24
32
48
64
96
(64, 32)
56.5
56.9
57.0
56.8
56.5
56.2
(128, 64)
57.1
57.5
57.7
57.7
57.2
56.8
(122, 96)
57.8
58.2
58.2
58.1
57.7
57.5
Table 6: Detailed ablation on the covering fold k for GQA and TextVQA.
To assess MoB's sensitivity to the covering-foll parameter k particularry under weak coupling with long prompts conduct a detailed ablation on using GQA and TextVQA.
As Table demonssddmssssrrtess, MoB is not overly sensitive to the choice of k, particularly within clear optimal range. For instance, in both two benchmarks, performance only varies by approximately 0.3% for l values between [2, under (K 64, Kp 32) setting.
There is also principled, theoretical reason for this robustness, which stems from the relationship between the covering fold k, the budget Kp. and the length L of prompt tokens P. From covering theory, every prompt token p P is covered by at least one visual token K, 2 kL, thereby ensuring the performance guarantee of MoB. Therefore, as selected is satisfies k S Kp/L. the performance will remain stable.
under the condition
Heuristic for estimating k. In practice, robust range for l can be inferred from the prompt length L. Given the analysis above, we expect an adaptive, per-sample search for a fine-grained k to yield only limited gains, so we rely on this length- based heuristic instead.
30

---

D.4 Real-life Application
MMB_CN
SQA
0.50
1.00
0.75
2.45
Donsity
Dansity
Damuity
0.59
0.25
0.35
0.00
147
143
260
350
2.50
147.0
248.5
150.5
251.5
0.00
147
143
360
355
353
Hausdorff Distance
Hauadorff Distance
Hausdorff Distance
VizWiz
GQA
MME
1.00
3.2
1.2
0.75
Deedit
0.50
0.6
0.6
0.3
0.8
0.03
1.7
163
1:9
352
353
0.0
1.40
149
150
151
152
0.0
1096
150.6
251.2
152.0
Hausdorff Distance
Haurdorff Distance
Hausdortf Distance
POPE
TextVQA
OCR
0.6
0.6
3.2
0.6
0.4
0.5
0.3
6.2
443
150.0
150.0
152.2
0.0
142.5
152.4
111.2
1520
0.0
140
150
251
352
Hausdorff Distancce
Hausdorff Distance
Hausdorff Distance
Figure 9: Observations of prompt-visual coupling v across 9 popular benchmarks.
Open-domain recipe. MoB is task-agnostic, which does not require pre-defined task labels and can operate online by classifying each sample's coupling pattern. For a given target model, we adopt a wwo-stage strategy:
Offline calibration. Analyze the empirical v) distributions on a set of representative benchmarks (as shown in Figure(9) and set robust threshold that separates weak vs. strong coupling.
Online classification and inference. For each incoming query, compute its Hausdorff distance using Algorithm 2 with tractable bilinear complexity O(N Classify the sample by comparing this value to t. then apply the corresponding budget configuration (e.g., Kp. k) and run MoB + forward inference. In practice, this online cost is negligible relative to the pruned forward pass.
Computational Overhead. We provide detailed cost breakdown for online computation of the Hausdorff distance using Algorithm 2 with complexity OdN on L.V5A- and LLa VA- Next-7B, where N, L, and denote the numbers of visual tokens, prompt tokens, and the feature dimension, respectively. As shown in Table 7 the measured cost (TFLOPs) of exact Hausdorff computation is orders of magnitude smaller than that of MoB itself and the model's forward pass, yielding negligible overhead.
31

---

LL.VA-I.5
576, L
10, d 4096)
Model
Vanilla
K = 64
K = 128
K = 192
Forward Compute dæ
8.2 2.3e -
1.0
1.9
2.8
2.3e -
2.3e 5
2.3e
MoB
1.7c
3.30 4
4.8c
LLaVA-Next (N
2880. L
10, d 4096)
Model
Vanilla
K = 320
K = 640
K = 960
Forward
40.5
4.6
9.1
13.6
Compute dи
1.2e -
1.2e -
1.2e 4
1.2e
MoB
3.9e -
7.6e -
1.le
Table 7: Computation cost in LLaVA-7B series (TFLOPs)
Concretely, computing dæ (e.g., 1.2 10 TFLOPs on LLaVA-Next) insignificant relative to the pruned forward pass (e.g., ~ 4.6 TFLOPs at K = 320) and, more importantly, to the savings from pruning (~ 35.9 TFLOPs). Thus, exact online estimation is not practical bottleneck; its cost is dwarfed by the efficiency gains of our method. Further acceleration is possible with standard techniques (e.g., heuristic support sampling or low-dimensional random projections), although unnecessary in our settings.
Heuristic Sampling: It computes the distance on smaller support sets of the tokens (V' C V, P C P), which can be constructed via random sampling (46) more advanced heuristics such as Key-Norm selection [3]. This reduces complexity to (N' L'd), where |V* = N', |P' L'
Random Projections: For a more theoretically grounded approach, the
(JL) lemma [19] allows project embeddings to much lower dimension (d' while
preserving geometric structure, reducing complexity to O(N
Potential Extensions. A natural extension is maintain online estimate of the coupling statistic during inference-4.. a running summary of an approximate r computed from shallow-layer tokens. As more samples are processed, we expect the empirical distribution of become bimodal (consistent with the benchmark patterns in Figure 9) enabling data-driven threshold to be derived on the fly that separates weak vs. strong coupling regimes. Using this live threshold, MoB could adapt K, (and k) per sample or per mini-batch by selecting from a small budget pool or by scheduling Ke function of f), with conservative warm-up and safeguards for distribution shift.
32

---

E
Omitted Technical Details
E.1 Proof of Lemma
Restatement of Lemma |
(An Error Bound for Visual Token Pruning), Under Assumption 1 given any token set with its pruned counterpart & VUP, X₂ SUP C Rd, the pruning error bound is given by:
(((x)=((Z)) & Ce max { min (du(S, V), du (V,P), min (dH(S,V), dn(S, (n(S,P))} PI}}
Remark. Here du (u,P)) P) and (S V) describe the prompt alignment and visual preservation, while du (V, P) is an inherent term that describes the prompt-visuce coupling of input data.
Proof. The intermediate input for any layer and its pruned counterpart are given by
= V UP and X₂ = SUP.
By Equation TT. the Hausdorff distance is symmetric, i.e.,
(H V) du (V.S).
(E1-1)
and induced by Euclidean distance.
Step 1. Bound the one-sided distances.
We analyze the distances by considering the membership of the points in the subsets.
Direction (X - X.) For any x: E X::
Case (i): If E P, then since P C Xr
inf || yll = 0.
yEN,
Case (ii): If x E V. then the candidate points in X₂ SUP can be chosen either from S or P. Thus,
inf
|| vll S ming inf sES
||x - s/1. inf ||
rll
pEp
Taking the supremum over x E yields
sup inf ||. vll < mink sup inf || a||, sup inf || (1a-pp) 211 PEP
vER,
xEv sES
sup inf || pll s max sup inf || pll, sup inf 1p =11} dz(VPP), pep sEV
NEV pEp
z pep
By Equation [ derive the distance in direction 1:
sup inf || vll s minfda dv(2,))} #EX vEA,
(E1-2)
Direction 2 (A₃ - X)) For any y E Xx:
Case (i): If P. then as P C X.
inf | - x|| TEX
Case (ii): If y E S, the candidate points in X = V.P can be chosen from either V or P; hence
inf |y - x|l $ minf inf lly - e||, inf lly - pll Lvey
#EX
peP
Taking the supremum over S yields
sup inf [yy x|| s mink sup inf lly all, yES wey
sup inf Ilv pl
PES TEX
wEs PEP
33

---

sup inf 1l pll s yEB pEp
max sup inf llv pll, sup inf llp vll = (S, pEp ves
yES PEP
By Equation 1 we derive the distance in direction 2:
sup inf lv xll I min( mii ds(s, wE zEx
(E1-3)
Step 2. Combine the bounds.
By Equation
[T, combining the bounds in (E1-2 and (E1-3) obtain
du (X, max { min (V, du (V,P), min (d ds(s
Based on (EI-1) we have
du (X X.) < max min (du(S, V), (V,P))), min (S, du (s,
Loading the Assumption [ we have the output discrepancy bounded by
[[F(X) - F(X4)l S C2 dn(xX,
C max min (d V), (V,P)))). min (d=(S, V). d)(S)
This completes the proof.
34

---

E.2 Proof of Lemma 2
Restatement Lemma (A Relaxed Error Bound under Practical Budgets). Under Assumptions 7 and 2 let X = VVPP, P. X. S S.P S IR with [S] = K & N. Partition the retained token set S into two disjoint subsets: S Sp L Sr devoted to prompt alignment du P) and visual preservation (S,,v, V), respectively. Then, the pruning error bound reduces 10
[[((2) - F(X))l I Ce max {d P), dS( V)} + Ce
Proof. By Lemma 1 we obtain
[FF(X) F(Xx)l & Ce max min (d=(S, V), (V(,P)), min (du(S, dH(S.P)))
Since min max D}, we have
[IF(X) F(Xg)l Ce max { dd( P), du du (V,))
(E2-1)
For any p E P, we have
inf
[p - a|| = min
inf || 8||, inf ||p +11 S inf |l 8||. SESp
#SS
sES.
SESp
Taking the supremum over E P yields
sup inf pEP sES
|pp - a|1 S sup inf |lp s||- pep SESp
Similarly, since S, C S,
sup inf ||s - pll sup inf ||s - pll. sES, PEP
sES pEp
Thus, by Equation
dn((SP)) S max (nn (Spp)) P), dn P)}.
Using Assumption 2 [d (V,P) < 1)) and the triangle incquality for Hausdorff distance, we have
dH((S,, P) - (S., V) + (v, P) < du (Sv, V) + 7, (S V) < ds p)) - dn (Sp P) + 1
Hence,
(s(S,P) < max (S P). (S, V) + m
(E2-2)
Similarly, one can show that
dH < max (S,, V), dh P) + n)
(E2-3)
Loading the maximum of (E2-2) E2-3 and du (V, P) into EE2- we obtain
[[F(X)) - F(X))] < Ct max (d((S (S.S) du du (V. P)
(S, P), du (S., V) n dn(S,, V), dn(Sp, P) + n, mn
< C2 max
Since (Sp. 2 0, (S., 1) 2 0, 2 0, we have
max (S (Sp, P), du (Sp: P) n7, nn dz(Sp: P) + 1
max (S. V). (S, V) + 9, n) dn V) + 17.
Hence
[F(X)) - F(2))) $ Ct max { (Sp P), du (s. + Ce n.
This completes the proof.
35

---

E.3 Proof of Lemma! 3
Restatement of Lemma (den- lower bound on covering numbers), Given P, P.V C R with
an effective dimension deff- Suppose their 5-dilations V. := -evv Be,, 6), Ps = U per ((p
(å & 11) satisfy dest dimensional covering regularity; thus, there exist constants b> a 0, 0 and > 6 such that
a (p((
d'c, SBC
cpp. Gv E (ő, col.
Remark Previous work suggests that both visual and language embeddings concentrate on low- dimensional manifold, so the effective covering dimension satisfies the typical relation dett & d.
Proof. We prove the two-sided bound for P; the argument for V is identical.
Notation.
(X) minimal number of closed balls of radius r covering X.
Xx = Uzex Bix, 8), with Bax, 6) {y lll x1 s 6}.
Step Transfer trick for small 6.
Fix 6 E (ó, co) and define = minfe 6, co)
If E S 60 d (so é' = o), then any e-cover { Jm of P satisfies for each Ps:
3x E P |yy x| م
3i: || z1|l Se = I1y zi|l S e c =
Hence
Pg C V 6(z,,')
= N(Pa,e) s N(P,c).
(E3-1)
i=1
Note: For > 8, the above transfer argument is not applied.
Step 2. Lower bound (),,
Split into two cases:
Case I: S co 8. Since Pa satisfies deft demedimmnioonal covering regularity; loading the lower- bound for Pa radius è' + 6, there exists a constant as such that
-IPo,e'=N(Pss.) 6) as (E + 8) dett
Based (E3-1)
we obtain
as(c + 55(c+5)-dat 5e- < N((P,.) & N(P,c)
Since 8 < it follows that + 5 2e; thus, we have
N(P,c) 2 as 2 dair c -dit datt
(E3-2)
Case II: e > 6n - 8. Define ă (eo 8ydatr, such that
(co 5) -ddit -a-1
Since 6n 5, we have
delt < (60 5) doit
Hence
dett
a
-c-datt dott < 1.
Since any nonempty set P has covering number at least one, the following holds
à den
S (((,e)).
(E3-3)
36

---

Therefore, set a : asn datt /} > O, combining (E3-2) and (E3-3) yields
c) ac acn
AC col
(E3-4)
Similarly, V holds (V),,) d'e dell, Ve co)
Step Upper bound on (P,,))).
Since P3 satisfies dest-dimensional covering regularity, there exists a constant by 2 2 0 such that
N(Pa, I by c d
Since P C Ps. we have (P,,e) - (Pss <); thus, the following holds
()),e) s (((s,, < by dder.
Based on the monotonicity of covering numbers, for every radius 2 8, we have
N(P,e) (P(,
Therefore, == mbx (V,, 8)), for all (6, col we have
((P,c) v be -dir
(E3-5)
Likewise for V, the following holds WV(,,) < cdst VCE (6.
Step 4. Combine the bounds.
Based (E3-4 (5-4) and (E3-5) for all col the following holds
a dets SN(P, s be ben det, -dat S <N(,e) S 5 -dit.
This completes the proof.
37

---

E.4 Proof of Theorem 1
Restatement of Theorem (Trade-off between Prompt Aligmment and Visual Preservation). Un- der Assumption 2 and the covering reguulrriy hypothesis of Lemma with constants a, a', deft > 0, there exist a radius-scale scaling factor
such that n/2 2 >> ó and K <N +VV, for every pruning results S = (Sp L1S,) C V with budget K satisfying
max 1 KD/Kaa/da, D2 S dd((p. P) dd(S.,
where D1 = a4a 00, D2 = 1/22
1/ O.
Remark (Optimal Attainment Level). The term D1 K-2/dett is completely determined by the pruning hudget, while D2 12 quantifies the effect of prompt-visual coupling. Hence, the optimal attainment level per objective is given by E" max(1/z, z, vD, K (11//ett) }. Any attempt to reduce one objective below e* forces the other above E", thereby increasing the overall pruning error.
Remark (Effect of Budget and Coupling Strength). As K decreases, a correspondingly shrinks ( (D2 growing as power function), ultimately making D22 12² dominate the bound; while as K increases, both of the terms reduce, thereby diminishing the trade-off and tightening the overall error bound.
Proof. We begin the proof by noting
Cp = dir (Sp: P), Ey = dn V), = 1sph Ky (Syb,
Kp + Kv = K.
Step 1. Quantify the impact of budget K.
By Lemma
3 for all Ep. Ev
co]. we have
acp" SN(P,, cp) S Kpp d' d' s SN(V, S Kv.
(E4-1)
By AM-GM inequality, we have K,Kv S {(K)² thus, loading (E4-1) we have
(aa') (x det < (
EpEv 2 (4ad) K 2/dea
Define D1 = a4 > 0, the K-bound is established by
Cp cv 2 K-2/orr
(E4-2)
Step Quantify the impact of prompt-visual coupling 1). Based on the budget condition, the scaling factor 2 holds
K <P(PP 2) + +V,
(F4-3)
For contradiction, suppose covering radii is simultaneously small, such that 6p n/2 and 6y < 1//2. Then, the monotonicity of covering numbers gives
N(P, (p) 2 N((, N(V.c)) ((),))2N(V,)
Hence
K Z + + 2 N((P.)) + +N((), 2), Therefore at least one of cp. 6v is n/2. Consequently
contradicting (E4-3)
cv
Define D2:= : 1/2 0, the boudd given by
cv 2 D2 122
(E4-4)
Step 3. Combine the impacts.
By (B4-2) and (E444) have
cpc, 2 D K 2/dait and Epr, 2 D2m = cp 2 max K 2/dais Dzm
This completes the proof.
38

---

E,5 Proof of Theorem 2
Restatement of Theorem (Performance Guarantee). Under Assumption 7 and the covering- regularity of Lemma 3 with constants a', denf > 0 and b>>a, R' > for any budget split (Kp. K
Kp), covering fold k, and token set X = VUP C Rd with |V] N, |P|= L, and dh (V, P) S
the following hold:
(a) Performance bound: The Performance degradation caused by MoB is upper bounded by
[[((X) - -(MoB((M))) 2 Ct max a maxfa(n ki, -1/dar, 3 (kp) Kk))- } + Cen:
where an,,, L) 17 (bb L/a) Udee, B 2(b))//dmr
(b) Multilinear complexity: The complexity of MoB is given by TMon O(N ( K)d).
Remark (Coupling Trade-off). Under weak coupling (large a(n, k. L)), minimizing the bound requires a larger Kp. Conversely, under strong coupling (small (x1], k, L)). the alignment term decays rapidly, favoring visual preservation (increasing K Kp). Specially, under perfect coupling (n 0), the bound simplifies to |Ayll S C2 3K - K2)) Kx))-ffaariic.. i.c, MoB reduces to pure visual preservation.
Remark (Budget Scaling). As the total budget K increases, the preservation term 3 (K K2) -1jdon decays, requiring a corresponding increase in Kp (and thus reduction in the alignment term) to rebalance the trade-off and further lower the overall error bound.
Remark (Scalability). MoB exhibits multilinear scalability with respect to visual tokens N, prompt tokens L, and retained tokens K (especially K, L & N), making it readily adaptable to more challenging scenarios, such as advanced MLLMs with higher-resolution inputs or multi-frame video.
Notation.
The intermediate input X' is formulated as
= V UP C Rd where
| N, P and N > L.
Particularly, V, P are compact sets with des effective dimensions.
We define the pruned intermediate input as
Mx = Xs. where 2== = S U P where [8] = K.
The budget configuration is given by (Kpr K.), where Kp K K.
Proof. We separately proof the Performance Guarantee & Complexity in Part A & Part B
Part A: Performance Guarantee
Part A-1: Performance Guarantee of prompt alignment
Step A-1.1: Bound of the radius derived by k-fold NN-covering
Given any union set before K K--truncation
S = U
arg argtop-k(cos(sppp p). where
|S = Ky and Kp S K. < kL,
PEP
*pEV
we define
6 = d P).
By previous work (15, NN-covering achieves a 1-approximation for the k-center problem with sufficient budget; i.e. specifically for any E P we have
inf ||p - s/ = inf llp vll. wev
5 ES
Thus,
sup inf ||p s
sup inf ||p w/|- pep nev
pEp E,E,
39

---

Based on Assumption
2 since Sp S V, the upper bound of the radius o is given by
c du (Sp, P) max sup inf lpp spll, sup inf |pp %01l) s,Es, PEP
pcP %
I max inf lpp 1 U||, sup inf llp =|]} vE PEP
(E5-1)
PEPvEV
du (V. P) m.
Step A-1.2: Impact of K--truncation on the radius
Based on Lemma
we have
or aor SN((Pr) s br dort
In particular:
1/d.
b (ep) -datt
2 Kp
cp
Kp
and also
1/dat
a (
I
Kgp
&
Rp
Combining the upper and lower bound for Ep and e,, respectively in terms of b, Kp. K
obtain
1/darr
1/datt
1/datr
bb.
1/datt
cp
Kp
aapp
akp
cp-
That is, truncating from K2 to Kp centers increases the radius by most the factor
cp s (bKg/akk 1/der
/
E
Since kL Ky. loading into above, we have
cp S (bkL/aK,) 1/datr
6p.
By loading (ES- into the above, the performance guarantee of prompt alignment is given by
Ep = dH P) s k, L) (K,) 1/der where a(n, k. L) = (bkL/a) 1/
(E5-2)
Part A-2: Performance Guarantee of Visual Preservation
By previous work 36 FPS achieves 2-approximation for the k-center problem:
cv 2<2c((K)))
(E5-3)
where e² (K,) the optimal radius with K, centers. Based on Lemma 3 we have
N(Vrr) r) S rar
thereby, the upper bound of optimal radius is given by
c'(K,) S (6//K,)!/44
By loading the above into (E5-3) the performance guarantee of visual preservation is given by cy d (S., V) < 3 (K.)
where 8= 8:=2b//dda 2/drr
(E5-4)
Part A-3: Performance Guarantee of MoB
By substituting (E5-2) and (E5-4) into Lemma
the performance guarantee of the MoB is given by:
[F(2)) - F(MMB((X)) < Ce max on k, L) (K₂) ( 11deet, B (Kw) 1/datt } + Ce nh
where a11 k, = (b L/a) 1/dat This completes the proof of Part A.
B = 2//
40

---

Part B: Complexity
Since k < Kp S Nx N, we restrict our complexity analysis to the leading-order terms.
Part B-1: Normalization
MoB do L2 normalization for each token E X C IR; thus, the complexity is given by Torm (( + d)..
(E5-5)
Part B-2: Selection of Prompt Center
Firstly, MoB calculates the cosine similarity with each E P and v E V via a matrix multiplication:
Msim PV E LLNN where V RRR and P E Baxdd
which leads a complexity of Tste 1-1 = O(N Ld). Subsequent, MoB do a top-k retrieval in the first dimension of Msim the select most closed centers for each prompt token p P. which can be reduced to a partial sorting, thereby leading to a complexity of Tate 1-2
ON L log k). Finally, MoB merge the selected result of each p E P, and truncated the top-Kp ones with largest similarity, OLL k log Kp). Consequently, the total complexity T ppseeeec of prompt
leading to Tstep 1-3 center selection given by:
T pp=eelect = step + step + Tstep 1-3,
ONN l.) + O(N T. log k) O log Kp),
(E5-6)
ON
Part B-3: Selection of Visual Center
Initially, MoB calculates the minimum distance (used in FPS) with each visual token VV,S, V and the selected prompt centers via a matrix multiplication together with an argmin operator:
dFps = arg min V Sp E N-K, where V E (NN)Kx and Sp E Kd thus, the complexity is given by
T-tep
O(N - Kp) K2 d) + +((N K2) Kp). Kp),
matrix multiplication
arginnn
= -(NN~ - K2) d).).
Subsequently, in K - Kp iterations, MoB add the tokens with largest minimum distance with an argmax operator in drps, and update the dyps with an inner production together with an N K dimensional element-wiss comparison; thus the complexity is given by
T Tstep 2-2 O(N (K)(K Kp)) O((K K2)
O((K K2)
argmax
inner productioin
ele ele-wise comparision
O(K KKK) Kp) N d).
Consequently, the total complexity T
Ty-select of visual center selection given by:
Ty-select = Tstop + Tstep 2-2,
=O((N- Kp) Kp d) + O((K -Kp) Nd).
(E5-7)
= ONN K dd).
Part B-4: Totally complexity
By (E5-5)
(E5-6) and (E5-7)
the totally complexity of MoB is given by
TMoR
Thorm Tp-select + Ty-solect,
O((N + L) d) + O(N Ld) O(N K d),
O(N Ld) + OON d),
O(N (L K)d).
This completes the proof of Part B.
Combining the Part & B, we complete the proof.
41


Reviews:
[{"review_id": "2QITA463bW", "reviewer": "Reviewer_k8zV", "summary": "This paper revisits the problem of visual token pruning in Multimodal Large Language Models, which is essential for reducing computational overhead from high-resolution visual input. While prior methods often integrate the two main pruning objectives, visual preservation (VP) and prompt alignment (PA), using static or heuristic strategies, this work introduces a principled, theoretically grounded framework for their integration. The authors propose a closed-form error bound for pruning based on the Hausdorff distance (Lemma 1), which ties pruning performance to the balance between VP and PA. Also, a new characterization of the trade-off between these two objectives under varying prompt-visual coupling conditions (Theorem 1) is demonstrated. Finally, a novel algorithm, Multi-objective Balanced Covering (MoB) is proposed which translates the trade-off into a budget allocation problem between prompt and visual token retention and is implemented with efficient greedy heuristics.", "questions": "1 - Could the authors provide more intuitive explanations or visual illustrations—particularly for the key theoretical insights in Section 3.2 (e.g., the trade-off in Theorem 1)? This would help clarify the core contributions for a broader NeurIPS audience.\n\n\n2 - Has a notation table or summary been considered to address the overloaded variables like \"X\", \"S\", \"K\", and “η”? This might significantly improve the clarity of mathematical sections.\n\n\n3 - What empirical evidence supports Assumption 1 (Lipschitz continuity of transformers with respect to Hausdorff distance)? Would it be possible to validate this assumption through ablation or probing experiments?\n\n4 - The experiments omit widely used benchmarks such as COCO and full VQAv2. Is there a reason for this exclusion, and could results on these datasets be included to better demonstrate generalizability?\n\n5 - How does pruning at deeper or multiple layers beyond ℓ = 2 impact performance and trade-off dynamics? Can additional ablations be provided to justify the pruning depth choice?\n\n6 - Given the dependency of MoB on the estimation of η, what method is used to compute or approximate η in practice? Is this estimation stable and efficient?\n\n7 - To what extent is MoB sensitive to the choice of covering fold parameter k, especially under weak coupling conditions with long prompts? Would an adaptive mechanism be feasible here?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n\n1 -  The paper derives an elegant, novel error bound for token pruning (Lemma 1 and 2), tying model output error to geometric distances in token embedding space. The application of covering theory and trade-off quantification (Theorem 1) is a significant conceptual leap over prior work.\n\n2 - The paper convincingly identifies that prior multi-objective token pruning methods do not adapt to variable prompt-visual coupling across tasks, explaining the failure of static integration schemes (e.g., MustDrop).\n\n3 - Extensive evaluations (Tables 1–3) across multiple MLLMs, datasets, and pruning budgets show consistent improvements, including outperforming strong baselines like FastV and MustDrop. MoB retains >95% of task performance with up to 88.9% token reduction.\n\n\n**Weaknesses**:\n\n1 - Despite having a solid technical background, the paper is extremely challenging to parse. The derivations, especially in Section 3.2 (covering regularity, trade-off derivations), are packed with notation and require significant effort to unpack. For a NeurIPS audience, clearer conceptual exposition with diagrams or more intuitive examples would greatly improve readability. The notation choices (e.g., reuse of \"X\", \"S\", \"K\", “η”) are overloaded and can be confusing. More thoughtful typographic separation or a notation table would help.\n\n\n2 - Assumption 1 (Lipschitz continuity of transformers w.r.t. Hausdorff distance) is plausible but unvalidated. While claimed to hold in practice, no empirical evidence (e.g., via probing or ablation) is shown. Similarly, Assumption 2 (bounded prompt-visual coupling η) might not hold in open-ended real-world prompts (e.g., in generative tasks). The robustness of MoB under such violations is not discussed.\n\n\n3- While the evaluation is extensive, it misses standard datasets like COCO, VQAv2 in full, or long-form visual reasoning tasks like ScienceQA. Evaluating on such benchmarks would further strengthen the case for generality. Some ablations, such as the role of pruning depth (Fig. 6), are shallow. For example, pruning at different layers beyond ℓ = 2 is not deeply explored, though it's likely to affect performance.", "was_revised": true, "final_justification": "After reading the authors rebuttal, my recommendation remains a borderline accept. The following points outline my final justification:\n\n* **Primary Weakness Addressed**: My main criticism was the paper's poor readability, stemming from the dense theoretical derivations and notation. The authors' clarifications and commitment to improving the manuscript's clarity have satisfactorily resolved this issue.\n* **Methodological Concerns Clarified**: The authors provided convincing justifications for their experimental design and theoretical assumptions. This includes their rationale for dataset selection, the choice of pruning at layer `l=2`, and the clarification that the coupling strength (η) is determined via an efficient offline analysis rather than a costly online estimation.\n* **Core Contribution is Significant**: The paper's primary strength is its novel theoretical framework, which derives a closed-form error bound for visual token pruning and quantifies the trade-off between visual preservation and prompt alignment. This represents a good conceptual advance over prior work.\n\nThe authors' rebuttal successfully clarified the paper's contributions and addressed its weaknesses. With the promised improvements to readability, the paper's technical strengths now more clearly justify its acceptance.", "rebuttal": "We sincerely acknowledge your responsible work and constructive comments, we provide our point-to-point feedbacks as follows\n\n**W1 & Q1: Could the authors provide more intuitive explanations or visual illustrations—particularly for the key theoretical insights in Section 3.2 ?**\n\n**A1:** Following your suggestion, we offer a more direct explanation of the theoretical insights in Section 3.2 here and will incorporate a new visual illustration and this discussion in the appendix of the revised manuscript.\n\nOur starting point is that optimal pruning must balance two fundamental objectives: **Visual Preservation (VP)** and **Prompt Alignment (PA)**, which are separately responsible for preserving the **global context** $\\mathcal{S}\\_{\\rm v}$ and the **local evidence** $\\mathcal{S}\\_{\\rm p}$. Just as humans need both context and specific evidence to reason accurately, a VLM without global context (VP) risks hallucination, while one without local evidence (PA) cannot answer the specific question.\n\nSection 3.2 reformulates token selection as a geometric covering problem, thereby establishing a relationship between the preservation quality of global context and the local evidence (measured by Hausdorff distance). Finally, **Theorem 1** *provides a principle for \"how much effort\" is separately required for pruning methods to preserve global context and local evidence with a fixed budget $K$ under a specific task type $\\eta$.*\n\n**W1 & Q2: Has a notation table or summary been considered to address the overloaded variables like \"X\", \"S\", \"K\", and “η”?**\n\n**A2:** Considering your constructive suggestion, we have compiled a notation table and supplied it to Appendix E of the revised manuscript to increase the readability and mathematical clarity of our manuscript. We believe this revision is beneficial to make readers better understand our works.\n\n**W2 & Q3: What empirical evidence supports Assumption 1? Would it be possible to validate this assumption through ablation or probing experiments?**\n\n**A3:** We justify Assumption 1 in three aspects:\n\n**Theoretical Grounding:** Prior work [1, 2] has shown that Transformers are approximately Lipschitz continuous w.r.t. standard matrix norms, e.g., the Frobenius norm, under the finite and compact input conditions we consider (Line 98). A standard inequality relates the Frobenius norm to the Hausdorff distance: $\\frac{1}{\\sqrt N}\\Vert\\mathcal X\\_1-\\mathcal X\\_2\\Vert_{\\_F}\\le d\\_H(\\mathcal X\\_1,\\mathcal X\\_2)\\le\\Vert\\mathcal X\\_1-\\mathcal X\\_2\\Vert_{\\_F}$. This implies that any model Lipschitz continuous w.r.t. the Frobenius norm is also Lipschitz continuous w.r.t. the Hausdorff distance.\n\n**Empirical Evidence:** Vignac et al. [3] have measured the empirical local Lipschitz constant of Transformers on real-world data and found it to be 100-1000x smaller than the theoretical worst-case bound. This indicates that practical models already lie in a well-behaved, near-Lipschitz regime. Besides, Kim et al. [4] have shown that the successful training of deep Transformers relies on controlling this constant. Architectures without such implicit or explicit constraints often fail to converge, suggesting that the well-trained VLLMs most likely adhere to a finite Lipschitz constant.\n\n**Consistency with Prior Work:** We wish to clarify that this assumption is not novel to our work but is a common theoretical premise in the visual token pruning community, having been adopted by prior methods such as DivPrune [5] and DART [6]. Our work builds upon this established foundation to derive deeper insights.\n\n**Regarding probing experiments:** We agree this is a valid consideration. However, computing the global Lipschitz constant is computationally challenging. The most feasible alternative, measuring local constants, has already been thoroughly executed by Vignac et al. [3, 4]. Their work provides strong empirical validation for Assumption 1, and we believe further experiments would likely reinforce their conclusion without fundamentally altering our analysis.\n\n**References**\n\n[1] Dasoulas G, et al. Lipschitz normalization for self-attention layers with application to graph neural networks. ICML, 2021.\n\n[2] Yudin N, et al. Pay Attention to Attention Distribution: A New Local Lipschitz Bound for Transformers. preprint, 2025.\n\n[3] Vignac C,  et al. Understanding the regularity of self-attention with optimal transport. preprint, 2023.\n\n[4] Kim H, et al. The lipschitz constant of self-attention. ICML, 2021.\n\n[5] Alvar S R, et al. Divprune: Diversity-based visual token pruning for large multimodal models. CVPR, 2025.\n\n[6] Wen Z, et al. Stop looking for important tokens in multimodal language models: Duplication matters more. preprint, 2025.\n\n**W3 & Q4: The experiments omit widely used benchmarks such as COCO and full VQAv2. Is there a reason for this exclusion, and could results on these datasets be included to better demonstrate generalizability?**\n\n**A4:** We want to explain that we have evaluated MoB in ScienceQA (SQA) (Appx. C.1), while the exclusion of COCO and the full VQAv2 benchmark is a deliberate choice based on their technical limitations for visual token pruning. We attribute it to the fact that COCO's language-based metrics (e.g., CIDEr) and VQAv2's strong language priors make VLLMs insensitive to the loss of key visual details. **These limitations are reflected in current community practice**: our evaluation protocol mirrors that of 13 recent SOTA pruning methods, none of which report results on COCO or full VQAv2. Hence, for a comprehensive comparison with prior works, we include VQAv2's validation set in our suite.\n\n**W3 & Q5: How does pruning at deeper or multiple layers beyond ℓ = 2 impact performance and trade-off dynamics? Can additional ablations be provided to justify the pruning depth choice?**\n\n**A5:** Thank you for the suggestion. We clarify that a comprehensive ablation on pruning depth ($\\ell=2,5,10,15,20$) is presented in Fig. 6. The results show a clear trade-off: pruning at deeper layers offers margin performance returns with notable latency increments. We chose $\\ell=2$ as the default setting for two primary reasons: First, it **offers a highly competitive performance-efficiency trade-off**. More importantly, it aligns with the community standard for single-stage pruning methods (e.g., FastV, DART), **ensuring a fair comparison** in algorithmic superiority rather than conflating results with higher computational cost. Finally, **multiple-layer pruning may introduce intractable high-order interactions besides the VP and PA**, which are beyond the scope of our theoretical analysis. Hence, our ablations focus exclusively on this single-stage scenario.\n\n**Q6: Given the dependency of MoB on the estimation of η, what method is used to compute or approximate η in practice? Is this estimation stable and efficient?**\n\n**A6:** We clarify that MoB relies on an identification of the coupling pattern (i.e., 'weak' vs. 'strong') rather than a precise estimation of the $\\eta$, which is much more tractable in practice. Specifically, we perform an offline analysis of a benchmark's $\\eta$ distribution using Alg. 2 ($O(NLd)$, Appx. B) to identify its coupling pattern.' During inference, we simply apply the predetermined $K_{\\rm p}$​ based on the coupling pattern, resulting in near-zero computational overhead at runtime. \n\nBesides, as reported in Fig. 1(b) & 9 (Appx. D.2), the $\\eta$ distributions for the two coupling patterns exhibit some overlap. This means our classification will inevitably be imperfect for samples in this overlapping region. In this case, our experimental results (Tables 1-4) show that MoB still consistently outperforms all baselines, highlighting the tolerance and stability of MoB to estimation of $\\eta$.\n\n**Q7: To what extent is MoB sensitive to the choice of covering fold parameter k, especially under weak coupling conditions with long prompts? Would an adaptive mechanism be feasible here?**\n\n**A7:** To address your concern, we have reorganized a portion of our ablation results, focusing specifically on the sensitivity of $k$ under weak coupling conditions with long prompts:\n\nTable 1, Ablation on fold $k$ in GQA\n|$\\langle K,K_{\\rm p}\\rangle$|0|2|4|6|8|12|16|24|32|48|64|96|\n|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|\n|$\\langle64,32\\rangle$|58.3|58.8|**59.0**||58.7||58.2||57.4||||\n|$\\langle128,64\\rangle$|60.2||60.5||**60.7**||60.6||60.0||59.5||\n|$\\langle192,96\\rangle$|60.6|||61.1||**61.2**||60.9||60.7||60.5|\n\nTable 2, Ablation on fold $k$ in TextVQA\n|$\\langle K,K_{\\rm p}\\rangle$|0|2|4|6|8|12|16|24|32|48|64|96|\n|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|\n|$\\langle64,32\\rangle$|56.5|56.9|**57.0**||56.8||56.5||56.2||||\n|$\\langle128,64\\rangle$|57.1||57.5||**57.7**||57.7||57.2||56.8||\n|$\\langle192,96\\rangle$|57.8|||58.2||**58.2**||58.1||57.7||57.5|\n\nAs the tables demonstrate, MoB is not overly sensitive to the choice of $k$, particularly within a clear optimal range. For instance, in both two benchmarks, performance only varies by approximately $0.3\\%$ for $k$ values between $[2, 8]$ under $\\langle K=64, K_{\\rm p}=32\\rangle$ setting.\n\nThere is also a principled, theoretical reason for this robustness, which stems from the relationship between the covering fold $k$, the budget $K_{\\rm p}$, and the length $L$ of prompt tokens $\\mathcal{P}$. From covering theory, every prompt token $p\\in\\mathcal{P}$ is covered by at least one visual token $v\\in\\mathcal{V}$ under the condition $K_{\\rm p}\\geq kL$, thereby ensuring the performance guarantee of MoB. Therefore, as selected $k$ satisfies $k\\leq K_{\\rm p}/L$, the performance will remain stable.\n\nThe above analysis provides a simple yet effective heuristic: one can easily determine a robust range for $k$ based on the prompt length $L$. Based on the above analysis, we believe a more delicate mechanism for adaptively searching for a fine $k$ value for each sample may bring limited potential gains."}, {"review_id": "LJyZ66wl3d", "reviewer": "Reviewer_oUKw", "summary": "This paper proposes MoB (Multi-objective Balanced covering), a novel pruning method for vision-language models that balances visual preservation (VP) and prompt alignment (PA) from a geometric perspective. The key idea is to leverage prompt-visual coupling (η) to statically allocate token budgets without requiring training. MoB achieves significant speedups and performance retention across diverse tasks, outperforming existing training-free and sparse pruning methods.", "questions": "1. The proposed MoB method hinges on the prompt-visual coupling strength (η) to allocate token budgets between visual preservation (VP) and prompt alignment (PA). However, η is not directly observable in real-world settings. How can η be inferred when the task type is unknown, such as in open-domain or user-driven inference? Does MoB require pre-defined task labels or dataset-specific heuristics to function, and how does this constrain its applicability in deployment?\n2. While η plays a central role in MoB's theoretical guarantees and pruning strategy, it is computed via Hausdorff distance between prompt and visual tokens—an operation that can be computationally expensive and dataset-dependent. The authors assume η is known a priori based on dataset labels, but this introduces additional supervision and cost that methods like FastV or SparseVLM do not require. Can the authors justify this additional cost, and is it fair to compare MoB against zero-cost or training-free baselines under this assumption? Furthermore, could η be efficiently approximated via lightweight dataset sampling (e.g., estimating η from a few examples), and if so, how robust would such estimation be across domains?\n3. Given that η reflects the geometric distance between prompt and visual token embeddings, it may vary across models of different sizes and architectures (e.g., LLaVA-7B vs. QwenVL-32B). Does η need to be re-estimated or recalibrated when switching to a different model? How transferable is the chosen Kp configuration across models, and to what extent does MoB’s pruning effectiveness rely on model-specific representations?\n4. The MoB framework assumes that all pruning-relevant information can be captured through the dual objectives of visual preservation (VP) and prompt alignment (PA). However, is this decomposition theoretically complete? Could there exist tokens whose value arises from higher-order interactions, global reasoning, or latent semantics not directly reflected in either VP or PA distances.", "limitations": "yes.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths:\n1. The method demonstrates strong empirical performance across a wide range of vision-language tasks, often outperforming previous training-free pruning approaches.\n﻿2. The experiments are relatively comprehensive, covering diverse datasets and settings (though lacking evaluation on larger models such as LLaVA-13B or QwenVL-32B).\n﻿3. The paper is well-written and clearly structured, making a technically dense topic accessible.\n﻿4. Importantly, the authors attempt to analyze the token pruning problem from a geometric and theoretical perspective, introducing prompt-visual coupling (η) and proving formal upper bounds on pruning error. This provides a novel and insightful angle to understand the pruning trade-off.\n﻿\nWeaknesses:\n﻿1. While theoretical analysis is a commendably ambitious direction, it is also risky—there is a concern that the theory may not be entirely complete. In particular, the paper assumes that all pruning-relevant information can be captured by VP and PA, which may overlook more complex token interactions, reasoning steps, or latent semantics.\n﻿2. The design and usage of η raise several questions. It is treated as a dataset-level prior, but not explicitly estimated during inference. Its dependency on model architecture, computational cost, and lack of deployable estimation strategy (especially in unseen, open-domain settings) limit the practicality of the approach.", "was_revised": true, "final_justification": "Most of the concerns I raised have been satisfactorily addressed in the rebuttal. In particular:\n﻿\n- The authors clarified key theoretical assumptions, such as the use of normalized embedding space and the model-invariance of their pruning strategy.\n- They also provided further explanation on the practicality of their method, including the online estimation of coupling strength and its overhead.\n- The methodology is solid, and the experimental results are comprehensive and convincing across multiple VLA models and benchmarks.\n﻿\nWhile there are still a few open questions (e.g., performance on larger models, more intuitive hyperparameter analysis), I view them as future directions rather than critical flaws.\n﻿\nOverall, I find this work to be technically sound, well-executed, and a valuable contribution to the field. I maintain my positive recommendation.", "rebuttal": "We sincerely appreciate your responsible, careful work and insightful questions! We have corrected the format problem of Fig. 3 and provide a point-to-point feedback as follows\n\n**W2 & Q1:  $\\eta$ is not directly observable in real-world settings. How can $\\eta$ be inferred when the task type is unknown, such as in open-domain or user-driven inference? Does MoB require pre-defined task labels or dataset-specific heuristics to function, and how does this constrain its applicability in deployment?**\n\n**A1:** Thank you for this critical question about real-world applicability. We provide an explanation in two aspects:\n\n**A Practical Strategy for Open-Domain Inference:** MoB does not require pre-defined \"task type\" labels. Instead, it can perform online classification of a sample's coupling pattern. The practical strategy is a two-stage process:\n\n- **Offline Calibration:** For a given target model, we first analyze the η distributions on a set of representative benchmarks to establish a robust classification threshold that distinguishes between weak and strong coupling patterns.\n- **Online Classification:** For each incoming query, we perform an online computation of its Hausdorff distance (Alg. 2, Appx. B, with a tractable bilinear complexity of $O(NLd)$). The sample is then classified based on this value and the pre-calibrated threshold, and the corresponding $K_{\\rm p}$ is applied.\n\nWe acknowledge the online computation introduces a manageable cost, but it does not fundamentally alter MoB's superior performance-efficiency trade-off.\n\n**An Advanced, Fully-Adaptive Framework:** Furthermore, we have explored this direction deeply in our subsequent work and developed a more elegant, fully-adaptive framework. It operates as follows:\n\n- During inference, it maintains a running statistic of an *approximated* $\\hat{\\eta}$ (e.g., from shallow-layer tokens) for the samples it has processed.\n- We observe that as more samples are seen, this statistic naturally reveals the same bimodal distribution pattern identified in our paper (**Fig. 1b & 9**).\n- The framework then uses this emerging distribution to **dynamically adjust its own classification threshold** for subsequent samples.\n\nThe theoretical framework presented in this paper is the essential foundation that enables and validates such future advancements.\n\n**W2 & Q2: The authors assume $\\eta$ is known a priori based on dataset labels, but this introduces additional supervision and cost that methods like FastV or SparseVLM do not require. Can the authors justify this additional cost, and is it fair to compare MoB against zero-cost or training-free baselines under this assumption? Furthermore, could  $\\eta$ be efficiently approximated via lightweight dataset sampling (e.g., estimating  $\\eta$ from a few examples), and if so, how robust would such estimation be across domains?**\n\n**A2:** Considering your concern, we offer the following points in response:\n\n**Justification of Experimental Settings:** We first argue that prior methods did not intentionally avoid using η, but were fundamentally unaware of the critical role of prompt-visual coupling. A core contribution of our paper is to first identify and formalize this factor. To experimentally validate our theory, it was necessary to introduce the benchmark's η pattern as a prior.\n\nWe acknowledge that this could potentially leverage information unavailable to other methods. *To ensure a fair comparison, we did not delicately fine-tune $K\\_{\\rm p}$ for each of the 14 benchmarks*. Instead, we use a coarse-grained approach: we classify benchmarks into one of two general coupling patterns and apply one of two corresponding strategies. We believe this setup is sufficiently fair and practical for real-world applications.\n\n**$\\eta$-Agnostic Evaluation:** In this experiment, we *apply this same budget configuration to **all benchmarks**, regardless of their coupling pattern.*\n\nTable 1 Comparative experiments on image understanding with the LLaVA1.5-7B under static $K_{\\rm p}$ \n|Setting|MMB|MMB-CN|SQA|VizWiz|GQA|MME|POPE|VQA-T|VQA-V2|OCR|\n|---|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|\n|$K = 192,K_{\\rm p} = 64$| | | | | | | | | | |\n|FastV|61.2|57.0|67.3|50.8|52.7|1612|64.8|52.5|67.1|291|\n|SparseVLM|62.5|53.7|69.1|50.5|57.6|1721|83.6|56.1|75.6|292|\n|MustDrop|62.3|55.8|69.2|51.4|58.2|1787|82.6|56.5|76.0|289|\n|DART|63.6|57.0|69.8|51.2|60.0|1856|82.8|57.4|76.7|296|\n|MoB|**63.8**|**57.5**|**70.0**|**52.4**|**61.2**|**1858**|**84.5**|**58.2**|**77.9**|**304**|\n|$K = 128,K_{\\rm p} = 48$| | | | | | | | | | |\n|FastV|56.1|56.4|60.2|51.3|49.6|1490|59.6|50.6|61.8|285|\n|SparseVLM|60.0|51.1|67.1|51.4|56.0|1696|80.5|54.9|73.8|280|\n|MustDrop|61.1|55.2|68.5|52.1|56.9|1745|78.7|56.3|74.6|281|\n|DART|**63.2**|**57.5**|69.1|51.7|58.7|1840|80.1|56.4|75.9|296|\n|MoB|**63.2**|57.3|**69.3**|**52.8**|**60.7**|**1842**|**81.7**|**57.5**|**77.2**|**299**|\n|$K = 64,K_{\\rm p} = 32$| | | | | | | | | | |\n|FastV|48.0|52.7|51.1|50.8|46.1|1256|48.0|47.8|55.0|245|\n|SparseVLM|56.2|46.1|62.2|50.1|52.7|1505|75.1|51.8|68.2|180|\n|MustDrop|60.0|53.1|63.4|51.2|53.1|1612|68.0|54.2|69.3|267|\n|DART|60.6|53.2|**69.8**|51.6|55.9|1765|73.9|54.4|72.4|270|\n|MoB|**61.7**|**54.2**|**69.7**|**52.0**|**59.0**|**1806**|**77.2**|**57.0**|**75.5**|**277**|\n\nAs the results demonstrate, even without the $\\eta$-based prior, **MoB consistently outperforms all baselines** in most cases. This highlights the intrinsic advantage of our approach, independent of the $\\eta$-based situation, and further strengthens the experimental results presented in our manuscript.\n\n**On Lightweight Estimation:** While sampling a few examples is more lightweight, it still uses prior information from the test set and risks sampling bias. In our subsequent work, we have developed a more elegant and fair solution, which maintains a statistic of an approximated $\\hat{\\eta}$ for the samples it has processed and adaptively identifies the coupling pattern of the following inputs, i.e., $K_{\\rm p}$, based on the statistic. \n\n**W2 & Q3: Does η need to be re-estimated or recalibrated when switching to a different model? How transferable is the chosen $K_{\\rm p}$ configuration across models, and to what extent does MoB’s pruning effectiveness rely on model-specific representations?**\n\n**A3:** We want to clarify the absolute numerical values of $\\eta$ can vary with model representations; however, it **does not** significantly affect the optimal $K_{\\rm p}$ configuration, which remains highly transferable. We justify this from both a theoretical and an empirical standpoint:\n\n**Theoretical Rationale:**  As stated in our manuscript (Lines 204-205), we apply L2 normalization to all token embeddings before pruning. Thus, *MoB operates in a normalized space*, where all tokens are projected onto a unit sphere, meaning our pruning decisions are based on the relative angular structure of the tokens, not their absolute Euclidean distances or magnitudes, which can vary across models. \n\n**Empirical Validation:** In our settings, the $K_{\\rm p}$ configuration is not tuned to the specific *absolute numerical value* of $\\eta$ for a benchmark. Instead, it is set based on the **coupling pattern**—that is, the *relative difference* between $\\eta$ distributions for different task types *within a single model's* embedding space. This is why we were able to apply the **exact same $K_{\\rm p}$ configurations across four distinct VLLMs**: LLaVA-1.5-7B, LLaVA-Next-7B, Qwen2-VL-7B and Video-LLaVA-7B. The results (Tables 1-4) show this fixed configuration is highly effective for all models. For instance, at an 88.9% reduction rate, MoB achieved average performance gains of 2.7% (LLaVA-1.5), 1.5% (LLaVA-Next), and 1.2% (Qwen2-VL) over the strongest baselines.\n\nThis provides powerful empirical evidence that *the configuration of $K_{\\rm p}$ is robust across models, and MoB's effectiveness stems primarily from these universal, task-driven geometric principles rather than relying heavily on model-specific representations.*\n\n**W1 & Q4: Is MoB's framework theoretically complete? Could there exist tokens whose value arises from higher-order interactions, global reasoning, or latent semantics not directly reflected in either VP or PA distances?**\n\n**A4:** Thank you for this profound question that addresses the core of our work. We agree that the theoretical completeness is a critical aspect. Our response is threefold:\n\n**Formulation Setting:** We clarify that higher-order interactions may influence multi-stage pruning pipelines, which prune tokens across multiple layers simultaneously. To keep the problem tractable and theoretically rigorous, *our analysis and methodology explicitly focus on the single-layer pruning setting (L. 94–111)*, where exclusively exist the first-order interactions of input tokens and the pruned counterpart. Within this well-defined scope, we argue that Visual Preservation (VP) and Prompt Alignment (PA) are sufficient to rigorously characterize the performance bounds of a pruning algorithm.\n\n**The Advantage of a Tractable Framework**: The challenge of balancing objectives in multi-stage methods, e.g., SparseVLM and MustDrop, highlights a key advantage of MoB's design. By focusing on a single, decisive pruning stage, MoB makes the pruning problem practicable and analyzable. This allows for an efficient and principled balance between VP and PA, thereby enabling us to derive a strict performance guarantee for MoB, as presented in Theorem 2.\n\n**Empirical evidence**: From an experiential perspective, our solid experimental results validate the correctness of our theoretical analysis. By solely considering the trade-off between these two objectives, MoB consistently and significantly outperforms a comprehensive suite of both single-stage and multi-stage methods across various benchmarks (Table 1, 4). We therefore believe this provides powerful evidence that VP and PA are the decisive factors for visual token pruning."}, {"review_id": "konJkuRM1h", "reviewer": "Reviewer_4yyW", "summary": "This paper investigates the performance differences between single-objective and multi-objective approaches in visual token pruning, with the following main contributions:\n\nThe study finds that multi-objective methods do not significantly outperform single-objective methods in visual token pruning. By introducing the Hausdorff distance, the authors derive the first closed-form error bound, quantifying the contribution of the Visual Preservation (VP) and Prompt Alignment (PA) objectives.\n\nBased on the ε-covering theory, the authors reveal an inherent trade-off between VP and PA, showing how to balance these two objectives under different pruning budgets to optimize performance.\n\nThe authors propose a Multi-Objective Balanced Covering (MoB) method, reformulating visual token pruning as a bi-objective covering problem. MoB employs a greedy radius-trading strategy for budget allocation, offering provable performance bounds and linear scalability.\n\nExperimental results show that MoB maintains 96.4% of the original performance in LLaVA-1.5-7B using only 11.1% of the original visual tokens, and achieves 1.3–1.5× speedup in LLaVA-Next-7B with negligible performance loss. Moreover, evaluations on Qwen2-VL and Video-LLaVA confirm MoB’s seamless integration into advanced multimodal large language models and diverse vision-language tasks.\n\nIn summary, this paper presents a novel visual token pruning method through theoretical analysis and experimental validation, emphasizing the importance of balancing different objectives during the pruning process and demonstrating its effectiveness in practical applications.", "questions": "See weakness", "limitations": "See weakness", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. The paper proposes the Multi-Objective Balanced Covering (MoB) method, systematically addressing the objective trade-off problem in visual token pruning. It provides provable performance bounds and linear scalability.\n\n   By introducing the Hausdorff distance, the authors derive the first closed-form error bound, quantifying the contribution of Visual Preservation (VP) and Prompt Alignment (PA) objectives, laying a solid theoretical foundation for future research.\n\n   The paper is well-structured and logically rigorous, guiding readers step-by-step through complex concepts and methods. Especially in the introduction of the MoB method, the working principle and implementation steps are explained in detail.\n\n    By reformulating visual token pruning as a bi-objective covering problem, the paper offers a novel perspective and approach, demonstrating high innovativeness.\n\n   Experimental results show that MoB performs exceptionally well on multiple benchmarks, achieving significant token reduction while maintaining high performance, indicating its practical applicability.\n\nWeaknesses：\n\nAlthough a new method is proposed, the performance improvement of MoB is relatively modest in some experimental settings, which may fall short of expectations in certain scenarios.\n\nThe interpretation of some experimental results could be more in-depth. The paper does not fully explore the applicability and limitations of different pruning strategies.\n\nWhile the method shows improvements, compared to existing approaches such as CrossGET, LLaVA-PruMerge, TokenPacker, Turbo, and AIM, which also demonstrate similar effectiveness, the paper lacks comparison and discussion with these methods.", "was_revised": true, "final_justification": "The authors partially provide comparative experiments in their manuscript. In the final version, they'll give explicit discussion of the key differences between the proposed method and existing approaches.", "rebuttal": "Thanks for your comments. To address your concerns, we provide our point-to-point clarification as follows\n\n**W1 & Q1: Although a new method is proposed, the performance improvement of MoB is relatively modest in some experimental settings, which may fall short of expectations in certain scenarios.**\n\n**A1:** Thank you for your feedback. We respectfully argue that the performance improvement of MoB is not modest, and we would like to highlight two key aspects of our contribution: its significant and robust empirical performance and its foundational theoretical insights.\n\n**1. Significant and Robust Empirical Performance:** We believe the term \"modest\" may overlook two critical patterns in our results.\n\n- **Consistent Superiority:** Across all three tested model families (LLaVA-1.5, LLaVA-Next, Qwen2-VL, and Video-LLaVA) and all pruning rates, MoB ***consistently outperforms the strongest competing baselines*** in average performance. For example, MoB achieves average performance gains of 2.7% (LLaVA-1.5), 1.5% (LLaVA-Next), and 1.2% (Qwen2-VL) in the image domain and achieves a **97.9%** average performance at a very aggressive 93.4% token reduction rate, surpassing the next-best method by 1.6% in the video domain.\n- **Widening Lead Under Aggressive Pruning:** Crucially, MoB's performance advantage becomes **more pronounced** under more challenging, high-ratio pruning scenarios. On LLaVA-1.5-7B, for instance, our lead over the strongest baseline grows from +1.4% at a 77.8% reduction rate to a more substantial **+2.7%** at an 88.9% reduction rate. This demonstrates the superior robustness of our method when efficiency demands are highest.\n\nWe contend that this consistent and robust outperformance, especially under aggressive compression, is a significant achievement.\n\n**2. Foundational Theoretical Contribution:** Beyond the performance numbers, we encourage the reviewer to consider the value of our theoretical contributions, which we believe are a core part of our work.\n\n- **Provable Guarantees:** Unlike most heuristic-based pruning methods, MoB is grounded in a formal theoretical framework that provides a **provable performance bound** (Theorem 2). This ensures its reliability.\n- **Novel Insights for the Community:** More importantly, this paper presents the first in-depth analysis of the key factors in visual token pruning—Visual Preservation (VP), Prompt Alignment (PA), prompt-visual coupling ($\\eta$), and budget ($K$)—and formalizes their interplay. MoB is a direct and effective application of these insights. We believe these foundational theories will motivate the community and inspire more powerful, principled pruning methods in the future, representing a profound potential impact.\n\n**W2 & Q2: The interpretation of some experimental results could be more in-depth. The paper does not fully explore the applicability and limitations of different pruning strategies.**\n\n**A2:** Thank you for your comment. We respectfully clarify that a central motivation and contribution of our work is precisely to provide an in-depth analysis of the applicability and limitations of different pruning strategies, and how to incorporate them to derive a robust visual token pruning method. *We have demonstrated this extensively in our manuscript from theoretical, empirical, and ablation perspectives.*\n\n**1. Theoretical Foundation:** Our key theoretical insight is that the effectiveness of a pruning strategy is contingent on the prompt-visual coupling pattern (η). We explicitly analyze this in our discussions of **Lemma 2 (Lines 143-147)** and **Theorem 2 (Lines 232-234)**. In these sections, we formally reveal that ***Prompt Alignment (PA) is crucial for weak-coupling tasks, while Visual Preservation (VP) is favored under strong-coupling conditions.***\n\n**2. Empirical Validation in Main Results:** This theoretical prediction is directly validated by the trends in our main experimental results in Tables 1 and 4. As we explicitly state in our analysis in **Lines 252-255:** ***\"...single-objective baselines exhibit complementary strengths under different coupling patterns...\".*** This discussion confirms that PA-based methods excel on weak-coupling benchmarks, while VP-based methods perform better on strong-coupling ones.\n\n**3. Targeted Ablation Studies:** Furthermore, we conducted a targeted ablation study (**Lines 284-299**), the results are presented in Figure 4, to isolate and verify this exact relationship. By systematically varying the budget allocation $K_{\\rm p}$ (considered as the quality of prompt alignment), we empirically confirmed that ***PA is more critical for weak-coupling tasks and VP for strong-coupling ones, thus validating our core theoretical premise.***\n\nIn summary, we believe our manuscript provides a comprehensive, multi-faceted analysis of the applicability and limitations of different pruning strategies. We are confident that these detailed discussions, located throughout the paper, fully address your concern.\n\n**W3 & Q3: While the method shows improvements, compared to existing approaches such as CrossGET, LLaVA-PruMerge, TokenPacker, Turbo, and AIM, which also demonstrate similar effectiveness, the paper lacks comparison and discussion with these methods.**\n\n**A3:** Thank you for your suggestion to compare with a broader set of baselines. We would first like to clarify that ***our original manuscript already contains a strong suite of 13 baselines (Appx. C.3), including the mentioned LLaVA-PruMerge [2]***, where MoB consistently demonstrates state-of-the-art performance (Tables 1-4). Thank you for suggesting these excellent works. To further address your concern, we have conducted additional experiments to provide a direct comparison against CrossGET [1] and AIM [5]. The comparison results are presented below:\n\n| Setting        |   MMB    |   SQA    |  VizWiz  |   GQA    |   MME    |   POPE   |  VQA-T   |  VQA-V2  |     Avg     |\n| :------------- | :------: | :------: | :------: | :------: | :------: | :------: | :------: | :------: | :---------: |\n| *FLOPs ↓31%*   |          |          |          |          |          |          |          |          |             |\n| CrossGET       | **64.7** |   66.7   |   47.7   |   61.4   |   1510   |   83.9   |   54.9   |   77.3   |    95.2%    |\n| **MoB**        |   64.5   | **70.0** | **52.6** | **61.7** | **1862** | **85.3** | **58.6** | **78.5** | **100.6 %** |\n| *FLOPs ↓73%*   |          |          |          |          |          |          |          |          |             |\n| LLaVA-PruMerge |   58.1   |   67.1   |   50.3   |   53.3   |   1554   |   67.2   |   54.3   |   68.8   |    88.8%    |\n| AIM            |   58.8   |   68.4   |    -     |   58.6   |   1772   | **85.7** |   53.8   |   75.4   |    95.3%    |\n| **MoB**        | **63.5** | **69.6** | **52.7** | **60.9** | **1845** |   82.1   | **57.8** | **77.5** | **99.4 %**  |\n| *FLOPs ↓88%*   |          |          |          |          |          |          |          |          |             |\n| LLaVA-PruMerge |   55.3   |   68.1   |   50.1   |   51.9   |   1549   |   65.3   |   54.0   |   67.4   |    87.4%    |\n| AIM            |   60.9   |   67.1   |    -     |   54.6   |   1573   | **79.5** |   48.4   |   69.0   |    89.6%    |\n| **MoB**        | **62.1** | **69.8** | **52.1** | **59.0** | **1806** |   77.2   | **57.0** | **75.5** | **96.4 %**  |\n\nAs the new results clearly demonstrate, ***MoB’s effectiveness is not \"similar\" but substantially superior.*** MoB achieves a **5.4%** average performance gain over the CrossGET under a ~31% FLOPs reduction, while \nAt a ~73% FLOPs reduction, MoB achieves a **4.1%** and **10.6%** average performance gain over the AIM and LLaVA-PruMerge under a ~73% FLOPs reduction. As for a more aggressive reduction, e.g., 88%, MoB still achieve **6.8%** and **9.0%** average performance gain over the AIM and LLaVA-PruMerge. We believe these significant margins validate the effectiveness of our approach.\n\n***We did not include Turbo [4] and TokenPacker [3] for the following principled reasons:***\n\n- The repository of Turbo's code is still invalid, and its does not report results on any of the standard benchmarks used by the community, making a fair and reproducible comparison impracticable. \n- Although TokenPacker is also designed for acceleration of VLLM, it belongs to a different research category. It is a novel efficient VLM instead of a visual token pruning method. TokenPacker requires extensive training with extra data, not only pre-training (1.2M data) but also STF (1.5M data). Comparing it directly with a training-free, inference-time visual token pruning method like MoB would be unfair. Crucially, these approaches are orthogonal; a pruning method like MoB could potentially be applied on top of an efficient VLM like TokenPacker for even more aggressive acceleration.\n\nFinally, we will add these new comparisons and a detailed discussion to the revised manuscript to provide readers with a more comprehensive understanding of MoB's contribution. We hope this response alleviates your concerns regarding MoB's performance.\n\n[1] Shi D, et al. Crossget: Cross-guided ensemble of tokens for accelerating vision-language transformers. preprint, 2023.\n\n[2] Shang Y, et al. Llava-prumerge: Adaptive token reduction for efficient large multimodal models. preprint, 2024.\n\n[3] Li W, et al. Tokenpacker: Efficient visual projector for multimodal llm. IJCV, 2025. \n\n[4] Ju C, et al. Turbo: Informativity-driven acceleration plug-in for vision-language large models. ECCV, 2024.\n\n[5] Zhong Y, et al. Aim: Adaptive inference of multi-modal llms via token merging and pruning. ICCV, 2025."}, {"review_id": "8jRuypWyVD", "reviewer": "Reviewer_HvnN", "summary": "This paper provides a systematic analysis of visual token pruning in Multimodal Large Language Models (MLLMs). The authors identify that existing methods often combine the two objectives of Visual Preservation (VP) and Prompt Alignment (PA) in a static manner, neglecting their varying importance across different tasks, which results in inconsistent performance. To address this, the authors derive the first closed-form error bound for visual token pruning based on the Hausdorff distance, and employ ε-covering theory to quantify the intrinsic trade-off between VP and PA under a fixed pruning budget. Building upon this theoretical insight, the paper proposes a novel training-free method—Multi-Objective Balanced Covering (MoB)—which reformulates the pruning process as a bi-objective covering problem. MoB translates the trade-off into a budget allocation problem via greedy radius trading. Extensive experiments demonstrate that MoB outperforms existing methods across LLaVA, Video-LLaVA, and Qwen2-VL, maintaining high performance even under aggressive pruning.", "questions": "Automatic Estimation of 𝐾𝑝 : Currently, MoB requires manual selection of 𝐾𝑝. Could the authors consider estimating 𝐾𝑝 from prompt length or coupling strength 𝜂? Would meta-learning or lightweight heuristic predictors help automate this?\nComputational Cost of Hausdorff Distance: While theoretically elegant, the computation of Hausdorff distance in high-dimensional spaces can be expensive. Have the authors considered approximations or alternatives that maintain performance?\nPerformance with Fine-tuning: As MoB is training-free, have the authors tested its combination with lightweight fine-tuning post-pruning? Would such a setup improve performance under aggressive compression?\nApplicability to Language Token Pruning: While MoB focuses on visual tokens, could the same methodology be adapted to prune language tokens, especially for long-context LLMs?", "limitations": "The authors explicitly discuss the reliance on theoretical assumptions and the potential cost of manually tuning Kp. They also suggest future work involving adaptive estimation of η. The discussion is honest and constructive.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\nTheoretical Contribution: The paper is the first to provide a closed-form error bound for visual token pruning, with rigorous theoretical analysis supported by a geometric interpretation using Hausdorff distance.\nNovel Methodology: The proposed MoB method addresses the bi-objective trade-off through a covering-based perspective, reformulating pruning as a budget allocation problem and offering provable performance guarantees.\nStrong Empirical Results: The method is evaluated across 14 benchmarks and three MLLMs, demonstrating superior performance. Notably, it retains 96.4% performance for LLaVA-1.5-7B with only 11.1% of tokens.\nPracticality: MoB is training-free and scales linearly with input size, making it applicable to high-resolution or multi-frame scenarios.\n\nWeaknesses:\nHyperparameter Sensitivity: MoB requires selecting the prompt center budget K_p, which may vary across tasks. Although the paper categorizes scenarios based on prompt-visual coupling strength (η), practical deployment may incur tuning overhead.\nAssumption Validity: The theoretical guarantees depend on the assumption of Lipschitz continuity of the model in the Hausdorff metric space. While reasonable in practice, this may not hold universally.\nLimited Discussion on Supervision Integration: While MoB is fully training-free, the paper does not explore the potential of incorporating supervised signals (e.g., task labels) to improve token selection relevance.", "was_revised": true, "final_justification": "Appreciate the thorough response. With most concerns addressed, I decide to keep my score.", "rebuttal": "Sincerely appreciate your responsible work and constructive questions! We provide a point-to-point feedback on the **weakness (W)** and **question (Q)**\n\n**W1, Q1: Automatic Estimation of $K_{\\rm p}$: Currently, MoB requires manual selection of $K_{\\rm p}$. Could the authors consider estimating $K_{\\rm p}$ from prompt length or coupling strength $\\eta$? Would meta-learning or lightweight heuristic predictors help automate this?**\n\n**A:** Thank you for this forward-thinking question, which is highly align with our ongoing research. In fact, deriving a analytical expression for the optimal $K_{\\rm p}$ from prompt length $L$ and coupling strength $\\eta$ is highly challenging at present. Therefore, to make the automatic estimation of $K_{\\rm p}$ more tractable, we have developed an elegant and practical framework in our follow-up work.\n\nInstead of solving a precise $K_{\\rm p}$ value, this framework learns to *adaptively classify* the coupling pattern of incoming samples and *applies a pre-configured budget configuration $K_{\\rm p}$*. It operates as follows:\n\n1. During inference, the framework maintains a running statistic of an **approximated** $\\hat{\\eta}$ for the processed samples. We observe that as more samples are processed, this statistic naturally reveals the same bimodal distribution pattern identified in our paper (Figs. 1b & 9).\n2. From this observed distribution, the framework **dynamically determines a classification threshold** that separates the two patterns.\n3. Each new sample is then classified into 'weak' or 'strong' coupling using this live, data-driven threshold, and the corresponding optimal $K_{\\rm p}$ is applied.\n\nThis approach directly leverages the **coupling strength** $\\eta$, as you suggested, in a more practicable way. It can be viewed as a form of **lightweight, online meta-learning** that adapts to the specific data distribution encountered during deployment. \n\n**W1, Q2: Computational Cost of Hausdorff Distance: While theoretically elegant, the computation of Hausdorff distance in high-dimensional spaces can be expensive. Have the authors considered approximations or alternatives that maintain performance?** \n\n**A2:** Thank you for this very important and forward-thinking question. We agree that the computational cost of the exact Hausdorff distance is a critical consideration. We address this with a quantitative analysis of the practical costs, followed by a discussion of efficient alternatives.\n\n**Quantitative Analysis:** We provide a detailed cost breakdown for LLaVA-1.5-7B and LLaVA-Next-7B below:\n\nComputational cost (TFLOPs) in LLaVA-1.5-7B, where $N=576, L=10, d=4096$.\n\n| Model         | Vanilla       | $K=64$        | $K=128$       | $K=192$       |\n| ------------- | ------------- | ------------- | ------------- | ------------- |\n| Forward       | 8.2           | 1.0           | 1.9           | 2.8           |\n| Compute $d_H$ | 2.3×10$^{-5}$ | 2.3×10$^{-5}$ | 2.3×10$^{-5}$ | 2.3×10$^{-5}$ |\n| MoB           |               | 1.7×10$^{-4}$ | 3.3×10$^{-4}$ | 4.8×10$^{-4}$ |\n\nComputational cost (TFLOPs) in LLaVA-Next-7B, where $N=2880, L=10, d=4096$.\n\n| Model        | Vanilla       | $K=320$       | $K=640$       | $K=960$       |\n| ------------ | ------------- | ------------- | ------------- | ------------- |\n| Forward      | 40.5          | 4.6           | 9.1           | 13.6           |\n| Compute$d_H$ | 1.2×10$^{-4}$ | 1.2×10$^{-4}$ | 1.2×10$^{-4}$ | 1.2×10$^{-4}$ |\n| MoB          |               | 3.9×10$^{-3}$ | 7.6×10$^{-3}$ | 1.1×10$^{-2}$ |\n\nAs the tables clearly show, while the complexity of computing Hausdorff distance, i.e., $O(NLd)$, is intuitively heavy in high-dimensional spaces, it remains **orders of magnitude smaller than both the MoB itself and the model's forward propagation**. Thereby, it cannot be a practical bottleneck of MoB. Considering your concern, we also present two powerful approximation techniques:\n\n- **Heuristic Sampling:** It computes the distance on smaller support sets of the tokens ($\\mathcal{V}'\\subset \\mathcal{V},\\mathcal{P}'\\subset\\mathcal{P}$), which can be constructed via random sampling [1] or more advanced heuristics like Key-Norm selection [2, 3]. This reduces complexity to $O(N'L'd)$.\n- **Random Projections:** For a more theoretically-grounded approach, the Johnson-Lindenstrauss (JL) lemma [4] allows us to project embeddings to a much lower dimension ($d'\\ll d$) while preserving geometric structure, reducing complexity to $O(NLd')$.\n\nIn summary, the cost of computing Hausdorff distance is already negligible in practice, as shown by our quantitative analysis, and can be further optimized via well-established approximation methods if needed. This ensures the broad applicability and efficiency of our framework.\n\n**References**\n\n[1] Zichen Wen, et al. Stop looking for important tokens in multimodal language models: Duplication matters more. preprint, 2025.\n\n[2] Akhauri Y, AbouElhamayed A F, Gao Y, et al. Tokenbutler: Token importance is predictable. preprint, 2025.\n\n[3] Guo Z, et al. Attention Score is not All You Need for Token Importance Indicator in KV Cache Reduction: Value Also Matters. CoRR, 2024.\n\n[4] Larsen K G, Nelson J. The Johnson-Lindenstrauss Lemma Is Optimal for Linear Dimensionality Reduction. LIPIcs, 2016.\n\n**W2:  Assumption Validity: The theoretical guarantees depend on the assumption of Lipschitz continuity of the model in the Hausdorff metric space. While reasonable in practice, this may not hold universally.**\n\n**A:** Thanks for your constructive concern; we justify Assumption 1 in three aspects:\n\n**Theoretical Grounding:** Prior work [1, 2] has shown that Transformers are approximately Lipschitz continuous w.r.t. standard matrix norms, e.g., the Frobenius norm, under the finite and compact input conditions we consider (Line 98). A standard inequality relates the Frobenius norm to the Hausdorff distance:  $\\frac{1}{\\sqrt N}\\Vert\\mathcal X\\_1-\\mathcal X\\_2\\Vert_{\\_F}\\le d\\_H(\\mathcal X\\_1,\\mathcal X\\_2)\\le\\Vert\\mathcal X\\_1-\\mathcal X\\_2\\Vert_{\\_F}$. This implies that any model Lipschitz continuous w.r.t. the Frobenius norm is also Lipschitz continuous w.r.t. the Hausdorff distance.\n\n**Empirical Evidence:** Vignac et al. [3] have measured the empirical local Lipschitz constant of Transformers on real-world data and found it to be 100-1000x smaller than the theoretical worst-case bound. This indicates that practical models already lie in a well-behaved, near-Lipschitz regime. Besides, Kim et al. [4] have shown that the successful training of deep Transformers relies on controlling this constant. Architectures without such implicit or explicit constraints often fail to converge, suggesting that the well-trained VLLMs most likely adhere to a finite Lipschitz constant.\n\n**Consistency with Prior Work:** We wish to clarify that this assumption is not novel to our work but is a common theoretical premise in the visual token pruning community, having been adopted by prior methods such as DivPrune [5] and DART [6]. Our work builds upon this established foundation to derive deeper insights.\n\n**References**\n\n[1] Dasoulas G, et al. Lipschitz normalization for self-attention layers with application to graph neural networks. ICML, 2021.\n\n[2] Yudin N, et al. Pay Attention to Attention Distribution: A New Local Lipschitz Bound for Transformers. preprint, 2025.\n\n[3] Vignac C,  et al. Understanding the regularity of self-attention with optimal transport. preprint, 2023.\n\n[4] Kim H, et al. The lipschitz constant of self-attention. ICML, 2021.\n\n[5] Alvar S R, et al. Divprune: Diversity-based visual token pruning for large multimodal models. CVPR, 2025.\n\n[6] Wen Z, et al. Stop looking for important tokens in multimodal language models: Duplication matters more. preprint, 2025.\n\n**W3, Q3: Performance with Fine-tuning: As MoB is training-free, have the authors tested its combination with lightweight fine-tuning post-pruning? Would such a setup improve performance under aggressive compression?**\n\n**A:** Thank you for this insightful suggestion. While our work focused on the training-free aspect, we agree this is a promising direction. A key advantage of MoB is that it provides a **principled, informative sparse input** to the LLM. A subsequent lightweight fine-tuning (e.g., LoRA) would then allow the model, originally trained on dense data, to **adapt** its attention mechanisms to this new distribution. We hypothesize this would significantly improve performance under aggressive compression and have added it as a key direction for future work.\n\n**Q4: Applicability to Language Token Pruning: While MoB focuses on visual tokens, could the same methodology be adapted to prune language tokens, especially for long-context LLMs?**\n\n**A:** Thank you for this constructive question. While naively applying MoB on language tokens might not yield the same degree of success, our **theoretical framework** has the fundamental generalizability for the language domain.\n\nIn the language domain, we can create a powerful analogy:\n\n- **Visual Preservation (VP)** is equivalent to preserving the broader **context** of long language representations.\n- **Prompt Alignment (PA)** is equivalent to extracting the **golden evidence**—the most critical sentences directly relevant to the facts.\n\nThe trade-off between VP and PA, a key insight of our theory, is therefore fundamentally the trade-off between **Context and Golden Evidence**. We believe this principle has significant implications for several typical applications in long-text LLMs:\n\n- **RAG:** Balancing **recall** (enough Context to reason) vs. **precision** (the Golden Evidence that answers the query).\n- **Summarization & LLM Memory:** Balancing **coherence** (the Context/flow) vs. **conciseness** (the Golden Evidence/key facts). \n\nOur framework provides a novel, principled view for analyzing and optimizing these fundamental trade-offs, highlighting its significance for the society of long-context LLMs."}]

Author rebuttal:

Meta-review: This paper introduces a solid theoretical framework for visual token pruning, quantifying the trade-off between visual preservation and prompt alignment. The methodology is sound, with comprehensive experiments and convincing clarifications provided in the original submission and rebuttal. As is raised by reviewers, the authors could further consider improving the readability of the original paper, scalability to larger models, and limited hyperparameter analysis. Overall, the work is technically strong, well-executed, and represents a meaningful contribution, meriting acceptance.
Decision: Accept (poster)