Beyond Attention or Similarity: Maximizing Conditional Diversity for Token Pruning in MLLMs
Authors: Qizhe Zhang
Mengzhen Liu
Lichen Li
Ming Lu
Yuan Zhang
Junwen Pan
Qi She
Shanghang Zhang
Venue: NeurIPS 2025 poster ()
Keywords: multimodal large language model
visual token pruning
training-free acceleration
Primary area: applications
TL;DR: We propose a training-free visual token pruning method CDPruner for MLLM inference acceleration by maximizing the conditional diversity of retained tokens.

Abstract: In multimodal large language models (MLLMs), the length of input visual tokens is often significantly greater than that of their textual counterparts, leading to a high inference cost. Many works aim to address this issue by removing redundant visual tokens. However, current approaches either rely on attention-based pruning, which retains numerous duplicate tokens, or use similarity-based pruning, overlooking the instruction relevance, consequently causing suboptimal performance. In this paper, we go beyond attention or similarity by proposing a novel visual token pruning method named **CDPruner**, which maximizes the conditional diversity of retained tokens. We first define the conditional similarity between visual tokens conditioned on the instruction, and then reformulate the token pruning problem with determinantal point process (DPP) to maximize the conditional diversity of the selected subset. The proposed CDPruner is training-free and model-agnostic, allowing easy application to various MLLMs. Extensive experiments across diverse MLLMs show that CDPruner establishes new state-of-the-art on various vision-language benchmarks. By maximizing conditional diversity through DPP, the selected subset better represents the input images while closely adhering to user instructions, thereby preserving strong performance even with high reduction ratios. When applied to LLaVA, CDPruner reduces FLOPs by **95\%** and CUDA latency by **78\%**, while maintaining **94\%** of the original accuracy. Our code is available at https://github.com/Theia-4869/CDPruner.

Beyond Attention or Similarity: Maximizing Conditional Diversity for Token Pruning in MLLMs
Qizhe Zhang'* Mengzhen Liu'
Lichen Li Ming Lu"
Yuan Zhang' Junwen Pan? Qi She Shanghang Zhang' State Key Laboratory for Multimedia Information Processing, School Computer Science, Peking University ByteDance (theia, shanghang) edu.
Abstract
In multimodal large language models (MLLMs), the length of input visual tokens is often significantly greater than that of their textual counterparts, leading to a high inference cost. Many works aim to address this issue by removing redundant visual tokens. However, current approaches either rely on attention-based pruning, which retains numerous duplicate tokens, or similarity- baaaed pruning, overlooking the instruction relevance, consequently causing suboptimal performance. In this paper, we go beyond attention or similarity by proposing novel visual token pruning method named CDPruner, which maximizes the conditional diversity retained tokens. We first define the conditional similarity between visual tokens conditioned on the instruction, and then reformulate the token pruning problem with determinantal point process (DPP) maximize the conditional diversity of the selected subset. The proposed CDPruner is training-free and model-agnostic, allowing easy application to various MIL.Ms. Extensive experiments across diverse MLLMs show that CDPruner establishes new state-of-the art on various vision- language benchmarks. By maximizing conditional diversity through DPP, the selected subset better represents the input images while closely adhering to user instructions, thereby preserving strong performance even with high reduction ratios. When applied to LLaVA, CDPruner reduces FLOPs by 95% and CUDA latency by 78%. while maintaining 94% of the original accuracy. Our code is available https::/github.com/ http://gwwbb.oooo///ceeeea////// 4ti.hhhheea466/DDPteneet
Introduction
Benefiting from the remarkable success of large language models (LLMs) Touvron et al.
2023a b
Jiang et al.] 20233] Bai et al. [023 Yang et al. 2024a Cai et al. 2024b). multimodll large lan guage
models (MLLMs) Liu et al. 2023 2024a Wang et al. 2024 Chen al. 2024d An et al.) 2025
have extended their powerful reasoning capabilities to more modalities, such as images or videos. To
fully leverage the strengths of L.Mss, MI..Ms typically encode visual inputs into form that language models can understand, known as tokens. Within the input sequence, the length of visual tokens often numbers in the hundreds, exceeding their textual counterparts by of times. And in video
streams iZhang et al. 2023 Lin et al. 2023 Zhang et al. [222cc] or high-resolution (Liu et al. 2024b
Luo al. 2024 Guo al. 2024 scenarios, this number can grow even larger. Since attention-base
models Vaswani et al. 2017 exhibit computational complexity that scales quadratically with token
length, an excessive number of visual tokens makes the use of MLLMs costly and impractical for
low-latency or resource- conuurcce-ottrrieeedd applications. [Team et al. 2024
Hu et al. 2024a].
Work done during an internship at Byte Project lead. "CCorreponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Arteation- based Praning
Similarity-hin Pruning
CDPruner (Ours)
Q: What brand is the restaurant?
Input Image
Q: the ramen spicy?
A close-up photo of meal at ICHIRAN, The clusha bowl with spocn placed in the center. The ramen seasoned with chili sauce, chapped scallions. and served with pieces chashu. Chopsticks placed the right of the bowl, still in their puper wrap, not opened. The ramen also served with nori on On top. from are served: bowl of orange spice (pessibly garlic sauce), plate smolee flavored stewed pork with chopped sea liions, and cup of marcha green tea.
right, following sides
special about spoon?
Description
Figure Comparison of different token pruning methods. Attention- based methods retain numerous duplicate tokens, failing to achieve effective visual token compression. Similarity-bas based methods neglect user instructions, always pruning the same tokens and paying insufficient attention to regions most relevant to the question. Our CDPruner considers the conditional diversity of the selected subset, dynamically adjusting pruning according to the user instructions and retaining maximal visual information. In this example, CDPruner successfully preserves tokens related to crucial details, such as the "ICHIRAN" logo on the bowl and chopsticks, the chili pepper on the ramen, and the anti-slip design on the spoon handle, while both alternative methods fail.
Abundant efforts have been made to reduce the inference cost of MLLMs by pruning visual tokens, and existing methods can be roughly divided into two categories. The first is to identify visual tokens with high attention scores as important and discard those deemed less critical [Chen et al. 2024a Xing et al.) 2024 Zhang al. 2024b]. The second to remove redundant parts based on feature similarity between visual tokens [en et al. 20256 Alvar et al.) 2025 Jeddi et al. 2025). As illustrated in Figure | both approaches suffer from inherent weaknesses, leading to suboptimal performance after pruning. Attention- baaed methods only consider the importance of visual tokens, resulting a large number of duplicate tokens being retained, while similarity- based methods neglect user instructions, failing to achieve dynamic pruning in alignment with the current question.
To address these issues, we propose CDPruner, plug-and-play method for MLLM inference ac- celeration by maximizing the conditional diversity of the selected subset. Conditional diversity simultaneously considers feature similarity and instruction relevance, maintaining considerable per- formance at high reduction ratios without the need for additional training. Specifically, we first calculate pairwise similarity between visual tokens conditioned on their relevance the input instruc- tion. To obtain the retained tokens, we reformulate the token pruning problem with determinantal point process (DPP), which is widely used for modeling iistwiie diversity based on pairwise similar- ity (Kulesza et al.) 2012 Chen et al. 2018 Celis et al.) 2018 [i et al.) 2024c] [Sun et al. 2025) Direct MAP inference for DPP Na-hard. To address this, we adopt a greedy algorithm with polynomial- time complexity that guarantees (1 - 1/e) approximation. By leveraging Cholesky decomposition, the additional latency introduced by solving the DPP remains within the limits required for real-time applications. In practice, the computational complexity can be further reduced through techniques such as sliding-windd [Chen et al.) 2018 or Markov chain Kang 2013] approximations.
As simple yet effective solution, CDPruner offers several practical advantages. First, in contrast attention-based methods [Chen al. 2024a Xing et al. 2024 Zhang al. 2024b), CDPruner does not require access to attention scores, which ensures its complete compatibility with efficient attention acceleration implementations (Dao et al./ [2022] Second, CDPruner does not depend on a specific visual encoder or language model, and can be readily implemented across any loken-based MLLM [Li et al. 2024a| Bai et al., 22055] Zhu et al., 20255. Extensive experiments across various MILMMs demonstrate the effectiveness and efficiency of CDPruner. When applied to LLaVA-NeXT- 7B, reduces FL.PPs by 95%, CUDA latency by 78%, and GPU memory by 17%, while maintaining 94% of the original performance in training-free manner.

---

In summary, the contributions of our work are three-fold:
1. We introduce CDPruner, a plug-and-play and model-agnostic solution for visual token pruning that maximizes conditional diversity.
2. We reformulate the token pruning problem with determinantal point process, which facilitates dynamic pruning by jointly considering feature similarity and instruction relevance.
3. We conduct extensive experiments on various vision- Saiionla benchmarks, demonstrating that CDPruner consistently achieves state-of-the-ars across different reduction ratios.
2 Related work
Multimodal large language models. The remarkable achievements of large language models
(LLMMs) Touvron et al. 2023a b. Jiang et al., [223]] Bai al. 2023 Yang al. 2024a Cai et al.
2024b] have lead to a growing trend of extending their powerful reasoning capabilities to other
modalities, eventually forming multimodal large language models (MLLMs) [Liu et al. 2023 Li
ctt al. 2024a| Wang et al. 22224] Bai et al., 2025 Chen et al. 2024c] Zhu et al., 2025 Liu et al.
2024c These models typically encode visual inputs tokens to fully leverage the capabilities
LLMs. However, the sparsity of visual signals results in a significantly larger number of visual
tokens compared their textual counterparts. For example, LLa [Liu et al. 2024a] converts 336 image into 576 tokens, while its high- high-resolution variant. LLa VA-VNNNTT [Liu et al. 2024b]. generates 2,880 tokens from an image with twice the resolution. In video understanding scenarios, LongVA VA (Zhang et al. [024a] transforms 2,000 frames into over 200K visual tokens, and Long VILA [Chen et al. 2024b can even handle up to 6,000 frames and produce an ultra-long input sequence of over 1M visual tokens, leading to enormous computational overhead. Therefore, achieving more efficient inference for MLLMs is becoming increasingly critical.
Visual token reduction. Reducing the number of input visual tokens is an effective way for MLLM
inference acceleration. Some works attempt to compress visual tokens via vision-text pre-fusion [Li
ct al. 2024d Hu et al. 2024b] Cai et al.] [024a] Zhang et al. [0255]] but these approaches require
architectural modifications and additional training, thereby increasing computational costs. Other
works adopt a training- free approach by removing redundant visual tokens during inference [Liu
et al. 2024d Yang al. 2025 Cao al.
2025 Ma et al. 2025 Zhang et al. 2025c], known as
token pruning. These methods can be broadly categorized into three groups.
The first group leverages text-visual attentions within the language model to assess the importance of visual tokens [Chen et al.] 2024a| Ye et al., 2025 Xing et al., 2024, /Zhng et al. 20246). However, as pointed out by Zhang al. [025aa] and Wen al. [2025a], such methods suffer from attention shift, which compromises pruning accuracy. Moreover, the reliance on attention scores makes them incompatible with efficient attention implementations like FlashAttention [Dao et al.
2022 The
second group avoids these issues by pruning before the language model (Shang et al. 2024 Yang et al. 2024b Song al.) 2024 ZZhan et al.. 2025a]. Nonetheless, these methods rely on specific visual encoder architectures and thus cannot be applied across different MLLMs. The third group directly prunes tokens based feature similarity among visual tokens [Wen et al.) 2025b Alvar et al., 2025 Jeddi et al., 2.25). However, like the second group, they fail to consider the relevance between visual tokens and user instructions during pruning. leading suboptimal performance. In this work, our CDPruner addresses all these challenges by jointly modeling feature similarity and instruction relevance through DPP, thereby ensuring both the diversity and quality of the retained token subset.
Determinantal point process. Determinantal Point Process (DPP) was first introduced describe the distribution of fermion systems in thermal equilibrium [Macchi 1975). where no two fermions can occupy the same quantum state, resulting an "anti-bunching effect that can be interpreted as diversity. Later, DPPs have been widely adopted in list-wise diversity modeling across various domains [Chen et al.) 2018 Celis et al.] 2018 Li et al. 2024c] [Sun et al.) 2025). Unlike Max- Min Diversity Problem (MMDP) [Porumbel et al. 2011], which also aims to maximize diversity, DPP emphasizes global diversity and typically yields more balanced and representative subset selections Kulesza et al. 2012). Traditional DPP focuses solely on feature similarity among samples. In this work, we extend this formulation by incorporating instruction relevance as condition, enabling unified consideration for superior visual token pruning performance in MLLMs.
3

---

Language Model f
x
3
4
x
X
CDPruner
Hg
H
DPP
Tokenizer
CDPruner
Language Instruction X H
Tokers
Projector
Text Tokens
Vision Encoder fo
Feature Similarity
Query Relevance
Input Image X
Figure 2: Overview of CDPruner. We first calculate the similarity between visual tokens conditioned on their relevance to the current instruction. Then, CDPruner uses a DPP to select the subset to keep. As training- training-free free and model-agnostic method, it ensures both the diversity and quality of the selected token subset, significantly reducing computational cost while maintaining considerable performance.
Method
In this section, we first review visual token pruning in MLLMs in Section 3.1
Then, we model the
feature similarity among visual tokens and their relevance to user instructions in Section 3.2 and Section 3.3 Finally, we present our CDPruner in Section 3.4 which maximizes the conditional diversity to obtain the optimal token subset. The overall design of CDPruner is shown in Figure 2
3.1 Visual token pruning
Existing MLLMs [Liu et al.) 2024a Wang et al. 2024 Chen et al. 2024c typically consist of three core components: vision encoder Jur multimodal projector g. and an LLM fa. The vision encoder encodes the input image X, into a sequence of visual tokens H, = (X,)) E R= whose length is significantly greater than that of their textual counterparts Hg. Visual token pruning aims to reduce the inference cost of MLLMs by decreasing the number of visual tokens:
H H.CH... |=m
arg min L (fo( H&),, S ( H,l)
Here, measures the discrepancy between the model outputs before and after visual token pruning, and m is the number of visual tokens retained (m < v)). Previous methods mainly rely on attention
scores for pruning [Chen al., 2024a Xing et al. 2024 Zhang et al., 2024b Shang al. 2024 Yang et al. 2022b)]. which often leads to sigmificant redundancy. Alvar et al. [2025] formulates the subset selection problem as a Max-Min Diversity Problem (MMDP) [Porumbel al. 2011. but this
approach overly focuses on extreme cases while neglecting global diversity.
3.2 DPP with token similarity
DPP was initially introduced model fermion repulsion in quantum physics Macchi 1975]. and has been widely applied in list-wise diversity modeling [Chen et al., 2018 Celis et al. 2018 Sun al. 2025). Formally, a DPP P on discrete set 7 = 2 2,...; n2 on the power set 22 When gives nonzero probability to the empty set, there exists positive semi-definite (PSD) kernel matrix L E RRxx indexed by elements of Z, such that for every subset
is probability measure defined
S C Z, the probability of sampling S is:
det (Ls) P(S) det (L I)
det (Ls),
where Lg is the principal submatrix of L corresponding to the subset S.
In the context of token pruning, we leverage DPP to model the diversity of the retained visual token subset. Given sequence of visual tokens H., the kernel matrix L is defined by the pairwise cosine

---

Is a bird the image?
image?
image?
knife imuge??
Figure 3: Visualization of relevance scores. We compute the relevance scores for several samples from the POPE benchmark using LLaVA-1.5-7B, with the instruction following the template: "Is there a [object) in the image?" Red indicates high relevance, while blue indicates low relevance.
similarity of visual features:
H H
Lig
(3)
| |H
According to the DPP sampling process, the optimal subset H. given by:
S arg max det (Ls), H. ( lee..
(4)
sC2.1
3.3 Instruction relevance
The above only considers the feature similarity among visual tokens, resulting in the same pruning result regardless of user instructions. We further introduce instruction relevance as a condition to achieve dynamic pruning. Given the visual embeddings H, E "x extracted from the input image and the text embeddings Hg E IRd derived from the user instruction, we calculate the cosine similarity measure the relevance r E R between each visual token and the instruction:
H H
H
[|
For MILMSs (Liu et al., 2023 2024b et al. 2024a that employ visual encoders paired with corresponding text encoders (e.g., CLIP [Radford et al. 202T)). D. we use features extracted from both as visual and text embeddings, respectively. For MLLMs [Bii al., 2025 Zhuu et al.) [025] only contain dedicated visual encoders, we instead use the output of the multimodal projector as the visual embeddings, and take the average of all token embeddings corresponding to the instruction from the language model the text embedding. For simplicity, we denote the visual and text embeddings obtained through both ways as H, and Hg. Figure 3 shows the relevance scores derived through the LLaVA-1.5-7B [Liu et al. 2024a for several samples from the POPE benchmark (Li et al. 2023
Furthermore, we apply min-max normalization to the obtained relevance scores to ensure the values are within the range of ( to 1:
min(r)
(6)
max(r) (ni(r))
3.4 CDPruner
Finally, we integrate feature similarity and instruction relevance for visual token pruning, leading to our proposed CDPruner, as shown in Figure Specifically, we modulate the original kernel matrix with the relevance scores to obtain new conditional kernel matrix:
L = diag (F) L diag (f).
The updated log-pro pegooabbbyilty of the subset S for DPP is:
log det (is)
> log (F?? log det (Ls).
(8)
5

---

Table 1: Performance comparison of different pruning methods on LLaVA-1.5-7B. Here, Acc. denotes the average performance across 10 benchmarks, Rel. represents the average percentage of per- formance maintained. Attention- based methods are shown with red background, attention& similarity- based methods with green background, and similarity based methods with blue background.
VQA
SQA tM
tha
MMB PN
MMB
MMVet
Upper Bound. 576 Tokens (T00%)
L1aVA-1.5-28
78.5
61.9
50.1
09.5
58.2
85.9
15066.5
64.7
58.1
31.3
63.4
100.0%
(7.8%%
FastV (R00V24)
69.2
364
68.2
1368.9
55.9
27.0
58.5
42.8%
"Drop(CVPE20)
74.3
57.1
49.4
70.1
56.7
77.5
1444.1
62.3
55.3
27.6
60.3
95.0%
PLarseVLMMIa
75.1
57.3
69.0
56.3
83.1
1399.3
62.6
56.9
29.7
61.0
96.3%
PruMerge+
1408.1
(COLTT20))
75.4
58.4
51.6
68.6
52.2
85.3
1413.4
63.0
52.3
29.9
60.7
8%
(ision2ip(crra22)
75.6
57.6
51.6
68.7
56.9
83.1
1436.9
62.1
57.0
31.6
61.6
97 .7
(9025
74.7
69.1
56.3
80.4
1408.7
60.7
57.3
30.9
61.1
96.
Div Prunc (OYPR25)
76.0
59.4
52.8
68.6
55.9
87.0
1405.1
61.5
54.8
30.6
61.7
516
CDPruner (Dana)
76.6
59.9
52.8
69.0
56.2
87.7
1451.4
63.1
55.0
32.8
62.5
99.0%
Retain
Tokens
88.888.. (1%))
(EUDV24)
50.1
18.9
(Drop(CVF226) (CVP226)
56.3
46.1
46.3
68.8
19.2
40.8
982.2
48.0
36.6
17.7
45.9
72.9%
66.9
$2.0
49.4
69.2
52.1
69.7
1 190.4
58.3
49.6
24.4
55.1
87. 57.1%
PruMerge+ (2024.05)
71.3
55.4
53.7
69.5
52.0
75.7
1316.8
59.6
52.1
28.0
58.3
92.4%
49.7
85.9
60.9
VisionZip(ccmaa5)
72.4
55.1
52.9
69.0
55.5
77.0
1365.2
60.1
55.4
29.4
59.5
44%
(2KT 02)
71.3
54.7
$3.5
69.3
54.7
73.8
1365,1
59.5
54.0
26.5
58.6
V2.65 6%
(Drru
75.4
58.6
53.4
68.1
55.3
87.5
1415,1
61.1
53.2
30.5
61.4
97.0%
Retain
Tokens
44.4%)
PruMerge+ (20rg, 06)
65.6
52.9
53.5
67.9
19.2
66.7
1236.6
55.1
45.9
24.7
54.3
$6.1%
(ROM(c0L
VisionZip(crraac)
67.1
51.8
52.4
69.1
53.1
69.4
1251.2
57.0
50.3
25.3
55.8
88.4%
(202
67.1
52.9
52.5
69.3
52.2
69.1
1273.3
58.5
50.0
25.0
56.0
58 6%
(OVPR2))
68.6
PDPruner(Oara)
09.5
53.2
87.9
59.6
49.6
27.8
We then obtain the optimal subset via MAP inference. Although MAP inference for DPP NP- hard, there exists a greedy algorithm with polynomial-time complexity that guarantees 1/e) approximation [Chen et al. 2018). By using Cholesky decomposition, the overall time complexity can be reduced to Onm The additional latency is negligible when < with less than 10ms per sample. The pseudocode for algorithm implementation is provided in the technical appendix.
4 Experiments
4.1 Experimental setup
Model architectures. We apply CDPruner to various MILM architectures, including the VA series such as LLLVA-1155 (Liu et al. 2024b for high-resolution inputs, and LLaVA Video [Zhang et al. 2024c] for video understanding,
2024a image understanding, I.LAVA-NeXT [L.iu al.)
as well as the current state-of-the--e ahe open- source model Qwen2.5 VL [Bai et al. 2025 Additional results more model architectures are provided in the technical appendix.
Evaluation benchmarks. We evaluate our method on 14 maageebaseed multimodal benchmarks, including 10 general VQA tasks such as VQAv2 [Goyal et al. 20 GQA [Hudson and Manning
2019], VizWiz [Guaai et al. 2018), ScienceQA-IMC [Lu et al.
2022]. HallBench [Guan et al., [0224] POPE [Li et al.) 2023), MME Fu et al., [024a], MMBench [Liu et al. 2025), MMBench-CN [Liu
et al. 2025 and MM- Vet Yu et al. 2023]. J. 4 text-oriented VQA tasks such as TextVQA [Singh
et al. 2019 ChartQ [Masry et al., 2022]. A12D [Kembhavi et al., 2016) and OCRBench [Liu et al.
2024c). and multi-turn dialog task MMDU [Lu et al.] 2024f]. We also conduct experiments on widely-used video understanding benchmarks, including MLVU [Zhou et al. 2024 MVBench [Li ct al. 2024b), Vide Wu et al. 2024] and Video-MME Fu et al. 2024b]. All experiments on these benchmarks follow the detault settings and evaluation metrics. Detailed descriptions of each task are provided in the technical appendix.
Comparison methods. We choose several recent works of different types comparsion methods, in- cluding attention based methods like FastV [Chen et al. 2024a], PrramidDrop [Xing et 2024 and Sparse VLM [Zhang et al.) 2024b), attentionkssimilarrtybbbar based methods like L.L.VA-Praii VA-Prunerge (Shang] jet al. 2024], TRIM [Song et al.) 2024) and VisionZip YYngg et al. 2024b) ) as well as similarity-based
methods like DART Wen et al. 2025b and DivPrune [Alvar et al. 2025

---

Table 2: Performance comparison of different pruning methods on LLaVA-NeXT-7B. Acc.
denotes the average performance across 10 benchmarks, Rel. represents the average percentage of per- formance maintained. Attention- based methods are shown with red background, attention& similarity- based methods with green background, and similarity -aase methods with blue background.
VQA
GQA
SQA RM
Toat
MME
MMB"N
MMB
Upper Bound All 2880 Tikons (1005))
LLavA-NeXT-78
81.3
62.5
55.2
61.5
60.3
86.8
1511.
66.8
57.3
40.0
65.2
100.0%
77.88
(7)
FastV (RO0V24)
77.0
67.4
58.1
79.5
1412.6
53.5
PDrop (CVP828)
79.1
60.0
53.8
66.7
57.8
83.8
1475.9
61.1
55.2
36.7
63.1
96.5%
PPanscVLMMIIoe
79.2
61.2
53.6
67.6
59.7
85.3
1456.8
65.9
58.6
36.
64.0
97.9%
PruMerge+
TRIM (CUL13025)
78.3
62.1
54.8
66.9
54,8
86.9
14718
66.8
55.8
37.8
63.8
97.65%
(C5F222
79.1
61.2
57.1
68.1
59.9
86.0
1493.4
65.8
58.1
38.9
64.9
99.5%
DARRT(2025.0
78.3
61.3
68.2
59.5
85.0
1450,2
61.9
57,1
36,9
61.1
98.2%
DivPrune
79.3
57.0
1469.7
57.3
CDPruner(Oara)
79.9
62.6
55.6
67.9
58.4
87.3
1474.5
66.3
57.5
41.9
65.1
100.1%
Retols
320 Thkrns
(ECOV24)
(
50.4
49.7
49.0
60.8
11711.5
55.5
SparscVLM (12222
74.5
$7.9
$1.2
67.2
56.5
76.9
1386.1
63.1
56.7
32.8
60.9
93.3%
PruMerge+ (ava4 06)
75.3
58.8
57.7
68.1
54.0
79.5
1444,3
63.0
55.6
31.4
61.6
94.0%
51.0
VissonZip(C/ree20
76.2
58.9
67.5
58.8
82.3
1397.1
63.3
55.6
35.8
62.4
95.7%
DART 02)
75.7
59.5
$6.8
67.5
57.6
$1.0
1419.5
64.2
55.7
35.7
62.5
95.8%
DivPrune (OVP325)
77.2
61.1
55.6
67.7
56.2
$4.7
1423.3
63.9
55.7
34,8
62.8
96.0%
(DPruner(Oarss)
87.2
Retain
160 Tokens
(44.4%)
PruMergex (2024.08)
20.5
56.2
51.2
66.9
50.3
71.1
1289.6
58.0
48.9
29.3
$7.3
87.7%
TRIM (COL118222
71.0
57.4
65.5
45.8
84.8
1275.8
61.6
45.2
57.8
87.7%
Vision/ip((CPPzz5)
71.4
55.2
55.5
67.9
55.0
74.9
1327.8
58.6
50.4
32.3
90.0%%
(202 02)
72.5
56.8
56.7
67.8
54,9
75.3
1325.1
62.0
53.6
32.2
59.8
91.7%
Div DivProncCee (OVPA25)
75.0
59.3
56.1
67.1
54.1
80.0
1356.6
62.9
53.7
32.0
60.8
92.9%%
(DPruner(Oars)
67.5
55.4
1425.3
53.8
4.2
Main results
We first apply CDPruner to LLaVA-1.5, which is widely adopted for evaluating visual token pruning strategies. Table Taalleppreseer presents the performance of different pruning methods on the LLaVA-1.5-7B model when retaining only 128, 64, or 32 visual tokens. With 77.8% tokens pruned, CDPruner remarkably maintains nearly all the original performance, surpassing VisionZip by 1.4%. When the number of visual tokens further decreases to 64, roughly one-tenth of the original token length, attention-baa pruning methods exhibit significant performance degradation of over 25% indicating that internal text-visual attention within the language model is not an ideal metric for pruning. Under the same reduction ratio, CDPruner only decreases the original performance by 3.4%. outperforming VisionZip and DivPrune by 2.6% and 2.3%, respectively. With only 5.6% of visual tokens retained, attention and similarity-based methods also encounter noticeable performance degradation because, despite selecting relatively important tokens, they include excessive redundancy and duplication. In this scenario, CDPruner still maintains 94.3% % of the original performance, significantly outperforming the best similarity-basee method, DivPrune, by 3 %, which fully demonstrates its effectiveness.
Among all 10 benchmarks, CDPruner achieves particularly strong performance POPE [Li et al. 2023), even exceeding the unprumed original VL-1-5 model. Since POPE is specifically designed to evaluate visual hallucination, this result suggests that appropriate pruming may help mitigate hallucination in MILMs,, which we believe a valuable direction for future research. On the other hand, CDPruner shows limited advantage VizWiz [Gurari et al.) 2018), primarily because questions in this benchmark often lack informative context (e.g.. "What is this?"), making them insufficiently effective conditional guidance for the DPP process.
4.3 CDPruner for high resolution inputs
Increasing the resolution of input images can improve the performance of MLLMs, but this improve- ment comes with substantial computational overhead. Higher resolutions introduce more visual tokens, inherently increasing redundancy and thus making it more suitable for pruning. To evaluate this, we apply CDPruner LLaVA-NeXT, a model specifically designed for handling high-resolution inputs. To ensure fair comparison by controlling the number of visual tokens, we fix the input resolution to x722, resulting 2,880 visual tokens. As shown in Table with 77.8% of tokens pruned, CDPruner maintains performance comparable to, slightly better than, the original LLa VA-
7

---

Table Performance comparison of different pruning methods on LLLVA-Video-77 with 64 frames per video. Here, Acc. denotes the average accuracy across 4 video-based benchmarks, and Rel. represents the average percentage of performance maintained. Attention- methods are shown with red background, and similarity-
methods are shown with blue background.
Method Metric
MLVU
MVBench test
Long VideaBench
Video-MME
m-ayg
val
perception
relation
w/o sub
short medium
long
Acc.
Rel.
Upper Bownd. All
169 (100%)
Vi
67.7
58.2
59.0
65.0
53.8
63.6
76.6
61.2
53.1
62.1
100.0%
Retain
x Tokens (1 62.1%))
FastV (BCCV24)
63.9
55.8
56.1
60,6
52.1
61.9
73.6
59.3
52.7
59.4
95.7%
PDrop(cPP225)
64,9
56.9
56.9
62.2
52.2
62.5
74.1
60.7
52.7
60.3
SparseVLM VLM(ICLL25)
65,5
56.8
56.0
61.0
51.7
61.0
73.0
58.8
51.2
59.8
96.3%
DAKT .22)
64.1
55.5
57.5
62.1
53.5
61.6
73.0
59.9
51.9
59.7
96.1%
DisPrune (CXP122))
64.1
55.1
58.6
64.2
53.7
61.1
72.9
59.3
51.2
59.7
96.2%
CDPruner (Ours)
66.3
57.4
$8.7
64,5
53.7
62.6
74.6
60.3
52,9
61.3
98.6%
Retain
FastV (BOCV24)
58.5
52.7
52.4
$7.0
48.5
56.0
63.8
55.9
48.4
$4.9
88.5%
PDrop(CVP125)
59.7
53.1
52.4
48.6
58.2
67.0
50.0
89.9%
SparseVLMMc (1C8L22))
60.7
54.1
53.7
$8.1
49.9
59.0
69.8
56.9
50.3
56.9
91.6%
DART
52.7
54.1
58.1
67.3
91.0%
(C772222))
61.5
53,7
56.4
62.1
51.4
59.3
69.9
57.9
50.2
57.7
93.09
CDPruner (Dura)
63.0
55.7
56.5
52.7
Retoin
64 x 16 Tokens (U 90.5%)
FastV (BCCV24)
52.8
46.7
46.6
48.8
44.7
50.0
55.0
50.0
45.0
49.0
79.0%
(Drop(cvPR26)
52.8
44.3
44.3
47.5
41.4
48.9
52.9
50.0
43.8
47.6
76.5%
SparseV1.M (ICNL2)
52.0
48.7
47.6
53.0
42.8
49.8
53.8
49.3
46.3
49.5
79.9%
DART
56.7
50.4
51.8
56.8
47.5
55.3
64.8
52.9
48.1
53.6
86.3%
DisPrume
58.6
52.0
52.1
$7.6
47.2
56.7
67.7
54.2
48.2
54.9
88.3%
CDPruner (Durs)
58.9
53.8
52.7
57.4
48.5
57.3
66.2
56.0
49.6
55.7
89.7%
NeXT, demonstrating the higher visual redundancy in high-resolution scenarios. As the reduction ratio further increases to 88.9% and 94.4%, CDPruner still retains up to 98% and 96% of the original performance, outperforming the second-best DivPruner by 2% and 3.1% respectively. These results highlight the strong effectiveness of CDPruner in high-resolution contexts.
4.4 CDPruner for video understanding
Video understanding is another task with high visual redundancy. To validate CDPruner in such a scenario, we apply it to LLaVA-Video, video frames to 64, each with a resolution of 384x38, resulting in over 10k tokens and considerable visual redundancy. As demonstrated in Table 3 with 62.1 % of visual tokens pruned, CDPruner maintains 98.6% % of the original performance, outperforming PDrop by 1.5%. As the reduction ratio increases 81.1%. CDPruner still preserves 95 % performance, significantly exceeding DivPrune's 93%. Furthermore, when only 16 visual tokens are retained per frame, text-based methods exhibit substantial performance degradation, while CDPruner able to maintain 89.7% performance, showing substantial 10% improvement over Sparse VLM. These results adequately demonstrate the effectiveness of CDPruner in video understanding applications.
advanced video MILMM. We set the maximum number of
4.5 CDPruner for advanced architectures
In addition the LLaVA series, we further apply CDPruner to the most advanced open-source MLLM architectures to validate its generalizability. Here, we select Qwen2.5-Vi as a representative model, with the input resolution fixed at 1008 1008, yielding 1,296 visual tokens. Due to the unique structure of its visual encoder and multimodal projector, pruning methods that require the [cls] token are no longer applicable. Therefore, we compare CDPruner only against representative methods from the other two categories, attention-base FastV and similarity-based DivPrune, with results summarized in Table Compared the LLaVA series, Qwen2.5-VL. exhibits more noticeable performance drop after pruning. This is because visual tokens are already compressed within its projector. Nevertheless, CDPruner consistently outperforms other methods under the same reduction ratios. With 60.5% and 80.2% of tokens pruned, CDPruner retains 97.5% % and 92.8% of the original performance, surpassing the second-best V by 0.5% and 2.0 2.0% respectively. When only 128 visual tokens remained, competing methods suffer from severe performance degradation. In contrast, CDPruner maintains 85.2% of the original performance, significantly higher than DivPrune's 79.9%. demonstrating the strong generalizability of CDPruner on advanced MLLM architectures.

---

Table 4: Performance comparison of different pruning methods on Qwen2.5-VL-77. Acc.
denotes the average accuracy, Rel. represents the average percentage of performance maintained. methods with blue.
Attention based methods are shown with red background, and similarity
Method
TexTVQA
ChartQa
AIED
OCRBench HallBench
MME
MMB-EN
MMB-CN
Acc.
Rel.
Upper Bound, All 1296 Tokens (100%)
Qwen2.5-V1-78
84.8
86.1
80.4
863
46.8
2304
8288
83.2
83.2
100.0%
Retain 512 Tokeus (1 60.5%)
FustV (BCCV24)
84.1
82.2
78.8
815
42,4
2317
82.0
81.8
81.1
97.0%
81.8
79.6
78.6
800
43,3
2279
81.6
82.1
80.1
96.0%
CDPruner (Ours)
84.2
82.8
78.9
827
42,5
2327
82.2
82.6
81.5
97.5%
() 80.2%)
(BCCV24)
81.5
70.9
76.2
703
39.0
2238
79.6
78.9
76.0
90.8%
DivPrune
76.5
692
36.4
80.0
79.6
74.0
88.2%
CDPruner (Oure)
82.4
73.0
77.5
749
40.1
2245
80.9
79.9
77.6
92.8% %
Renui 128 Tokous
90.11%
Fast V (BCDV24)
73.8
52.2
71.4
531
33.8
2008
72.9
72.2
66.2
79.0%
DivPrune (CVP826)
67.0
50.4
72.1
549
32.6
2108
77.8
77.8
67.3
79.9%
CDPraner (Oure)
77.8
59.2
74.0
632
37.2
2127
76.2
76.5
71.3
85.2%
Table Performance comparison of different pruning methods on multi-turn dialogues. Acc. denotes the average accuracy, Rel. represents the average percentage of performance maintained. GPT-4o used for evaluation from six dimensions: Creativity (C), Richness (R). Visual Perception (VP), Logical Coherence (L.C), Answer Accuracy (AA), Image Relationship Understanding (IRU).
Method
C
LC
AA
IRU
Acc.
Rel.
Upper Bound, All 576 Tokens (100%)
LLaVA-1.5-BB
34,8
32.7
39.4
65.3
47.4
39.5
42.9
100.0%
Retain 128 Tokens 77.8%)
(CIM
35.7
34.2
38.7
64.6
46.8
39.2
42.8
99.8%
CDPruner (Gurs)
36.2
34.9
40.0
66.2
48.0
40.8
44.0
102.6% %
Retain 64 Tokens (1 88.9%)
(CLL
35.6
34.1
37.1
63.8
44.8
37.7
41.7
97.2%
CDPruner
36.1
34.4
38.6
64.5
46.2
39.0
42.8
99.8%
Retain 32
Tokens (L 94.4%)
(RIM(COLING25)
35.4
34.0
36.1
62.8
44.2
36.9
41.2
96.0%
CDPruner 35.6
34.0
36.7
62.9
44.6
38.0
41.5
96.7%
4.6 CDPruner for multi-turn dialog understanding
Multi-turn dialog understanding presents major challenge for visual token pruning. In single-turn scenarios, pruning methods only need to retain visual tokens most relevant to the current question. In contrast, multi-turn dialogues require preserving more holistic visual semantics avoid losing information that may be crucial for answering future questions. To evaluate our proposed method, we adopt the MMDU benchmark (Liu et al. 2024f], which includes both multi-turn and multi-image dialogues, and compare our CDPruner against TRIM, which prunes tokens solely based on their relevance the current query. Both pruning methods are applied to LLa model for evaluation. To prevent dialogues from exceeding LLaVA's context length limit, we select a subset of 100 samples from MMDU, each containing five dialogue turns and no more than 12 images. GPT-4o is used for evaluation across the six dimensions defined in the original paper.
As shown in Table CDPruner consistently outperforms TRIM across various reduction ratios, demonstrating its superior adaptability to multi-turn dialogues. With 77.8% of visual tokens removed, CDPruner achieves better performance than the LLaVA-1.5 baseline. Even when 88.9% of visual tokens are pruned, it maintains 99.8% % of the original performance, surpassing TRIM by 2.6% When only 32 tokens per image are retained, CDPruner still preserves 96.7% of the original performance. TRIM prunes tokens based on their relevance to the current instruction, which is suboptimal for multi-turn dialogues. If the subsequent question differs significantly from the previous one, the retained tokens may no longer be relevant, resulting in degraded performance, In contrast, CDPruner incorporates diversity modeling via DPP, which enables to preserve more informative and compre- hensive visual content while still considering relevance. As a result, it maintains better performance even in multi-turn scenarios. Exploring a visual token pruning framework that can effectively handle both single- and multi-turn dialogues is valuable research direction, which we leave for future work.

---

Table Efficiency analysis of different pruning methods on LLaVA-NeXT-77B The performance is evaluated on POPE. Attention- based methods are shown with red background, attention&ssimilarity
based methods with green background. and similarity- methods with blue background.
FLOPs
Prefill Time
Decode Time
KV Cache
GPU Memory
Score (FI)
Method
Token
(T)
(ms/token)
(ms/token)
(MB)
(GB)
LLaVA-NeXT-7B
2880
41.7
246
29
1440.0
16.7
86.8
(BCCV24)
320
4.4 (x9.5)
54 (x4.6)
23
(x 2
160.3
15.6
49.5
(CVop(CV))25)
320
4.5 (x9.3)
(x4.5)
24 (x
160.2
15.6
60.8
Sparse (LCCM22)
(x9.3)
(x3.5)
25 (x11))
161.2
76.9
VisionZip(CVPRRC
320
4.2 (x9.9)
38 (x6.6)
22 (x1.3)
160,0
14.8
82.3
DivPrune (CVPR26)
320
4.2 (x9.9)
38 (x6.6)
22 (x1.3)
160.0
13.8
84.7
CDPruner (Burs)
320
4.2 (x9.9)
38 (xx6.6
22 (x1.3)
160.0
13.8
87.2
VO4V2
G0A
POPS
MMVet
El StoiE (N)
ScoSoo (NN)
Aecrrey
Contea Dffraa
Cithene
Coieee Offraac
Visual Sumbher
10
230
36 Tolen
15
20
130
Token
3M0
Figure 4: Ablation study of CDPruner design. DPPruner denotes applying DPP to visual token pruning without conditioning on instruction relevance, as degraded variant of CDPruner.
4.7 Efficiency analysis
To demonstrate the efficiency of CDPruner, we conduct comparative analysis against other pruning methods in terms of FL FLOPs, CUDA latency, KV cache, and GPU memory on the high-resolution MILMM VALaVA-NeeT-- We choose POPE for evaluating inference efficiency, it contains questions of similar length and happens to contain only one prefill and one decode stage. As shown Table 6 when the number of visual tokens is reduced from 2,880 to 320, CDPruner achieves nearly a x 10 reduction in FLOPs. Regarding CUDA latency, CDPruner reduces the time for prefill and decode stages by x6.6 and 1.3, respectively, significantly improving real-worle inference efficiency. In addition to runtime latency, CDPruner also reduces KV cache and GPU memory. Compared to all other pruning methods, CDPruner consistently achieves the best efficiency while maintaining the highest performance.
All experiments are performed on a single NVIDIA A A100-80GB GPU.
4.8
Ablation study
We further conduct an ablation on the design of CDPruner, as illustrated Figure We compare the performance of different pruning strategies on LLaVA-1.5-77 across four benchmarks, under varying numbers visual tokens. Here, DPPruner refers to a variant that directly applies DPP to visual token pruning without any condition. This version consistently outperforms DivPrune, demonstrating that the global modeling
token diversity via DPP more effective than MMDP. When instruction relevance is further incorporated condition. CDPruner achieves additional performance gains, validating the benefit of jointly modeling feature similarity and instruction relevance.
5 Conclusion
In this paper, we introduce a novel training-free visual token pruning method CDPruner, for MLLM inference acceleration. Specifically, it first defines the conditional similarity between visual tokens based on the instruction, and then reformulates the token pruning problem with DPP to maximize the conditional diversity of the selected subset. Extensive experiments on diverse image and video benchmarks demonstrate that CDPruner achieves state-of-the-art performance across various MLLM architectures, including the LLaVA series and the advanced Qwen2.5-VV Efficiency analysis further shows that CDPruner significantly reduces inference latency and memory usage while maintaining competitive performance, facilitating the practical deployment of MLLMs in real-world applications.
10

---

Acknowledgements
This work was supported by the National Natural Science Foundation of China (62476011) 1), and by the National Science and Technology Major Project (No. 00222D00118001
References
Saced Ranjbar Alvar, Gursimran Singh, Mohammad Akbari, and Yong Zhang. Divprune: Diversity-based visual token pruning for large multimodal models. arXiv preprint arXiv:2503.02175, 2025.
Ruichuan An, Kai Zeng, Ming Lu, Sihan Yang. Renrui Zhang. Huitong Ji, Qizhe Zhang, Yulin Luo, Hao Liang, and Wentao Zhang. Concept-as-tree: Synthetic data is all you need for vlm personalization. arXiv preprint arXiv:2503. /2999, 2025.
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEFE international conference on computer vision, pages 2425-2433, 2015.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Huang, et al. Qwen technical report. arXiv preprint vrXiv:2309.16609 2023.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Siho Song. Kai Dang, Peng Wang, Shijie Wang, Jun Tang, al. Qwen2.5-vl technical report. arXiv preprint arXiv:2502. 13992,, 2025.
Mu Cai, Jianwei Yang. Jianfeng Gao, and Yong Jac Lec. Matryoshka multimodal models. In Workshop Video-Language Models@ NeurlPS 2024, 2024a.
Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, et Internlm2 technical report. arXiv preprint arXiv:2403. 100003177999 17297, 2024b.
Jiajun Cao, Qizhe Zhang. Peidong Jia, Xuhui Zhao, Bo Lan, Xiaoan Zhang, Xiaobao Wei, Sixiang Chen, Zhuo Li, Yang Wang, et al. Fastdrivevla: Efficient end-to-end driving via plug-and-pla reconstruction-bot based token pruning, arXiv preprint arXiv: 233318 2025.
Elisa Celis, Vijay Keswani, Damian Straszak, Amit Deshpande, Tarun Kathuria, and Nisheeth Vishnoi. Fair and diverse dpp-hased data summarization. In International conference on machine learning, pages 716-725.
PMIR, 2018
Laming Chen, Guoxin Zhang, and Eric Zhou. Fast greedy map inference for determinantal point process to improve recommendation diversity. Advances in Neural Information Processing Systems, 31, 2018.
Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baohao Chang. An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models. In European Conference on Computer Vision, pages 19-35. Springer, 2024a.
Yukang Chen, Fuzhao Xue, Dacheng Li, Qinghao Hu, Ligeng Zhu, Xiuyu Li, Yunhao Fang, Haotian Tang, Shang Yang, Zhijian Liu, et al. Longvila: Scaling long-context visual language models for long videos. arXiv preprint arXiv:2408. 10188, 2024b.
Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling, arXiv preprint arXiv:2412.05271, 2024c.
Zhe Chen, Jiannan Wu, Wenhai Wang. Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning generic visual-linguistic tasks. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 25185-24199 2024d.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL https://lmsys org/ org/blog// blog/
Tri Dao. Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattentio: Fast and memory-efficien exact attention with io awareness. Advances in Neural Information Processing Systems, 15:16944-16339, 2022.

---

Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang. Xiawu Zheng. Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. Mme: A comprehensive evaluation benchmark for multimodal large language models, 2024a.
Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang. Chenyu Zhou, Yunhang Shen, Mengdan 7hang, et al. Video-mme: The first-ever comprehensive evaluation henchmark of multi-modal Ilms in video analysis. arXiv preprint arXiv:2405..1075. 2024b.
Yash Goyal, Tejas Khot, Douglas Summers-
SSaay Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6904-6913, 2017.
Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen,
Furong Huang, Yaser Yacoob, et al. Hallusionbench: an advanced diagnostic suite for entangled language hallucination and visual illusion large vision- -angugge models. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 143355, 2024.
Zonghao Guo, Ruyi Xu, Yuan Yao, Junbo Cui, Zanlin Ni, Chunjiang Ge, Tat-Seng Chua. Zhiyuan Liu, and Gao Huang. Llava-uhd: an Imm perceiving any aspect ratio and high-resolution images. In European Conference on Computer Vision. pages 490-406.60 Springer, 2024.
Danna Gurari, Qing Li, Abigale J Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3608-3617, 2018.
Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang. Yuxiang Huang, Weilin Zhao, et al. Minicpm: Unveiling the potential of small language models with scalable training strategies. arXiv preprint (00i:::2044063955 2024a.
Wenbo Hu, Zi-Yi Dou, Liunian Li, Amita Kamath, Nanyun Peng, and Kai-Wei Chang. Matryoshka query transformer for large vision-language models. Advances in Neural Information Processing Systems, 37: 50168-50188, 88, 2024b.
Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 6700-6709, 2019.
Ahmadreza Jeddi, Negin Baghbanzadeh, Flham Dolatabadi, and Babak Taati. Similarity-aware token pruning: Your vim but faster. arXiv preprint arXiv:2503.1154, 2025.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, al. Mistral 7b. arXiv preprint arXiv:2310.0663 2023.
Byungkon Kang. Fast determinantal point process sampling with application to clustering, Advances in Neural Information Processing Systems, 26, 2013.
Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Sco, Hannaneh Hajishirzi, and Ali Farhadi. diagram is worth a dazen images. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11 11-4, 2016, Proceedings, Part IV 14, pages 235-251. Springer, 2016.
Ivan Krasin, Tom Duerig, Neil Alldrin, Vittorio Ferrari, Sami Abu-El-Haija, Alina Kuznetsova, Hassan Rom, Jasper Uijlings, Stefan Popov, Andreas Veit, et al. Openimages: A public datasct for large-scale multi-label and multi-class image classification. Dataset available from httpps//gghhab..oooopnnnneee 2017.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David Shamma. et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International jourrna of computer vision, 123:32-73, 2017.
Alex Kulesza, Ben Taskar, al. Determinantal point processes for machine learning. Foundations and Trends@ in Machine Learning. 5(2 (22-3)223-2221 2012.
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, ct al. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03:3 2024a.
Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang. Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. Mybench: A comprehensive multi modal video understanding benchmark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22195-22200 2024b.
12

---

Xueqi Li, Gao Cong, Guoqing Xiao, Yang Xu, Wenjun Jiang. and Kenli Li. On evaluation metrics for diversity- enhanced recommenddtions. In Proceedings of the 33rd ACM International Conference on Information and Knowledge Management, pages 1286-1295, 2024c.
Yanwei Li, Chengyao Wang, and Jiaya Jia. Llama-vid: An image worth 2 tokens in large language models, In European Conference on Computer Vision. pages 323-340. Springer, 2024d.
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucina- tion in large vision-language models. arXiv preprint arXiv:2305. 10355, 2023.
Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning. Peng Jin, and Li Yuan. Video- Ilava: Learning united visual representation by alignment before projection. arXiv preprint axX1v2331.10112, 2023.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Fision-ECCy 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740-755. Springer, 2014.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jac Lee. Visual instruction tuning. Advances in neural information processing systems, 36:34892 34916, 2023.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296-26306, 2024a.
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024b. httL:/tttte//laaaalll.ggtt io/ blog/ 2024- 01 30- 1lava- -nxt//
Mengzhen Liu, Mengyu Wang, Henghui Ding, Yilong Xu, Yao 7hao, and Yunchao Wei. Segment anything with precise interaction. In Proceedings of the 32nd ACM International Conference on Multimedia, pages 3790-3799, 2024c.
Ting Liu, Liangtao Shi, Richang Hong, Yue Hu, Quanjun Yin, and Linfeng Zhang. Multi-stage vision token dropping: Towards efficient multimodal large language model. arXiv preprint arXiv:2411.J0S00 2024d.
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangho Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? In European Conference on Computer Vision, pages 21 216-233. Springer, 2025.
Yuliang Liu, Zhang Li. Mingxin Huang, Biao Yang, Wenwen Yu, Chunyuan Li, Xu- Xu-Cheng Yin, Cheng- Liu, Lianwen Jin, and Xiang Bai. Ocrbench: the hidden mystery of ocr in large multimodal models, Science China Information Sciences, 6111)22201 2024e.
Ziyu Liu, Tao Chu, Yuhang Zang, Xilin Wei, Xiaoyi Dong. Pan Zhang. Zijian Liang, Yuanjun Xiong, Yu Qiao, Dahua Lin, et al. Mmdu: A multi-ture multi-image dialog understanding benchmark and instruction-tuning datasct for Ivims. Advances in Neural Information Processing Systems, 37:8698-877 8873,, 2024f.
Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances Neural Information Processing Systems, 35:2507- 2521, 2022.
Gen Luo, Yiyi Zhou, Yuxin Zhang. Xiawu Zheng, Xiaoshuai Sun, and Rongrong Ji. Feast your eyes: Mixture- of-resolution adaptation for multimodal large language models. arXiv preprint arXiv:2403.0300.. 2024.
Junpeng Ma, Qizhe Zhang, Ming Lu, Zhibin Wang, Qiang Zhou, Jun Song, and Shanghang Zhang. Mmg- vid: Maximizing marginal gains at segment-level and token-level for efficient video llms. arXiv preprint arXiv:2508.21044 2025.
Odile Macchi. The coincidence approach stochastic point processes. Advances in Applied Probahility, 7(1): 122, 1975.
Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: A benchmark for question answering about charts with visual and logical reasoning. arXiv preprint arXiv:2203. 10244, 2022.
Daniel Cosmin Porumbel, Jin Kao Hao, and Fred Glover. A simple and effective algorithm for the maxmin diversity problem. Annals of Operations Research, 186:275-295 2011.
13

---

Alec Radford, Jong Wook Kim. Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry. Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 9748-8763. PmLR, 2021.
Yuzhang Shang. Mu Cai, Bingxin Xu, Yong Jae Lee, and Yan Yan. -lava-prumerge Adaptive token reduction for efficient large multimodal models. arXiv preprint arXiv:2403. 15388, 2024.
Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8317-8326, 2019.
Dingjie Song, Wenjun Wang. Shumian Chen, Xidong Wang, Michael Guan, and Benyou Wang. Less is more: simple yet effective token reduction method for efficient multi-modal Ilms. arXiv preprint arXiv:2409. 10994, 2024.
Hui Sun, Shiyin Lu, Huanyu Wang, Qing-Gur Guo Chen, Zhao Xu. Weihua Luo. Kaifu Zhang. and Ming Mdp3: A training-fre free approach for list-wise frame selection in video-llms. arXiv preprint arXivv2501.0288 2025.
Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju. Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini research and technology- arXiv preprint arXiv:2403.0888 08295, 2024.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint 2302 / 13971, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.092288 2023b.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Peng Wang, Shuai Bai, Siman Tan, Shijic Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. arXiv preprint arXiv:2409. 1211, 2024.
Zichen Wen, Yifeng Gao, Weijia Li. Conghui He, and Linfeng Zhang. Token pruning in multimodal large language models: Are solving the right problem? arXiv preprint 250.::25022.1011,, 2025a.
Zichen Wen, Yifeng Gao, Shaobo Wang, Junyuan Zhang. Qintong Zhang. Weijia Li. Conghui He, and Linfeng Zhang. Stop looking for important tokens in multimodal language models: Duplication matters more. arXiv preprint arXiv:2502. 11494 2025b.
Haoning Wu, Dongxu Li, Bei Chen, and Junnan Li. Longvideobench: A benchmark for long long-context interleaved video-language understanding. Advances in Neural Information Processing Systems, 37:28828 -28277, 2024.
Long Xing, Qidong Huang, Xiaoyi Dong, Jiajie Lu, Pan Zhang, Yuhang Zang, Yuhang Cao, Conghui Hc, Jiaqi Wang, Feng Wu, al. Pyramiddrop: Accelerating your large vision-language models via pyramid visual redundancy reduction. arXiv preprint arXiv:2410.17247 2024.
An Yang, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng. Bowen Yu, Chengyuan Li. Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2.5 technical report. arXiv preprint arXiv:2412.55115. 2024a.
Cheng Yang. Yang Sui, Jinqi Xiao, Lingyi Huang, Yu Gong, Chendi Li, Jinghua Yan, Yu Bai, Ponnuswamy Sadayappan, Xia Hu, al. Topv: Compatible token pruning with inference time optimization for fast and low- memory multimodal vision language model. In Proceedings of the Computer Vision and Patern Recognition Conference, pages 19803-19813, 13, 2025.
Senqiao Yang, Yukang Chen, Zhuotao Tian, Chengyao Wang, Jingyao Li, Bei Yu, and Jiaya Jia, Visionzip: Longer is better but not necessary in vision language models. arXiv preprint arXiv:2412.04467, 2024b.
Weihan Ye, Qiong Wu, Wenhao Lin, and Viyi Zhou. Fit and prune: Fast and training- free visual token pruning for multi-modal large language models. In Proceedings of the AAAI Conference on Artificial Intelligence. volume 39, pages 22128-211 20-8222136...2 2025.
Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang. Kevin Lin, Zicheng Liu, Xinchao Wang, and Li- juan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308.02444 02490, 2023.
14

---

Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre- training. Proceedings the IEEE/CVF international conference on computer vision, pages 11975-11986. 2023.
Hang Zhang, Xin Li, and Lidong Bing. Video-Ilama: An instruction-tuie tunde audio-visual language model for video understanding. arXiv preprint N.N::2200.00888, 2023.
Peiyuan Zhang, Kaichen Zhang, Bo Li, Guangtao Zeng. Jingkang Yang, Yuanhan Zhang. Ziyue Wang, Hao- ran Tan, Chunyuan and Ziwei Liu. Long context transfer from language to vision. arXiv preprint arXiv:2406.16880 2024a.
Qizhe Zhang, Aosong Cheng, Ming Lu, Renrui Zhang. Zhiyong Zhuo, Jiajun Cao, Shaobo Guo, Qi She, and Shanghang Zhang, Beyond text-visual attention: Exploiting visual cues for effective token pruning in vlms. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 20857-20867, 2025a.
Shaolei Zhang, Qingkai Fang. Zhe Yang, and Yang Feng. Llava- mini: Efficient image and video large multimodal models with one vision token, arXiv preprint arXiv:2501 25ti:25500000855. 2025b.
Yuan Zhang, Chun-Kai Fan, Junpeng Ma, Wenzhao Zheng. Tao Huang, Kuan Cheng, Denis Gudovskiy, Tomoyuki Okuno, Yohei Nakata, Kurt Keutzer, et al. Sparsevlm: Visual token sparsification for efficient vision-language model inference. arXiv preprint arXiv:2410.04417. 2024b.
Yuan Zhang, Junpeng Ma. Qizhe Zhang. Chun-Kai Kai Fan, Wenzhao Zheng, Kuan Cheng, Jiwen Lu, and Shanghang Zhang. Sparsevlm+: Visual token sparsification with improved text-visual attention pattern. 2025c.
Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. Video instruction tuning with synthetic data. arXiv preprint 241002210.. 2024c.
Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Shitao Xiao, Xi Yang, Yongping Xiong, Bo Zhang, Tiejun Huang, and Zheng Liu. Mlvu: A comprchensive benchmark for multi-task long vidco understanding. arXiv preprint arXiv:2406.0 04264, 2024.
Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu. Yuchen Duan, Hao Tian, Weijie Su, Jie Shao, Intervl3:: Exploring advanced training and test- eest-time recipes for open-source multimodal models, arXiv preprint Xixv:22004. 10479, 2025.
15

---

Beyond Attention or Similarity: Maximizing Conditional Diversity for Token Pruning in MLLMs
Appendix
Section A describes the fast greedy MAP inference algorithm used in this work, along with the corresponding pseudocode. Section provides some details of the experimental setup, including information about model architectures, evaluation benchmarks, comparison methods and implemen- tation. Sections C and present additional experimental and visualization results respectively. And sections and discuss the limitations and broader impacts of this work.
A Fast greedy MAP inference algorithm for DPP
The direct MAP inference for DPP is NP-hard. Therefore, we adopt the fast greedy algorithm proposed by Chen et al. [2018). Specifically, given the kernel matrix L E ą2*x" indexed by elements of Z and the current selected subset S Z, the next index j to be added in the iteration satisfies:
arg max log det (Lsu(i)) - log det (Ls).
(9)
Since L is PSD matrix, all of its principal minors are also PSD. Suppose det(Ls) > and the Cholesky decomposition Lg VVT, where E |[SSs(S| is an invertible lower triangular matrix. For any i E Z \ S, the Cholesky decomposition of Lsufs can be derived as:
Ls
Ls
V C
V
0 d:
[su(i) =
Li,s
Li
=
d
C
(10)
where the row vector E R1 and scalar d; 2 0 satisfies:
Vcl Ls.s
(11)
d L, - || 112.
(12)
In addition, according to eq. det dct(L))()) = det(VV
(T0), it can be derived that
det(Ls)
(13)
Therefore, eq. p is equivalent to
arg maxlog (Ls) d)) - log (det(Ls)) arg max log (d)) i
(14)
iE
Once eq. (14) is solved, the Cholesky factor of Ls can be efficiently updated after new item is added S. For each item c; and d, can updated incrementally. Define d' and d' as the new vector and scalar of i E Z (S U (j}). According to eq. (10) and eq. (11) we have
V
0
T = LS =
Ls.s L
C,
d
c
(15)
Combining eq. (15) with eq.
111
get
c [c,
(Liz - (e,, ci))/d/]== ic; ei].
(16)
And eq. 12 implies
d.2 - - ||c = Li - ||c; - - dc of.
(17)
Initially, S 0, and eq. Algorithm [ In the k-th iteration, for each item i E Z \ S, updating e; and d, involve the inner product of two vectors of length k resulting in overall complexity Okk)). Therefore, the greedy algorithm runs (nm² time. After parallelizing the for-loop over i using CUDA, the additional inference latency introduced for each sample can be reduced to less than 10ms, which is negligible.
13 implies d? - det L.:) = Lis- The complete algorithm is shown in
16

---

Algorithm Fast greedy MAP inference algorithm for DPP
Input: kemel matrix L, index list Z, retained size m Output: selected subset S
I: C: = [. de Lis
2: j = arg : log(d?), {j}
3: while S < m do
4: for E Z \ do
5:
(cj. /d) C; [c ei), di
= مز
6:
= di e
7:
end for
8:
j arg 2 S=
9:
end while
10: return
B Details of experimental setup
B.1 Model architectures
B.1.1 LLaVA series
VA-155 [Luu et al. 2024a] The LLaVA series is one of the most widely used open-source vision-language models, and its simple design, low training cost, and outstanding performance have made it cornerstone in the field of open-source MLLMs. Specifically, the original LLa adopts a pretrained CLIP [Rafford al., 2021] as the visual encoder and Vicuna [Chiang et al., 2023] as the language model. A simple linear projector connects these two modules, enabling the LLM to accept image grid features as input. And the visual instruction tuning enables LLaVA to handle tasks. Compared to the original version, LLa updates the linear connector with an MLP, increases the input image resolution, and incorporates a broader set of instruction tuning data, resulting in significant performance improvements. The model processes image inputs at a resolution of 336x336, leading to 576 visual tokens per image.
LLaVA-NeXT [Liu et al. 2024b] To further enhance the visual perception capabilities of the model, LexV also known as VL. adopts a dynamic resolution design increase the input image resolution. Specifically, the model selects the optimal aspect ratio based on the original resolution of the input image, increasing the resolution by up to 4x. Without replacing the visual encoder, LLaVA-NeXT splits the high-resolution image into several sub-images of the same size as the original image. These sub-images are individually encoded and then concatenated as input to the LLM,, leading to improved performance in reasoning, OCR, and world knowledge. To ensure fair comparison by controlling the number of visual tokens, we fix the input resolution to 672x672, 4x the original resolution, resulting in totally 2,880 visual tokens.
LLa VidVieo [Zhang al., 2024c variant in the LLaVA series specifically designed for video understanding tasks. It introduces the SlowFast representation to balance the number of video frames and the count of visual tokens. The model employs SigLIP [Zhai et al.) [023] as the visual encoder and accepts video inputs with a resolution of 384 x 384, encoding each frame into 729 visual tokens. To further reduce computational cost, LLaVA- -ideo applies 2x2 average pooling to the grid visual features, reducing the number of visual tokens by 4x. During evaluation, we sample 64 frames per video, resulting a total of 10,816 visual tokens.
B.1.2 Advanced architectures
Qwen2.5-VL (aai al. 2025 The most advanced model of the Qwen-VL series. Building upon its predecessor, Qwen2-Vi VI. it introduces significant enhancements in visual understanding, document parsing, and video comprehension. The model employs a redesigned Vision Transformer architecture with window attention, SwiGLU activation, and RMSNorm, aligning with the Qwen2.5 language model structure. Notably, it supports dynamic resolution and frame rate processing, enabling the comprehension of videos up to an hour long with precise event localization. (wwn22.5VVL. excels in tasks such as object detection, OCR, and structured data extraction from documents, making it versatile visual agent capable of reasoning and tool usage across various domains.
17

---

InternVL3 [Zhu et al., 2025| One of the most advanced open-source MLLMs at present. Building upon its predecessor, VL2.5 it retains the ViT-MLP-LLM architecture, integrating a Vision Transformer with a large language model through an MLP connector. InternVI.3 features a native multimodal pre-training paradigm. jointly acquiring linguistic and multimodal capabilities single stage. It incorporates Variable Visual Position Encoding to handle extended multimodal contexts and employs advanced training techniques like supervised fine-tuning and mixed preference optimization. InternVL3 demonstrates superior performance across wide range of multimodal tasks, including tool usage, GUI agents, industrial image analysis, and 3D vision perception.
B.2
Evaluation benchmarks
B.2.1 General image benchmarks
VQAv2 [Goyal et al., 2017] The second version of the VQA benchmark [Antol et al. 2015] for open-ended visual question answering, designed to evaluate the ability to understand images, natural language, and commonsense knowledge. It includes 265,016 images from COCO (Lin et al. 2014] and abstract scenes, each paired with average of 5.4 questions. Each question is annotated with 10 ground truth answers and 3 plausible alternatives. We use the test-dev split for evaluation.
GQA Hudson and Manning, 2019] A large-scale visual question answering benchmark built on real images from the Visual Genome dataset [Krishna et al. 2017], designed to assess compositional reasoning and visual understanding. It provides over 22 million balanced question-answe pairs, with each image annotated by a detailed scene graph describing object classes, attributes, and relationships in the scene. We use the test-dev balanced split for evaluation.
VizWiz (uurri et al., 2018 visual question answering benchmark collected in a real-world accessibility setting, where blind users captured images and asked spoken questions about them. Each visual question paired with 10 crowdsourced answers. It introduces two key tasks: answering visual questions and predicting whether question is unanswerable based on the image, highlighting challenges such as poor image quality and ambiguous content, We use the test split for evaluation.
ScienceQA [Lu et al. 2022] A large-scale multimodal multiple-choice question answering bench- mark focused on diverse scientific domains. It contains 21,208 questions spanning natural science, language science, and social science, categorized into 26 topics, 127 categories. and 379 skills. Among them, 48.7% include image context, 48.2% include text context, and 30.8% include both. A majority of questions are annotated with grounded lectures (83.9%) and detailed explanations (90.5%), offering external knowledge and reasoning to support the correct answer. We use the test split that includes image context (also known as CantnceQQ-IMGG for evalution.
POPE [Li et al. 2023) A benchmark designed to assess object hallucination in large vision-language models. The images are sourced from COCO [Lin et al., 2014), and the questions focus on whether a specific object is present in the image, assessing the degree of object hallucination. Precision, recall, and Fl score are used to quantify hallucination rates, and we use the test split for evaluation.
HallusionBench [Guan et al. 2024] An image-context reasoning benchmark crafted to expose two frequent failure modes of large vision-language models: language hallucination (answers driven by strong linguistic priors that contradict the image) and visual illusion (misleading visual features that produce confident yet wrong responses). Comprising carefully designed examples that remain challenging for GPT-4V and LLa,A-1.55 it enables fine-grained diagnosis of how VLMs over-trust language or under-exploit vision, offering insights for building more faithfully grounded models.
MME Fuu et al., 2024a] A comprehensive benchmark evaluating both perception and cognition abilitics of multimodal large language models. It contains a total of 14 subtasks. The perception tasks include coarse- and fine-grad fine-grained recognition as well as OCR. Coarse gaarse-grained recognition primarily focuses on the presence, count, position, and color of objects, while fine-grained recognition involves identifying specific posters, celebrities, scenes, landmarks, and artworks. The cognition tasks include commonsense reasoning, numerical calculation, text translation and code reasoning.
MMBench [Liu et al., 2025] A comprehensive multimodal benchmark designed to evaluate wide range of vision-language capabilities. It features a carefully curated dataset with a larger number and greater diversity of evaluation questions and skills compared to existing benchmarks. MMBench also introduces a novel CircularEval strategy, leveraging ChatGPT to convert open-ended model responses into structured choices, enabling more consistent and robust evaluation of model predictions.
18

---

MM- (Yu et al. 2023] A benchmark focuses on the integration of different multimodal capa- bilities. It defines core capabilities through 218 challenging examples. including recognition. OCR, knowledge, language generation, spatial awareness, and mathematics. This benchmark utilizes ChatGPT assistant for evaluation, providing unified metrics for assessing answers of varying styles.
B.2.2
Text-oriented benchmarks
AI2D [Kembhavi et al. 2016] A diagram- based question answering benchmark consisting of over 5,000 grade school science diagrams, annotated with more than 150,000 structured labels and ground-truth syntactic parses. It also includes over 15,000 multiple-choice questions aligned with the diagrams, enabling research on visual reasoning and diagram understanding in scientific contexts. We use the test split with mask for evaluation.
Tex [Singh et al. 2019) A benchmark designed to evaluate model's ability to read and reason about text within images. The images are primarily sourced from the Open Images v3 dataset Krasin it al. [2017]. containing various scenarios such signs, billboards, and product packaging with rich text information. introduces a new modality, scene text, that models must recognize and interpret in order to answer questions accurately. This benchmark emphasizes the integration of OCR and visual reasoning for effective multimodal understanding. We use the validation split for evaluation.
ChartQA [Masry et al. 2022] A large-scale benchmark designed for question answering over charts, focusing on complex reasoning that involves both visual interpretation and logical arithmetic operations. It includes 9.6K human-written questions and 23. IK questions generated from chart summaries. Unlike prior template-based benchmarks, ChartQA challenges models to perform multi- step reasoning using both the visual content and underlying data tables of charts, highlighting the need for advanced multimodal understanding. We use the test split for evaluation.
OCRBench [Liu et al., 2024e A comprehensive evaluation benchmark assessing the OCR ca- pabilities of large multimodal models. It comprises 29 datasets across diverse text-related visual tasks, including text recognition, scene text-centric VQA, document-oriented VQA, key information extraction, and handwritten mathematical expression recognition.
B.2.3 Video benchmarks
MLVU [Zhou et al. 2024 The first comprehensive benchmark designed to evaluate multimodal large language models on long video understanding tasks. It features a diverse set of long videos ranging from 3 minutes to 2 hours in length and includes nine evaluation tasks spanning multiple-choic and free-form generation formats. These tasks are categorized into holistic understanding, single-detail understanding, and multi-detail understanding, challenging models to process both global and local information across long video content. We use M-Avg as the evaluation metric.
MVBench Li et al., 2024b A comprehensive benchmark for evaluating the temporal understanding abilitics of multimodal large language models in video comprehension. It consists of 20 challenging tasks specifically designed to require dynamic video analysis beyond single-framae understanding. MVBench introduces a novel static-to-dynamic transformation approach, converting static tasks into temporally grounded ones, thus systematically testing a wide range of temporal reasoning skills from low-level perception to high-level cognition. We use the test split for evaluation.
SingVideoBench (Wu et al. 2024) A large-scale benchmark for evaluating long-form video un- derstanding large multimodal models. It features 3,763 varying-length web-collected videos (up to an hour long) with subtitles, across diverse topics. This benchmark introduces novel referring reasoning task, where questions include explicit references to specific video segments, requiring models to retrieve and reason over detailed multimodal context. It includes 6,678 multiple-choice questions in 17 fine-graines categorics. We use the validation split for evaluation.
Video-MME [Fu et al., 2024b) The first comprehensive benchmark specifically designed to evaluate multimodal large language models in video understanding. It includes 900 manually selected and annotated videos totaling 256 hours, covering 6 primary domains and 30 subfields. This benchmark supports diverse temporal lengths (from 11 seconds to hour) and integrates multiple modalities such as video frames, subtitles, and audio. With 2,700 expert-annotated question-answer pairs, Video-MME provides a high-quality, fine-grained assessment of MI. ability to reason over complex sequential and multimodal information. We do not use subtitles during evalution.
19

---

B.3
Comparison methods
B.3.1 Text-based methods
FastV [Chen et al., 2024a] The first work identify the inefficient visual attention phenomena in MLLMs. Based on this observation, FastV proposes straightforward solution, that is, to prune the part of visual tokens with the lowest visual-text attention score after layer of the model, thereby achieving MLLM inference acceleration in training-free frreing-freeeee manner.
Prop ( et al., 2024 Building upon FastV, PrramidDrop further observes that pruning in shallow layers has a larger impact on model performance, while the redundancy of visual tokens tends to increase with model depth. Based on this insight, it proposes hierarchical pruning strategy that divides the MLLM into multiple stages and prunes a certain proportion of visual tokens at the end of each stage, leading to improved performance.
SparseVLM [Zhang et al. 2024b] Similar to PyramidDrop. Sparse SparseVLMy VLM also adopts multi-stage token pruning strategy. However, unlike previous approaches, it focuses the impact of the instruction tokens on vision- vision-language attention. It argues that not all text tokens contribute to the visual token pruning, only those highly relevant to the visual content are important. Therefore, it first selects the text tokens most related to the visual input as raters, and uses their attention to the visual tokens to guide the pruning process, leading further performance improvements.
B.3.2 Vision-based methods
LLa Prrumeree [Shang et al. 2024 The first work to perform token pruning solely based on visual information. It first selects important visual tokens using attention scores from the visual encoder, and then merges each of the remaining tokens with its most similar selected token through clustering-basse approach. Building on this, LLaVA-Prumest sampling to further enhance performance.
introduces spatially uniform
TRIM
[Song al., 2024] Pruning only based visual information while ignoring user instructions may lead to suboptimal performance. TRIM addresses this by leveraging CLIP metrics for pruning. Specifically, it computes the cosine similarity between image tokens from the visual encoder and text tokens from the text encoder, and uses these similarities to estimate the importance of each visual token. Tokens with lower similarity scores are pruned accelerate inference.
VisionZip (Yang al., 2024b] Similar L.LVA- Prummerge, VisionZip also relies on visual informa- tion for token pruning. It observes that attention within the visual encoder is highly concentrated, and therefore first selects several dominant tokens based on visual attention. Then, among all the remaining tokens, set of contextual tokens is obtained through clustering. These two groups are combined and fed into the language model, aiming to preserve as much visual information as possible.
B.3.3 Similarity-based methods
DART Wen et al. 2025b This work argues that in token pruning, duplication matters more than importance. Based on this insight, it first selects a small set of pivot tokens, and then iteratively retains the most diverse tokens from the remaining ones by selecting those with the lowest similarity to the already selected tokens. Finally, a group of the most diverse visual tokens is obtained.
DivPrune Alvar et al. 2025] This work also focuses token diversity. However, unlike previous approaches, DivPrune reformulates the token pruning problem as MMDP, aiming to retain the most diverse subset by maximizing the minimum pairwise distance among the selected tokens.
B.4 Implementation details
For image benchmarks, we use the official implementation of LLaVA For video benchmarks, we adopt the official codebase of LLaVA-NeXT for the model architecture and utilize Imms-evaf " evaluation. For advanced architectures like Qwen2.5-VI. we employ VI.MEal for evaluation.
https//github.com/nattian
https://gthhbb.oo///aaaAAA// NeXT
https:/ggthnh.com///oo//nnn. Laa.mM-ess-aaammm-aaaaaaaa
20

---

Table 7: Performance comparison of different pruning methods on LLaVA-1.5-133, Acc. denotes the average performance across 10 benchmarks, Rel. represents the average percentage of perfor- mance maintained. Attention- methods are shown with red background, attention&&smilarity based methods with green background, and similarity baseed methods with blue background.
VQA
SQA tM
tha
MMB
MMB
MMVet
Upper Bound. 576 Tokens (T00%)
L1aVA-1.5-13
80.0
63.3
53.6
Y2.8
61.2
86.0
1551.2
68.3
63.3
36.2
66.2
100.9%
876)
FastV (R00V24)
74.2
38.6
75.5
1460.6
66.1
32.8
"Drop(CVPE22)
78.2
61.0
53.8
73.3
60.2
83.6
1489.5
67.5
62.8
32.1
64.7
97.4%
VLarseVLMMIR
77.6
59.6
51.4
74.3
59.3
85.0
1487.9
68.4
62.6
35.2
64.8
97
PruMerge+
(OLLT20)))
76.4
59.4
49.7
72.4
55.0
86.8
1426.9
67.1
58.4
35.1
63.2
95. 95.2%
Vision2ip<crra223
76.8
57.9
52.3
73.8
58.9
82.7
1449.2
67.4
62.5
36.0
64.1
97.0%
(9025
75.7
74.2
$8.7
80.4
1395.D
65.4
34.8
63.2
95.
Div Prunc (OVPR25)
77.1
59.2
53.5
72.8
$8.0
86.8
1457,7
66.3
60.7
34.4
64.2
8%
CDPruner (Oans)
77.7
39.7
32.9
75.2
3X.4
87.3
147K.D
67.5
61.3
36.2
64.88
98.0%
Retain
Tokens
88 9%))
(EUOV24)
(Drop(CVF226) (CVP226)
70.8
54.1
50.5
73.1
55.3
66.1
1247.0
63.1
56.6
21.9
57.4
85.9%
(ICCL22)
73.2
55.9
52.1
73.0
$7.1
77.9
1574.3
65.2
60.3
32.9
61 A
25.2%
PruMerge+ (2024.05)
72.6
56.3
52.4
73.5
54.4
75.7
1338.2
65.0
59.3
30.3
60.6
91.5%
$2.0
86.5
1406.22
52.7
27.8
VisionZip(ccmaa5)
73.7
56.2
53.2
74.2
57.4
75.7
1379.6
64.9
61.3
33.4
61.9
93.8%
DAKT 02)
72.4
55.7
$3A4
73.8
57.3
72.8
1380.0
64.7
60.6
32.8
61.3
92.8%
71.7
(Orrs)er(Oars)
76.7
39.4
53.6
72.5
$7.6
87.1
1466.8
65.5
58.8
35.2
64.00
96.6%
Retain
Tokens
94..%
PruMerge+ (20rg, 06)
66.8
54.1
52.3
71.7
52.4
67.4
1269.1
61.1
53.5
28.7
57.1
$6.5%
(ROM(c0L
57.9
VisionZip(crraae)
68.4
52.7
53.0
72.9
55.2
66.8
1257.7
61.2
55.8
29.3
57.8
87.6%
DART (2025. .22)
68.1
53.9
$2.0
73.2
55.1
66.9
1282.8
61.9
56.2
29.4
58.1
58. .0%
(OVPR2))
CDPruner (Oura))
71.9
55.3
87.6
63.7
56.6
30.9
Additional experimental results
C.1 CDPruner for larger language model
To evaluate the effectiveness of our proposed method on larger language models, we apply CDPruner to two models equipped with 13B LLMs: --VV--1---13B and LLaVA-NeXT-13B. The results are presented Tables and 8 The larger language models lead to significant performance improvements and also make MLLMM less sensitive to visual token pruning. Among various pruning strategies, text-attention based methods benefit the most from scaling up the language model, indicating that larger L.LM brings more accurate attention. Across different types of pruning methods, CDPruner consistently outperforms all other approaches under various reduction ratios. With 77.8% visual tokens removed, our method retains 98.0% and 99, 99.9% of the original performance LLaVA--15-11 and LLaVA-NeXT-I3B, respectively, demonstrating its effectiveness on larger language models.
C.2 CDPruner for advanced open-source MLLM
In addition to Qwen2.5-VV. VL, we further apply CDPruner to one of the most advanced open- open-source MLLMM to date, VL.3. The results are shown Table 9 Here, we fix the input resolution to 896x896, yielding 1, 1,280 visual tokens. Notably, unlike its performance on the LLaVA series, DivPrune exhibits significant performance drop on Intern as does not account for the relevance to user instructions during pruning. In contrast, our CDPruner jointly considers both diversity and relevance, consistently achieving the best performance across different reduction ratios. Specifically, even when 90% of the visual tokens are removed, our method retains 83,9% of the original performance, 3% higher than the second-best FastV, demonstrating its effectiveness and adaptability in advanced MLLM architectures.
C.3 Efficiency analysis on larger language model
Here, we conduct an additional efficiency analysis on LLLVA--NeT--13B, which has higher computa- tional demands. As shown in Table 10 the combination of higher input image resolution and larger L.M results in significantly increased inference cost. Our CDPruner effectively reduces the number of visual tokens from 2,880 to 320, achieving a 10x reduction in FLOPs, along with 6.8x and 1.4x
21

---

Table 8: Performance comparison of different pruning methods on LLaVA-NeXT-133. Acc.
denotes the average performance across 10 benchmarks, Rel. represents the average percentage of per- formance maintained. Attention- based methods are shown with red background, attention& similarity- based methods with green background, and similarity based methods with blue background.
VQA
SQA 1M6
the
MMB AM
MMB
MMVet
Upper Bound, All 2800
Tikkons (100%)
LLaVA-NeXT-78
82.3
64.4
59.1
73.1
63.2
85.3
1539.5
68.3
61.2
45.0
9
100.9%
FastV (RGDV24)
71.7
60.7
80.2
1516.7
59.9
43.8
DDrop(CVPE20
$1.1
62.8
58.1
71.7
62.1
84.4
1559.1
66.6
60.8
39.7
66.5
97.6 .7.6%
PLarseVLMM (CCLL2
79.9
62.7
57.5
72.5
62.8
85.6
1562.7
68.8
64.0
41.3
67.3
98.9%
PruMerge+
70.6
(COL
79.4
63.1
$4.1
71.2
57.6
87.3
1534.6
68.7
61.2
42.3
66.3
97.2%
(ision2ip<crra223
79.7
62.9
56.2
70.8
62.1
85.8
1549.2
68.1
62.6
46.8
67.2
99 2962%
(9025
79.3
71.0
61.3
85.2
1542.4
45.5
66.8
Div Prunc (OYPR25)
80.4
72.2
59.2
86.5
1526.1
67.5
62.9
39.0
CDPruner (Oars)
81.0
64.01
57.1
71.8
61.0
87.55
1545.6
68.9
62.1
47.3
67.8
99.9%
Retnin
Tokens
88, (%)
(EUDV24)
(Drop(CVF226) (CVP226)
75.4
57.7
52.1
72.1
56.2
74.6
1386.3
62.8
55.3
29.5
60.5
88.2%
SparseVLM (M
76.7
60.9
$4.7
70.9
60.0
81.5
1491.6
68.0
63.5
39.3
65.0
55.5%
PruMerge+ (2024.05)
75.9
53.6
70.7
55.9
79.1
1426.5
66.6
60.6
36.5
63.1
92.6%
87.2
VisionZip(ccmaa5)
76.8
60.7
54.8
70.2
60.7
82.3
1487.3
66.5
62.3
41.1
65.0
95.6%
DAKT 02)
76.4
60.9
$4.2
69.8
19.7
81.1
1457.4
65.9
61.9
41.4
64.4
94 8%6
72.3
(Orrs)er(Oars)
79.6
63.1
55.1
71.6
58.7
87.6
1498.5
66.3
61.8
42.4
66.1
97.1% %
Retain
160 Tokens []] (44.4%)
PruMerge+ (202e 06)
71.6
57.9
50.8
70.1
52.8
72.1
1315.9
63.2
57.1
30.6
59.3
86.8%
(C0.220))))25)
VisionZip(crraac)
72.4
57.8
52.5
69.7
58.6
76.8
1393.9
64.8
60.0
35,9
61.8
90.8%
DART (2025,205.00) .02)
72.8
58.7
52.1
20.1
57.2
75.7
1389.3
64.6
60.8
35.0
61 &
90.5%
(OVPR2))
CDPruner
71.7
36.7
60.1
Table 9: Performance comparison of different pruning methods on Intern VL3-8B. Acc. denotes the average accuracy, Rel. represents the average percentage performance maintained. Attention- based methods are shown with red background, and similarity-based methods with blue.
Method
A12D
TextVQA
ChartQA
OCRBench
HallBench
MME
MMB-EN
MMB-CN
Acc.
Rel.
Upper Bound, All /280 Takens (100%)
VL.3-8B
85.2
81.5
85.1
853
50.0
2394
83.9
82.6
84.2
100.0%
Retain 256 Tokens () 80.0%)
FastV (BC3V24)
82.2
74.4
70.7
632
48,5
2348
83.6
82.0
77.8
92.4%
DivPrune (CVP926)
57.5
38.7
80.2
70.4
82.8%
CDPruner (Dura)
82.7
75.7
72.0
640
48.8
2334
83.5
81.7
78.1
92.9%
90.0%)
FastV (BC0V24)
77.3
63.7
46.9
426
42,5
2250
81.3
80.2
68.4
80.9%
DivPrune (CVP125)
76.4
42.7
378
37.7
2166
78.4
77.6
64.3
75.7%
CDPrumer (Qura)
79.9
67.5
50.8
471
44.6
2282
82.1
80.3
70.8
83.9%
decreases in prefill and decode time, respectively. Meanwhile, it maintains competitive performance, demonstrating the efficiency of CDPruner for larger MLLM inference.
C.4 Ablation study on balance factor
Since the amount of information contained in the instructions of different benchmarks varies, introduce balance factor 9 to control the trade-off between diversity and relevance. Specifically, from eq. (8), we derive (/(2(1 0), which is then used transform the relevance vector 7 and
construct a new conditional kernel matrix:
İ' diag (exp (af)) diag (exp (ar)).
(18)
The updated log-prole seg-prbbbbility of subset S for DPP is given by:
log det
(is)
2x
Σ n; SES
log (Lg) x Σ f + (1-10 (1-
llgg det (Ls)
(19)
iES
By adjusting 0, we can modulate the relative importance relevance and diversity in the modeling process. As shown in Table 11] the ablation results for 0 indicate that the optimal value varies across benchmarks. Selecting the best factor value for each dataset leads to performance improvements. worth noting that even without introducing this balancing factor (i.c., the version used in the main
22

---

Table 10: Efficiency analysis of different pruning methods on LLaVA-NeXT-13B, The per-
formance is evaluated on POPE. Attention-basee methods are shown with red background, atten-
mon&&similarity-baeee methods with green background, and similarity-base methods with blue background.
FLOPs
Prefill Time
Decode Time
KV Cache
GPU Memory
Score (F1)
Method
# Token
(T)
(ms/token)
(ms/token)
(MB)
(GB)
LLaVA-NeXT-13B
2880
79.9
434
44
2250.0
30.1
85.3
FastV (BCCV24)
320
(x9.4)
(x5.8)
33 1.3)
250.8
28.0
63.6
(CVPR22))
320
8.5 (x9.4)
86 (x5.0)
34 (x1.3)
250.7
28.0
74.6
Sparse (
320
8.5 (x9.4)
101 (x4.3)
38 (x122
254.8
31.6
81.5
VisionZip
320
8.2 (xx.7)
64 (x6.8)
32 (x1.4)
250.0
26.6
82.3
DivPrune (CVPP22))
320
8.2 (x9.7)
(x6.8)
32 (x1.4)
250.0
25.9
85.2
CDPruner (Oura)
(x9.7)
(x6.8)
32 (x1.4)
250.0
25.9
87.6
Table 11: Ablation study of balance factor on LLa LL-V---,--7B,, 64 visual tokens retained.
0 VQA V2
GQA
VizWiz
SQA IMG
VQA
Tru
POPE
MME
MMB
MMB
MMVet
Ace.
Rel.
0.0
74.6
57.6
53.9
67.9
55.8
86.2
1358.7
59.3
53.4
29.2
60.6
95.7%
0.2
74.8
58.2
53.8
68.2
55.7
86.6
1362.1
59.4
53.3
29.3
60.7
95.9%
87.2
1378.2
0.6
75.5
58.9
54.1
68.5
55.6
87.3
1396.3
60.3
52.9
30,7
61.4
97.0%
0.8
75.2
58.5
53.3
68.4
55.0
87.4
1415.3
61.6
52.8
29.4
61.2
96.5%
best
75.5
58.9
54.1
68.5
55.8
87.4
1415.3
61.6
53.4
30.7
61.7
97.5%
experiments), method already achieves strong results. Therefore, in practical applications, one may choose whether to introduce and tune this additional hyperparameter based on specific needs.
D Additional visualization results
Here, provide additional visualizations of relevance scores in Figure 5 It can be clearly observed that models with language-image pre-training are able to effectively capture the correspondence between instructions and regions of interest in the image, which is crucial for instruction-guided visual token pruning in multimodal large language models.
E
Limitations
One limitation of our work that the proposed method can only be applied to open-source MLLMs, where the encoded visual tokens can be accessed during inference. However, there exist many black- box models, including ChatGPT, Gemini, and Claude, which also require significant computational resources for visual reasoning. Moreover, although our method is applicable to state-of-the-art open- source MILMM architectures such Qwen2.5-VL and InternVI.3 and achieves superior performance compared to existing approaches, these models generally more sensitive to visual token pruning. can be observed that, compared to the LLaVA series, these advanced models tend to suffer greater performance degradation after pruning. This is likely due to the fact that their architectures already incorporate visual token compression techniques like pixel unshuffle. Exploring how to enable efficient inference within these architectures will be important direction of our future work.
F
Broader impacts
Recently, MLLMs have been widely applied across various industries, thanks to their powerful reasoning capabilities. However, redundant visual inputs bring high computational complexity and significantly increases its usage cost. In this work, we propose a simple yet effective solution that accelerates MILM inference by visual token pruming without the need of any additional training. We believe this approach can facilitate the practical application of MLLMs by reducing deployment costs, lowering inference latency, and enabling usage on resource-constrained edge devices. It is important to note that this work does not mitigate the potential misuse of MILMMs by malicious actors.
23

---

Input Image
Is there bird?
Is there boat?
Is there rock?
there lake??
Input Image
Is there phone?
Is there plate?
Is there bottle?
Is there a glass?
Input Image
Is there a banana?
Is there a bowl?
Is there a kettle?
Is there a table?
Input Image
blender? Is there a knife?
there lime?
Is there a lequila?
Figure 5: Additional visualizations of relevance scores. We compute the relevance scores for several samples from the POPE benchmark using LLaVA-1.5-7B, with the instruction following the template: "Is there [object] in the image?" Red indicates high relevance, while blue indicates low.
24

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The abstract and introduction clearly state the claims made, including the contributions made in the paper. And the claims made accurately match experimental results, which can be well generalized to other MLLMs.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: There a separate "limitations" section in the technical appendices located in the technical appendix.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and prool's
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
25

---

Answer:
[NA]
Justification:
There is no theoretical result in the paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The detailed methods are described in the paper. All the pre-trained models and datasets used the experiments are publicly available, and the code is provided in the technical appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
26

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
The paper provides the code with sufficient instructions faithfully reproduce the main experimental results in the technical appendix. The authors also plan to release the code open-source websites such GitHub after the publication of the paper.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/
mublic/guider CodeSubmissioonollee.
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips wwtpccccccccccueeeeeeeeeeeeppeee
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: All the evaluation details are clearly provided in the paper. The proposed method does not contain any hyperparameters, and the settings on all benchmarks follow the previous work. The authors also provide the corresponding implementation details in the code and technical appendices located in the technical appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: The paper focuses training-free model inference acceleration, and thus does not involve experimental statistical significance.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
27

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: All the experiments were conducted on 8 NVIDIA A A100-80GB GPUs. There is also separate analysis section for comparing the inference efficiency different methods.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
The authors have reviewed and ensured compliance with the Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. if there special consid- eration due to laws or regulations in their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is negative socictal impact of the work performed
Guidelines:
The answer NA means that there is no societal impact of the work performed.
28

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
The authors have appropriately cited the original papers that produced the code package or datasct.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
29

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
datasets.papersstt
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: appendix.
The paper provides the code with detailed documentation in the technical
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mooel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
30

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM for what should or should not be described.
31


Reviews:
[{"review_id": "ZeziLXEm0U", "reviewer": "Reviewer_w7Zd", "summary": "This paper tackles a critical and practical problem in MLLMs: the high inference cost caused by a large number of visual tokens. The authors introduce CDPruner, a novel, training-free pruning method that intelligently reduces visual tokens by maximizing the \"conditional diversity\" of the retained set. The paper is well-motivated, the proposed method is elegant, and the experiments are comprehensive and impressive.", "questions": "N/A", "limitations": "N/A", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strength**:\n\nThe work astutely identifies the core trade-off in token pruning (relevance vs. diversity) and proposes an elegant solution. The concept of \"conditional diversity,\" modeled via a Determinantal Point Process (DPP), is a novel and powerful way to unify these competing goals, providing a fresh perspective that moves beyond simpler heuristics.\n\nThe authors validate CDPruner across diverse MLLM architectures and challenging tasks (including high-resolution and video inputs). The results are compelling, demonstrating remarkable performance retention (e.g., >94% accuracy with only 5.6% of tokens) and significant efficiency gains in latency and FLOPs. The breadth of experiments strongly supports the method's effectiveness.\n\n\n**Weakness**:\n\nWhile the paper claims the additional overhead of CDPruner is \"negligible\" (<10ms), a more detailed analysis would be beneficial. The paper states the complexity is O(nm²), but it would be helpful to see how this overhead scales in practice with the number of initial tokens n and kept tokens m. In scenarios with fewer visual tokens or in real-time applications with extremely strict latency requirements, this \"fixed cost\" might become a non-trivial factor. A more granular analysis would help readers better assess the cost-benefit trade-off in different settings.", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the time and effort in evaluating our work. We are encouraged by the recognition of our method as *elegant* and our experiments as *comprehensive and impressive*. After carefully reading the comments, we would like to address the reviewer’s concerns regarding the computational overhead of DPP in our CDPruner as follows:\n- **Computational overhead with respect to the initial token number $n$**:\n\n  We fix the number of kept tokens to 64 and measure the runtime of a single DPP step (averaged over 10,000 runs) as a function of the initial token number $n$:\n\n  |Initial Token Number ($n$)|144|288|576|1152|2304|\n  |:---|:---:|:---:|:---:|:---:|:---:|\n  |Latency (ms)|0.86|1.09|1.37|2.00|3.64|\n\n  The results show that the overall computational overhead does not increase significantly with the initial token number $n$. Even when the number of visual tokens exceeds 2k, the latency of a single run remains below 4ms, which is well within an acceptable range for practical applications.\n\n- **Computational overhead with respect to the kept token number $m$**:\n\n  We fixed the number of initial tokens to 576 and measured the runtime of a single DPP step (averaged over 10,000 runs) as a function of the kept token number $m$:\n\n  |Kept Token Number ($m$)|16|32|64|128|256|\n  |:---|:---:|:---:|:---:|:---:|:---:|\n  |Latency (ms)|0.45|0.77|1.37|2.73|5.84|\n\n  The results show that while the runtime increases more noticeably with $m$ compared to $n$, the latency per run still remains very low. In practical scenarios, the number of kept tokens $m$ is typically much smaller than the initial number $n$, so the overall DPP runtime remains negligible and does not affect the real-time application of the algorithm.\n\n- **Further optimization of algorithmic complexity**:\n\n  Our current implementation adopts fast greedy MAP inference for DPP, with a theoretical complexity of $\\mathcal{O}(nm^2)$. However, for MLLMs with dynamic high-resolution inputs (e.g., LLaVA-NeXT, InternVL3), the input image is first divided into equally sized patches, and the DPP can be performed in parallel across different patches. This significantly reduces the computational complexity: for an image with $k$ patches, the global complexity $\\mathcal{O}((kn)(km)^2) = \\mathcal{O}(k^3nm^2)$ can be reduced to a local complexity of $\\mathcal{O}(knm^2)$.\n\n  Additionally, by applying a sliding window of width $w (w < m)$ during DPP computation, the complexity can be further reduced to $\\mathcal{O}(wnm)$ [1]. Furthermore, incorporating a Markov chain approximation into the DPP sampling can reduce the complexity to the ideal $\\mathcal{O}(m^3)$ [2].\n\n- **Further reduction of runtime latency**:\n\n  Given that the overall computing process of DPP is sequential, in real-time applications with extremely strict latency requirements, we can further optimize inference latency by implementing custom operators, enabling low-level and system-level acceleration.\n\nIn summary, we believe the computational overhead of DPP does not hinder the real-time applicability of CDPruner. Moreover, various strategies exist to further reduce the inference latency under different deployment scenarios. We hope the above responses effectively address your concerns, and we sincerely hope you would consider increasing the rating if your concerns have been resolved.\n\n&nbsp;\n\n[1] Chen, Laming, Guoxin Zhang, and Eric Zhou. \"Fast greedy map inference for determinantal point process to improve recommendation diversity.\" Advances in neural information processing systems 31 (2018).\n\n[2] Kang, Byungkon. \"Fast determinantal point process sampling with application to clustering.\" Advances in Neural Information Processing Systems 26 (2013)."}, {"review_id": "TNLdOGX2NU", "reviewer": "Reviewer_pdN1", "summary": "The paper proposes CDPruner, a novel visual token pruning method for multimodal large language models (MLLMs) to reduce inference cost. It defines conditional similarity between visual tokens based on instructions and reformulates the token pruning problem using determinantal point process (DPP) to maximize conditional diversity. CDPruner is training-free and model-agnostic, achieving state-of-the-art performance on various vision-language benchmarks while significantly reducing computational cost.", "questions": "- The paper states that CDPruner is model-agnostic. Could the authors provide further information on potential challenges and corresponding strategies when adapting it to different types of visual encoders and language models?\n- The effectiveness of the CDPruner method is significant. Could the authors provide an in-depth analysis of the reasons behind CDPruner’s effectiveness? Such analysis would greatly benefit research in this field.\n  \nIf the authors address these issues, I will further increase my score.", "limitations": "The author discusses the limitations in supplementary materials.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "- Strengths:\n  - This paper proposes a novel method, CDPruner, which surpasses traditional attention-based or similarity-based token pruning approaches. By defining conditional similarity and leveraging Determinantal Point Processes (DPP) to maximize conditional diversity, it offers a unique perspective for selecting diverse and relevant visual tokens. This represents a significant advancement in improving the inference efficiency of MLLMs.\n  - CDPruner is model-agnostic and requires no additional training, granting it high flexibility and allowing easy application to various MLLMs without further training. This feature saves time and computational resources, facilitating seamless integration into different models and architectures.\n  - The authors conduct a comprehensive evaluation of CDPruner across multiple image and video understanding tasks, covering various benchmark datasets. This thorough assessment provides strong evidence of the method’s robustness and generalizability in different scenarios and datasets.\n    \n- Weaknesses:\n  - The visual and textual embeddings vary across different models, which may affect the applicability of the CDPruner method. It may be beneficial to consider optimizing the generalizability of CDPruner from the perspectives of visual and textual embeddings, such as by introducing additional embedding models.\n  - Although this is understandable, the paper does not consider applications in multi-turn dialogue scenarios. Is it possible to dynamically adjust the pruned tokens in multi-turn contexts? I suggest investigating the extent to which pruning affects the performance of multi-turn dialogues.\n  - Although CDPruner achieves excellent performance, further exploration of its underlying principles, such as the semantic contribution of key tokens, is warranted.", "was_revised": true, "final_justification": "After reading the authors' rebuttal, I find that my main concerns have been satisfactorily addressed.\nThe authors clarified how CDPruner maintains generalizability across different embedding models, alleviating my concern about its applicability beyond specific architectures.\nWhile multi-turn dialogue was not a focus of the current work, the authors provided reasonable justification and future directions for extending CDPruner to such scenarios.\nThe explanation regarding token-level contributions and pruning rationale was helpful in clarifying the interpretability aspect of the method.\nGiven these responses, I now have greater confidence in the soundness and potential of the proposed approach. I will raise my score accordingly. I encourage the authors to further explore generalization and multi-turn capabilities in future work.", "rebuttal": "We thank the reviewer for the detailed review and valuable feedback. We are encouraged by the recognition of our method as *novel and offering a unique perspective*, as well as the acknowledgment that our evaluation is *comprehensive and thorough*. After careful consideration of the comments, we provide the following point-by-point responses:\n1. **Additional embedding models for instruction relevance**:\n\n   To compute the relevance between visual tokens and user instructions, it is necessary to obtain visual and textual embeddings of the same dimension and ensure they are situated in a unified semantic space. Therefore, the most suitable choice is a model that has been pre-trained through language-image contrastive pretraining. If the visual features in the MLLM are already well aligned with text, an additional embedding model may not be required. Here, we introduce the CLIP model for Qwen2.5-VL to validate the effectiveness of an additional embedding model. The corresponding experimental results are as follows:\n\n   |Method|AI2D|HallBench|MMB-EN|MMB-CN|Acc.|Rel.|\n   |:---|:---:|:---:|:---:|:---:|:---:|:---:|\n   |*Upper Bound, All 1296 Tokens* **(100%)**|||||||\n   |Qwen2.5-VL-7B|84.4|46.8|83.9|83.4|74.6|100.0%|\n   |*Retain 512 Tokens* **(-60.5%)**|||||||\n   |CDPruner|82.9|42.5|82.2|82.6|72.6|96.5%|\n   |CDPruner+CLIP|**83.7**|**43.6**|**83.1**|**82.1**|**73.1**|**97.5%**|\n   |*Retain 256 Tokens* **(-80.2%)**|||||||\n   |CDPruner|80.5|40.1|80.9|79.9|70.4|93.3%|\n   |CDPruner+CLIP|**81.7**|**41.5**|**82.6**|**81.1**|**71.7**|**95.3%**|\n   |*Retain 128 Tokens* **(-90.1%)**|||||||\n   |CDPruner|76.0|37.2|76.2|76.5|66.5|88.0%|\n   |CDPruner+CLIP|**79.3**|**38.1**|**81.3**|**78.1**|**69.2**|**91.5%**|\n\n   After incorporating CLIP into Qwen2.5-VL, CDPruner consistently achieves improved performance across various benchmarks. This result indicates that accurate instruction relevance estimation plays a crucial role in effective token pruning and provides new insights for adapting CDPruner to different MLLM architectures. We will continue to explore the value of additional embedding models for token pruning in our future work.\n\n2. **Extension to multi-turn dialogues**:\n\n   In multi-turn scenarios, it is possible to dynamically adjust the pruned tokens based on the current instruction by caching the key-value pairs of all visual tokens. Specifically, during the prefill stage, instead of discarding all pruned tokens, we can encode and store their key and value representations. In subsequent dialogue turns, we can then retrieve relevant tokens from the cache based on the current instruction, enabling dynamic token adjustment. Following the reviewer’s suggestion, we directly apply CDPruner to the MMDU benchmark [1] without performing dynamic token adjustment to evaluate the effect of pruning under this challenging setting:\n\n   |Method|C|R|VP|LC|AA|IRU|Acc.|Rel.|\n   |:---|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n   |*Upper Bound, All 576 Tokens* **(100%)**|||||||||\n   |LLaVA-1.5-7B|34.8|32.7|39.4|65.3|47.4|39.5|42.9|100.0%|\n   |*Retain 128 Tokens* **(-77.8%)**|||||||||\n   |TRIM|35.7|34.2|38.7|64.6|46.8|39.2|42.8|99.8%|\n   |CDPruner|**36.2**|**34.9**|**40.0**|**66.2**|**48.0**|**40.8**|**44.0**|**102.6%**|\n   |*Retain 64 Tokens* **(-88.9%)**|||||||||\n   |TRIM|35.6|34.1|37.1|63.8|44.8|37.7|41.7|97.2%|\n   |CDPruner|**36.1**|**34.4**|**38.6**|**64.5**|**46.2**|**39.0**|**42.8**|**99.8%**|\n   |*Retain 32 Tokens* **(-94.4%)**|||||||||\n   |TRIM|35.4|34.0|36.1|62.8|44.2|36.9|41.2|96.0%|\n   |CDPruner|**35.6**|**34.0**|**36.7**|**62.9**|**44.6**|**38.0**|**41.5**|**96.7%**|\n\n   TRIM prunes tokens based on their relevance to the instruction, which is suboptimal for multi-turn dialogue. If the subsequent question differs significantly from the previous one, the retained tokens may no longer be relevant, resulting in degraded performance. In contrast, CDPruner incorporates diversity modeling via DPP, which enables it to preserve more informative and comprehensive visual content while still considering relevance. As a result, it maintains better performance even in multi-turn scenarios. Experimental results show that CDPruner consistently outperforms TRIM under different reduction ratios, demonstrating its adaptability to multi-turn dialogues.\n\n3. **In-depth analysis of CDPruner’s effectiveness**:\n\n   This work identifies visual diversity and instruction relevance as two key factors for token pruning in MLLMs. Previous methods typically considered only one of these factors. Methods focusing solely on diversity (e.g., DART, DivPrune) tend to retain too many tokens irrelevant to the current question, while a large number of tokens in key areas are discarded, leading to loss of critical information. On the other hand, methods focusing only on relevance (e.g., FastV, TRIM) often retain redundant tokens while discarding informative ones from low-relevance regions, also resulting in performance degradation. Naively combining both factors in a two-stage pruning strategy does not effectively address this issue, as each stage may still retain unimportant tokens. In contrast, CDPruner jointly models both relevance and diversity using DPP, and selects the optimal subset from a global perspective. This enables it to retain the most relevant tokens while removing redundancy, achieving efficient pruning without sacrificing performance.\n\n4. **Potential challenges across different architectures and corresponding strategies**:\n\n   Since CDPruner performs token pruning before the language model, the visual encoder is more crucial to our method. For MLLMs whose visual encoders are paired with a corresponding text encoder (e.g., LLaVA-1.5, LLaVA-OneVision), we directly compute cosine similarity between visual and textual embeddings for relevance estimation. For models without a paired text encoder (e.g., Qwen2.5-VL, InternVL3), alternative strategies can be adopted, such as using embeddings from the language model’s embedding layer to compute similarity, leveraging text-visual attention from shallow layers of the language model, or introducing an additional embedding model (e.g., CLIP) to assist relevance computation. Designing a general pruning strategy for diverse MLLM architectures remains a promising direction, and we will continue to explore it in future work.\n\n5. **Font size issue**:\n\n   Thanks for pointing this out. We will increase the font size in the corresponding figures and tables in the revised version to improve the readability of the paper when printed.\n\nWe sincerely appreciate the reviewer’s recognition of our work. And we hope that the above analysis of CDPruner’s effectiveness and the discussion on adaptation strategies to various model architectures have addressed your concerns and can offer valuable insights for future research in this field.\n\n&nbsp;\n\n[1] Liu, Ziyu, et al. \"Mmdu: A multi-turn multi-image dialog understanding benchmark and instruction-tuning dataset for lvlms.\" Advances in Neural Information Processing Systems 37 (2024): 8698-8733."}, {"review_id": "5Osny4h1YU", "reviewer": "Reviewer_5bbK", "summary": "This paper proposes a novel token pruning method to maximize the conditional diversity of retained tokens for efficient VLMs. Based on the conditional similarity between visual tokens conditioned on the instruction, the token pruning problem is reformulated with determinantal point process (DPP). Experimental results demonstrate the effectiveness of the proposed method.", "questions": "Overall, I think the proposed method is novel with extensive evaluation. I am concerned about the additional computation cost by extracting textual features and the compatibility with other token pruning methods. If the authors well address my concerns, I will raise the score accordingly.", "limitations": "N/A", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The paper is well written and easy to understand. The figure is helpful to understand the whole framework of the proposed method.\n2. The proposed conditional similarity is novel and efficient to apply without the need of calculating the attention score.\n3. Experiments are extensive with various model variants and, image and video benchmarks, which demonstrate the effectiveness of the proposed method.\n\nWeaknesses:\n1. The proposed method needs the additional text embedding for modeling the conditional similarity. Line 145-150, extracting the textual features output by the corresponding text encoders would introduce additional computational cost. I would like to see the analysis of such additional cost.\n2. Would the proposed method be combined with token pruning methods within LLMs (such as FastV) for further pruning?", "was_revised": true, "final_justification": "I think the rebuttal well address my concerns. I raise the score to Accept.", "rebuttal": "We sincerely thank the reviewer for the effort and constructive feedback. We are encouraged by the recognition of our method as *novel and effective*, the acknowledgment of our experiments as *extensive,* and the appreciation of the paper as *well written and easy to understand*. After thoroughly reading the comments, we provide the following point-by-point responses:\n\n1. **Computational cost of text feature extraction**:\n\n   For text feature extraction, we follow the implementation of TRIM. Since the extraction of visual and textual features is entirely independent, we allocate **separate CUDA streams** for each, allowing both processes to run in parallel. As text feature extraction is generally faster than visual feature extraction, this component **does not introduce additional overhead in practice**. The corresponding pseudocode is as follows:\n   ```python\n   image_stream = torch.cuda.Stream()\n   text_stream = torch.cuda.Stream()\n   \n   with torch.cuda.stream(image_stream):\n       image_features = self.vision_tower(images)\n   \n   with torch.cuda.stream(text_stream):\n       text_inputs = self.text_tokenizer(texts)\n       text_features = self.text_tower(**text_inputs)\n   \n   torch.cuda.synchronize()\n   ```\n\n2. **Compatibility with token pruning methods within LLM**:\n\n   Our proposed method is fully compatible with any pruning method within LLMs. To demonstrate this, we combine CDPruner with three representative in-LLM pruning methods: FastV, PDrop, and SparseVLM. To retain $m$ visual tokens, we first apply CDPruner before the LLM to reduce the token number to $2m$, and then apply the in-LLM pruning method to further prune it down to $m$ tokens. The experimental results are as follows:\n\n   |Method|VQAV2|GQA|VizWiz|SQA-IMG|TextVQA|POPE|MME|MMB|MMB-CN|MM-Vet|Acc.|Rel.|\n   |:---|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n   |*Upper Bound, All 576 Tokens* **(100%)**|||||||||||||\n   |LLaVA-1.5-7B|78.5|61.9|50.1|69.5|58.2|85.9|1506.5|64.7|58.1|31.3|63.4|100.0%|\n   |*Retain 128 Tokens* **(-77.8%)**|||||||||||||\n   |CDPruner|76.6|59.9|52.8|69.0|56.2|87.7|1431.4|63.1|55.0|32.8|**62.5**|**99.0%**|\n   |CDPruner+FastV|76.4|58.2|51.3|68.3|57.4|81.8|1469.6|62.8|56.3|32.3|61.8|98.1%|\n   |CDPruner+PDrop|76.9|59.6|50.9|67.9|57.4|85.2|1473.9|63.6|55.9|32.3|62.3|98.7%|\n   |CDPruner+SparseVLM|76.8|59.5|51.3|67.7|57.4|86.1|1440.4|63.1|56.3|32.6|62.3|98.7%|\n   |*Retain 64 Tokens* **(-88.9%)**|||||||||||||\n   |CDPruner|75.4|58.6|53.4|68.1|55.3|87.5|1415.1|61.1|53.2|30.5|61.4|97.0%|\n   |CDPruner+FastV|75.3|57.4|52.6|69.6|56.6|82.2|1417.1|62.4|53.1|30.3|61.0|96.5%|\n   |CDPruner+PDrop|74.3|57.9|52.3|69.9|56.1|84.9|1406.6|62.2|54.2|30.0|61.2|96.7%|\n   |CDPruner+SparseVLM|75.7|58.3|52.4|69.5|55.9|86.3|1434.4|61.8|54.3|30.6|**61.7**|**97.4%**|\n   |*Retain 32 Tokens* **(-94.4%)**|||||||||||||\n   |CDPruner|73.6|57.0|53.1|69.5|53.2|87.9|1373.0|59.6|49.6|27.8|60.0|94.3%|\n   |CDPruner+FastV|73.4|56.1|52.9|68.7|54.8|81.1|1368.9|60.0|50.9|28.3|59.5|93.9%|\n   |CDPruner+PDrop|73.5|56.5|52.5|69.8|54.4|83.2|1366.1|60.1|51.2|28.9|59.8|94.4%|\n   |CDPruner+SparseVLM|74.1|56.9|52.8|69.3|54.5|85.5|1392.7|60.1|52.4|29.3|**60.5**|**95.4%**|\n\n   These combined methods yield further performance improvements over the original CDPruner, clearly demonstrating that our approach is fully compatible with various in-LLM pruning approaches.\n\nIn summary, through the implementation of dual CUDA streams, **CDPruner does not introduce additional computational cost for text feature extraction**, confirming its efficiency. Meanwhile, experimental results show that **CDPruner is also compatible with a wide range of pruning methods within LLMs** and achieve improvements, fully demonstrating its compatibility. We hope the above response addresses your concerns."}, {"review_id": "49yYTAWKFr", "reviewer": "Reviewer_4nPZ", "summary": "This paper proposes CDPruner, a model-agnostic approach to condense sequence length in multimodal LLMs  by maximizing diversity of the retained tokens conditional on the query and the other retained tokens.", "questions": "- Runtime: Can you expand on runtime at the beginning of the introduction where you mention the relate improvements in FLOPs and memory? Or even better add an extra column in one of the result tables comparing runtime of all methods. This would make this work even more comprehensive\n- Clarify better how you arrive at K tokens from a given input size. I thought this is a controllable hyperparameter but in multiple occasions  it sounds like it is static (\"we fix the input resolution to 672×672, resulting in 2,880 visual tokens\" and similarly later for videos)\nAbbreviating Large Language Models as LMMs in the first sentence gave me a good laugh", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Positive\n- Pruning is conditional on the question, unlike for most (all?) attention or similarity based pruning methods\n- CDPruner is rigorously  compared against half a dozen of other methods on 14 datasets for both visual Q&A and video understanding\n- Performance is consistently superior in all settings\n- Method is novel\n\nNegatives:\n- The method is presented generically for MLLMs but it seems that it is limited to Image/Video QA\n- The \"Rel.\" metric seems wrong or unintuitive. Table 2, it seems that it is normalized to the first row, but e.g., 2nd row 62.2/65.2 = 0.954 (not 0.956) and especially the CDPruner value of 100.1% does not make sense", "was_revised": true, "final_justification": "I am convinced that this is a solid and impactful paper and thus retained my original score of 5 through the rebuttal phase.", "rebuttal": "We sincerely thank the reviewer for the thorough review and valuable time. We are encouraged by the recognition of our method as novel and our performance as superior. After carefully considering the comments, we provide the following responses:\n1. **Evaluation benchmarks**:\n\n   We follow the most commonly adopted benchmarks used in open-source MLLMs, covering **general visual question answering, document understanding and OCR, as well as video understanding**. We plan to extend our evaluation to a broader range of tasks in future work.\n\n2. **Calculation of the “Rel.” metric**:\n\n   The “Rel.” metric is calculated as the average of the relative accuracies for each benchmark, rather than using a single relative average accuracy across all tasks. Thanks for pointing this out — we will clarify the calculation method of the metrics in the revised version.\n\n3. **Runtime**:\n\n   We have already described the runtime in **L18** of the abstract and **L57** of the introduction. Here, the reported **CUDA latency** refers to the actual runtime during inference. Furthermore, we provide a detailed analysis of runtime and efficiency in **Section 4.6** and **Table 5**. Following the reviewer’s suggestion, we will include runtime latency as an additional column in Tables 1–4 in the revised version.\n\n4. **Explanation of token number $K$**:\n\n   The number of retained tokens $K$ here is not a hyperparameter but rather an input to the algorithm. The goal of our work is to improve the inference efficiency of MLLMs to meet various deployment constraints. Therefore, $K$ can be set according to the available computational budget. For example, the original LLaVA-NeXT-7B model has an inference latency of approximately 280 ms per sample. In latency-sensitive scenarios requiring < 60 ms inference time, the number of visual tokens must be reduced from 2880 to 320. In this case, we should set $K \\le 320$. In short, users can flexibly set the retained token number $K$ based on their specific computational constraints, and our CDPruner is capable of adapting to different values of $K$ while consistently achieving strong performance. The values of $K$ presented in the tables are intended to illustrate performance under different reduction ratios, following previous works. The high-resolution input of 2880 visual tokens, as well as the token number in video scenarios, are fixed for fair comparison and do not directly determine the value of $K$.\n\n5. **Abbreviation typo**:\n\n   We thank the reviewer for the careful review. We will correct this typo in the revised version."}]

Author rebuttal:

Meta-review: This paper proposes a novel visual token pruning method, called CDPruner, for multimodal LLMs based on determinantal point process (DPP). In specific, CDPruner tries to maximize both of the diversity between visual tokens and the relevance to input text tokens, especially without any additional training. Experimental results on various image and video understanding benchmarks with LLaVA series and Qwen2.5-VL show that CDPruner maintains performances while significantly reducing visual tokens and corresponding computational cost.

Overall, the proposed CDPruner seems to be technically sound and novel with clear motivation. Moreover, rigorous empirical validation supports its feasibility and powerfulness. Most concerns raised by the reviewers are well addressed by the authors, including the additional runtime for pruning. Based on the consensus among the reviewers, I would recommend this paper to be accepted.

From the perspective of optimal pruning, I think it would be beneficial to reflect on the dependency of the MLLM backbone model and its underlying reasoning process. In addition, it would strengthen the work to experimentally evaluate whether further finetuning of the MLLM with the proposed pruning method leads to additional performance improvements.
Decision: Accept (poster)