LearnPruner: Rethinking Attention-based Token Pruning in Vision Language Models
Authors: Rinyoichi Takezoe
Yaqian Li
Zi-Hao Bo
Anzhou Hou
Mo Guang
Kaiwen Long
Venue: ICLR 2026 Poster (2026)
Keywords: Vision-Language Model
Token Reduction
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Vision-Language Models (VLMs) have recently demonstrated remarkable capabilities in visual understanding and reasoning, but they also impose significant computational burdens due to long visual sequence inputs. Recent works address this issue by pruning unimportant visual tokens, achieving substantial computational reduction while maintaining model performance. The core of token pruning lies in determining token importance, with current approaches primarily relying on attention scores from vision encoders or Large Language Models (LLMs). 
In this paper, we analyze the effectiveness of attention mechanisms in both vision encoders and LLMs. We find that vision encoders suffer from attention sink, leading to poor focus on informative foreground regions, while in LLMs, although prior studies have identified attention bias toward token positions, text-to-vision attention demonstrates resistance to this bias and enables effective pruning guidance in middle layers. 
Based on these observations, we propose $\textbf{LearnPruner}$, a two-stage token pruning framework that first removes redundant vision tokens via a learnable pruning module after the vision encoder, then retains only task-relevant tokens in the LLM's middle layer. 
Experimental results show that our LearnPruner can preserve approximately 95\% of the original performance while using only 5.5\% of vision tokens, and achieve 3.2$\times$ inference acceleration, demonstrating a superior accuracy-efficiency trade-off.

Published as a conference paper at ICLR 2026
LEARNPRUNER: RETHINKING ATTENTION-BASED TOKEN PRUNING IN VISION LANGUAGE MODELS
Rinyoichi Takezoe, Yaqian Li, Zihao Bo, Anzhou Hou, Mo Guang, Kaiwen Long"
Li Auto Inc.
longka wen x xxiangg.. com
ABSTRACT
Vision-Language Models (VLMs) have recently demonstrated remarkable capa- bilities in visual understanding and reasoning. but they also impose significant computational burdens due to long visual sequence inputs. Recent works address this issue by pruning unimportant visual tokens, achieving substantial computa- tional reduction while maintaining model performance. The core of token pruning lies in determining token importance, with current approaches primarily relying on attention scores from vision encoders or Large Language Models (LLMs). In this paper, we analyze the effectiveness of attention mechanisms in both vision en- coders and LLMs. We find that vision encoders suffer from attention sink, leading poor focus on informative foreground regions, while in LLMs, although prior studies have identified attention bias toward token positions, text-to- visiin atten- tion demonstrates resistance to this bias and enables effective pruning guidance in middle layers. Based on these observations, we propose LearnPruner, two- stage token pruning framework that first removes redundant vision tokens via a learnable pruning module after the vision encoder, then retains only task-relevant tokens in the LLM's middle layer. Experimental results show that our Learn- Pruner can preserve approximately 95% of the original performance while using only 5.5% of vision tokens, and achieve 3.2x inference acceleration, demonstrat- ing superior accuracy-efficiency trade-off.
INTRODUCTION
In recent years, with the rapid advancement of Large Language Models (LLMs) Guo et al. (2025) Achiam et al. (2023): Bai et al.) (2023), Vision-Language Models (VLMs) have achieved remark- able progress. By extending the comprehension and reasoning capabilities of LLMs to the visual
modality, VLMs have demonstrated unprecedented performance across various multimodal tasks, including visual question answering, image captioning, and visual reasoning Li et al. (023a); Liu
et al. 2023): y
Wang et al. 2024).
Existing VLMs typically convert images into discrete token sequences through vision encoders and then feed them into LLMs after modal alignment modules. However, the massive number visual tokens poses significant computational challenges for VL.M inference, particularly in high-resolution image or long-video input scenarios. This computational burden severely limits the practical deploy- ment of VLMs in resource-constrained environments and real-time applications.
To mitigate these computational challenges, token pruning has emerged as a promising solution. The core idea of token pruning is to assign importance scores to individual visual tokens through specific criteria, then keep only the top-k most important tokens while discarding the rest during the inference phase. Since visual content typically has much lower information density than text, substantial reductions in visual token count through pruning techniques yield minimal performance degradation.
Obviously, the effectiveness of token pruning heavily depends on accurate visual token importance assessment. Recent studies primarily rely on attention scores from visual encoders or LLMs as
"Corresponding author.

---

Published as a conference paper at ICLR 2026
Vision Encoder
Image
Query
[CLS]]
Vision
Text
All
Are cats inside?
buran wearing jacket?
What are the two visible numbers?
Figure 1: Attention maps vision encoder and LLM decoder in LLaVA-1.5 Liu et al. (202)). The former leverages the attention scores of the [CLS] token derived from the second last layer of the vision encoder, while the latter shows the average attention received by vision tokens from vision, text, and all tokens in the middle layer (12th layer) of the LLM.
the metrics, e.g., [CLS] attention Zhang et al. 2025a) Yang et al. (2025) or average LLM atten- tion Chen al. (2024): Zhang al. (2025b). To investigate whether attention mechanisms accu- rately reflect token importance, we visualize the attention heatmaps of LLaVA-1.5 Liu et al. 2003)).
widely-used VLM, from both the vision encoder and LLM.
As shown in Figure while the [CLS] token can partially focus on foreground objects, it often allocates excessive attention to low-informative background regions. This observation consistent with the prior al.) (202T) tend generate high-norm outlier tokens, i.e., artifacts that aggregate global image information while discarding spatial detail. On the other hand, while prior work Zhang et al. (2025a) identified the attention shift phenomenon in LLMs, where LLMS exhibit a bias toward the lower half of images due to the locality of positional encoding and causal masking mechanisms, this bias is mainly observed in tisionntovission or all-token attention patterns. In contrast, text-to-vision attention appears effectively focus gulevatelevant regions.
et al. (2023) that shows Vision Transformers CaansfoorrrrOQquubbb et al. (202)) Radford
The above observations motivate us to conduct an in-depth analysis on the effectiveness of current baaeed token pruning methods, revealing critical insights: (1) In vision encoder, [CLS] token fails to adequately attend to salient foreground objects, which leads to suboptimal pruning results, particularly under limited token budgets. (2) In LLM,, text-to-vision attention demonstrates robustness in resisting attention shift, and can provide reliable guidance for selecting query-relevant vision tokens in the middle layer. However, delaying pruning until the middle layers still involves substantial redundant computations, resulting in marginal acceleration gains.
To address these limitations, we propose LearnPruner, two-stage token pruning framework that enhances the efficiency of VLMs by sequentially removing unnecessary vision tokens after the vi- sion encoder and in the LLM. Specifically, we first eliminate inherent visual redundancy by em- ploying a lightweight learnable module to directly predict the importance scores of vision tokens, replacing the conventional [CLS] attention scores. Moreover, a small set of diversity tokens are re- tained to provide complementary visual information. The remaining tokens then pass into the LLM for cross-modal interaction with the query instructions. Subsequently, we perform query-aware to- ken selection in the middle layer of LLM, further discarding tokens that are irrelevant to the given query.
Benefiting from the refined pruning strategy and importance measures, LeamPrrner achieves fa- vorable accuracy-efficiene trade-off. Extensive experiments on various VLM benchmarks demon- strate that LearPrrnner outperforms previous state-of-the-art methods. With only 5.6% of tokens
2

---

Published as a conference paper at ICLR 2026
retained, LearnPruner preserves 94.8% of the original performance and achieves 2.3x and 1.5x speedup in prefill and total time, respectively.
2
RELATED WORK
2.1
VISION-LANGUAGE MODELS.
The evolution of Vision-Languaal Models (VLMs) has progressed from early joint embedding ap- proaches Kiros et al. (2014); Karpathy & Fei-Fei (2015) to sophisticated architectures leveraging Large Language Model capabilities. Modern VLMs such LLa
al. (2023). MiniGPT- 4 /Zuu et al. (202)), and Owen-VL Bai et al. 2023) have demonstrated remarkable performance in multimodal understanding by integrating visual encoders with powerful language models through
modal alignment modules.
Moreover, recent VLM inputs have expanded to high-resolution images and video sequences, dra- matically increasing visual token sequences. For example, LLaVA-1.5 Liu et al. (2023) generates 576 tokens per image, VA
al. (2024a) divides the high-resolution images into grid of sub-images, generating up to 5 576 = 2, 880 tokens, and VA One Vision Li et al. (2024) applies pooling operations to video frames, creating up 32 x 196 6, 272 tokens. This trend toward longer visual sequences has not only increased computational burden but also created a sig- nificant imbalance in multimodal inputs, where visual tokens often dominate the input composition. However, visual modalities exhibit substantially higher redundancy compared to language, leading to a mismatch between token quantity and information density, which motivates the necessity for visual token reduction research in VLMs.
2.2 VISUAL TOKEN REDUCTION FOR VLMS.
As mentioned above, visual token reduction aims to improve inference efficiency by removing re- dundant information in images. FastV Chen et al.] (2024) is a pioneering work in this field, which computes the average attention scores one token received from all other tokens within the LLM to determine token importance, and prunes unimportant tokens at the shallow layer of the LLM. Pyra- midDrop Xing et al. (2024) observes that redundancy increases progressively within LLMs, thus proposing hierarchical pruning strategy. Sparse VLM VLanseg al. (2025b) leverages text-to-viss tivvisoon attention scores to achieve text-aware guidance, and utilizes the rank of attention matrices to adap- tively adjust the pruning ratio. Meanwhile,
studies apply pruning strategies directly after the vision encoder. VisPruner [Zhang et al.) (2025a) argues that attention shift and attention dispersion issues exist within LLMs, therefore replaces the importance criteria with [CLS] token attention scores. VisionZip Yang et al.] (2025) further incorporates the token merging technique to avoid losing any small but potentially important information. Alternatively, from diversity perspective, DivPrune Alvar et al. (2025) and DART Wen et al. (2025) select tokens based on feature similarity to maintain diverse set that captures comprehensive visual context.
In addition to the above training- frree methods, recent studies have introduced training-ban -baeed pruning methods to further improve accuracy, the expanse of acceptable additional training overhead. ATP-LLaVA Ye et al. (2025) utilizes the global attention distribution of images to predict instance- specific pruning thresholds. TwigVLM [Shao et al. (2025) inserts additional decoder blocks in the shallow layers of the LLM, which not only perform more reliable pruning, but also enable decoding stage acceleration through self speculacce decoding. Despite existing works relying on attention results to remove redundant tokens, the attention mechanism exhibits inherent limitations, and its effectiveness warrants further exploration.
METHOD
3.1 PRELIMINARY
Existing works typically rely attention map derived from the transformer block to achieve token pruning. The attention mechanism Vaswani et al. (2017) widely applied in both key components of VLMs (the vision encoder and the I.LM) facilitate token interaction. Formally, given token
3

---

Published as a conference paper at ICLR 2026
Radera (17
Viice
Vikeo Teat
-d--
-3-
Rankrng
10
100
Led
Porkenuree (1)
Performance
Averge 60
80
70-
40
10
15
75%
9095
55%
100
200
300
400
300
515
50-
12
16
20
24
28
32
Pruning Rate
Visual Token Indis
Layer
(6)
Figure 2: Analysis of attention in VLMs. (a) Performance comparison of token pruning strategies. 'Random' and [CLS]' denote random and [CLS] ] attention- -bsed selection from the entire image (all) or foreground regions (fg). (b) Distribution of attention received by vision tokens from both vision and text modalities. The bar chart presents the average values across multiple instances, while the dashed line chart shows the distribution of individual samples. (c) Performance comparison of importance criteria derived from different layers of LLM attention maps. Pruning is performed at the second layer with a 90% pruning ratio.
sequence X = [x1, T2, INN] RNxd, the attention computation first transforms the input into query Q key K, and value V respectively:
Q = XW K V XW.
where W WR. and W, are projection matrices, N the sequence length and is the hidden dimension. Then, the attention map A and output O are computed as:
(Aogit + M
logit = QK", A softmax
AV
(2)
dd
where A denotes the attention map and M RNNN is optional mask matrix. For vision encoders adopting global attention, M is a zero matrix, while for LLMs employing causal attention, M an upper triangular matrix of oo values ensure each token can only attend to previous tokens.
3.2 STUDY ATTENTION VLMS
Attention in Vision Encoder. Vision encoders commonly include an additional [CLS] token at the beginning the token sequence, serving interact with patch tokens and aggregate global formation. The [CLS] token is expected to focus on the visually salient regions. Consequently, existing works naturally utilize the attention score from the [CLS] token as the importance estima- tion of the patch tokens. However, recent studies Darcet et al. 22023) Yang al. (2024) reveal that most vision encoders tend to generate artifacts uniform, background areas. These artifacts receive disproportionately high attention from other tokens, despite containing limited semantic informa- tion. Therefore a natural question arises: whether the [CLS] token adequately altend to salient foreground regions?
To answer this question, we design comparative experiment using LLaVA-1 1A-V-------- Liu et al. (2023) five benchmarks: GQA Hudson & Manning (2019) POPE Li al. (023b). MME Fu et al. 2003). TextVQA Singh et al. (2019) and VQAv2 Goyal et al. (017) (all subsequent experiments maintain the same setup unless otherwise specified). Specifically, ly. we employ LangSAM" an open- source visual grounding tool built on top of SAM-2 Ravi et al. (2224) and GroundingDINO Liu et al. (2024b), to segment the foreground objects of the image. We pre-deline a of common foreground and background categories, and obtain the complete foreground mask based on the comprehensive segmentation results.
We then evaluate two pruning strategies: global token selection and foreground-constraaned selec- tion, the average performance shown in Figure (a). For [CLS] attention-based pruning, con- straining token selection to informative regions (1 ([CLS) fg) consistently outperforms selection from
http:////tttttt..o/////aaaaaaaaa wwgps///nnnnnngggntttaaaamm -nnyhhing
4

---

Published as a conference paper at ICLR 2026
the entire image ([CLS) alt. especially under aggressive pruning ratios. Furthermore, random foreground token selection (Randomyg) fg) can achieve comparable performance to [CLS]ate all- This suggests that the [CLS] ] token may not effectively focus salient foreground regions, particularly that the most attended tokens show poor alignment with their importance, which is consistent with the observation in Figure
Attention in LLM. In LLMs, visual tokens not only interact within the modality but also with text tokens. Existing works typically utilize the average attention received by other tokens the last instruction token to estimate the importance score, and pruning usually occurs in the shallow layers of LLMs to achieve lower computational costs. However, recent studies Zhang al. (2025a) have identified the attention shift phenomenon in LLMs, where visual tokens with higher indices tend to receive higher attention scores. Intuitively, the information flow within LLMS should serve as guidance for token pruning, hence we ask: Is there a more effective way to leverage LLM attention for pruning assistance?
To answer this question, first ex- tend the analysis of attention shift based Zhang et al.] (2025a). We randomly select 1,000 image- text pairs from VQAv2 as inputs to the base VI.M and compute the av- erage attention received at each vi- sion token index. The attention decomposed into visual attention (from vision tokens) and text atten- tion (from text tokens). As shown in Figure (b), while both modali- ties demonstrates attention shift, the increasing trend in text attention is much more gradual than visual at- tention across token indices. This is mainly due to the decay property of positional encoding and the causal masking in visual attention. Meanwhile, although text attention overall exhibits positive correlation with token index, atten- tion patterns vary considerably across individual instances and tokens with lower indices can also be highly attended. In contrast, vision attention consistently shows bias, which may be harmful for pruning decisions.
Shallow Layer
Middlic Layer
Deep Layer
bothemandadd socees ballajin in this image?
Figure 3: Attention heatmaps from specific text tokens to images in shallow, middle and deep layers of LLM.
Based the observation, further conduct experiments to compare different importance criteria within LLMs. The criteria include the average attention from visual modality, text modality, both modalities, as well the attention from the last instruction token. For fair comparison of these criteria's 's effectiveness, we follow the approach of [Shao et al. (2025) by pre-computing attention maps from specific layer as guidance for token selection, then performing token pruning at the second layer. The pruning ratio is set to 90% and the results are shown in Figure
We can find that text attention and last token attention consistently outperform the other criteria across almost all layers. Incorporating visual attention leads to significant performance degradation, indicating that it is heavily affected by attention shift and fails to provide complementary informa- tion. Moreover, an interesting trend can be observed: attention reliability gradually increases from shallow layers (over 95% the baseline performance is retained from layer onwards) and shows stable performance in middle layers, but significantly declines as the layer goes deeper. more concrete example is shown in Figure(3 In both shallow and deep layers, uninformative regions tend absorb most of the attention from text tokens, while in the middle layer, different text tokens able to focus on their semantic-related regions, hence allowing for effective token selection.
3.3 LEARNPRUNER
Based on the above study, we propose LearnPruner, which leverages learnable pruning criteria to replace the attention-basee criteria in vision encoder and adopts progressive pruning strategy, per- forming pruning after the vision encoder and within the LLM respectively. The overall framework shown in Figure
5

---

Published as a conference paper at ICLR 2026
Is he swinging the but?
LLM Devoder Layer 1+1
LLM Decoder Layer
TexieGrideed
0000
Leurnable Pruning Module
Vision Encoder
Projector
Diversity- -hase
Token Selection
Remove Visual Redundancy
Remove Text Inrllevntt Consent
Figure 4: Overview of LearnPruner. After vision encoder, we select informative tokens based on importance socre predicted by learnable pruning module. Besides, limited number of diverse kens are selected based on similarity to minimize background information loss. Finally, we perform text-guided token selection in the middle layer of the LLM, further discarding tokens that are irrele- vant to the given query.
Remove Visual Redundancy. Visual redundancy is inherently present in images, thus it is intuitive perform pruning after the vision encoder. The stage aims to preserve the original image infor- mation through more compact token representations. Although [CLS] attention is widely used as the pruning criterion, as analyzed before, [CLS] ] token fails accurately attend to visually salient regions. Inspired by Rao et al. (202T); Huang et al. (2024), we employ a learnable pruning module (LPM) to directly predict the importance score of each vision token.
Specifically, we feed the token features from the vision encoder into lightweight MLP that per- forms binary classification to determine whether each token should be preserved or pruned. Since discrete binary decisions are non-differentiable, we employ Straigh-TThrough Estimator (STH) Ben- gio al. 2013) to enable backpropagation during training. This process can be formulated as:
M Softmax(MLP(XX
(3)
M hand = argmax(MM..)
(4)
where x!(O) denotes the token features from the vision encoder. Through the STE trick, the binary mask Mhard ensures that pruned tokens do not participate in subsequent computations during the forward pass, while the soft mask Moof provides gradients during the backward pass. Implementa- tion details of end-to-end training can be found in the supplementary material. At inference time, Mooft serves as the importance score for each token. More implementation details about LPM can be found in Appendix A.I Considering that LPM tends focus on semantically rich foreground regions, it might ignore background information that could also be crucial in certain VQA tasks, therefore we introduce diversity-basee based token selection module during inference to complement comprehensive visual con- text. After LPM selects a set of informative tokens, for each remaining token, we compute its cosine similarity with all selected tokens and identify its maximum similarity value. We then iteratively add the one with the smallest maximum similarity score, thereby ensuring maximum diversity in the selected set. This process continues until the token budget is reached.
Remove Text-Irrelevant Content. Although images contain vast information, not all visual content necessary for the given query, hence we perform a second pruning within LLM to further remove text-inrelevant tokens. As demonstrated in our analysis, text attention is less affected by attention shift and exhibits strong response to relevant regions. Inspired by this, we directly leverage text attention to guide token pruning. The text attention is computed the average attention from query

---

Published as a conference paper at ICLR 2026
Method
Params.
GQA
SQA'
VQA"
POPE MIME VQA
MMB
MMB
RelAcc.
Upper Bound, 576 Tokens (100%)
Vanilla
61.9
69.5
58.2
85.9
1862
78.5
64.7
58.3
100.0%
Retain Averaged
128 Tokens 77.8%)
FastV
49.6
60.2
50.6
59.6
1490
61.8
56.1
51.4
82.1
VLL
56
67.1
54.9
80.5
1696
73.8
60.0
51.1
92.6%
DivPrune
59.3
69.0
56.1
86.7
1718
76.0
62.0
54.8
96.4%
DART
57.9
69.1
56.3
80.4
1721
74.7
60.7
57.3
95.4%
VisPruner
58.2
57.0
84.6
1794
75.8
62.7
57.3
97.3%
VisionZip
57.6
68.9
56.8
83.2
1762
75.6
62.0
56.7
96.3%
VisionZip $
20.9M
58.9
68.3
57.0
83.7
1823
76.6
62.6
97.3%
TWigVL.M
610M
60.6
69.5
57.8
86.6
1818
77.9
63.5
99.0%
LearmPruner
0.53M
60.3
68.5
57.3
86.7
1820
77.3
63.8
56.8
98.5%
Retain
Averaged
64 Tokens (4 R8.
9%)
FastV
46.1
51.1
47.8
48.0
1256
$5.0
48.0
42.7
71.4%
VLM
52.7
62.2
51.8
75.1
1505
68.2
56.2
46.1
85.6%
DivPrune
57.8
68.2
54.7
85.6
1674
74.1
59.3
52.3
93.9%
DART
54.7
69.3
54.7
73.9
1705
71.3
59.5
54.0
91.9%
VisPruner
55.4
69.1
55.8
80.4
1689
72.7
61.3
55.1
94.3%
VisionZip
55.1
69.0
55.5
77.0
1690
72.4
60.1
55.4
93.0%
VisionZip I
20.9M
57.0
68.8
56.0
80.9
1756
74.2
61.5
95.1 1%
TWigVLM
610M
58.8
70.0
55.8
82.7
1760
75.6
60.4
96.0%
LearnPruner
0.53M
58.9
86.8
1750
62.6
55.7
96.9%
Retain Averaged 32 Tokens (L 94.4%)
FastV
41.5
42.6
42.5
32.5
1090
43.4
37.8
33,2
58.6%
VLM
48.3
57.3
46.1
67.9
1290
58.6
51.4
40.6
77.5%
DivPrune
54.9
68.6
52.9
81.5
1611
71.2
57.6
49.1
90.5%
DART
52.9
69.3
52.2
69.1
1615
67.1
58.5
50.0
88. 0%
VisPruner
52.2
69.2
53.9
72.7
1567
67.7
58.4
52.7
89.7%
VisionZip
51.8
68.8
53.1
68.7
1536
67.1
57.7
50.3
87.8%
LearnPruner
0.53M
57.2
68.2
56.1
84.5
1672
74.0
60.8
55.5
94.8%
Table 1: Performance comparisons of different pruning methods on LLaVA-v1.5-7B. 'Params." de- notes the number of learnable parameters and 'RelAce.' denotes the average relative accuracy across all benchmarks compared to the vanilla model. Bold and underlined values indicate the best and second-best performance, respectively.
tokens over all heads:
N
A (k)
1
(k) x
(k)).
(5)
Ng
where x and x denote query tokens and vision tokens at the k-th layer, respectively. Ng is the number of query tokens. We only keep the top-k tokens with the highest attention scores to participate in further interactions within the LLM.
4 EXPERIMENTS
In this section, evaluate the performance of LeamPrunnr widely-used VL.Ms across ious benchmarks, to compare with the state-of-the-ar token pruning methods. Then we conduct ablation studies analyze the effectiveness of each component.
4.1 EXPERIMENTAL SETUP
In the experiments, train model using 10% of the LLaVA-665K dataset Liu et al. (023) We keep the base VLM weights frozen to preserve the original performance and only train the LPM. During inference, the first stage retains R₁ tokens after vision encoder, where X 10% of R tokens are kept as diversity tokens. The second pruning stage is performed in the A = 12-th layer of the LLM,, retaining Ra tokens. For evaluation, fix the average number of retained vision tokens

---

Published as a conference paper at ICLR 2026
Method
GQA
SQA!
VQA
MME VQA
MMB
RelAcc.
Upper Bound, 2,880
Tokens (100%)
Vanilla
64.2
70.2
61.3
1842
81.2
67.9
100%
Retain Averaged 640
Tokens 77.8%)
Sparse VLM
60.3
67.7
57.8
1772
77.1
65.7
95.4%
DivPrune
61.9
67.8
57.0
1760
79.3
65.8
96.0%
VisionZip
61.3
68.1
60.2
1787
79.1
66.3
97.1 %
VisionZip
62.4
67.9
60.8
1778
79.9
65.9
97.5%
TwigVLM
63.4
69.9
58.6
1864
81.2
67.4
99.
LearnPruner
63.8
70.0
60.5
1837
80,5
67.5
99.3% %
Retain Averaged 320
Tokens
88.9%)
VL.
57.7
67.3
55.9
1694
73.4
64.3
92.3%
DivPrune
61.1
67.7
56.2
1724
77.2
63.9
94.3%
VisionZip
59.3
67.3
58.9
1702
76.2
63.1
93.9%
VisionZip
61.0
67.5
59.3
1770
78.4
64.4
95.9%
TwigVLM
62.2
68.7
57.4
1758
79.7
65.0
96.3%
LearnPruner
62.2
68.6
58.4
1845
78.3
66.8
97.5%
Retain Averaged 160
Tokens
94.44%)
VLM
51.2
67.5
46.4
1542
66.3
63.1
85.0%
DivPrune
59.3
67.1
54.1
1643
75.0
62.9
91.7%
VisionZip
55.5
68.3
56.2
1630
71.4
60.1
90.1 %
VisionZip
58.2
67.5
57.3
1699
75.6
63.9
93.3%
LearnPruner
58.7
67.6
55.0
1784
76.2
65.3
94.0%
Table 2: Performance comparisons of different pruning methods on LLaVA-NeXT-7B.
across all LLM layers ensure fair comparison. Given the target token budget, we the ratio of R2 and Rg to 3 to determine their specific values.
TGIF-QA
MSVD-QA
MSRYTT-QA
Average Ace.
Method
Acc.
Score
Acc.
Score
Acc.
Score
Score
Upper Bound, 2,048 Tokens (100%)
Vanilla
18.1
1.27
64.1
3.41
56.1
2.97
46.1
2.55
Retain Averaged 456 Tokens (1 77.8%)
FastV
16.4
1.17
60.3
3.24
53.0
2.84
43.2
2.42
LeamPrrunr
17.5
1.25
63.6
3.36
55.3
2.93
45.5
2.51
Retain Averaged 228 Tokens
88. 88.9%)
FastV
12.4
1.00
2.93
50.4
2.70
39.3
2.21
LearnPruner
16.3
1.17
61.5
3.30
53.8
2.87
43.9
2.45
Table 3: Video understanding performance comparisons of different pruning methods on Video- LLa VA-7B. Performance is evaluated on the first Ik samples of each benchmark. We use GPT-4.1 to assist in evaluating the accuracy.
4.2 MAIN RESULTS
To demonstrate the effectiveness of LearnPruner, we present a comprehensive evaluation of Learn- Pruner various VLMs, including LLaVA-v1.5-7B, LLaVA-NeXT-7B for high resolution input, Video-LLaVA VA-BB for video input and Qwen2.5-VL-77 (beyond the LLaMA family).
Results on LLaVA--1.5-7B. We first deploy our method on LLaVA-v1.5-7BB and then eval- uate the performance on eight image understanding benchmarks. The compared methods in- clude Vast'V Chen et al. (2024), SLM Zhang et al. (20256), DivPrune Alvar al. (2025). DART Wen et al. (2025). VisionZip Yang et al. (2025). VisionZip Yang et al. (2025). Vis- Pruner Zhang et al. (2025a) and TwigVLM Shao et al. (2025). Among them, VisionZipt and TwigVLM require additional training, while the rest training-free. As shown in Table when the number of vision tokens reduced from 576 to 128, LearPPruner only decreases the average accuracy by 1.5%, showing comparable performance to TwigVLM. It is worth noting that Leam- Pruner is more lightweight, as TLi introduces additional decoder layers for training, leading to increased model complexity. Moreover, method demonstrates greater advantages in scenarios
8

---

Published as a conference paper at ICLR 2026
Method
GQA
SQA'
VQA
MME
VQA
MMB
Avg
Upper Bound, 1280
Tokens (100%)
Vanilla
59.7
77.4
76.7
2324
83.9
83.8
100%
Retain Averaged 426
Tokens 66. 7%))
FastV
58.4
79.1
75.7
2330
81.4
81.4
98.9%
LearnPruner
59.3
77.4
75.2
2339
82.4
83.2
99.3%
Retain Averaged 284 Tokens (4 77.8%)
FastV
53.6
77.8
72.7
2228
77.8
79.1
94.7%
LearPrruner
59.0
77.0
74.4
2358
81.7
82.2
98.7% %
Retain Averaged 142
Tokens ( 88. 9%)
FastV
42,8
70.5
54.8
1565
57.2
56.2
71.4%
LearnPruner
51.9
76.1
70.8
2292
79.3
81.8
94.1 %
Table Performance comparisons different pruning methods on Qwen2.5-VL-7B.
KV Cache (MB)
GPU Memory
Method
Token
TFLOPs
Prefill Time
Total Time
(GB)
LLaVA-v1.5-76
576
8.6 (1.0x)
463.2 (1.0x)
761.6 (1.0))
318.9 (1.0x)
14.7 (1.0x)
128
2.8 (3.1x)
262.6 (1.8x)
569.8 (1.3x)
95.0 (3.4x)
13.5 (1.1x)
+ -LeamPrunerr
32
1.6 (5.4x)
201.4 (2.3x)
507.3 (1.5x)
47.0 (688)
13.3 (1.1x)
I.aVA-NeXT-7b
2880
31.9 (1.0x)
1891.5 (1.0x)
2307.6 (1.0x)
1156.1 (1.0x)
(1.0%)
640
11.4 (2.xx)
683.0 (288))
1083.6 (2.1x)
369.4 (3.1x)
16.3 (1.3x)
+ LearnPruner
160
5.2 (6.1x)
315.6 (6.0x)
711.7 (3.2x)
129.4 (8.9x)
13.5 (15x)
Table 5: Performance comparisons on computational efficiency.
with more constrained token budgets. With only 64 or 32 tokens retained, LearnPruner maintains 96.9% and 94.8% of the original performance, respectively, significantly outperforming other meth- ods. This result suggests that our method can more effectively identify and preserve the most critical visual information.
Results on I.L.V--NeT--7S In Tabel 2 we further evaluate the performance of LearnPruner on LLaVA-NeXT-7B. LLaVA-NeXT splits the original image into multiple sub-images and feeds them together to the VLM to enable high-resolution input, resulting vision token sequences that can reach up to 2880 tokens. Although LLaVA-NeXT improves the ability of image understanding, it also increases computational burden and introduces more visual redundancy. As we can observed, LearnPruner consistently maintain strong performance under different settings. When remov- ing 88.9% of tokens and retaining only 320 tokens, LearnPruner preserves 97.5% of the original performance.
Results on Video-LLaVV-7R. LearnPruner is a general pruning method that can be extended to video domain. We conduct experiments using Verco-LLaVA-BBBLE. al. (2023) as the base model on three widely-used video understanding benchmarks: TGIF-QA Jang et al. (2017) MSVD-QA Xu al. (2017), and MSRVTT-QA Xu et al. (2017). Following previous work Chen al. 2024 and due to resource constraints, we evaluate on the first 1,000 samples from each benchmark and employ GPT Assistant to score the model responses. As shown in Table 3 LearnPruner achieves better accuracy and matching scores than FastV on all evaluated video QA tasks, further demonstrating the generalization across different application scenarios.
Results on Qwen2.5-VI-77B. To further demonstrate the generalization ability of LearnPruner, we conduct additional experiments on Qwen2.5-VL. VL-77B Bai et al. (2025). powerful VLM with a dis- tinct architecture from the LLaVA series. We follow the same training settings and the results are shown in Table LearnPruner still consistently outperforms FastV at different reduction ratios. When removing 88.9% of vision tokens, FastV's performance drops dramatically to 71.4% relative accuracy, while LearnPruner maintains 94.1 % of the original performance, showing the superiority of our method.

---

Published as a conference paper at ICLR 2026
4.3 ABLATION STUDIES
To validate the impact of different components in LearnPruner, we conduct ablation experiments on -LaVA-vI.--7B. We evaluate the model on benchmarks mentioned in Table [ and report the average performance, the number of retained tokens fixed 64, More ablation studies for pruning strategies are presented in Appendix A.2
Effectiveness of importance criteria. The core of LeamPrrunr lies in replacing [CLS] attention- based importance criteria with learnable prediction scores, and implementing a two-stage pruning strat- egy. We conduct comparison experiments to ver- ify the effectiveness of these design choices and the result is shown in Table o In the stage of remov- ing visual redundancy, compared to selecting tokens by [CLS] attention, LPM improves performance by 1.7%, demonstrating that LPM more effectively focus on salient foreground regions. Notably, us- ing LPM alone in Stage is conceptually similar
Stagel
Stage2
RelAce. %)
[CLS] Atm
94.6
LPM
96.1
LPM
96.9
LPM
Text Attn
96.9
Table 6: Effectiveness of different impor- tance criteria.
Dynamic-LLaVa Huang et al.] (024). which also employs learned importance predictors for token selection. However, can be seen that incorporating Stage? further improves performance, demon- strating that our two-stage pruning strategy enables more effective token budget allocation. In the stage of removing text-irrelevant content, we attempt to insert an additional LPM module in the LLM as well, combining token features and attention distributions predict importance scores. However, the results show no further performance improvement, indicating that attention signals are already sufficiently reliable and token features cannot provide complementary information. Therefore, finally decide to directly utilize the attention results for pruning, avoiding the need to train multiple LPMs..
4.4 EFFICIENCY ANALYSIS
To demonstrate the efficiency of LearnPruner, conduct a comparative analysis of computational cost and memory usage on VA-v.--113 and LLaVA-Next-7BB As shown in Table [5] we use NVIDIA A100-80GB to perform the inference POPE dataset. When the number of retained to- kens is reduced from 576 to 32 in LLaVA-vI.5-7BB the prefill time and total time achieve 5,4 x and 2.3x speedup respectively, while the KV cache storage is reduced by 6.8x. Due our lightweight design, the computational overhead and memory usage of LPM are negligible. Moreover, the effi- ciency improvements become more pronounced the visual token sequence length increases. When the number of retained tokens is reduced from 2,880 to 160 in LLaVA-Next-7B. the prefill time and total time achieve 6.0x and 3.2x speedup respectively.
5 CONCLUSION
In this paper, we conduct an in-depth analysis on attention mechanisms from both vision encoders and LLMs. Building the observations, propose LearnPruner, two-stage pruning framework that first removes redundant vision tokens via a learnable pruning module after the vision encoder, then further discards text-irrelevant tokens in the LLM's middle layer. Extensive experimental sults show that our LearnPruner outperforms previous state-of-the-ate methods and achieves a better accuracy-efffiienng trade-off.
6 ETHICS STATEMENT
The research presented in this paper focuses on token pruning for Vision-Language Models (VLMs) to improve their computational efficiency. The research process of this paper does not violate ICLR ethics. There are no discrimination, bias, or faimess issues that need to be addressed. Our models not expected generate potentially harmful content.
10

---

Published as a conference paper at ICLR 2026
7 REPRODUCIBILITY STATEMENT
This article proposes novel token pruning approach for VLMs to improve their inference efficiency. The base model and dataset used in this article are all from open-source and well-referenced, so this aspect does not affect the reproducibility. To further ensure reproducibility, we describe the implementation details in the main text Section [4.] and the Appendix A.I We will release source code and model checkpoints to support reproducibility.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint arXiv:2303.0877/ 2023.
Saeed Ranjbar Alvar, Gursimran Singh, Mohammad Akbari, and Yong Zhang. Divprune: Diversity- based visual token pruning for large multimodal models. In Proceedings of the Computer Vision and Puttern Recognition Conference, pp. 9392-9401, 2025.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309. 16609, 2023.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang. Peng Wang, Shijie Wang, Jun Tang. Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang. Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report. arXiv preprint ar arXiv:2502. 13923, 2025.
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating propagating gradients through stochastic neurons for conditional computation. arXiv preprint / 2013.
David Chen and William Dolan. Collecting highly parallel data for paraphrase evaluation. In Proceedings of the 49th annual meeting of the association for computational linguisties: human language technologies, pp. 190-200, 2011.
Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang- An image is worth 1/2 tokens after layer 2: Plug-and-plas inference acceleration for large vision- language models. In European Conference on Computer Vision, pp. 19-35. Springer, 2024.
Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers. arXiv preprint arXiv:2309. 16588, 2023.
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, et al. Mme: comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306. 13394, 2023.
Yash Goyal. Tejas Khot, Douglas Summers- Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6904-6913, 2017.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in lims via reinforcement learning. arXiv preprint arXiv:2501.12945 2025.
Wenxuan Huang, Zijie Zhai, Yunhang Shen, Shaosheng Cao, Fei Zhao, Xiangfeng Xu, Zheyu Ye, Yao Hu, and Shaohui Lin. Dynamic-llava: Efficient multimodal large language models via dy- namic language context sparsification. arXiv preprint arXiv:2412.0088 2024.
Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6700-6709, 2019.

---

Published as a conference paper at ICLR 2026
Yunseok Jang, Yale Song, Youngjae Yu, Youngjin Kim, and Gunhee Kim. Tgif-qa: Toward spatio- temporal reasoning in visual question answering. In Proceedings of the IEEE/CVE Conference Computer Vision and Pattemn Recognition, pp.
2017.
Andrej Karpathy and Li Fei-Fei. Deep visual-semantic alignments for generating image descrip- tions. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 3128-3137, 2015.
Ryan Kiros, Ruslan Salakhutdinov, and Richard S Zemel. Unifying visual-semantic embeddings with multimodal neural language models. arXiv preprint arXiv:1411.2539 2014.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International Journal of Computer Vision, 123(1):32-73, 2017.
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang. Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408. 03326, 2024.
Junnan Li, Dongxu Li. Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International Conference on Machine Learning, pp. 19730-19742. PMLR, 2023a.
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. arXiv preprint arXiv:2305. 10355, 2023b.
Yuncheng Li, Yale Song, Liangliang Cao, Joel Tetreault, Larry Goldberg, Alejandro Jaimes, and Jiebo Luo. Tgif: A new dataset and benchmark on animated gif description. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 4641-4650, 2016.
Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. Vidco-llava: Learning united visual representation by alignment before projection. arXiv preprint arXiv:2311.10122, 2023.
Tsung-Y! Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European Conference Computer Vision, pp. 740-755. Springer, 2014.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, (6:34892-344916 2023.
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lec. Llavanext: Improved reasoning, ocr, and world knowledge, 2024a.
Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang, Hang Su, al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. In European Conference on Computer Vision, pp. 38-55. Springer, 2024b.
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model all-around player? In European Conference on Computer Vision, pp. 216-233. Springer, 2024c.
Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems. 55:2507-2521 2022.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.077/93, 2023.
12

---

Published as a conference paper at ICLR 2026
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp. 8748-8763. PmLR, 2021.
Yongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu, Jie Zhou, and Cho-Jui Hsich. Dynamicvit: Efficient vision transformers with dynamic token sparsification. Advances in Neural Information Processing Systems, 34:13937-13949, 2021.
Nikhila Ravi, Valentin Gabeur, Yuan- Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, et al. Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408.007/4 2024.
Zhenwei Shao, Mingyang Wang, Zhou Yu, Wenwen Pan. Yan Yang. Tao Wei, Hongyuan Zhang, Ning Mao, Wei Chen, and Jun Yu. Growing twig accelerate large vision-language models. 2025.
Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 8317-8326, 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, tukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Informa- fion Processing Systems, 30, 2017.
Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et Qwen2-vl: Enhancing language model's 's perception of the world at any resolution. arXiv preprint arXiv:2409. /2191, 2024.
Zichen Wen, Yifeng Gao, Shaobo Wang, Junyuan Zhang, Qintong Zhang, Weijia Li, Conghui He, and Linfeng Zhang. Stop looking for important tokens in multimodal language models: Duplica- tion matters more. arXiv preprint arXiv:2502.1 1494, 2025.
Long Xing, Qidong Huang, Xiaoyi Dong, Jiajie Lu, Pan Zhang, Yuhang Zang, Yuhang Cao, Conghui He, Jiaqi Wang, Feng Wu, et al. Pyramiddrop: Accelerating your large vision-language models via pyramid visual redundancy reduction. arXiv preprint arXiv:2410. 17247. 2024.
Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Yueting Zhuang. Video question answering via gradually refined attention over appearance and motion. In Pro- ceedings of the 25th ACM International Conference Multimedia, pp. 1645-1653, 2017.
Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 5288-5288 0288-599,777 2016.
Jiawei Yang. Katie 7. Luo, Jiefeng Li, Congyue Deng, Leonidas Guibas, Dilip Krishnan, Kilian Q Weinberger, Yonglong Tian, and Yue Wang. Denoising vision transformers. In European Confer- ence on Computer Vision, pp. 453-469, Springer, 2024.
Senqiao Yang, Yukang Chen, Zhuotao Tian, Chengyao Wang, Jingyao Li, Bei Yu, and Jiaya Jia. Visionzip: Longer better but not necessary in vision language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19792-19802, 2025.
Xubing Ye, Yukang Gan, Yixiao Ge, Xiao-Ping Zhang, and Yansong Tang. Atp-llava: Adaptive token pruning for large vision language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 24972-24982, 2025.
Qizhe Zhang, Aosong Cheng, Ming Lu, Renrui Zhang, Zhiyong Zhuo Jiajun Cao, Shaobo Guo, Qi She, and Shanghang Zhang. Beyond text-visual attention: Exploiting visual cues for effective token pruning in vlms. Proceedings of the IEEE International Conference on Computer Vision, 2025a.
13

---

Published as a conference paper at ICLR 2026
Yuan Zhang, Chun-Kai Fan, Junpeng Ma, Wenzhao Zheng, Tao Huang. Kuan Cheng, Denis Gu- dovskiy, Tomoyuki Okuno, Yohei Nakata, Kurt Keutzer, et al. Sparsevlm: Visual token sparsi- fication for efficient vision-language model inference. In International Conference Machine Learning, 2025b.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En- hancing vision-language understanding with advanced large language models. arXiv preprint orXiv:2304.10592 2023.
14

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.1 TRAINING IMPLEMENTATION DETAILS
In this section, introduce the training method of learnable pruning module in LearnPruner.
End-to-end Optimization. Unlike inference phase that we directly discard tokens based on the importance score M predicted by LPM, need to preserve the complete token sequence to optimize LPM in end to end training process.
off the interactions between pruned tokens and retained tokens in the attention layers to simulate the pruning process, enabling the LPM learn how assess token importance for the task. More specifically, token mask by substituting the visual token positions in an all-ones vector with the binary mask Mord to create the full token sequence mask M. Then we implement an attention masking strategy by modifying the Softmax(-) function:
Following Rao et al. 2021), given binary mask Mhand
first obtain the complete
1,
i=j
Gú
Mj,
i i
(6)
cxp( (xp(Q.K] (Q.K))
A Σn exp( Q.K G
G is constructed as graph adjacency matrix, where G5 = 1 denotes that the j-th token participates in the attention computation of the i-th token. With this design, Á is equivalent the attention matrix calculated by actually discarding masked tokens.
Objective Function. Our goal is to encourage the model use the least number of vision tokens to produce the correct answer. To achieve this, we introduce a pruning loss to constrain the number of kept tokens, which defined as:
1 Σ Mhard - NV
2
Eprine
where N₂ is the number of vision tokens and denotes the target ratio of kept tokens. We set to encourage minimal token usage. With the regularization term, the final objective is:
[ Lmp + A Eprune
(8)
where Cmp is the original VLM loss and we set x = 1 in all our experiments.
A.2 MORE EXPERIMENTAL RESULTS
Ablation on pruning position k. The pruning position within the LLM affects the reliability of attention distribution and the number of tokens retained each stage. Table 7a varies k from 8 to 16 to investigate the impact on LearnPruner. We observe that when pruning at shallow layers, with
tokens retained, performance still significantly decreases, indicating that attention distribution from shallow layers is not reliable, which aligns with the observations in Figure 2 Then the performance begins to stabilize from 12-th layer onwards, and we finally select k=12 which achieves the best performance.
Ablation on retention ratio between R2 and Rg. The ratio between R2 and Rg determines the number of tokens retained at each stage. A smaller ratio leads to more visual information loss, while a larger ratio leads to a limited token budget allocated text-related regions. As shown Table the best result is achieved at R R₂ 3. which is chosen in our default setting. It indicates that is important to preserve the complete visual information the shallow layers, while only a small number of tokens are required in the middle layers, which is consistent with our two-stage pruning strategy.
Ablation on proportion of diverse tokens X. The hyper-parameter À controls the proportion of diverse tokens retained in the first stage. A low À risks losing comprehensive visual context, partic- ularly background information that could also be crucial in cetrain scenarios. Conversely, high À may allocate more tokens to less informative regions, leaving insufficient budget to adequately pre- serve salient foreground objects. As shown in Figure (a), performance drops when À approaches
15

---

Published as a conference paper at ICLR 2026
k
R1
R2
RelAcc.
R1
R2
R1
R2
RelAcc. (%)
8
129
43
96.2
85
85
96.1
10
120
96.3
1.5
102
96.6
12
111
96.9
2
114
57
96.8
14
105
35
96.8
3
129
43
96.9
16
96
32
96.9
4
136
34
96.8
(a) Ablation study on pruning position &o in the LLM. Ri and Ra the retained token bers of first and second pruning stages, respec- tively.
(h) Ablation study retention ratio between R1 and Rl.
Table 7: Ablation experiments for LearnPruner. We LLaVA-v1.5-7B as the base model and
evaluate the average performance on benchmarks mentioned in Table | Under default settings, pruning position & set 12 and the number of retained tokens between two pruning stages R1 : R2 is fixed to 3.
44.5
1050
1/aV
1060
LLaVA NeXT
128
C-.
Ray 64
95.5
Reg f4
99.5-
Reg 320
Pofommnce (Ka)
98.5
68.0
95.8-
98.0
67.5
97.5
95.5
96.5
61
96.8
50
110
96.0
Prrguttion Thiining Data (94)
Progortion of Training Datta
(a)
(b)
Figure 5: Ablation experiments for LearnPruner. (a). Ablation study proportion of diverse tokens in the first pruning stage. We use LLaVA-v1.5-7B as the base VLM. (b). Ablation study on different training data scales. We use LLaVA-v1.5-7B and LLaVA-NeXT-7B the base VLM and train multiple models with different proportions 15%-100% the LLaVA-665K dataset.
1, most VQA tasks focus foreground objects. Finally, empirically set À 0.1, which achieves the optimal balance between preserving semantic-rich foreground content and maintaining necessary background context across diverse visual scenarios.
Data efficiency. To evaluate the impact of training data scale on LearnPruner, we train multiple models with different proportions (5%-100%) of the LLLVA-66555 dataset. As shown in Figure (b), LearnPruner achieves strong performance with only 5% of the training data, and the model maintains consistent performance across different data ratios. This demonstrates the robustness and efficiency of LearnPruner in training. which can be attributed to the lightweight and concise design of our LPM module. Such data efficiency makes LearnPruner highly practical and flexible for deployment in real-world scenarios.
A.3 QUALITATIVE RESULTS
To better understand the effectiveness of LearnPruner, we visualize the pruning results on the exam-
ples chosen from GQA and TextVQA in Figure Figure We select Fast'V [Chnn et al. (2024) and VisionZip Yang et al.] (2025) for comparison, which represent pruning approaches conducted after the vision encoder and within the LLM, respectively. VisionZip relies on [CLS] attention for pruning, which fails properly focus on foreground objects and even selects meaningless padding tokens. On the other hand, limited by token budget, FastV prunes tokens after the second layer the
LLM, showing bias toward preserving tokens in the lower half of the image and weak responsive- ness to queries. In contrast, LearPrruner first removes visual redundancy after the vision encoder, accurately focusing semantically rich regions through learning-based based importance criterion, and
16

---

Published as a conference paper at ICLR 2026
supplements background information with small number of diversity tokens. In the second stage, leveraging reliable attention results from middle layers of the LLM, only the most query-relevant tokens are retained, enabling the VLM to answer questions using the minimal number of vision tokens. Futhermore, we show several failure cases in Figure 7 When dealing with complex visual scenes, the first-staee pruning may fail adequately preserve critical visual information within the constrained token budget, leading to information loss that propagates through subsequent layers. Alternatively, even when query-relevant visual tokens are well retained, prediction failures can still occasionally occur, which is mainly due to the inherent reasoning limitations of the base VLM.
A.4 EVALUATION BENCHMARKS
In this section, we provide a brief introduction about the evaluation benchmarks in our experiments.
GQA Hudson & Manning (201)). The GQA benchmark focuses visual scene understanding and reasoning. leveraging scene graph structures from Visual Genome Krishna et al.) (2017). involves spatial relations and object attributes, making it challenging for models achieve precise visual reasoning in complex scenes.
ScienceQA Lu et al. (2022). The ScienceQA benchmark uses multiple-choice questions to evaluate the zero-shot generalization on diverse science topics. The dataset contains rich domain diversity across three subjects: natural sciences, languaes science, and social science. Since some questions are not related to the image, we only evaluate the performance on the samples with images, denoted as "SQA!" in the experimental tables.
VaQA Singh et al. (2019). The TextVQA benchmark evaluates model's ability to process and understand text information in images. Answers the questions may be directly derived from the text in the images or require contextual reasoning. Moreover, optical character recognition (OCR) results is provided to assist the model to recognize text in the images. The dataset is denoted as "TextVQA" TM in the experimental tables.
POPE Li et al. (2023b). The POPE benchmark focuses on severe object hallucination issues in VLMs hence the questions mainly concern the presence of objects in the images. The reported result is calculated by the mean Fl score over the three indicators: adversarial, random, and popular.
MME Fu et al. (2023). The MME benchmark measures both perception and cognition abilities of VLMs on a total of 14 subtasks. Apart from OCR, the perception includes the recognition of coarse-grained and fine-grained objects. The former identifies the existence, count, position, and color of objects. The latter recognizes movie posters, celebrities, scenes, landmarks, and artworks. The cognition includes commonsense reasoning, numerical calculation, text translation, and code reasoning.
VQA-v2 Goyal (017)). VQA-v2 is large-scale benchmark consisting of 265,016 images from MSCOCO datasel Lin et al. (2014). Each image is paired with open-ended questions and 10 human-provided ground truth answers.
MMBench Liu et al. (2024c). MMBench is bilingual benchmark for assessing the multi-modal capabilities of VLMs, which incorporates multiple-choice questions in both English and Chinese versions. It defines three levels of ability dimensions: Level-1 (Perception and Reasoning), Level-2 (six sub-abilities), and Level-3 (twenty specific tasks). We use 'MMB' and 'MMBCN- to denote the English and Chinese versions of MMBench, respectively.
TGIF-Q -AIF-QQIIA Jang et al. (0117). The TGIF-QA benchmark extends the bmsee- VQA tasks to the video domain, requiring the model to understand and reason about spatial-temporal relationships in dynamic visual content. It includes 72K animated GIFs from the Tumblr GIF dataset Li et al. (2016) and 165K QA pairs. We employ GPT-4.1 to assist in evaluating the accuracy of the model's answers (same for the following two benchmarks).
MSVD-QA Xu et al. (2017). The MSVD-QA benchmark is constructed from the Microsoft Re- search Video Description Corpus Chen & Dolan (2011) and comprises 1,970 short video clips with 50,500 corresponding question-answe pairs.
MSRYTT-QA Xu et al. (2017). The MSRVTT-QA QA benchmark is based on the Microsoft Research Video to Text dataset Xu al. (2016), it contains 10,000 video clips and 243,000 QA pairs. The
17

---

Published as a conference paper at ICLR 2026
videos in MSRVTT-QA depict more complex scenes and activities than those in MSVD-QA, requir- ing models to effectively process both visual and temporal information.
A.5 LLM USAGE DISCLOSURE STATEMENT
We utilized OpenAT's GPT-4 for assistance with language editing (improving grammar and clar- ity) and for generating/debugging experimental code and evaluating some benchmarks. All LLM- generated content was thoroughly reviewed, verified, and edited by the authors, who take full sponsibility for the accuracy and integrity of this submission.
18

---

Published as a conference paper at ICLR 2026
LearnPrunee
Image
FastV
VisionZip
Stagel
Stage2
Are there pots?
X
X
yes
Are there chairs or TV stands?
no x
no X
yes
Does the rug in front of cabinet look and blue?
yes
yes
no
same
yes X
yes
X
color as the shoes?
What's the made of?
glass
X
glass X
wood
V
Figure 6: Visualization of the pruning results. Yellow patches indicate retained tokens, with an average of 32 tokens preserved.
19

---

Published as a conference paper at ICLR 2026
LeamPruner
Image
FastV
VisionZip
Stagel
Stage2
kind bag the right of the chair?
purse X
purse X
backpack
V
What is the number of the player yellow on the right?
10 X
10 X
5
am it. Build it Live
Live
B
Imagine X
A V
A
V
shirt brand start with?
Failure kind of food black and white?
dessert
X
cupcake X
cupcake X
Failure case the hydrant on the right?
yes X
yes X
yes X
Figure 7: Visualization of the pruning results. Yellow patches indicate retained tokens, with an average of 32 tokens preserved.
20


Reviews:
[{"review_id": "DnnCDNyTUR", "reviewer": "Reviewer_iSWN", "summary": "This paper proposed a new token pruning pipeline, LearnPruner, with a learnable module to determine token importance. Despite the prior concern with the bias in vision-to-vision attention, the proposed method leverage the more robust text-to-vision attention for the second stage token pruning. In the experiments, the proposed pruning framework is able to better preserve accuracy compared many prior baselines while providing similar benefit in computational cost reduction.", "questions": "See weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The learnable module provides a more robust way to acquire the visual token importance.\n- The overall performance is improved compared to prior baselines.", "weaknesses": "- There is no failure case analysis.\n- Ablation study on the LearnPruner training: what will happen if more or fewer training data samples are used?\n- Extra ablation study on the diversity-based token selection is needed.\n- The use of text-to-vision attention remain a bottleneck in the pruning process. The challenge of attention bias and drifting is not tackled with.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "n4rzKZE1jV", "reviewer": "Reviewer_uDiD", "summary": "LearnPruner is a two-stage token pruning framework for Vision-Language Models. It first uses a learnable module to remove redundant vision tokens, then prunes text-irrelevant tokens in LLM middle layers. It preserves 95% performance with only 5.5% tokens, achieving 3.2× faster inference and superior accuracy-efficiency trade-off.", "questions": "see weaknesses", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The writing is clear and easy to follow.\n2. Experimental results show strong performance on both LLaVA-1.5 and LLaVA-Next.", "weaknesses": "1. Experiments on LLaVA-1.5 and LLaVA-Next provide limited persuasiveness; please include comparisons on Qwen2.5-VL.\n2. The design of Remove Text-Irrelevant Content does not appear novel, it seems to be a straightforward reuse of existing ideas.\n3. Additional training is required compared with training-free methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VgrMzJiuHK", "reviewer": "Reviewer_FyhA", "summary": "This work proposes LearnPruner, a two-stage pruning approach for VLMs. The proposed approach first removes redundant vision tokens with a learnable pruning module after the vision encoder, and then retains only task-relevant tokens in the LLM's middle layer. These are based on some analyses on the VLM's attention patterns: in visual encoder, [CLS] fails to adequately attend to salient foreground objects, while in LLM, text-to-vision attention can provide reliable guidance for token selection. Experiments with multiple models and a variety of datasets show the effectiveness of the proposed approach.", "questions": "- I'm wondering how some of the hyper-parameters (such as k=12-th layer of the LLM for the second stage) are selected? Will these settings be model- or benchmark-sensitive?\n- The approach seems to be query sensitive, if we would like to perform pruning in a query-agnostic way, will the proposed approach be adaptable to this setting?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The direction of making VLMs more efficiency is important for its real-world deployment.\n- The method is well-motivated by preliminary analyses.\n- The proposed approach is shown to obtain good performance.", "weaknesses": "- It is unclear how well the Learnable Pruning Module can generalize to different tasks, that may be very different to the ones that have been seen in the training of the module.\n- The ablation study seems a little thin, where it would be better if more settings can be investigated. For example, what will be a good ratio of the compression of the vision encoder and the LLM? (The ratio is set to 3, but how would the overall efficiency be influenced if we have other settings?)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iZgszYLF2f", "reviewer": "Reviewer_ui91", "summary": "This work presents a two-stage pruning framework designed to make VLM inference more efficient while preserving accuracy. The first stage employs a learnable module after the vision encoder to estimate and discard redundant visual tokens, and the second stage applies a text-guided pruning process in the LLM’s middle layers, selectively retaining only tokens that are relevant to the query through text-to-vision attention.", "questions": "Please refer to the Weaknesses section.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- This study offers an insightful examination of the attention behavior in both the vision encoder and the LLM, pointing out the shortcomings of [CLS]-based attention and demonstrating how LLM attention can be effectively exploited for token selection.\n- Unlike prior works relying on attention heuristics ([CLS] or average attention), the learnable pruning module introduces a differentiable and data-driven importance predictor.\n- The clear organization and presentation make the paper easy to follow.", "weaknesses": "- The concept of the learnable pruning module appears highly similar to the learnable image-token predictor proposed in Dynamic-LLaVA (ICLR 2025), which leverages image token features to predict a binary mask for selecting tokens during prefill computation. However, the current work does not include a proper comparison or discussion of this prior work. It would considerably strengthen the paper to include Dynamic-LLaVA as a baseline in the experiments and to explicitly discuss methodological differences and advantages, thereby clarifying the novelty and demonstrating the superiority of the proposed approach.\n  * https://openreview.net/forum?id=hzVpZDrW73\n\n- The paper briefly mentions a diversity-based token selection module to preserve background information during inference. However, the description of this component is insufficient and does not clearly explain how the module is implemented. Moreover, I was not able to find an ablation study quantifying its contribution, making it difficult to assess the actual impact of the diversity-based selection compared to the LPM.\n\n- Diversity-based vision token pruning methods are not properly discussed or compared. It would strengthen the paper to include these approaches as baselines in the comparison experiments to better demonstrate the superiority of the proposed method.\n  * DivPrune: Diversity-based Visual Token Pruning for Large Multimodal Models ( https://arxiv.org/abs/2503.02175 )\n  * DART: Stop Looking for Important Tokens in Multimodal Language Models: Duplication Matters More ( https://arxiv.org/abs/2502.11494 )\n\n- I am concerned about the practical applicability of the proposed method in conjunction with modern high-performance inference engines and optimization techniques. The use of middle-layer text-to-vision attention in the second stage may not be directly compatible with implementations such as FlashAttention. Furthermore, introducing additional parameters for the LPM in the first stage and performing token pruning within intermediate layers in the second stage could require substantial implementation effort and modification for popular inference frameworks such as vLLM or TensorRT. It would be beneficial for the authors to discuss these integration challenges and clarify how their method can be efficiently deployed in real-world inference systems.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)