FlowCut: Rethinking Redundancy via Information Flow for Efficient Vision-Language Models
Authors: Jintao Tong
Wenwei Jin
Pengda Qin
Anqi Li
Yixiong Zou
Yuhong Li
Yuhua Li
Ruixuan Li
Venue: NeurIPS 2025 poster ()
Keywords: Vision Language Model
Efficiency
Visual Token Pruning
Primary area: deep_learning
TL;DR: To address inefficiency from excessive visual tokens in LVLMs, we propose an information-flow perspective revealing dynamic redundancy emergence and introduce a method aligned with the model’s inherent behavior, outperforming all existing approaches.

Abstract: Large vision-language models (LVLMs) excel at multimodal understanding but suffer from high computational costs due to redundant vision tokens. Existing pruning methods typically rely on single-layer attention scores to rank and prune redundant visual tokens to solve this inefficiency. However, as the interaction between tokens and layers is complicated, this raises a basic question: Is such a simple single-layer criterion sufficient to identify redundancy? To answer this question, we rethink the emergence of redundant visual tokens from a fundamental perspective: information flow, which models the interaction between tokens and layers by capturing how information moves between tokens across layers. We find (1) the CLS token acts as an information relay, which can simplify the complicated flow analysis; (2) the redundancy emerges progressively and dynamically via layer-wise attention concentration; and (3) relying solely on attention scores from single layers can lead to contradictory redundancy identification. Based on this, we propose FlowCut, an information-flow-aware pruning framework, mitigating the insufficiency of the current criterion for identifying redundant tokens and better aligning with the model's inherent behaviors. Extensive experiments show FlowCut achieves superior results, outperforming SoTA by 1.6% on LLaVA-1.5-7B with 88.9% token reduction, and by 4.3% on LLaVA-NeXT-7B with 94.4% reduction, delivering 3.2$\times$ speed-up in the prefilling stage.  Our code is available at https://github.com/TungChintao/FlowCut.

FlowCut: Rethinking Redundancy via Information Flow for Efficient Vision-Language Models
Jintao Tong' Wenwei Jin
Anqi Li Yuhong Li Yuhua Li" Ruixuan Li
Pengda Qin
Yixiong Zou
School of Computer Science and Technology. Huazhong University of Science and Technology
"Xiaohongshu Inc. "Shanghai Jiao Tong University
{jintaotong, yixionga) Chust. www.steuuuuu.te.eeeeeeeeeeeeeeee "(vensei1217.jjal mumiel.yyhongeeee
Abstract
Large vision-language models (LVLMs) excel at multimodal understanding but suffer from high computational costs due to redundant vision tokens. Existing pruning methods typically rely on single-layer attention scores to rank and prune redundant visual tokens to solve this inefficiency. However, as the interaction between tokens and layers is complicated, this raises basic question: simple single-layer criterion sufficient to identify redundancy? To answer this question, we rethink the emergence of redundant visual tokens from fundamental perspective: information flow, which models the interaction between tokens and layers by capturing how information moves between tokens across layers. We find (1) the CLS token acts information relay, which simplify the complicated flow analysis; (2) the redundancy emerges progressively and dynamically via layer-wise attention concentration; and (3) relying solely attention scores from single layers can lead to contradictory redundancy identification. Based on this, we propose FlowCut, insufficiency of the current criterion for identifying redundant tokens and better aligning with the model's inherent behaviors. Extensive experiments show that FlowCut achieves superior results, outperforming SoTA by 1.6% on LLaVA-1.55 7B with 88.9% token reduction, and by 4.3% LLaVA-NeXT-7B with 94.4% reduction, delivering 3.2x speed-up in the prefilling stage. Our code is available at https: / thub. www.comm//aaa.. http///nghhnttt//oooooooooo
such
information-foon wirtooonnfowwwwwwwrww-e pruning framework, mitigating the
Introduction
Large vision-language models (LVLMs)) [4, 26, 53] have achieved remarkable progress, demon- strating impressive abilities in multimodal perception and reasoning by effectively bridging visual and linguistic modalities. However, LVLMs face significant challenges computational costs and memory demands, primarily caused by the massive amounts of vision tokens, especially for high-resolution images [39, 55] and multi-frame videos [36, 44], limiting their practical applications.
As visual signals inherently contain more spatial redundancy than information-des dense text [12, 45, 52], surge of recent efforts [6. 57, 59, 62] introduce token reduction strategies solve the inefficiency of LVLMs by pruming visual tokens in a training- free manner. These methods typically rely on attention scores rank and prune tokens either specific layer or several L.LM layers with a pre-defined prune ratio for each layer. However, regardless the pruning schedule, redundancy is assessed solely based the attention scores from the current layer. This raises basic question: Are redundancy defined through such a simple single-layer single-criterion score, as the core of current pruning methods, rigorous enough to support pruning decisions?
"Coresponding author. !rroeect leader.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

PrefiLL Tine ILLaWW-RaKT-28
CLS token gathers information globally
FlowCat
spend
Tokon Reduction 88.9%
aFfeccirt
MUX
92%
patch tokens draw information fron ne weiighorhoods and @ CLS token
LL.05-1.3
LLaW-RoKT
Qven2-VL
Vi08F-LiaMA
Figure 1: (Left) Performance across various visual understanding benchmarks on diverse LVLMs, FlowCut significantly outperforms other methods. (Right) We provide unified, bottom-up per- spective based on information flow by modeling inter-token interactions across layers, revealing the dynamic emergence of redundancy and guiding pruning decisions to align with this inherent behavior.
In this work, we rethink the emergence of redundant visual tokens from a fundamental perspective: information flow. Specifically, the information flow for each token is defined as the inflow (the source token that transmits the most information to this token) and outflow (the destination token that this token transmits the most information to), which is reflected in the attention map in each layer of ViT and fundamentally models the interaction between tokens and layers. Through systematic analysis of information flow patterns, we uncover a of key insights that reveal how redundancy emerges. guiding the identification of important tokens: 1 The CLS token acts as an information relay and its behavior can largely represent overall information flow, allowing the complex loken-to-tokes interactions to be approximated and validating it an effective proxy for identifying redundant tokens. @ Redundancy emerges progressively via layer-wise attention concentration, suggesting dynamic pruning strategy aligned with the natural evolution of attention to match the tokens' inherent behavior. Relying solely on attention score from single layers can lead to contradictory redundancy identification. indicating the limitation of single-criterion scoring and single-layer view, which demonstrates the insufficiency of the current criterion for identifying redundancies.
Driven by these insights, we propose FlowCut, an information weatttioo-flo--aae alow-aware pruning framework. Specifically, introduce layer-wise adaptive pruning ratio module based on attention entropy. adjusting the pruning strength according to the inherent concentration level of attention distributions. We also design a multi-criteria scoring strategy to jointly consider attention strength, information density, and semantic relevance, providing a more reliable estimation of token importance. Further, we develop cumulative importance evaluation mechanism that aggregates importance scores over layers, mitigating the insufficiency of the current criterion through continuity across layers. Our designs ensure that pruning decisions align with the inherent dynamic information flow of tokens, preserving critical information while effectively eliminating redundancy (Fig. I left).
In summary, our contributions can be listed
We provide a unified, bottom-up perspective (information flow) by modeling token-to-token interaction explore the propagation pattern of visual information and understand the emergence of redundant, revealing redundancy as natural consequence of asymmetric attention.
Based on comprehensive analysis of the information flow, we verify both the irrationality and the insufficiency of detecting redundant tokens based on a single layer or a single criterion.
Based on the analysis, we propose FlowCut, an information-flow-awwe -lawe pruning framework that better aligns with the model's inherent flow of information, mitigating the insufficiency of the current criterion of identifying redundant tokens.
Extensive experiments show FlowCut achieves superior results: outperforming SoTA by 1.6% on LLLVA-1.55777 (96.0% vs 94.4%) with 88.9% token reduction, and by 4.3% on LLaVA-NeXT-7B (91.9% vs 87.6%) with 94.4% reduction while delivering 3.2 x speed-up in prefilling stage.

---

Tokes
outflon Low
Token
CLS Token
Layer
Layer
Layer
Layer
Layer 20
Figure 2: (Left) Information outflow and inflow across vision encoder layers. (Right) Attention map of various tokens across vision encoder layers. Patch tokens engage in sparse, selective information interactions while the CLS token acts as context relay that gathers and provides information globally.
2 Rethinking Visual Redundancy from Information Flow
Information flow offers unified, bottom-up up perspective for understanding token interactions. this section, we systematically delve into information flow to analyze the emergence of redundancy.
2.1 Preliminaries
Architecture of LVLMs. The architecture of LVLMs generally comprises three key components: pre-trained vision encoder, which transforms input images into visual tokens; 2) a modality connector, serving as a bridge between the vision encoder and the LLM by aligning visual tokens with the LLM's word embedding space; and 3) a pre-trained LLM, which integrates the aligned visual and textual information to perform reasoning and generate responses.
Visual Token Processing. LVLMs process visual tokens in three stages: vision encoding, prefilling, and decoding. Images are first divided into patches and encoded into visual tokens by a vision backbone. These tokens are then fused with textual tokens during the prefilling stage to form joint representations, while key-value pairs are cached for generation. In the decoding stage, new tokens generated autoregressively using the cached information, enabling efficient multimodal inference.
2.2 Delve into Information Flow of Visual Tokens
i) Analysis of Information Flow
We begin by analyzing information inflow and outflow patterns across vision encoder layers. Specifi- cally, for an attention map A E IR XNN we define information inflow for each token as its primary source of information (argmax (A[i, :I)) and outflow as the primary destination of its information (argmax(A) A|::]) Figure (Left) reveals two distinct information flow patterns: CLS tokens consis- tently engage in global information exchange while providing information to patch tokens, whereas patch tokens exhibit short-range, local, uniform flows in shallow layers and long-range, global, deterministic flows in deeper layers.
To further understand information flow, we analyze the attention map of both the CLS token and patch tokens across layers [1, 65). As shown in Figure 2 (Right, rows 1-3), each patch token consistently focuses on small subset of tokens. While this focus is initially on the CLS token and its neighbors in shallow layers, deep layers, each patch token uniformly directs substantial attention to specific group of non-neighboring, non-CLS tokens, which we identify as "hub tokens' due to

---

this concentrated attention. In contrast, the CLS token (Figure 2, Right, row 4) imitially distributes its attention broadly across most tokens, engaging with a wide spectrum of tokens. As layers get deeper and the "hub tokens' emerge as the focus for patch tokens, the CLS token's 's attention pattern also gradually converges towards these hub tokens.
Insight 1
Information flow offers unified, bottom-up perspective for understanding token interactions, agnostic to model architectures. It reveals redundancy as a natural consequence of asymmetric attention: layers deepen, attention concentrates fewer tokens, and tokens that no one focuses on are naturally the redundant ones.
However, while conceptually powerful, modeling all token-to-token flows across all layers is costly and analytically intractable, necessitating a simplified yet representative proxy.
ii The CLS Token Serves as the Relay that Represents the Overall Information Flow
As in Fig. 2 (Right), the attention patterns of both patch and CLS tokens gradually converge to the same hub token in deeper layers. This trend implies an information relay mechanism: for given patch token, since shallow-layer patch tokens primarily attend only their local neighbors and the CLS token, it has minimal direct access to information from distant patch tokens; therefore, the CLS token, as it can gather global information from most patch tokens, must have transmitted information of distant patches to the given patch token to direct this patch's 's attention in deep layers to these distant patch tokens, i.e., the CL.S token serves as the relay to transmit information across patch tokens. Moreover, as patch tokens derive information from the CLS token, their attention gradually mirrors its preferences, therefore, their attention converges to the same hub tokens.
Furthermore, we measure the average distance between each token and the token that it attends the most in Fig. 3 (Left), where the attention distance exhibits an overall increasing trend as the layer deepens. This further verifies that patch tokens initially gather information primarily from nearby neighbors and rely on the CLS token for capturing distant semantic context.
The CLS token acts as global information relay, broadcasting distant-patch informa- tion to each patch token. As layers deepen, patch tokens increasingly mirror the CLS token's 's focus, converging toward the same set of hub tokens, suggesting we can take the CLS token as a proxy to simplify the complex information flow and identify critical tokens.
Insight 2
125
-n-attention distance
attttettion estropy
305
35
1.
85
7
11
13
35
17
19
21
23
22
17
2
Figure 3: (Left) Average attention distance of patch tokens generally increases as the layer deepens. (Middle) Attention entropy across vision encoder layers. (Right) The CLS token attention across layers: dynamic evolution of attention distributions.
tii) Progressive Attention Concentration and Emergence of Redundancy
Our above analysis reveals that the CLS token's 's behavior can largely represent the overall information flow. Therefore, we visualize the overall information flow tendency by measuring the CLS token's attention across the vision encoder layers. Fig. 3 (Right) demonstrates that attention pattern distribu- tions dynamically evolve across layers, and as information traverses deeper layers, attention generally narrows, progressively concentrating of attended tokens inherently implies that other tokens become less critical to the information flow. We interpret this diminishing attention as the progressive emergence of semantic redundancy. Thus, redundancy appears not as a static feature but as an emerging property, developing through complex, layer-by-layer interactions during the encoding process.
smaller group of tokens. This increasing concentration
To quantify the trend of attention concentration, we measured the entropy of the attention distribution at each layer. As in Fig. 3 (Middle), the attention entropy exhibits overall decreasing trend.

---

Notably, sharp decline in entropy is observed between layers 11 and 15, signifying period of rapid attention focusing. This result quantitatively supports the progressive concentration of attention, verifying that redundancy naturally emerges as attention becomes more concentrated.
Redundancy emerges progressively as attention concentrates over depth, not abruptly at the end but layer by layer, driven by increasingly focused semantic interactions. This layer- specific development of redundancy suggests that optimal pruning should be dynamic and layer-adaptive rather than applying fixed ratios across layers pruning only at the final stages.
Insight 3
Vale
Someric Stetterite
Nitertie Serr
Figure Different criteria for identifying critical tokens can show contradictory results, implying the instability of each single criterion in each single layer.
iv) Beyond High Attention: Identifying Truly Imporiant Tokens
Given the complex interactions between tokens and layers modeled by information flow, critical question naturally arises: Is the simple single-layer criterion, as has been adopted by current works, sufficient capture such a complex generation of redundant tokens? To investigate this issue, analyze several metrics at different layers of the vision encoder (selected for feeding into LLM). Alongside attention scores, we measured: 1) the LI norm of the Value vectors, which serves as an indicator of their information density [16, 22], where small norm suggests a weak signal and thus less information conveyed by Value vectors; and 2) the semantic similarity between each token and the CLS token. Given that the CLS token aggregates global information, this similarity score reflects the token's 's semantic alignment with, or relevance to, the overall global context [34].
The results presented in Fig. demonstrate that some tokens highly attended to by the CLS token can exhibit remarkably low information density (as measured by the Value LI norm) and low semantic relevance to the global context (as measured by semantic similarity to the CLS token). This means that applying different criteria, such the high attention scores of the CLS token, in identifying critical tokens can lead to contradictory results, even though all these criteria are reasonable, This problem may be the result of the noisy information in the information flow. Since the flow is transmitted through every two tokens and every connected layers, every small perturbation (noise) in the token/attention could be amplified lead to wrong identification of critical tokens.
High attention score from single layer is not bad criterion, but is still insufficient in measuring token importance due to the complexity in token interactions. An ideal criterion should take networks' inherent behavior into account and resist the instability of current criteria.
Insight 4
3 Methodology
In this section, based on the above analysis, we introduce our method for visual token reduction, which better fits the model's inherent information flow, including flow distribution-auu prune ratio, cumulative flow importance tracking, and multi calti-criteria evaluator. The overall framework is in Fig. 5.
3.1 Attention Distribution-Awart Prune Ratio
Redundancy emerges progressively attention becomes more concentrated. Based on this insight, since the CLS token acts as the relay of information flow, we adopt the CLS token's attention entropy to adaptively adjust pruning ratios: higher entropy indicates widespread token contribution to information flow, necessitating conservative pruning, while lower entropy reflects concentrated attention and emerging redundancy, permitting more aggressive pruning. Specifically, the attention score with heads A E N is computed as:
QK
A Attention Q, K) = Softmax
(I)
VDr
5

---

Layer
attn
35
Large Language Model
3.6
4.2
current
91
182
system tokens
visual
instruction tokens
block,
history
5.9
67
lil..
block,.,
17 17
23
current
:
Embedding
Connector
history
Adsptive compute prune nun by CLS taken's 's attention entropy
Tokenizer
Vision Encoder
bll.
h
Crtti-ritteia Evaluator
block,
history
message> man doing in this picture?
bth.
blocke
score
lll.. calculate cumulative scores
Yo adaptive compute prune
visual tokens
The overview of FlowCut, an information flow-aware pruning framework. The process involves: (1) adaptively determining pruning ratios based on attention concentration; (2) evaluating token importance via multiple criteria; and (3) pruning tokens based on combined current and historical scores, with historical values updated accordingly.
Figure 5:
where Q and K denote query and key matrices, D3 is the key dimension. By averaging A across all heads, we obtain aggregated attention matrix Ang E RNN here N is the sequence length.
For vision encoders with a CLS token, we extract the CLS token's s attention map as global attention vector A For those without CLS token, we first average all tokens to create global token, then compute attention scores between this global token and all sequence tokens to derive A The global attention vector A [R 1glxN is then used to compute the attention entropy H (AB) follow:
Hmx log,, H(A#)
Σ
A log AQ
where, Hmaz denotes the maximum possible entropy when attention is uniformly distributed. The final visual tokens' prune number P is computed as:
N T
H(A*)
P
(1 (1-ri) (1-))),, where TH H
(3)
VL
where N the current visual token number, T is the target number, I. the remain prune layers, and P will be rounded to the nearest integer by round(
3.2 Multi-Criteria Evaluator
Existing methods typically rely single criterion, such the attention of the CLS token or the last instruction token, which verified by us to be insufficient to accurately capture the role of each token in information flow. To address this limitation, propose multi-criteria evaluation framework that incorporates attention strength, semantic similarity, and information density.
Specifically, for attention strength Ie, use A as the metric. For semantic similarity F, we establish a global value vector V by using cither the CLS token's value vector (for encoders with CLS token) or the average of all tokens' value vectors (for encoders without a CLS token), and then calculate the similarity between V and patch token's 's value vector. For information density 1s, we measure by computing the L1 norm of patch token's value vector. The formula follow:
V V
A Softmax
I
(4)
VD,
where D, is the dimension of V. The final importance score S is caculate
I
S =
+
r
(5)
E
J
Σ
1

---

Table I: Performance of Flo on LLaVA-1.5 under different visual token settings. The vanilla number of visual tokens is 576. The last column is the average accuracy proportion relative to the upper limit. FlowCut achieves optimal performance on various image understanding benchmarks. The comparison with additional methods and setting is provided in Appendix A.3.
Method
GQA MMB MMB
MME POPE SQA VQA VQA
VizWiz SEED MMVel LLaVA-B
Avg.
Upper Bound, 576 Tokens (100%)
Vanilla
61.9
64.7
58.1
1862
85.9
69.5
78.5
58.2
50.0
59.3
31.1
66.9
100%
Retoin /92
Tokens [] 66.7%)
ToMe (2CUR28)
54.3
60.5
1563
72.4
65.2
52.1
53.1
27.9
88.7%
FastV (600V44)
52.7
61.2
57.0
1612
64,8
67.3
67.1
52.5
50.8
57.1
27.7
49.4
89.4%
SparseVLM (rema5))
57.6
62.5
53.7
1721
68.9
50.5
55.8
66.1
96.6%
PDrop (ewsas)
57.3
62.9
56.8
1766
82.3
68.8
75.1
56.5
51.1
54.7
30.5
65.8
96.7%
VisionZip (ewazs)
59.3
63.0
57.3
1778
68.7
76.6
57.3
51.2
56.4
31.7
67.7
98.5%
FlowCut (Ours)
60.1
63.2
57.8
1836
86.1
68.6
77.1
57.5
51.8
57.5
34.3
68.4
100.2%
Retain 128
Tokens
(1) 77 .86)
ToMe (2CER23)
52.4
53.3
1343
62.8
59.6
63.0
49.1
50,9
27.2
81.8%
FastV (B2224)
49.6
56.1
56.4
1490
59.6
60.2
61.8
50.6
51.3
55,9
28.1
52.0
85.9%
VLM (108225)
56.0
60.0
51.1
1696
80.5
67.1
73.8
54.9
51.4
53.4
30.0
62.7
93.8%
PDrop (owazs)
57.1
61.6
56.3
1664
82.3
68.3
72.9
56.6
51.0
53.3
30.8
61.9
95.1%
VisionZip (ewazs)
$7.6
62.0
56.2
1759
83.2
68.9
56.8
51.6
54.9
32.1
64.8
97.2%
Flow Cut (Ours)
58.5
62.1
56.5
1792
85.2
68.6
76.0
57.3
52.2
56.2
32.5
67.5
98.5% %
Retain 64
Tokens (L 88.9%)
ToMc (XCER28)
48.6
43.7
1138
52.5
50.0
57.1
45.3
44.0
24.1
71.4%
FastV (B00224)
46.1
48.0
52.7
1256
48.0
51.1
47.8
50.8
51.9
25.8
46.1
77.5%
VLM (ros25))
52.7
56.2
46.1
1505
75.1
62.2
68.2
51.8
50.1
51.1
23.3
57.5
84.6%
PDrop (cvPos))
47.5
58.8
50.5
1561
55.9
68.6
69.2
50.6
50.7
40.0
30.7
59.2
86.1%
VisionZip (evmos)
55.1
60.1
55.3
1687
77.0
69.0
72.4
55.5
52.8
52.2
31.5
62.9
94.4%
Flow Cut (Ours)
55.6
60.8
55.4
1744
80.2
69.1
72.8
55.6
53.2
53.5
32.3
65.1
96.0% %
3.3
Cumulative Flow Importance Tracking
Based on the observation that attention distribution dynamically and continually shift across layers, and the single-layer criterion insufficient to capture a token's importance, we track cumulative importance by aggregating both historical and current criteria across layers, enabling more accurate and stable assessment of each token's 's overall contribution. Specifically, we gain multi-criteria score Sour each layer and update cumulative score Semm using both the current and historical values:
gt) 0.5 x I 0.5 ((-1) where ( [ if layer 0
(6)
We prune every two layers, relying on the cumulative score to ensure that the importance value takes into account both current and historical values, which implicitly filters out noise in flows.
4 Experiments
Experiment Setting. To demonstrate FlowCut's effectiveness, we conducted extensive experiments on diverse open-source LVLMs. For image understanding tasks, we implemented our designs on the widely-used LLaVA sapily-speciffialll I.L [39] and the high-resolution LLaVA-Ner 7B (sppporrggg up 2880 image reduction ratios. To further validate the universality of our method, we extended evaluation to the more advanced Dwen2-VL [55] model. For video understanding tasks, employed Video-LLaVa [36] as a baseline and verified our method across three video benchmarks.
across twelve benchmarks with varying
Implement Details Our method can be seamlessly integrated into models in a training free manner, or training from scratch, resulting substantial improvements inference speed and training efficiency. For VA--55, we complete the pruning at the penultimate layer of the vision encoder. For LLa VA- NeXT and Qwen2-VI. VL. which contain a larger number of tokens, our pruning process spans from the vision encoder to the second layer of the L.M,, which we prune tokens to twice the target number in the vision encoder, then further reduce the final target number the second layer of the LLM. All of our experiments conducted on Nvidia A800-80G GPU.

---

Table 3: Performance of Flow on Qwen2-VL-7B for image understanding. As the original token num- bers is dynamical, the reduction ratio is approximate.
Table 4: Performance of FlowCut video understand- ing tasks. The original Video-LLaVA's video token number is 2048, while we only retain the 256 tokens.
Method
GQA
MMB MMB MME POPE SQA VQA Tcu
Avg.
Method
TGIF
MSVD
MSRVT
Avg.
Upper Bound, All Tokens (100%)
Asc
Score
Acc
Score
Asc
Soore
Score
Vanilla
61.9
79.9
79.5
2338 87.2 85.1
82.2
100%
Vidco-LLaVA
46.9
3.34
69.8
3.91
57.1
3.49
100%
100%
Token Reduction (1 66.7%)
FastV
44.2
3.29
60.3
3.72
40.6
3.18
83.9% 94.9%
FastV
58.0
76.1
73.9 2130 82.1 80.0
77.3
93.6%
94.2%
98.5%
86.4%%
95.19)
77.1%
91.1%
45.9
3.32
68.6
3.90
32.9
3.02
FlowCut
60.5
79.2
78.2 2335 86.0
84.0
81.1
98.7%
LL
84.6%
95.258
98.9%
99.4%
98.3%
99.7%
57.6%
86.5%
Token Reduction (1 77.8%)
40.3
3.21
61.5
3.74
41.8
3.19
FastV
56.7
74.1
72.4
2031 79.2 78.3
72.0
90.4%
PDrop
85.9%
96.
88.1%
95.7%%
73.2%%
91.4%)
82.4% 94.4%
FlowCut
59.2
77.8
76.9
2310
84.6 80.5
78.3
96.5%
44.3
3.29
65.2
3.83
54.5
3.43
Token Reduction (1 88.98.9
88.9%)
VisionZip
94.5%
98.5%
93,408
98.01)
95.11%
98.3%
94144% 98.2%
FastV
51.9
70.1
63.8
1962
76.1 75.8
60.3
83.6%
46.8
3.38
68.8
3,91
55.7
3,46
FlowCut
98.6%
FlowCut
56.4
72.6
72.5
2252
81.8
78.2
68.9
91.3%
99.8%
100.3%
98.6%
100%
97.5%%
99.1%)
4.1
Effectiveness of FlowCut in Inference
Table 2: Comparison of FlowCut on LLaVA-NeXT- 7B with other methods.
Image understanding tasks We apply Flow- Cut to LLaVA-1.5-7B during inference without additional training and compare with other approaches. The results in Table 1 demonstrate that FlowCut achieves optimal results on almost all benchmarks, with its average performance significantly exceeding all existing methods. we conduct further experiments on V--Ne-- BB. As shown in Table 2. Flow Cut outperform- ing other methods by a significant margin. We even surpassing VisionZip by 4.3% while retain- ing only 160 tokens. Furthermore, as shown in Table 3, we extended the evaluation to Qwen2- VL to validate that FlowCut can be applied seam- lessly to different architectures.
Method
GQA MMD MMB MME POPE VQA" VQA
SEED
Arg.
Upper Bound, 2850 Tokens (100%)
Vanilla
64.2
67.9
60.6
1846
86.4
81.8
61.3
63.6
100%
Rehale 640
Tokens (L (7.88)
SpurseVLM
60.3
65.8
58.5
1773
84.2
77.1
57.8
59.3
95.3%%
PDrup
60.6
65.5
58.5
1781
57.4
60.5
95.79
VisionZip
61.3
66.2
57.8
1787
85.9
79.1
60.2
60.9
96.9%
FlowCot
61.9
66.7
59.2
1837
86.1
79.8
60.6
61.9
98.2%
Tokeny
SpuraVVLM
57.7
63.2
54.4
1685
82.2
73.4
55.9
56.9
991.2%
Piroop
58.3
63.9
56.8
1736
80.2
75.2
55.3
57.8
92.34%
VisionZip FlowCut
59.2
55.3
1702
76.2
58.9
58.2
93.0%
59.8
65.3
57.5
1791
83.4
77.8
60.1
59.5
95.6%
Renale 160
Tokens (L 94.4%%
SpurseVLM
51.2
52.1
48.6
1542
72.7
66.3
46.4
19.2
79.8%
PDrop
54.9
61.8
54,9
1513
72.3
70.2
52.7
53.9
96.2%
VisionZip
55.5
60.1
52.7
1628
74.8
71.4
56.2
54.2
87.66
FlowCut
57.6
1746
74.6
57.6
56.8
Video understanding tasks applied Video-LLaVA, which encodes each video into 8 frames with 256 visual tokens per frame (totaling 2048 visual tokens). We compared FlowCut to other methods under the setting of retaining only 256 tokens in total (32 per frame). As shown in Table 4, FlowCut achieves 98, 88.6% accuracy across three benchmarks, outperforming the previous state-of-the- art VisionZip by 4.2%. This notable gain highlights it's superior reasoning capability handling complex multimodal video data.
To assess our method's effectiveness in video understanding, we
Efficiency Analysis Our proposed FlowCut substantially improves inference efficiency in LVLMs while maintaining comparable performance. As shown in Table 5, we compare total inference time, prefilling time, and FLOPS on both LLaVA-1.5-7BB and LLa VA-NeXT-BB against several existing methods. FlowCut consistently achieves the best efficiency gains. Notably, it delivers a 3.2x speedup in prefilling and 3.0x speedup in the entire inference on LLaVA-NeXT
Table 5: Efficiency analysis of FuowCut on LLaVA-1.5-7B (left) and LLaVA-NeXT-7BB (right). The detailed metrics include practical total running time (minssec), prefilling time and TFLOPs. A denotes the inference speedup. The results are tested for one A800 GPU on POPE benchmark.
Total Timel,
Prefilling Timel,
Total Time]
Prefilling Time.
Methods
Token
A1
At
TELOPs
Methods
Token
A
At
TELOPs
LLaVA-1,5-78
576
17:05
1.0x
102ms
1.0x
8.82
I.aVA-NeXT778
2880
35:16
1.0%
206ms
1.0x
31.03
FastV
64
15:32
1.1x
87.3ms
1.2x
2.26
FFstV
160
28:27
1.3x
117ms
1.8x
7.35
SLarseVLM
64
15:57
1.1x
90.1ms
1.1x
2.31
SparseVLM
160
30:26
1.2x
136ms
1.5x
7.62
PDrop
64
12:56
1,3x
72.5ms
1.4x
2.16
PDiop
160
13:45
2.6x
79.5ms
2.6x
6.78
-VisionZip
64
12:10
1.4x
69.2ms
1.5x
2.03
VisionZip
160
12:32
2.8x
69.9mm
9ms
2.9x
4.72
CulwCut
64
11:17
1,5x
62.6ms
1.6x
1.95
FlowCut
160
11:40
3.0x
63.8ms
3.2x
4.29
8

---

4.2 Effectiveness of FlowCut in Training
Our method can also be seamlessly integrated into models during training to reduce the number of visual tokens, thereby saving memory and shortening training time. As shown in Table 6, we evaluate FlowCut with different token configurations across 10 benchmarks on LLaVA-1.55 1.5-7B. The results demonstrate that with 192 tokens, FlowCut achieves a 1.5x training speedup while actually improving performance to 101.1% of the original model. Even with further reduction to 128 tokens, training speedup reaches 1. 1.6x while still retaining 99.8% of the original performance.
Table 6: Performance and Training Time of FlowCut on LLa The first line of each method shows the raw benchmark accuracy, and the second line is the proportion relative to the upper limit. The time refers to the practical training duration (pretrain time + fine-tuning time), while A indicates training acceleration factor. All experiments are conducted on 8 Nvidia A800 GPUs.
Method
Tokens
Timel
A1
GQA
MMB
MMB
MME
POPE
SQA
VQA
VQA
te
SEED
MMVet
Avg.
1.a 7B
576
12,97h
1.0x
61.9
64.7
58.1
1862
85.9
69.5
78.5
58.2
59.3
31.1
100%
62.0
66.4
59.2
1840
86.0
68.9
78.7
58.1
59.2
33.6
FlowCut
192
8.65h
1.5x
101.5
100.2%
102.6%
901
98.8%
100.1%
99.1%
100.3%
99.8%
99.8%
108.0%
60.8
66.9
59.5
1835
84.5
69.2
77.9
57.9
58.6
31.2
FlowCut
128
7.98h
1.6x
99.8%
98.2%
103.4%
102.4%
98.5%
98.4%
99.6%
99.2%
99.5%
98.8%
100.3%
4.3 Ablation Studies
Effectiveness of each desigh To assess the contribution of each design in FlowCut, we conduct ablation studies on three benchmarks under a setting that retains only 128 visual tokens (177.8%)), as shown Table 7. Naively pruning a single layer leads to significant performance drops. Introducing either the cumulative importance evaluation or the multi-criteria scoring strategy individually brings consistent gains across benchmarks, confirming their effectiveness. Moving from single-layer to multi-layer pruning yields notable improvements, yet applying uniform prune ratio across layers remains suboptimal. Enabling adaptive prune ratio notably enhances performance (e.g., +1.8% on POPE). The best results are achieved when all components are integrated in FlowCut, yielding minimal performance degradation compared to the unpruned baseline. These results verify that each component contributes independently and synergistically to performance.
Adaptive layer-wise pruning outperforms single-layer pruning As shown in Figure we measure information density across layers by the L.1 norm of value vectors, where lower norms indicate sparser and less informative representations [22]. In single-layer pruning, the information density drops sharply after pruning, indicating substantial information loss. In contrast, FlowCut aligns pruning decisions with the token's dynamic inherent information flow, effectively preserving essential content.
Table 7: Ablation study of various designs with 128 tokens retain across three benchmarks.
Layer23
Flowau (Curs]
Mathode
Token
Adaptire
Commlatire Muttiple TraVQA GQA POPE
Prune
Inforration Density Plaweets
L..VA-115577 L.578
576
58.3
61.9
85.9
Single Lover
128
55.3
54.9
76.6
Single-Layer
56.3
0.45
Single Layer
128
55.9
57.3
$2.0
Singe-Layer
56.8
57.8
0.33
Mulii Layer
36.2
56.3
Muti- Layer
128
V
$7.1
58.0
833
033
Layer
56.3
Matei-Layer
128
56.7
57.8
83.5
430
Muli Layer
57.0
53.3
$4.0
MultiLLyer
57.1
Figure 6: Information density measured by Value vector's L1 norm shows that layer-wise progres- sive pruning results in less information loss.
Mahi-L
128
36.
57.9
84.3
PlawCat
4.4 Analysis and Discussion
FlowCut effectively preserves critical information. According to the attention computation, larger value vectors (V) contribute more to the output, indicating richer information content [17]. We use

---

Figure 7: Box plot of Value vector (V) across layers before (left) and after (right) adopting FlowCut.
box plots (Figure 7) compare the distribution of value vectors before and after applying FlowCut. After pruning, the distribution shifts toward higher magnitudes, suggesting that although fewer tokens remain, each retains
information, effectively preserving overall content richness.
FlowCut guides model focus more effectively. Figure 8 compares the pruning results and VQA answers of Clo and the previous SoTA. VisionZip relies single-layer attention scores, resulting in most tokens retained after pruning being concentrated in the center of the image. In contrast, FlowCut leverages multi-layer and multi-criteria evaluations, leading to more uniform spatial distribution of retained tokens. Moreover, by effectively eliminating redundant and distracting tokens, FlowCut produces more accurate answers than even the non-pruned baseline in some cases.
Related Work
helled,
Visual Token Compression In LVLMs, visual tokens typically contain substantial redundant in- formation. Various methods [5, 32, 55J have been proposed to address this inefficiency. FastV [6] leverages attention scores at the LLM's second layer for pruning, while Sparse VLM [62] intro- duces text-guided pruning through cross-modal attention. VisionZip [59] employs CLS token attention at the vision encoder's final layer for compression, and PDrop [57] uses the last in- struction token's s attention map to drop visual tokens several pre-selected layers according to manually designed fixed pruning ratios. However, these methods evaluate token importance using attention scores from single layer, limiting their comprehensiveness. Our method, instead, accu- mulates diverse criteria across multiple layers and pruning ratios, achieving efficient inference while preserving performance. (More in appendix A.2)
The oxtilt acconts. the hellee let prodmiientty candaann in seerly alass.
afills
designe.
The outtl: rad, the hetact klash
LUVA
I Connect Incorrect
Partiallyy Cornect
FlavOit
The extfut se 21855, and the Reouboard white.
hetest
outfit rst and the Ielnet
14 alit, and the seouasand ix black.
Figure 8: Example comparison of FlowCut and previous SOTA visual question answering.
performs progressive token reduction with adaptive
Conclusion
In this work, we provide novel perspective (information flow) for understanding the emergence of redundant tokens, revealing redundancy natural consequence of asymmetric attention. Through comprehensive analysis, we demonstrate the limitations of single-layer, single-criterion metrics in identifying redundancy, and propose FlowCut, an information-flow-aww aligns pruning decisions with the inherent dynamic information flow of tokens. Extensive experiments show that FlowCut achieves efficient inference while preserving performance.
pruning framework that
Acknowledgments
This work is supported by the National Natural Science Foundation of China under grants 62206102: the National Key Research and Development Program of China under grant 2024YFC3307900; the
10

---

National Natural Science Foundation of China under grants 62436003, 62376103 and 62302184; Ma- jor Science and Technology Project of Hubei Province under grant 2025BAB01 and 2024BAA008; Hubei Science and Technology Talent Service Project under grant 2024DJC078; and Ant Group through CCF-Ant Research Fund. The computation completed in the HPC Platform of Huazhong University of Science and Technology.
References
[1] Samira Abnar and Willem Zuidema. Quantifying attention flow in transformers. In Proceedings of the 58th Anmual Meeting of the Association for Computational Linguistics, pages 4190-4197, 2020.
[2] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint axXiv:3033.08774 2023.
[3] Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang. Xiaodong Deng. Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309. 16609, 2023.
[4] Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilitics. arXiv preprint arXiv:2308. 12966, 12))3, 2023.
[5] Daniel Bolya, Cheng Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffmann. Token merging: Your vit but faster. In ICLR, 2023.
[6] Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang, An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models. In European Conference on Computer Vision, pages 19-35, Springer, 2024.
[7] Yi Chen, Jian Xu, Xu-Yao Zhang, Wen-Zhuo Liu, Yang-Yan Liu, and Cheng-L.in Liu. Recoverable compression: A multimodal vision token recovery mechanism guided by text information. Proceedings of the AAAI Conference on Artificial Intelligence, volume 39. pages 2293-2301, 2025.
[8] Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 24185 24198, 2024.
[9] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90% * chatgpt quality, march 2023. URL htps://mmsys. org/hlagg2022--03330--ii -00-iiia 3(5), 2023.
[10] Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. arXiv preprint arXiv:2307.0869) 2023.
[11] Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with to-awarensss. Advances in neural information processing systems, 35:16344 16359, 2022.
[12] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner. Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010. 19909..199,,,.. 2020.
[13] Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun. et al. MME: A comprehensive evaluation benchmark for multimodal large language models. arXiv:2306. 13394, 2023.
[14] Suyu Ge, Yunan Zhang, Liyuan Liu, Minjia Zhang. Jiawei Han, and Jianfeng Gao. Model tells you what discard: Adaptive kv cache compression for llms. arXiv preprint arXiv:2310.01801, 2023.
[15] Yash Goyal, Tejas Khot, Douglas Summers-Stay Dhruv Batra, and Devi Parikh. Making the in vqa matter: Elevating the role image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6904-6913, 2017.
[16] Zhiyu Guo, Hidetaka Kamigaito, and Taro Watanabe. Attention score is not all you need for token importance indicator in ky cache reduction: Value also matters. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 21158 1111158-11666 21166, 2024.

---

[17] Ankit Gupta and Jonathan Berant, Value-aware approximate attention. In Proceedings of the 2021 Conference on Empirical Methods Natural Language Processing, pages 9567-9574, 2021.
[18] Danna Gurari, Qing Li, Abigale J Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In Proceedings of the IEFE conference on computer vision and pattern recognition, pages 3608-3617, 2018.
[19] Chi Han, Qifan Wang, Wenhan Xiong, Yu Chen, Heng and Sinong Wang. Lm-infinite: Simple on-the-fly length gencralization for large language models. 2023.
[20] Drew A Hudson and Christopher Manning. GQA: A new dataset for real-world visual reasoning and compositional question answering. Conference on Computer Vision and Pattern Recognition (CVPR). 2019.
[21] Yunseok Jang, Yale Song, Youngjae Yu, Youngjin Kim, and Gunhee Kim. Tgif-qa: Toward spatio-temporal reasoning in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2758-2766, 2017.
[22] Goro Kobayashi, Tatsuki Kuribayashi, Sho Yokoi, and Kentaro Inui. Attention is not only weight: Analyzing transformers with vector norms. In 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, pages 7057-7075. Association for Computational Linguistics (ACL), 2020.
[23] Dan Kondratyuk, Lijun Yu, Xiuye Gu. José Lezama, Jonathan Huang. Grant Schindler, Rachel Hornung. Vighnesh Birodkar, Jimmy Yan, Ming Chang Chiu, et al. Videopoet: A large language model for zero-shot vidco generation. Proceedings of Machine Learning Research, 335:25105525115 2024.
[24] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium Operating Systems Principles, pages 611-626, 2023.
[25] Bohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang, Rui Wang, Ruimao Zhang, and Ying Shan. Seed-bench: Benchmarking multimodal large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 13299-13308, 2024.
[26] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19730-19742. PMLR, 2023.
[27] Shiwei Li, Huifeng Guo, Lu Hou, Wei Zhang, Xing Tang, Ruiming Tang, Rui Zhang, and Ruixuan Li. Adaptive low- www-precision training for embeddings in click throuuhh rate prediction. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 4435 4443, 2023.
[28] Shiwei Li. Xiandi Luo, Xing Tang, Haozhao Wang. Hao Chen, Weihong Luo, Yuhua Li. Xiuqiang He, and Ruixuan Beyond zero initialization: Investigating the impact of non-zero initialization on lora fine-tuning dynamics. In Forty-second International Conference on Machine Learning, 2025.
[29] Shiwei Li, Xiandi Luo, Haozhao Wang, Xing Tang, Ziqiang Cui, Dugang Liu, Yuhua Li, Xiuqiang He, and Ruixuan Li. Beyond higher rank: Token-wise input-output projections for efficient low-rank adaptation. In Advances in Neural Information Processing Systems, 2025.
[30] Shiwei Li, Xiandi Luo, Haozhao Wang, Xing Tang, Ziqiang Cui, Dugang Liu, Yuhua Li, Xiuqiang He, and Ruixuan Li. Bora: Towards more expressive low-rank adaptation with block diversity, arXiv preprint arXiv:2508.0m 06953, 2025.
[31] Shiwei Li, Wenchao Xu, Haozhao Wang, Xing Tang, Yining Qi. Shijie Xu, Weihong Luo, Yuhua Li, Xi- uqiang He, and Ruixuan Li. Fedbat: communication-effiicee federated learning via learnable binarization. In Proceedings of the 1 International Conference on Machine Learning, pages 29094,29095, 2024.
[32] Yanwei Li, Chengyao Wang. and Jiyy Jia. Llama-vid: An image is worth 2 tokens in large language models. In European Conference on Computer Vision, pages
Springer, 2024.
[33] Yanwei Li, Yuechen Zhang, Chengyao Wang, Zhisheng Zhong, Yixin Chen, Ruihang Chu, Shaoteng Liu, and Jiaya Jia. Mini-gemini: Mining the potential of multi-modality vision language models. arXiv:2403. 18814 2024.
[34] Li, Hualiang Wang. Yiqun Duan, Jiheng Zhang, and Xiaomeng Li. A closer look at the explainability of contrastive inguuage-image pre-training. Pattern Recognition, page 111409, 2025.
12

---

[35] Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang. Wayne Xin Zhao, and Ji- Ri-Rong Wen. Evaluating object hallucination in large vision-language models. arXiv preprint S.Ki::2305.133555 2023.
[36] Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning. Peng Jin, and Li Yuan. Video-Ilava: Learning united visual representation by alignment before projection. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 5971-5984, 2024.
[37] Zhihang Lin, Mingbao Lin, Luxi Lin, and Rongrong Ji. Boosting multimodal large language models with visual tokens withdrawal for rapid inference. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 5334-5342, 2025.
[38] Hao I Liu, Matci Zaharia, and Pieter Abheel. Ring attention with blackwise transformers for near-infinite context. arXiv preprint arXiv:2310.0135 2023.
[39] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jac Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296 26306, 2024.
[40] Haotian Liu, Chumyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jac Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024.
[41] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Lee. Visual instruction tuning. Advances in neural information processing systems, 36:34892-3444 34916, 2023.
[42] Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? In European conference computer vision, pages 216-233. Springer, 2024.
[43] Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai- Chang, Long-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances Neural Information Processing Systems, 35:2507-2521, 2022.
[44] Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. arXiv preprint arXiv:2306.05422 2023.
[45] David Marr. Vision: computational investigation into the human representation and processing of visual information. MIT press, 2010.
[46] Yuzhang Shang. Mu Cai, Bingxin Xu, Yong Jac Lee, and Yan Yan. Llava prameee: Adaptive token reduction for efficient large multimodal models. arXiv preprint arXiv:2403. 153888 2024.
[47] Dachuan Shi, Chaofan Tao, Anyi Rao, Zhendong Yang, Chun Yuan, and Jiaqi Wang. Crossget: Cross- guided ensemble of tokens for accelerating vision-language transformers. In International Conference on Machine Learning, pages 44960-44990. PMLR, 2024.
[48] Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Mareus Rohrbach. Towards vqa models that can read. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8317-8326, 2019.
[49] Gemini Team, Rohan Anil, Sebastian Borgeaud, -ean-Baptiste Alayrac. Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312. 11805, 2023.
[50] Jintao Tong, Ran Ma, Yixiong Zou, Guangyao Chen, Yuhua Li, and Ruixuan Li, Adapter naturally serves as decoupler for cross-domain few-shot semantic segmentation. arXiv preprint arXiv:2506.0733 2025.
[51] Jintao Tong, Yixiong Zou, Guangyao Chen, Yuhua Li, and Ruixuan Li. Self-disentanglement and re- composition for cross-domain few-shot segmentation. arXiv preprint arXiv:2506.00667, 2025.
[52] Jintao Tong, Yixiong Zou, Yuhua Li, and Ruixuan Li. Lightweight frequeney masker for cross-domain few-shot semantic segmentation. Advances in Neural Information Processing Systems, 37:96728-96749, 2024.
[53] Peter Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Adithya Jairam Vedagiri IYER, Sai Charitha Akula, Shusheng Yang. Jihan Yang, Manoj Middepogu, Ziteng Wang, et al. Cambrian-1: A fully open, vision-centric exploration of multimodal llms. Advances in Neural Information Processing Systems, 97:87310-87535 2024.
13

---

[54] Hugo Touvron. Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix. Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv: 222223322.9971.. 13971, 2023.
[55] Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu. Jialin Wang. Wenbin Ge. et al. Qwen2-vl: Enhancing vision- saanugg model's perception the world any resolution. arXiv preprint www.v:2449..2211101 2024.
[56] Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks, 2024. URL htts::/arxxvv org/abs/2339.1122 17.55393..7,,, page 1, 2024.
[57] Long Xing. Qidong Huang. Xiaoyi Dong. Jiajie Lu. Pan Zhang, Yuhang Zang, Yuhang Cao. Conghui He. Jiaqi Wang, Feng Wu, et al. Pyramiddrop: Accelerating your large vision-language models via pyramid visual redundancy reduction. arXiv preprint arXiv:2410.1722 2024.
[58] Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Yueting Zhuang. Video question answering via gradually refined attention over appearance and motion. In Proceedings of the 25th ACM international conference on Multimedia, pages 1645-1653, 2017.
[59] Senqiao Yang, Yukang Chen, Zhuotao Tian, Chengyao Wang. Jingyao Li, Bei Yu, and Jiaya Jia. Visionzip: Longer is better but not necessary in vision language models. arXiv preprint arXiv:2412.04467 2024.
[60] Weihao Yu, Zhengyuan Yang. Linjie Li. Jianfeng Wang. Kevin Lin, Zicheng Liu, Xinchao Wang. and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308. 02490, 2023.
[61] Qizhe Zhang, Aosong Cheng. Ming Lu, Zhiyong Zhuo Minqi Wang, Jiajun Cao, Shaobo Guo, Qi She, and Shanghang Zhang. [cls] attention you need for training-fie ffee visual token pruning: Make vlm inference faster. arXiv e-prints, pages arXiv-2412, 2024.
[62] Yuan Zhang, Chun-Ka Kai Fan, Junpeng Ma, Wenzhao Zheng, Tao Huang, Kuan Cheng, Denis Gudovskiy, Tomoyuki Okuno, Yohei Nakata, Kurt Keutzer, et al. Sparsevim: Visual token sparsification for efficient vision-language model inference. arXiv preprint arXiv:2410.04// 2024.
[63] Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, et al. H2o: Hcavy-hitter oracle for efficient generative inference of large language models. Advances in Neural Information Processing Systems, 36:34661-344 34710, 2023.
[64] Jiedong Zhuang, Lu Lu, Ming Dai, Rui Hu, Jian Chen, Qiang Liu, and Haoji Hu. St3: Accelcrating multimodal large language model by spatial-temporal visual token trimming. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 11049-11057 2025.
[65] Yixiong Zou, Shuai Yi, Yuhua Li, and Ruixuan Li. A closer look at the els token for cross- dossin few-shot learning. Advances in Neural Information Processing Systems, 37:85523-855455 2024.
[66] Yixiong Zou, Shanghang Zhang, Jianpeng Yu, Yonghong Tian, and José MF Moura. Revisiting mid-level patterss for cross-domain few-shot recognition. In Proceedings of the 29th ACM International Conference on Multimedia, pages 741-749, 2021.
14

---

A Technical Appendices and Supplementary Material
A.1 Dataset
We evaluated our method across fifteen benchmarks: twelve for image understanding tasks and three for video understanding tasks, each assessing different aspects of multimodal intelligence.
GQA [20] The GQA benchmark is structured around three parts: scene graphs, questions, and images. It enriches visual content with comprehensive spatial information and object-level attributes. Questions are specifically designed to evaluate models' ability to comprehend visual scenes and reason about diverse aspects of images.
MMBench [42] The MMBench Benchmark evaluates models through three hierarchical levels of abilities. The first level (L-1) focuses fundamental perception and reasoning capabilities. The second level (L-2) expands into six distinct sub-abilitics, while the third level (L-3) further refines these into 20 specific dimensions. This structure enables a granular and comprehensive assessment of a model's ' various capabilities. Additionally, MMB-CN is the Chinese version of the benchmark.
MME [13] MME comprehensively evaluates models' perceptual and cognitive abilities across 14 subtasks. By employing manually constructed essructiion-aswer pairs and concise instructions, effectively minimizes data leakage and ensures fair assessment of model performance.
POPE [35] The POPE benchmark systematically evaluates object hallucination in models through series of binary questions about object presence in images. Using accuracy, recall, precision, and FI score as metrics, precisely measures hallucination levels across different sampling strategies.
ScienceQA [43] ScienceQA encompasses a wide array of domains, including natural, language, and social sciences, with questions hierarchically organized into 26 topics, 127 categories, and 379 skills. Through this comprehensive structure, it offers diverse scientific questions that effectively evaluate multimodal understanding, multi-step reasoning capabilities, and interpretability.
VQA-V2 [15] VQA-V2 evaluates models' visual perception capabilities through open-ended ques- tions about 265.016 real-world scence images. Each question contains 10 numan-annotated ground truth answers, enabling thorough assessment of model's ability to interpret and respond visual queries.
TeX [48] TextVQA focuses on the integration of textual information within images. It evaluates model's 's ability to interpret visual elements and embedded text in images through tasks, requiring both visual and textual comprehension to answer questions accurately.
VizWiz [18] The VizWiz benchmark contains over 31,000 visual questions created by blind individu- als who photographed scenes with mobile phones while recording spoken questions. Each question includes 10 crowdsourced answers. The dataset presents distinct challenges: lower-quality images from visually impaired photographers, conversational spoken questions, and some questions that remain unanswerable due to content limitations.
SEEDBench [25] SEEDBench contains 19,000 ing models across 12 distinct aspects. It comprehensively assesses capabilities in recognizing spatial and temporal patterns within both images and videos.
multiple-choice questions evaluat-
MMVet [60] MMVet defines six core abilities (recognition, OCR, knowledge, language generation, spatial awareness, and math) and evaluates models through 16 specific capability combinations. These integrations enable comprehensive assessment of models' performance across complex scenarios.
LLa VA- -Bnnch [39] LLLVA- categorizes questions into three types: conversational (simple QA), detailed description, and complex reasoning tasks, featuring 24 diverse images with 60 questions spanning indoor and outdoor scenes, memes, paintings. sketches, and more. Each image contains manually curated detailed description and carefully selected questions, designed assess model robustness across various prompts.
TGIF-QA [21] TGIF-QA extends the image question answering task to videos with 165,000 question- answer pairs based on GIFs. It introduces four task types: three requiring spatio-tempore reasoning (repetition count, repeating action, state transition) and frame QA answerable from single frames.
MSVD-QA [58] The MSVD-QA benchmark, based on the MSVD dataset, features 1,970 video clips with approximately 50.5K question- answer pairs. It presents open- onde- questions across five
15

---

categories (what, who, how, when, where) covering diverse video content aspects, serving both video question answering and captioning tasks.
MSRVTT-QA [58] MSRVTT-QA comprises 10K video clips with 243K question-answer pairs, challenging models to integrate visual and temporal elements of video content. Similar to MS MSVD-QA, it includes five question types, evaluating models' ability to comprehend complex video content.
A.2 Related Work
Large Vision- Language Models Recently, building on the success of large language models (LLMs) [2, 3, 9, 54], LVLMs [4, 39, 41, 49, 53] have demonstrated impressive performance in multimodal reasoning by bridging visual and linguistic modalities. However, a critical challenge persists in processing images: the quadratic growth of visual tokens as image resolution increases, which leads to prohibitive computational costs during training and inference. High-resolution image models like VA--NNN [40] and Mani-Gemini-Be Hemiini-HHDIA [33] process thousands of tokens per image, while video models such as VideoLLa [36] and VideoPoet [23] demand even more tokens for multiple frames. This highlights the need for more efficient methods [28, 30, 29, 50J to extract information from visual tokens, instead of merely extending their length.
Efficient Large Language Models Efficient inference [11, 10, 24, 38) in LLMs is challenged by their autoregressive nature, where each token prediction depends on the full preceding context. Recently, variety of token reduction strategies have been proposed to accelerate inference and compress the key-value (KV) cache [19]. For instance, StreamingLLM [56] retains only attention sinks and most recent tokens to shrink the KV cache, while FastGen [14] adaptively manages KV cache based on the behavior of attention head. Heavy- Hitter Oracle (H20) [63] employs accumulated
attention-bel based scoring to prune key-valus pairs during the generation stage. These methods primarily target textual redundancy to improve inference efficiency [31, 27].
A.3 Evaluation under Additional Settings and Comparison with More Methods
Retain less visual tokens To further validate our method under highly constrained token budgets, we conduct experiments with only 32 visual tokens. As shown in Table 8, our method outperforms existing methods by notable margin under aggressive token reduction, validating its effectiveness in preserving informative content despite severe compression.
Table 8: Performance of FlowCut on LLaVA-1.5 under 32 visual token setting.
Method
Tokens
GQA
TextVQA
VQA-V2
POPE
SQA
VizWiz
Avg
LLaVA-1.5 7B (upper limit)
576
61.9
58.2
78.5
85.9
69.5
50.0
100%
FastV
32
46.2
51.5
56.0
35.7
68.3
49.1
78.7%
VisionZip
32
51.5
51.7
65.1
68.3
68.2
49,9
88.7%
FlowCut (Ours)
32
52.1
53.0
66.9
69.6
68.7
$2.3
90.8%
Compare with more methods We have already compared with several representative methods in the main text. Here, we further include more prior works for comparison: ALLaVA-PruMeerg [46] clusters the visual tokens via k-nearest merging algorithm and merges them. ST [64] prunes low- importance tokens progressively across layers using token attention maps. VTW [37] removes unimportant visual tokens based attention score at specific layers of the L.M.. RCom [7] merges unimportant patch tokens based on their similarity to the CLS token and the text tokens. Faster- VLM [61] prunes vision tokens based CLS attention scores at vision encoder's output layer.
These methods all rely on lingle-layer, single- meteec scores (e.g., attention or similarity) assess token importance. In contrast, our method employs multiple metrics and aggregates scores across layers using an EMD-based strategy, which helps mitigate single-metric bias and layer bias [66, 51]. We also discuss our differences with CrossGET [47] here. CrossGET injects learnable cross tokens to capture cross-mon modsl information and merges less important tokens based on cosine similarity. In contrast, our method leverages CLS attention, performs multi-layer and multi-metric evaluation, and applies dynamic layer-wise pruning. Moreover, CrossGET requires re-training due its learnable cross tokens while our approach is entirely training-frec.
16

---

As shown in Table 9, our method better preserves performance under similar FLOPs, achieving consistently stronger results across wide range of benchmarks. Note that for ST as the code not publicly available, we follow their benchmark protocol for fair comparison.
Table 9: Comparison of FlowCut on LLa VA-15-B with more methods.
Method
TFLOPs
Te VQA-V2
POPE
MMB
Metric
ST [64]
FlowCut (Ours)
LLaVA-1.5 7B
8.82
58.2
78.5
85.9
64.7
69.5
A12D
55.4
55.8
PruMerge [46]
1.95
53.5
65.9
70.7
56.8
68.5
SQA
68.9
69.2
VTW [37] (K=5)
2.01
42.7
46.0
21.3
65.3
MMMU
35.3
36.2
RCom [7]
70.4
72.0
69.0
MMB
63.8
64.3
FasterVLM [61]
1.97
55.2
71.9
76.1
60.4
68.9
POPE
85.2
86.1
FlowCut (Ours)
1.95
55.6
72.8
80.2
60.8
69.1
TFLOPs
4.27
4.25
A.4 More Sensitivity Analyses of Hyperparameters
Cumulative importance mechanism's weighting coefficients We default to a setting of 0.5 for historical scores and 0.5 for current scores. As shown in Table 10, further conduct sensitivity analysis. When only the current score is used (i.e., without the cumulative strategy), the POPE performance is 83.8. The best performance is achieved at 0.5:0.5 and 0.6:0.4, while all other settings still outperform the non-cumulative baseline. This demonstrates that our method is robust to changes in the weighting coefficients and validates the effectiveness of the cumulative strategy.
Table 10: Sensitivity analysis of cumulative importance mechanism's 's weighting coefficients.
history:current 0:10 (w/o cumulative strategy
1:9
2:8
3:7
4:6
5:5
6:4
7:3
8:2
9:1
POPE benchmark
83.8 (baseline)
84.6
84.8
84.6
85.0
85.2
85.2
85.1
85.0
84.7
Pruning frequency For pruning frequency, we default to pruning every two layers with dynamic prune ratios. As shown in Table 11: 1) Pruning every two layers yields the best performance. Performance decreases as the interval increases, and pruning every layer (n=1) performs slightly worse than n=2, which we attribute to the fact that the first pruning step under n=l cannot leverage historical scores and must rely solely single-layer pruning (i.e., only pruning at the last layer), further demonstrating both robustness to this hyperparameter and the benefit of pruning redundancies as they emerge. 3) These results confirm that our method is not sensitive to the prune frequency, highlighting its robustness.
the current layer. 2) Multi-layer pruning outperforms
Table 11: Sensitivity analysis of pruning frequency.
Pruning each layer
n=2
n=6
n=8
n=12
only prune at last layer
POPE benchmark
84.9
85.2
84.6
84.6
84.3
84.3
84.0
83.9 (baseline)
A.5
Pseudocode
Here, we provide the pseudocode of our method. And note that method is indeed compatible with FlashAttention because it does not require the full N x N attention map. Instead, it only needs the 1 x N attention value of single CLS token or global token. When using FlashAttention, we can obtain the necessary attention values with small, separate computation: 1) For models with a CLS token, we separately compute its 1 x N attention map with the patch tokens; 2) For models without CLS token, we derive global token (by averaging patch tokens) and then compute its x N attention map. This targeted computation highly efficient, with complexity of just (()), adding virtually overhead. This allows method and FlashAttention to be used together effectively.
17

---

Algorithm Pseudocode for FlowCut
Input: current layer's hidden_states and QKV
Output: selected tokens for current layer QXV shape: [B, N, DJ (batch size, sequence length, embedding dim)
If the architecture without CLS token, use globally pooled token a aubstitute
cle_token output. hidden mean [B,D] elae: cla_token output seeput..idden,sstteee::::
architecture
without Setaitteture.withutt..oom
[B,D]
patch_ token - output. hidden_statess(:
Q_cla = Q(:, 0] [B, DJ, first token is CLS token
K_patch k[:, 1:] [B, N-1, D], exclude CLS token
Compute CLS-to-token attention (0(N) complexity, compatible with FlashAttention) cla_attn softmax (dot (q_cla, (aatch.transpose(11 -2)) sqrt(D)) [B, Nt1]
Compute semantic similarity evaluate token-level informativeness
V_cls v[:, 0] [B, D]
V_patch v(:, 1:] # [B, N-1, D]
semantic_attn = coftmax(dot
V_patch. tranph..ranspose(-1,, -2)) aqrt(()) [B, X-1]
Compute value norm as proxy for token information capacity
v_acore LL_nor_(V_patch) [B, N-1]]
multiple importance criteria
importance (normalizo
normalize (araalii(sseaantccaatta v_acore [B, N-1]
Accumulate importance across layers (ENA-like smoothing)
cumulative_score = 0.5 * curulative_score + 0.5 importance
Estimate pruming ratio based on CLS attention entropy
entropy compute_ (atpuuee_ettroycle_stttt # [B]
entropy_ratio entropy (Ng(N)
prune_ratio target_num) / pgrt(remain_layer
entropy_ratio 2)
Select top-X tokens based cumulative importance
k_keep round((1 prune_ratio) N)
keep_idx topk(cumulative_ecore. K-K_keep) # [B, K_keep]
Retain tokens
if not final_step: tokens (cls_tccls_tokee filter (patch_tokena, keep_idx))
elif final_step: tokens filter (patch, keep_idx)
A.6 Broader Impact
Our research rethink the essence of visual token reduction from a fundamental information flow perspective. Through systemical analysis of information flow patterns, we reveal that redundancy as a natural consequence of asymmetric attention. Through comprehensive analysis, we demonstrate the limitations of single-layer, single-criterion metrics in identifying redundancy, and propose FlowCut, pruning framework that aligns pruning decisions with the inherent dynamic
an iitormatioo-flow-veee information flow of tokens. In some cases, observe that token pruning, by removing redundant and distracting tokens, produces more accurate answers than the non-pruned baseline. This suggests that token pruning may hold promise for mitigating hallucinations in multimodal models. Therefore, is worth exploring in future work why token pruning helps reduce hallucinations, and how we can better leverage efficient techniques such token pruning and token merging not only to accelerate inference but also to suppress hallucinations.
A.7 More Visualization Results
18

---

Layer 1
Layer
Layer 3
Layer 4
Layer
Layer 6
Layer
Layer 8
Layer
Layer 10
Layer
Layer 12
Layer 13
Layer
Layer 15
Layer 16
Layer 17
Layer 18
Layer 19
Layer
Layer 21
Layer 22
Layer 23
Layer 24
Figure Visualization of information inflow main streamline.
19

---

Layer
Layer
Layer 3
Layer
Layer
Layer
Layer
Layer
Layer
Layer 10
Layer
Layer 12
Layer 13
Layer 14
Layer 15
Layer 16
Layer 17
Layer 18
Layer 19
Layer 20
Layer 21
Layer 22
Layer 23
Layer 24
Figure 10: Visualization of information outflow main streamline.
20

---

original image
192 tokens
128 tokens
tokens
Figure 11: Sparsification Visualization examples of FlowCut on LLaVA-L.5-7B.
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our main claims made in the abstract and introduction accurately reflect the paper's 's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of the work in the conclusion.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution low images taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers grounds for rejection, worse outcome might be that reviewers discover limitations that aren't tt acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
22

---

Justification:
We do not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material. Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We have reported the detailed settings and hyper-parameters. We will release our codes.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data provided not.
If the contribution a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to hosted model (e.g.. in the case of a large language model), releasing of model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
23

---

Answer: [Yes]
Justification: We have reported the detailed settings and paper-paramttess. We will release our codes.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (uttps://nips.. co/ http://gddeeeS/duduuunsssssslsss
for more details.
While we encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed reproduce the results. See the NeurIPS code and data submission guidelines (https: /tips.cc/pputeeee wwwpppblccgguieeeeeeeeeeooooeeoe Seeii//guuddee/ooeuuueeeee /ww.ee/ubmiiiiioooooooooooo
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized ver- sions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification:
We have reported the detailed settings and hyper-papee parameters. We will release our codes.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [NA]
Justification: Experiments are conducted based on the open-source LVLMs, and we follow the same random seed to conduct experiments and compare with other methods without the error bar.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, statistical significance tests, least for the experiments that support the main claims the paper.
The factors of variability that the emor bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors). should clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1 1-sigma error bars, but one should state it. The authors should preferably report 2-sigma error bar than state that they have 96% CI, if the hypothesis Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (c.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We have reported the computer resources in the implementation details.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make it into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: / //nuurrrrsp.. wwwpblic//thissuuiddeeoo
Answer:
[Yes]
Justification:
We conform, in every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consideration due laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We have discussed the broader impact the appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
25

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses addition to attacks, mecha- nisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility MI.).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (c.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: Our work does not have the risk for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We have respected the license of assets used in this paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset.
For scraped data from particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package
should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's creators.
26

---

13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We have respected the license of assets used in this paper.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their sub- missions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This paper does not include human subjects information.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should paid at least the minimum wage the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not include human subjects information.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact core methodology, scientific rigorousness, originality of the research, declaration is not required.
27

---

Answer:
[NA]
Justification:
We only used LLM for writing.
Guidelines:
The answer NA means that the core method development in this research does not involve
for LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: h/teer/iss...//// cc/Conferencces/ /202 what should or should not be described.
28


Reviews:
[{"review_id": "nmRYCxIPGm", "reviewer": "Reviewer_4FwU", "summary": "This paper introduces FlowCut, which is a token pruning method for reducing visual tokens in multimodal LLMs. While previous methods typically use attention scores from a single layer to identify redundant tokens, FlowCut proposes an information flow approach that identifies redundant tokens progressively through attention across multiple layers. Specifically, FlowCut suggests a layer-wise adaptive pruning ratio, a multi-criteria token scoring system, and a cumulative importance evaluation mechanism to aggregate importance over layers. Experiments are conducted on widely used multimodal LLMs such as LLaVA-1.5 and LLaVA-NeXT, showing that FlowCut outperforms some previous baselines.", "questions": "See Weaknesses please", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "### Strengths \n* This paper is well-organized\n* Extensive visualization is provided\n\n### Weaknesses\n* It seems that this method is incompatible with FlashAttention since it depends on the attention value. \n\n* For better clarity, I'd suggest the authors include pseudocode or an algorithm to describe their method precisely.\n\n* It would be better to have experimental results under a very small token number, as there is too much redundancy in the visual domain.\n\n* This paper shares some similar designs, such as the utilization of the CLS token with a prior work Crossget [1]. I’d suggest the authors provide a comparison or discussion.\n\n[1] CrossGET: Cross-Guided Ensemble of Tokens for Accelerating Vision-Language Transformers.  (ICML 2024).", "was_revised": true, "final_justification": "Thank the authors for providing a rebuttal. My concerns have been addressed, and I have raised my score. I'd suggest the authors carefully revise and include the new experiments and discussion to their final version.", "rebuttal": "**Thank for your valuable input.**\n\n### 1. Our method is indeed compatible with FlashAttention (for weakness 1)\n\nOur method is indeed compatible with FlashAttention because it does not require the full N×N attention map. Instead, it only needs the 1×N attention value of a single cls token or global token. When using FlashAttention, we can obtain the necessary attention values with a small, separate computation: 1) For models with a `CLS` token, we separately compute its 1×N attention map with the patch tokens; 2) For models without a `CLS` token, we derive a global token (by averaging patch tokens) and then compute its 1×N attention map. This targeted computation is highly efficient, with a complexity of just O(n), adding virtually no overhead. This allows our method and FlashAttention to be used together effectively.\n\nThe training process for our results in Appendix Table 7 utilized both our method (FlowCut) and FlashAttention, achieving a significant speedup. We copy the result here for convenient lookup.\n\n| Method (using flash attention) | Token | Training Time (w/ flash attention) |\n| ------------------------------ | :---: | :--------------------------------: |\n| LLaVA-1.5 7B (upper limit)     |  576  |               12.97h               |\n| Ours                           |  128  |               8.02h                |\n\n### 2. Pseudocode (for weakness 2)\n\nThank you for the valuable suggestion. Due to markdown limitations, we provide a simplified pseudocode here. A more complete version will be added in the final paper, and we commit to releasing the full codebase.\n\n```\n# Input: current layer's hidden_states and QKV\n# Output: selected tokens for current layer\n# QKV shape: [B, N, D] — batch size, sequence length, embedding dim\n# If the architecture without CLS token, use a globally pooled token as a substitute\n\ncls_token = output.hidden_states[:,0] # [B,D]\npatch_token = output.hidden_states[:,1:] # [B,N-1,D]\n\nQ_cls = Q[:, 0]             # [B, D], first token is CLS token\nK_patch = K[:, 1:]          # [B, N-1, D], exclude CLS token\n\n# Compute CLS-to-token attention (O(N) complexity, compatible with FlashAttention)\ncls_attn = softmax(dot(Q_cls, K_patch.transpose(-1, -2)) / sqrt(D))  # [B, N-1]\n\n# Compute semantic similarity to evaluate token-level informativeness\nV_cls = V[:, 0]             # [B, D]\nV_patch = V[:, 1:]          # [B, N-1, D]\nsemantic_attn = softmax(dot(V_cls, V_patch.transpose(-1, -2)) / sqrt(D))  # [B, N-1]\n\n# Compute value norm as a proxy for token information capacity\nv_score = l1_norm(V_patch)  # [B, N-1]\n\n# Fuse multiple importance criteria\nimportance = (normalize(cls_attn) + normalize(semantic_attn)) * v_score  # [B, N-1]\n\n# Accumulate importance across layers (EMA-like smoothing)\ncumulative_score = 0.5 * cumulative_score + 0.5 * importance\n\n# Estimate pruning ratio based on CLS attention entropy\nentropy = compute_entropy(cls_attn)          # [B]\nentropy_ratio = entropy / log(N) \t\t\t\t\t\t\nprune_ratio = (N - target_num) / sqrt(remain_layer) * (1 - entropy_ratio ** 2)\n\n# Select top-K tokens based on cumulative importance scores\nkeep_idx = topk(cumulative_score, K=round((1 - prune_ratio) * N))  # [B, K]\n\n# Retain cls token and selected tokens\nif not final_step: tokens = concat(cls_token, filter(patch_tokens, keep_idx))\n# Retain selected tokens\nelif final_step: tokens = filter(patch_tokens, keep_idx)\n```\n\n### 3. Retain less token (for weakness 3)\n\nThank you for the valuable suggestion. To further validate our method under highly constrained token budgets, we conduct experiments with **only 32 visual tokens**. Our method outperforms existing methods by a notable margin under aggressive token reduction, validating its effectiveness in preserving informative content despite severe compression.\n\n|                            | Token |   GQA    | TextVQA  |  VQA-V2  |   POPE   |   SQA    |  VizWiz  |    Avg    |\n| :------------------------- | :---: | :------: | :------: | :------: | :------: | :------: | :------: | :-------: |\n| Llava-1.5 7B (upper limit) |  576  |   61.9   |   58.2   |   78.5   |   85.9   |   69.5   |   50.0   |   100%    |\n| FastV                      |  32   |   46.2   |   51.5   |   56.0   |   35.7   |   68.3   |   49.1   |   78.7%   |\n| VisionZip                  |  32   |   51.5   |   51.7   |   65.1   |   68.3   |   68.2   |   49.9   |   88.7%   |\n| Ours                       |  32   | **52.1** | **53.0** | **66.9** | **69.6** | **68.7** | **52.3** | **90.8%** |\n\n### 4. Compare with CrossGET (for weakness 4)\n\nCrossGET injects learnable *cross tokens* (distinct from the CLS token) into each modality to capture cross-modal information, and evaluates token importance using cosine similarity with respect to these tokens, followed by merging less important tokens via complete-graph soft matching. \n\nOur method differs from CrossGET in three key aspects: **1) Theoretical Foundation:** CrossGET proposes a token reduction technique for inference acceleration. Beyond merely proposing a method, our work introduces a novel perspective (information flow) for understanding the emergence of redundant tokens. We find that the CLS token acts as an information relay and thus leverage it to simplify the modeling of token-to-token interaction, offering a comprehensive analysis of information flow to guide method design (as recognized by reviewer T8rC & NDVm). **2) Methodological Superiority:** CrossGET does not adopt the CLS token but injects learnable *cross tokens* to capture cross-modal information and utilizes *cross tokens* as a single criterion to assess token importance. In contrast, our method performs multi-layer and multi-criteria evaluation (CLS attention serves as one of the metrics), enabling more accurate identification of redundant tokens. We also introduce dynamic layer-wise pruning to ensure that pruning decisions align with the evolving information flow of tokens. **3) Training-Free Application:** CrossGET requires re-training due to its learnable *cross tokens*. In comparison, our method is entirely training-free while still achieving superior performance.\n\nAs shown below, despite being training-free, our method reduces FLOPs to 50% and consistently outperforms CrossGET in accuracy and speed. Note that our method can achieve even better performance with training.\n\n|                              | FLOPs $\\downarrow$ |   GQA    | TextVQA  |  VQA-V2  |   POPE   |   SQA    |  VizWiz  |\n| :--------------------------- | :----------------: | :------: | :------: | :------: | :------: | :------: | :------: |\n| Llava-1.5 7B (upper limit)   |        100%        |   61.9   |   58.2   |   78.5   |   85.9   |   69.5   |   50.0   |\n| CrossGET (requires training) |        66%         |   61.4   |   54.9   |   77.3   |   83.9   |   66.7   |   47.7   |\n| Ours (w/o train)             |      **50%**       | **61.5** | **57.8** | **77.6** | **86.1** | **68.7** | **52.1** |\n| Ours (w/ train)              |      **43%**       | **62.0** | **58.1** | **78.7** | **86.0** | **68.9** | **52.1** |"}, {"review_id": "reHNinQCbX", "reviewer": "Reviewer_T8rC", "summary": "This paper first analyzes the information propagation in Transformers from shallow to deep layers using attention scores. It finds that the CLS token functions as a relay, helping most patch tokens gradually expand their attention from neighboring positions in shallow layers to longer-range dependencies in deeper layers, eventually imitating the CLS token by focusing on a few specific tokens. This insight inspires a dynamic pruning strategy. By integrating attention scores, similarity to the CLS token, and the L1 norm, the authors propose a layer-wise importance metric, which is further accumulated across layers. This leads to an efficient and powerful dynamic pruning method that outperforms traditional static pruning approaches.", "questions": "1. During forward propagation, is dynamic pruning performed simultaneously at each layer? If so, does it mean that pruning decisions made in earlier layers will influence the pruning results in later layers?\n2. Additionally, if that's the case, since the number of pruned tokens per layer is related to attention entropy, how is it ensured that the final number of tokens matches the target? Is the final pruning step forcibly adjusted to meet the target count?\n3. In Section 2 of the main paper, the analysis of information propagation is based on a tokenizer that includes a CLS token. If a tokenizer without a CLS token is used, and the result of global pooling is treated as the CLS token, would similar conclusions still hold?\n4. What is the design principle behind the combination of multiple metrics? Why are both addition and multiplication used?\n5. In Equation (4) of the main paper, what does $I^r$ represent? It is not mentioned earlier in the text—should it instead be $I^d$?", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n1. The proposed method not only achieves faster computation but also delivers better performance.\n2. It is motivated by the perspective of information propagation, providing a certain theoretical foundation and offering potential for further improvement.\n3. The idea of dynamic pruning offers a more flexible and elegant solution.\n\n**Weaknesses**\n1. The paper does not provide a detailed rationale behind the design choices for combining multiple metrics.", "was_revised": true, "final_justification": "The rebuttal clarified my main questions. I have no remaining concerns and will maintain my original score.", "rebuttal": "**Thank for your valuable input and appreciation on our work**\n\n### 1. More detailed rationale behind the design for multiple metric (for weakness 1 & question 4)\n\nMost existing methods adopt a single metric, typically the attention score, as the sole criterion for token importance:\n$\\mathrm{cls\\_attn} = \\text{softmax}\\left( \\frac{Q_{\\text{cls}} \\cdot K_{\\text{patch}}^\\top}{\\sqrt{D}} \\right).$ \nHowever, we argue that this is insufficient. From an information flow perspective, attention alone does not reliably indicate how much information a token conveys. According to the attention computation formula: \n\n$\\text{Attn} = \\text{softmax}(QK^\\top/\\sqrt D), \\text{output}=Attn\\cdot V$\n\neven if a token receives a high attention weight, it does not necessarily contribute significant information to the final representation. Its actual contribution also depends on the magnitude of its value vector $V$. If the attention score is high but $V$ is close to zero, the token’s contribution is negligible (as shown in main text Figure 4, where some tokens exhibit high attention but nearly zero $V$). Therefore, we introduce the $l_1\\text{-}norm$ of $V$ as an additional metric, capturing the actual information carried by the token. \n\nHowever, some tokens may exhibit large value magnitudes while lacking semantically useful information (e.g., register tokens), making it unreliable to assess their true contribution based solely on attention and value magnitude. To this end, we introduce a semantic similarity-based metric to assess whether a token carries meaningful information: $\\mathrm{semantic\\_attn} = \\text{softmax}\\left( \\frac{V_{\\text{cls}} \\cdot V_{\\text{patch}}^\\top}{\\sqrt{D}} \\right),$ which measures the alignment between the CLS token and each patch token in the value space. Since the CLS token aggregates semantically important content, a high similarity indicates that the token likely contributes useful information.\n\nBy aligning with the structure of the attention output computation: $\\text{softmax}(QK^\\top / \\sqrt D) \\cdot V$, we design our combined metric as: $ ( \\mathrm{semantic\\_attn} + \\mathrm{cls\\_attn}) \\times \\|V\\|_1.$ This formulation ensures that the preserved tokens are both highly attended and semantically informative, with non-trivial content. That's why we use both addition and multiplication. As demonstrated in the ablation studies in the main text, this multi-metric design consistently outperforms single-metric baselines, confirming its effectiveness.\n\n### 2. More details about adaptive pruning strategy (for question 1, 2)\n\nFor the adaptive pruning strategy, we calculate the number of tokens to prune and perform pruning every two layers. This design ensures that each pruning step considers both current and historical token importance, avoiding layer bias caused by inconsistent token relevance across layers.\n\nMoreover, pruning decisions in later layers are influenced by earlier ones. Since redundancy emerges progressively as attention becomes more concentrated, pruning tokens as soon as they become redundant is more effective than deferring pruning to the final layer. The ablation study in the main text (Table 6) confirms this. Early removal prevents redundant tokens from continuously consuming attention, allowing more information to flow toward useful tokens.\n\nIn addition, the number of tokens pruned in each layer depends on how many have already been removed. As shown in Equation (3) in the main text, we calculate the pruning ratio based on the difference between the target number of tokens and the cumulative number already pruned.\n\nA final pruning step forcibly removes the remaining tokens to exactly meet the target count.\n\n### 3. Conclusions still hold in non-CLS architecture (for question 3)\n\nOur conclusions remain valid for non-CLS architectures. To support this, we additionally report the attention entropy of the CLS token in *LLaVA-1.5 (with CLS)* and the global token in *LLaVA-1.5 (without CLS)*, where the global pooled token is treated as the CLS equivalently, on the same input sample. While the absolute values differ slightly, **the overall trends across layers are highly consistent**, indicating that the global token plays a similar role in capturing information flow.\n\n| Layer                     | 2    | 4    | 6    | 8    | 10   | 12   | 14   | 16   | 18   | 20   | 22   |\n| ------------------------- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- |\n| CLS token attn entropy    | 6.2  | 6.1  | 5.9  | 5.8  | 5.3  | 3.8  | 4.4  | 3.8  | 3.9  | 4.2  | 4.3  |\n| global token attn entropy | 6.2  | 6.0  | 5.9  | 5.9  | 5.5  | 3.7  | 4.2  | 3.6  | 3.6  | 4.1  | 4.2  |\n\nFurthermore, as shown in Table 3 of the main text, we apply our method to Qwen2-VL, whose vision encoder is CLS-free. The model still achieves strong performance, demonstrating the effectiveness of our approach in CLS-free architectures.\n\n### 4. Minor Writing error (for question 5)\nThank you very much for your careful correction. Equation 4 should indeed be $I^d$ instead of $I^r$. We will take extra care to correct this in the final version to prevent any minor writing errors."}, {"review_id": "jcwGL39WdR", "reviewer": "Reviewer_NDVm", "summary": "This paper targets the inference inefficiency of large-scale vision–language models (LVLMs) caused by redundant visual tokens by proposing an information-flow-based pruning method. The authors dynamically adjust pruning strength at each transformer layer according to the CLS token’s attention entropy, score tokens using attention weight, semantic similarity to CLS, and Value-vector norm, and aggregate importance across layers to avoid the instability of single-layer criteria. On multiple LVLM architectures and 12 image + 3 video benchmarks, FlowCut dramatically reduces token count—up to 94.4% fewer tokens—while preserving 91.9% inference accuracy. Under fixed final-token budgets, its layer-wise adaptive pruning achieves the greatest cumulative computation reduction compared to prior methods.", "questions": "1. The multi-criteria evaluation and cumulative flow tracking rely on several hyperparameters (e.g., weighting coefficients, pruning frequency). Could the authors include ablation studies or sensitivity analyses to demonstrate robustness across a reasonable range of these parameters?\n\n2. FlowCut currently uses attention entropy and the Value-vector L₁ norm to characterize information flow. How can it avoid mistakenly pruning critical tokens in dense multi-object or heavily occluded scenes?\n\n3. Some readers may not be familiar with Vision Transformer details; terms like \"CLS\" should be defined at first mention or briefly explained to improve clarity.\n\n4. Long video inputs dramatically increase token counts and introduce large inter-frame semantic shifts. Can FlowCut's layer-wise adaptive pruning maintain high-quality inference under these conditions?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Pros\n\n1. The framework is hierarchically structured and methodologically novel, achieving significant speed–accuracy gains with an intuitive combination of strategies.\n\n2. The Insights module rigorously analyzes visual token redundancy from a dynamic information-flow perspective, presenting core observations step by step with clear logic.\n\n3. Experiments are comprehensive across multiple LVLM architectures and 12 image + 3 video benchmarks, using diverse metrics to fully demonstrate the efficiency–effectiveness trade-off.\n\nCons\n\n1. The multi-criteria evaluation and cumulative flow tracking involve several hyperparameters (e.g., weighting coefficients) without ablation or sensitivity analyses to validate robustness.\n\n2. Evaluation is limited to the LLaVA family and Qwen2-VL; applicability to other vision encoders or CLS-free architectures remains unverified.\n\n3. Reliance solely on attention entropy and Value-vector L₁ norm to characterize information flow may misjudge redundancy in complex scenes (e.g., dense multi-object overlaps).", "was_revised": false, "final_justification": "", "rebuttal": "**Thank for your valuable input and appreciation on our work**\n\n### 1. More sensitivity analyses of hyperparameters (for weakness 1 & question1)\n\nFor the cumulative importance evaluation mechanism's weighting coefficients, we default to a setting of 0.5 for historical scores and 0.5 for current scores. We further conduct a sensitivity analysis, with three key observations: 1) When only the current score is used (i.e., without the cumulative strategy), the POPE performance is 83.8. Incorporating historical scores consistently improves performance across all coefficient settings, validating the effectiveness of our cumulative strategy. 2) The best performance is achieved at 0.5:0.5 and 0.6:0.4, while all other settings still outperform the non-cumulative baseline, indicating robustness to coefficient changes. 3) These results confirm that our method is not sensitive to the weighting coefficients, demonstrating its robustness.\n\n| history : current | 0:10 (w/o cumulative strategy ) | 1:9  | 2:8  | 3:7  | 4:6  |   5:5    |   6:4    | 7:3  | 8:2  | 9:1  |\n| :---------------: | :-----------------------------: | :--: | :--: | :--: | :--: | :------: | :------: | :--: | :--: | :--: |\n|  POPE benchmark   |         83.8 (baseline)         | 84.6 | 84.8 | 84.6 | 85.0 | **85.2** | **85.2** | 85.1 | 85.0 | 84.7 |\n\nFor pruning frequency, we default to pruning every two layers with dynamic prune ratios. Additional sensitivity experiments show: 1) Pruning every two layers yields the best performance. Performance decreases as the interval increases, and pruning every layer (n=1) performs slightly worse than n=2. We attribute this to the fact that the first pruning step under n=1 cannot leverage historical scores and must rely solely on the current layer. 2) Multi-layer pruning outperforms single-layer pruning (i.e., only pruning at the last layer), again demonstrating both robustness to this hyperparameter and the benefit of pruning redundancies as they emerge.  3) These results confirm that our method is not sensitive to the prune frequency, demonstrating its robustness.\n\n| Pruning each n layer | n=1  |   n=2    | n=3  | n=4  | n=6  | n=8  | n=12 | only prune at last layer |\n| :------------------: | :--: | :------: | :--: | :--: | :--: | :--: | :--: | :----------------------: |\n|    POPE benchmark    | 84.9 | **85.2** | 84.6 | 84.6 | 84.3 | 84.3 | 84.0 |     83.9 (baseline)      |\n\n### 2. Our method can apply on cls-free architecture (for weakness 2)\n\nOur method is fully applicable to CLS-free architectures. We would like to clarify that Qwen2-VL is a CLS-free architecure, and as shown in main text Table 3 our method demonstrates strong performance on this architecture as well (we copy result here for convenient lookup). This confirms that the effectiveness of our approach is not limited to CLS-based models.\n\n| Method                    |         Tokens          |   GQA    |   MMB    | MMB$^{CN}$ |   MME    |   POPE   |   SQA    | TextVQA  |   Avg.    |\n| ------------------------- | :---------------------: | :------: | :------: | :--------: | :------: | :------: | :------: | :------: | :-------: |\n| Qwen2-VL 7B (upper limit) | 100%|   61.9   |   79.9   |    79.5    |   2338   |   87.2   |   85.1   |   82.2   |   100%    |\n| FastV                     |          33.3%          |   58.0   |   76.1   |    73.9    |   2130   |   82.1   |   80.0   |   77.3   |   93.6%   |\n| Ours                      |          33.3%          | **60.5** | **79.2** |  **78.2**  | **2335** | **86.0** | **84.0** | **81.1** | **98.7%** |\n| FastV                     |          22.2%          |   56.7   |   74.1   |    72.4    |   2031   |   79.2   |   78.3   |   72.0   |   90.4%   |\n| Ours                      |          22.2%          | **59.2** | **77.8** |  **76.9**  | **2310** | **84.6** | **80.5** | **78.3** | **96.5%** |\n| FastV                     |          11.1%          |   51.9   |   70.1   |    63.8    |   1962   |   76.1   |   75.8   |   60.3   |   83.6%   |\n| Ours                      |          11.1%          | **56.4** | **72.6** |  **72.5**  | **2252** | **81.8** | **78.2** | **68.9** | **91.3%** |\n\n\n\n### 3. More complex scenes  (for weakness3 & question 2)\n\nOur method not only utilizes attention-based signals (attention entropy and attention score) and the Value-vector L₁ norm to characterize information flow, but also introduces a semantic effectiveness metric: $\\mathrm{semantic\\_similarity} = \\text{softmax}\\left( \\frac{V_{\\text{cls}} \\cdot V_{\\text{patch}}^\\top}{\\sqrt{D}} \\right)$, which evaluates whether a token contains semantically meaningful content. Compared to prior work relying solely on attention scores, our multi-criteria approach ensures that preserved tokens are both highly attended and semantically informative.\n\nFor more complex scenes, we would like to point out that VQA-v2 benchmark includes images with dense multi-object scenes, where our method performs strongly (main text Table 1).\n\nHere, we evaluate on the more challenging Referring Expression Comprehension (REC) task from the RefCOCO benchmark, which requires the model to understand textual instructions and detect target objects in images that often contain dense multi-object and heavily occluded scenes. For example, given an image containing multiple people in different outfits and the instruction “woman with green shirt,” the model must correctly locate the intended target. Our method performs well under such conditions, demonstrating its ability to prune redundant tokens while preserving those critical for fine-grained understanding.\n\n| RefCoCo                     |               Tokens               | val       | testA     | testB     |\n| --------------------------- | :--------------------------------: | --------- | --------- | --------- |\n| Qwen2.5-VL-7B (upper limit) | 100%  | 89.45     | 92.56     | 85.16     |\n| FastV                       |             retain 50%             | 73.85     | 73.38     | 74.21     |\n| Ours                        |             retain 50%             | **87.29** | **90.93** | **83.06** |\n\n### 4. Long video scenes (for question 4)\n\nAs shown in Table 4 of the main text, we have already evaluated our method on three video understanding benchmarks, demonstrating that FlowCut can effectively accelerate inference with negligible performance drop in video scenarios. To further assess its robustness on long-form inputs, we additionally evaluate on EgoSchema, a long-form video question answering dataset. The results show that FlowCut continues to perform well under long video settings:\n\n|                              | Tokens | EgoSchema |\n| ---------------------------- | :----: | :-------: |\n| Video-LLaVA 7B (upper limit) |  2048  |   38.4    |\n| FastV                        |  256   |   33.7    |\n| Ours                         |  256   | **37.9**  |\n\n### 5.  Additional explanation of the CLS token (for question 3)\n\nThank you very much for your suggestion. We will add an additional explanation of the CLS token to help readers unfamiliar with Vision Transformers better understand it: \n\n> The CLS (classification) token is a special learnable token prepended to the input sequence in Vision Transformers, whose final representation is typically used for classification or as a summary of the image."}, {"review_id": "gL5N87waFK", "reviewer": "Reviewer_TJih", "summary": "This paper provide a novel perspective (information flow) for understanding the emergence of redundant tokens, revealing redundancy as a natural consequence of asymmetric attention. Based on authors' insights,  they propose a combined criteria to measure the importance of tokens. The method is tested on multiple benchmarks, demonstrating robustness.", "questions": "1.What is the overhead of the proposed work？And how do authors reproduce the prior works in Tab. 5, using flash attention or not?\n\n2. In Eq. 6, the authors combine historical and current criteria by a equal weight 0.5, what about others weights? Moreover, the token importance in last layer is used in the current layer, does it make sense?\n\n3. Token pruning is implemented in every two layers，why？How about three or more layers?\n\n4.What about the performance on image caption task like coco caption, nocaps and flickr30k?", "limitations": "The authors argue that they discuss the limitations of the work in the conclusion, but in fact not. For limitations, please refer to weaknesses and questions.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n\n1. The figures are visually good.\n2. The experimental evaluations are conducted on several MLLMs and video benchmarks, the results are promising.\n3. The method is clear and easy to follow.\n\nWeaknesses\n\n1. The author proposed a new concept called information flow, but it is essentially just the attention in the model. This concept actually diminishes the readability and comprehensibility of the paper. \n2. The insights are neither novel nor appealing, as similar observations have been made in many previous articles on Vision Transformers.\n3. The proposed method is just attention-based and not aligned well with insights, what is the relationship between it and information flow?\n4. Some benchmarks are missed, see questions.\n5. Literature review is not comprehensive enough, some prior works should be discussed in related work or compared in experiments:\n\n[1] LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Models. Arxiv24\n\n[2] ST3: Accelerating Multimodal Large Language Model by Spatial-Temporal Visual Token Trimming. AAAI25\n\n[3] [CLS] Attention is All You Need for Training-Free Visual Token Pruning: Make VLM Inference Faster. Arxiv24\n\n[4] Recoverable Compression: A Multimodal Vision Token Recovery Mechanism Guided by Text Information. AAAI25\n\n[5] Boosting Multimodal Large Language Models with Visual Tokens Withdrawal for Rapid Inference. AAAI25", "was_revised": true, "final_justification": "Thanks for authors' rebuttal,  I have raised my score. I suggest the authors include the new experiments and discussion in their revision.", "rebuttal": "### 1. Clarify misunderstanding of our work's insight and method (for weakness 1,2,3)\nThere may be some misunderstandings about our work. We introduces a unified, bottom-up perspective (*information flow*) by modeling token-to-token interaction to explore the propagation pattern of visual information and understand the emergence of redundant tokens. **Attention serves as only one component,** our perspective on information flow extends beyond attention: we analyze and visualize the evolution of information flow across layers, quantify each token's information content by Value-vector magnitudes, and assess whether tokens contain semantically meaningful information by semantic similarity. All these analyses are grounded in our proposed *information flow* view. As Reviewer T8rC noted: *“providing a certain theoretical foundation and offering potential for further improvement”*, and Reviewer NDVm said: *“The Insights module rigorously analyzes visual token redundancy from a dynamic information-flow perspective”*.\n\n**For the insight:** Our proposed *information flow* goes beyond merely referring to the attention in the model. 1) We model the token interaction process from an information flow perspective and primarily analyze the patterns and trends of information interaction, with some analyses illustrated through attention map visualizations rather than focusing on attention itself. 2) We analyze and visualize the evolution of information flow across layers, showing that lower layers exhibit local, neighborhood-level interactions, while higher layers engage in long-range semantic interactions. 3) We incorporate Value-vector magnitudes to quantify the amount of information each token carries and semantic similarity to assess whether tokens contain semantically redundant or meaningful information.\n\n**For the method:** Each module of our method is motivated by phenomena uncovered through our information flow analysis: 1) Based on the observed layer-wise differences in information interaction patterns, we propose an EMD-like cumulative scoring strategy that iteratively updates token importance by aggregating both current and historical interaction evidence; 2) Based on the finding that redundancy emerges progressively as attention concentrates over depth, driven by increasingly focused semantic interactions, we introduce an adaptive mechanism to determine the pruning ratio for each layer accordingly; 3) Based on the analysis that attention alone does not fully capture token contribution, we additionally incorporate the Value-vector L1 norm and semantic similarity to estimate each token’s contribution to information gain.\n\n### 2. Experiments on more benchmarks (for weakness 4 & question 4)\n\nIn the main text, we have already evaluated our method on 12 image benchmarks and 3 video benchmarks. Here, we further include results on three widely-used image captioning benchmarks: COCO 2017 Caption, NoCaps, and Flickr30k. We evaluate our method under both 192 and 128 token settings. The results show that our method achieves strong performance on these benchmarks, offering substantial inference acceleration with minimal performance degradation.\n\n|                     |Token|COCO|Nocaps|Flickr30k|\n| ------------------ | :---: | :------: | :------: | :------: |\n| LLaVA-1.5 7B (upper limit) |576|110.5|105.5|75.2|\n| FastV                      |192|109.1|104.2|73.5|\n| Ours                       |192|**110.7**|**105.3**|**74.9**|\n| FastV                      |128|105.9|101.2|71.0|\n| Ours                       |128|**107.8**|**102.4**|**73.6**|\n\n### 3. Compare with more works (for weakness 5)\n\nWe have already compared with several representative methods in the main text. Here, we further include more prior works for comparison: PruMerge clusters the visual tokens via k-nearest neighbormerges algorithm and merge them. ST$^3$ prunes low-importance tokens progressively across layers using token attention maps. VTW removes unimportant visual tokens based on attention score at specific layers of the LLM. RCom merges unimportant patch tokens based on their similarity to the CLS token and the text tokens. FasterVLM prunes vision tokens based on CLS attention scores at vision encoder's output layer.\n\nThese methods all rely on single-layer, single-metric scores (e.g., attention or similarity) to assess token importance. In contrast, our method employs multiple metrics and aggregates scores across layers using an EMD-based strategy, which helps mitigate single-metric bias and layer bias.\n\nAs shown in the tables below, our method better preserves performance under similar FLOPs, achieving consistently stronger results across a wide range of benchmarks. Note that for ST$^3$, as the code is not publicly available, we follow their benchmark protocol for fair comparison.\n\n| Methods            | TFLOPs $\\downarrow$ | TextVQA  |  VQA-V2  |   POPE   |   MMB    |   SQA    |\n| ------------------ | :------------: | :------: | :------: | :------: | :------: | :------: |\n| Llava-1.5 7B       |8.82|58.2|78.5|85.9|64.7|69.5|\n| PruMerge [1]       |1.95|53.5|65.9|70.7|56.8|68.5|\n| VTW [5] (K=5)      |2.01|8.1|42.7|46.0|21.3|65.3|\n| RCom [4]           |1.96|55.5|70.4|72.0|57.9|69.0|\n| FasterVLM [3]      |1.97|55.2|71.9|76.1|60.4|68.9|\n| **FlowCut (Ours)** |1.95|**55.6**|**72.8**|**80.2**|**60.8**|**69.1**|\n\n|                    | TFLOPs $\\downarrow$ |   AI2D   |   SQA    |   MMMU   |   MMB    |   POPE   |\n| ------------------ | :-----------------: | :------: | :------: | :------: | :------: | :------: |\n| ST$^3$ [2]         |  4.27  |   55.4   |   68.9   |   35.3   |   63.8   |   85.2   |\n| **FlowCut (Ours)** |  4.25  | **55.8** | **69.2** | **36.2** | **64.3** | **86.1** |\n\n[1] LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Models. Arxiv24\n\n[2] ST3: Accelerating Multimodal Large Language Model by Spatial-Temporal Visual Token Trimming. AAAI25\n\n[3] [CLS] Attention is All You Need for Training-Free Visual Token Pruning: Make VLM Inference Faster. Arxiv24\n\n[4] Recoverable Compression: A Multimodal Vision Token Recovery Mechanism Guided by Text Information. AAAI25\n\n[5] Boosting Multimodal Large Language Models with Visual Tokens Withdrawal for Rapid Inference. AAAI25\n\n### 4. Overhead of our work (for question 1)\n\nWe have reported the overhead of our method in Table 5 of the main text. The FLOPs reported for both our method and other baselines in Table 5 are measured without using Flash Attention. This is because prior works (e.g., SparseVLM**, **FastV, and VisionZip) are not compatible with Flash Attention. Therefore, to ensure a fair comparison, we report all FLOPs under the same setting (without Flash Attention).\n\nIt is worth noting that our method is compatible with Flash Attention, and using them together can further accelerate inference. We report this combination in Appendix Table 7, where we observe a significant speed-up in training time when both are used.\n\n### 5. More explanation of cumulative importance strategy (for question 2)\nAs shown in main text Figure 2 and Figure 4, the information contribution of the same token varies across layers. Therefore, determining token importance based solely on the current layer introduces layer bias, as a token might appear unimportant in the current layer but be crucial in others. To address this, we adopt an EMD-like cumulative scoring strategy, where token importance is updated as: cumulative = 0.5 × current + 0.5 × previous cumulative. This approach effectively integrates both local and historical importance signals. Its effectiveness is supported by the ablation study in Table 6 of the main text, as well as the sensitivity analysis discussed in our Answer 6.\n\n### 6. More sensitivity analyses of hyperparameters (for question 2, 3)\nFor the cumulative importance evaluation mechanism's weighting coefficients, we default to a setting of 0.5 for historical scores and 0.5 for current scores. We further conduct a sensitivity analysis, with three key observations: 1) When only the current score is used (i.e., without the cumulative strategy), the POPE performance is 83.8. Incorporating historical scores consistently improves performance across all coefficient settings, validating the effectiveness of our cumulative strategy. 2) The best performance is achieved at 0.5:0.5 and 0.6:0.4, while all other settings still outperform the non-cumulative baseline, indicating robustness to coefficient changes. 3) These results confirm that our method is not sensitive to the weighting coefficients, demonstrating its robustness.\n\n| history : current | 0:10 (w/o cumulative strategy ) | 1:9  | 2:8  | 3:7  | 4:6  |   5:5    |   6:4    | 7:3  | 8:2  | 9:1  |\n| :---------------: | :-----------: | :--: | :--: | :--: | :--: | :------: | :------: | :--: | :--: | :--: |\n|  POPE benchmark   |83.8 (baseline)| 84.6 | 84.8 | 84.6 | 85.0 | **85.2** | **85.2** | 85.1 | 85.0 | 84.7 |\n\nFor pruning frequency, we default to pruning every two layers with dynamic prune ratios. Additional sensitivity experiments show: 1) Pruning every two layers yields the best performance. Performance decreases as the interval increases, and pruning every layer (n=1) performs slightly worse than n=2. We attribute this to the fact that the first pruning step under n=1 cannot leverage historical scores and must rely solely on the current layer. 2) Multi-layer pruning outperforms single-layer pruning (i.e., only pruning at the last layer), again demonstrating both robustness to this hyperparameter and the benefit of pruning redundancies as they emerge.  3) These results confirm that our method is not sensitive to the prune frequency, demonstrating its robustness.\n\n| Pruning each n layer | n=1  |   n=2    | n=3  | n=4  | n=6  | n=8  | n=12 | only prune at last layer |\n| :------------------: | :--: | :------: | :--: | :--: | :--: | :--: | :--: | :-----------: |\n|    POPE benchmark    | 84.9 | **85.2** | 84.6 | 84.6 | 84.3 | 84.3 | 84.0 |83.9 (baseline)|"}]

Author rebuttal:

Meta-review: The paper received three borderline accepts and one accept recommendations. The reviewers felt that the rebuttal addressed most of their initial concerns regarding some lack of experiments, background literature, and issues with clarity.  Based on the paper's good approach, final comprehensive experiments, and promising results, the ACs agree with the reviewers and recommend acceptance.  Please revise and include the new experiments and discussion in the final version.
Decision: Accept (poster)