Title: FlowCut: Rethinking Redundancy via Information Flow for Efficient Vision-Language Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: Vision Language Model

Efficiency

Visual Token Pruning
Abstract: Large vision-language models (LVLMs) excel at multimodal understanding but suffer from high computational costs due to redundant vision tokens. Existing pruning methods typically rely on single-layer attention scores to rank and prune redundant visual tokens to solve this inefficiency. However, as the interaction between tokens and layers is complicated, this raises a basic question: Is such a simple single-layer criterion sufficient to identify redundancy? To answer this question, we rethink the emergence of redundant visual tokens from a fundamental perspective: information flow, which models the interaction between tokens and layers by capturing how information moves between tokens across layers. We find (1) the CLS token acts as an information relay, which can simplify the complicated flow analysis; (2) the redundancy emerges progressively and dynamically via layer-wise attention concentration; and (3) relying solely on attention scores from single layers can lead to contradictory redundancy identification. Based on this, we propose FlowCut, an information-flow-aware pruning framework, mitigating the insufficiency of the current criterion for identifying redundant tokens and better aligning with the model's inherent behaviors. Extensive experiments show FlowCut achieves superior results, outperforming SoTA by 1.6% on LLaVA-1.5-7B with 88.9% token reduction, and by 4.3% on LLaVA-NeXT-7B with 94.4% reduction, delivering 3.2$\times$ speed-up in the prefilling stage.  Our code is available at https://github.com/TungChintao/FlowCut.

FlowCut: Rethinking Redundancy via Information Flow for Efficient Vision-Language Models

Abstract

Large vision-language models (LVLMs) excel at multimodal understanding but suffer from high computational costs due to redundant vision tokens. Existing pruning methods typically rely on single-layer attention scores to rank and prune redundant visual tokens to solve this inefficiency. However, as the interaction between tokens and layers is complicated, this raises a basic question: Is such a simple single-layer criterion sufficient to identify redundancy? To answer this question, we rethink the emergence of redundant visual tokens from a fundamental perspective: information flow, which models the interaction between tokens and layers by capturing how information moves between tokens across layers. We find (1) the CLS token acts as an information relay, which can simplify the complicated flow analysis; (2) the redundancy emerges progressively and dynamically via layer-wise attention concentration; and (3) relying solely on attention scores from single layers can lead to contradictory redundancy identification. Based on this, we propose FlowCut, an information-flow-aware pruning framework, mitigating the insufficiency of the current criterion for identifying redundant tokens and better aligning with the model's inherent behaviors. Extensive experiments show that FlowCut achieves superior results, outperforming SoTA by 1.6% on LLaVA-1.57B with 88.9% token reduction, and by 4.3% on LLaVA-NeXT-7B with 94.4% reduction, delivering 3.2 × speed-up in the prefilling stage. Our code is available at https://github.com/TungChintao/FlowCut .

1 Introduction

Large vision-language models (LVLMs) [4, 26, 41, 53] have achieved remarkable progress, demonstrating impressive abilities in multimodal perception and reasoning by effectively bridging visual and linguistic modalities. However, LVLMs face significant challenges in computational costs and memory demands, primarily caused by the massive amounts of vision tokens, especially for high-resolution images [39, 55] and multi-frame videos [36, 44], limiting their practical applications.

As visual signals inherently contain more spatial redundancy than information-dense text [12, 45, 52], a surge of recent efforts [6, 57, 59, 62] introduce token reduction strategies to solve the inefficiency of LVLMs by pruning visual tokens in a training-free manner. These methods typically rely on attention scores to rank and prune tokens either at a specific layer or several LLM layers with a pre-defined prune ratio for each layer. However, regardless of the pruning schedule, redundancy is assessed solely based on the attention scores from the current layer. This raises a basic question: Are redundancy defined through such a simple single-layer single-criterion score, as the core of current pruning methods, rigorous enough to support pruning decisions?

Vanilla

206ms

CLStokengathersinformationglobally

FastV

117ms

(LLaVA-NeXT-7B)

FlowCut

63.8ms

3.2× speed up

CLS

→

Vanilla(100%)

100%

TokenReduction↓88.9%

98.6%

FlowCut

VisionZip

96.0%

95.6%

Pdrop

95%

94.4%

94.4%

93.0%

92.3%

FastV

CLS

91.2%

91.3%

%06

86.1%

85%

84.6%

83.6%

84.6%

83.9%

CLS

82.4%

80%

77.5%

75%

patchtokensdrawinformationfrom

LLaVA-1.5

LLaVA-NeXT

Qwen2-VL

Video-LLaVA

neighborhoodsand

2CLS token

In this work, we rethink the emergence of redundant visual tokens from a fundamental perspective: information flow. Specifically, the information flow for each token is defined as the inflow (the source token that transmits the most information to this token) and outflow (the destination token that this token transmits the most information to), which is reflected in the attention map in each layer of ViT and fundamentally models the interaction between tokens and layers. Through systematic analysis of information flow patterns, we uncover a set of key insights that reveal how redundancy emerges, guiding the identification of important tokens: ➊ The CLS token acts as an information relay and its behavior can largely represent overall information flow , allowing the complex token-to-token interactions to be approximated and validating it as an effective proxy for identifying redundant tokens. ➋ Redundancy emerges progressively via layer-wise attention concentration , suggesting a dynamic pruning strategy aligned with the natural evolution of attention to match the tokens' inherent behavior. ➌ Relying solely on attention score from single layers can lead to contradictory redundancy identification , indicating the limitation of single-criterion scoring and single-layer view, which demonstrates the insufficiency of the current criterion for identifying redundancies.

Driven by these insights, we propose FlowCut, an information-flow-aware pruning framework. Specifically, we introduce a layer-wise adaptive pruning ratio module based on attention entropy, adjusting the pruning strength according to the inherent concentration level of attention distributions. We also design a multi-criteria scoring strategy to jointly consider attention strength, information density, and semantic relevance, providing a more reliable estimation of token importance. Further, we develop a cumulative importance evaluation mechanism that aggregates importance scores over layers, mitigating the insufficiency of the current criterion through continuity across layers. Our designs ensure that pruning decisions align with the inherent dynamic information flow of tokens, preserving critical information while effectively eliminating redundancy (Fig. 1 left).

In summary, our contributions can be listed as

We provide a unified, bottom-up perspective ( information flow ) by modeling token-to-token interaction to explore the propagation pattern of visual information and understand the emergence of redundant, revealing redundancy as a natural consequence of asymmetric attention.

Based on a comprehensive analysis of the information flow, we verify both the irrationality and the insufficiency of detecting redundant tokens based on a single layer or a single criterion.

Based on the analysis, we propose FlowCut, an information-flow-aware pruning framework that better aligns with the model's inherent flow of information, mitigating the insufficiency of the current criterion of identifying redundant tokens.

Extensive experiments show FlowCut achieves superior results: outperforming SoTA by 1.6% on LLaVA-1.5-7B (96.0% vs 94.4%) with 88.9% token reduction, and by 4.3% on LLaVA-NeXT-7B (91.9% vs 87.6%) with 94.4% reduction while delivering 3.2 × speed-up in prefilling stage.

Layer 8

Token 1

Layer 22

Token 301

outflow

Token 576

Layer 8

Layer 22

CLS Token

inflow

一

Layer 2

Layer 6

Layer 10

Layer 15

Layer 20

2 Rethinking Visual Redundancy from Information Flow

Information flow offers a unified, bottom-up perspective for understanding token interactions. In this section, we systematically delve into information flow to analyze the emergence of redundancy.

2.1 Preliminaries

Architecture of LVLMs. The architecture of LVLMs generally comprises three key components: 1) a pre-trained vision encoder, which transforms input images into visual tokens; 2) a modality connector, serving as a bridge between the vision encoder and the LLM by aligning visual tokens with the LLM's word embedding space; and 3) a pre-trained LLM, which integrates the aligned visual and textual information to perform reasoning and generate responses.

Visual Token Processing. LVLMs process visual tokens in three stages: vision encoding, prefilling, and decoding. Images are first divided into patches and encoded into visual tokens by a vision backbone. These tokens are then fused with textual tokens during the prefilling stage to form joint representations, while key-value pairs are cached for generation. In the decoding stage, new tokens are generated autoregressively using the cached information, enabling efficient multimodal inference.

2.2 Delve into Information Flow of Visual Tokens

i) Analysis of Information Flow

We begin by analyzing information inflow and outflow patterns across vision encoder layers. Specifically, for an attention map A ∈ R N × N , we define information inflow for each token as its primary source of information ( argmax( A [ i, :]) ) and outflow as the primary destination of its information ( argmax( A [: , j ]) ). Figure 2 (Left) reveals two distinct information flow patterns: CLS tokens consistently engage in global information exchange while providing information to patch tokens, whereas patch tokens exhibit short-range, local, uniform flows in shallow layers and long-range, global, deterministic flows in deeper layers.

To further understand information flow, we analyze the attention map of both the CLS token and patch tokens across layers [1, 65]. As shown in Figure 2 (Right, rows 1-3), each patch token consistently focuses on a small subset of tokens. While this focus is initially on the CLS token and its neighbors in shallow layers, in deep layers, each patch token uniformly directs substantial attention to a specific group of non-neighboring, non-CLS tokens, which we identify as 'hub tokens' due to

this concentrated attention. In contrast, the CLS token (Figure 2, Right, row 4) initially distributes its attention broadly across most tokens, engaging with a wide spectrum of tokens. As layers get deeper and the 'hub tokens' emerge as the focus for patch tokens, the CLS token's attention pattern also gradually converges towards these hub tokens.

Insight 1 Information flow offers a unified, bottom-up perspective for understanding token interactions, agnostic to model architectures. It reveals redundancy as a natural consequence of asymmetric attention: as layers deepen, attention concentrates on fewer tokens, and tokens that no one focuses on are naturally the redundant ones.

However, while conceptually powerful, modeling all token-to-token flows across all layers is costly and analytically intractable, necessitating a simplified yet representative proxy.

ii) The CLS Token Serves as the Relay that Represents the Overall Information Flow

As in Fig. 2 (Right), the attention patterns of both patch and CLS tokens gradually converge to the same hub token in deeper layers. This trend implies an information relay mechanism: for a given patch token, since shallow-layer patch tokens primarily attend only to their local neighbors and the CLS token, it has minimal direct access to information from distant patch tokens; therefore, the CLS token, as it can gather global information from most patch tokens, must have transmitted information of distant patches to the given patch token to direct this patch's attention in deep layers to these distant patch tokens, i.e., the CLS token serves as the relay to transmit information across patch tokens. Moreover, as patch tokens derive information from the CLS token, their attention gradually mirrors its preferences, therefore, their attention converges to the same hub tokens.

Furthermore, we measure the average distance between each token and the token that it attends the most in Fig. 3 (Left), where the attention distance exhibits an overall increasing trend as the layer deepens. This further verifies that patch tokens initially gather information primarily from nearby neighbors and rely on the CLS token for capturing distant semantic context.

Insight 2 The CLS token acts as a global information relay, broadcasting distant-patch information to each patch token. As layers deepen, patch tokens increasingly mirror the CLS token's focus, converging toward the same set of hub tokens, suggesting we can take the CLS token as a proxy to simplify the complex information flow and identify critical tokens.

12.5

6.5

-attentiondistance

attentionentropy

11.5

6.0

10.5

5.5

年

9.5

5.0

8.5

4.5

7.5

4.0

6.5

3.5

1

3

5

7

9

11

13

15

17

19

21

23

1

3

5

7

6

11

13

15

17

19

21

23

iii) Progressive Attention Concentration and Emergence of Redundancy

Our above analysis reveals that the CLS token's behavior can largely represent the overall information flow. Therefore, we visualize the overall information flow tendency by measuring the CLS token's attention across the vision encoder layers. Fig. 3 (Right) demonstrates that attention pattern distributions dynamically evolve across layers, and as information traverses deeper layers, attention generally narrows, progressively concentrating on a smaller group of tokens. This increasing concentration of attended tokens inherently implies that other tokens become less critical to the information flow. We interpret this diminishing attention as the progressive emergence of semantic redundancy. Thus, redundancy appears not as a static feature but as an emerging property, developing through complex, layer-by-layer interactions during the encoding process.

To quantify the trend of attention concentration, we measured the entropy of the attention distribution at each layer. As in Fig. 3 (Middle), the attention entropy exhibits an overall decreasing trend.

Notably, a sharp decline in entropy is observed between layers 11 and 15, signifying a period of rapid attention focusing. This result quantitatively supports the progressive concentration of attention, verifying that redundancy naturally emerges as attention becomes more concentrated.

Insight 3 Redundancy emerges progressively as attention concentrates over depth, not abruptly at the end but layer by layer, driven by increasingly focused semantic interactions. This layerspecific development of redundancy suggests that optimal pruning should be dynamic and layer-adaptive rather than applying fixed ratios across layers or pruning only at the final stages.

1.0

1.0

0T

0.8

8'0

0.8

0.6

0.6

0.6

0.4

0.4

0.4

0.2

0.2

0.2

Layer 2 Attention Score

Layer 2 Value L1 Norm

Layer 2 Semantic Similarity

Layer 23 Attention Score

0.0

Layer 23 Value L1 Norm

0.0

Layer 23 Semantic Similarity

0.0

iv) Beyond High Attention: Identifying Truly Important Tokens

Given the complex interactions between tokens and layers modeled by information flow, a critical question naturally arises: Is the simple single-layer criterion, as has been adopted by current works, sufficient to capture such a complex generation of redundant tokens? To investigate this issue, we analyze several metrics at different layers of the vision encoder (selected for feeding into LLM). Alongside attention scores, we measured: 1) the L1 norm of the Value vectors, which serves as an indicator of their information density [16, 22], where a small norm suggests a weak signal and thus less information conveyed by Value vectors; and 2) the semantic similarity between each token and the CLS token. Given that the CLS token aggregates global information, this similarity score reflects the token's semantic alignment with, or relevance to, the overall global context [34].

The results presented in Fig. 4 demonstrate that some tokens highly attended to by the CLS token can exhibit remarkably low information density (as measured by the Value L1 norm) and low semantic relevance to the global context (as measured by semantic similarity to the CLS token). This means that applying different criteria, such as the high attention scores of the CLS token, in identifying critical tokens can lead to contradictory results, even though all these criteria are reasonable. This problem may be the result of the noisy information in the information flow. Since the flow is transmitted through every two tokens and every connected layers, every small perturbation (noise) in the token/attention could be amplified to lead to wrong identification of critical tokens.

Insight 4 High attention score from a single layer is not a bad criterion, but is still insufficient in measuring token importance due to the complexity in token interactions. An ideal criterion should take networks' inherent behavior into account and resist the instability of current criteria.

3 Methodology

In this section, based on the above analysis, we introduce our method for visual token reduction, which better fits the model's inherent information flow, including flow distribution-aware prune ratio, cumulative flow importance tracking, and multi-criteria evaluator. The overall framework is in Fig. 5.

3.1 Attention Distribution-Aware Prune Ratio

Redundancy emerges progressively as attention becomes more concentrated. Based on this insight, since the CLS token acts as the relay of information flow, we adopt the CLS token's attention entropy to adaptively adjust pruning ratios: higher entropy indicates widespread token contribution to information flow, necessitating conservative pruning, while lower entropy reflects concentrated attention and emerging redundancy, permitting more aggressive pruning. Specifically, the attention score with h heads A ∈ R h × N × N is computed as:

PruneEachTwoLayer

00000

attnentropy

3.7

3.5

3.6

阝

4.2

system

visualtokens

instruction

1

blockn

history

current

+

5.9

5.3

91

65

77

67

102

tokens

tokens

,

6.1

5.9

5.7

40

6.2

blockn-1

1717

23

5

8

current

Embedding

Connector

history

+

13579111315171921

Adaptivecomputeprunenumby

CLS token'sattention entropy

Tokenizer

Vision

，

阝

Encoder

current

Multi-Criteria Evaluator

block1

+

history

<sysmessage>

What is the

man doing in

this picture?

attentionscore

semanticscore

informationscore

[ll., calculate cumulative scores

adaptive computeprune num&prune visual tokens

where Q and K denote query and key matrices, D k is the key dimension. By averaging A across all heads, we obtain an aggregated attention matrix A avg ∈ R N × N , here N is the sequence length.

For vision encoders with a CLS token, we extract the CLS token's attention map as global attention vector A g . For those without a CLS token, we first average all tokens to create a global token, then compute attention scores between this global token and all sequence tokens to derive A g . The global attention vector A g ∈ R 1 × N is then used to compute the attention entropy H ( A g ) as follow:

where, H max denotes the maximum possible entropy when attention is uniformly distributed. The final visual tokens' prune number P is computed as:

where N is the current visual token number, T is the target number, L is the remain prune layers, and P will be rounded to the nearest integer by round( P ) .

3.2 Multi-Criteria Evaluator

Existing methods typically rely on a single criterion, such as the attention score of the CLS token or the last instruction token, which is verified by us to be insufficient to accurately capture the role of each token in information flow. To address this limitation, we propose a multi-criteria evaluation framework that incorporates attention strength, semantic similarity, and information density.

Specifically, for attention strength I a , we use A g as the metric. For semantic similarity I s , we establish a global value vector V g by using either the CLS token's value vector (for encoders with a CLS token) or the average of all tokens' value vectors (for encoders without a CLS token), and then calculate the similarity between V g and patch token's value vector. For information density I d , we measure it by computing the L1 norm of patch token's value vector. The formula as follow:

where D v is the dimension of V . The final importance score S is caculate as:

Method

GQA MMB MMB

CN

MME POPE

SQA VQA

V2

VQA

Text

Avg

.

(100%)

Vanilla

61.9

64.7

58.1

1862

85.9

69.5

78.5

58.2

50.0

59.3

31.1

66.9

100%

Retain 192 Tokens

(

↓

66

.

7

%)

ToMe

(ICLR23)

54.3

60.5

-

1563

72.4

65.2

68.0

52.1

-

53.1

27.9

-

88.7%

FastV

(ECCV24)

52.7

61.2

57.0

1612

64.8

67.3

67.1

52.5

50.8

57.1

27.7

49.4

89.4%

(ICML25)

57.6

62.5

53.7

1721

83.6

68.9

75.6

56.1

50.5

55.8

31.5

66.1

96.6%

PDrop

(CVPR25)

57.3

62.9

56.8

1766

82.3

68.8

75.1

56.5

51.1

54.7

30.5

65.8

96.7%

VisionZip

(CVPR25)

59.3

63.0

57.3

1778

85.2

68.7

76.6

57.3

51.2

56.4

31.7

67.7

98.5%

FlowCut (Ours)

60.1

63.2

57.8

1836

86.1

68.6

77.1

57.5

51.8

57.5

34.3

68.4

100.2%

Retain 128 Tokens

(

↓

77

.

8

%)

ToMe

(ICLR23)

52.4

53.3

-

1343

62.8

59.6

63.0

49.1

-

50.9

27.2

-

81.8%

FastV

(ECCV24)

49.6

56.1

56.4

1490

59.6

60.2

61.8

50.6

51.3

55.9

28.1

52.0

85.9%

(ICML25)

56.0

60.0

51.1

1696

80.5

67.1

73.8

54.9

51.4

53.4

30.0

62.7

93.8%

PDrop

(CVPR25)

57.1

61.6

56.3

1664

82.3

68.3

72.9

56.6

51.0

53.3

30.8

61.9

95.1%

VisionZip

(CVPR25)

57.6

62.0

56.2

1759

83.2

68.9

75.6

56.8

51.6

54.9

32.1

64.8

97.2%

FlowCut (Ours

)

58.5

62.1

56.5

1792

85.2

68.6

76.0

57.3

52.2

56.2

32.5

67.5

98.5%

Retain 64 Tokens

(

↓

88

.

9

%)

ToMe

(ICLR23)

48.6

43.7

-

1138

52.5

50.0

57.1

45.3

-

44.0

24.1

-

71.4%

FastV

(ECCV24)

46.1

48.0

52.7

1256

48.0

51.1

55.0

47.8

50.8

51.9

25.8

46.1

77.5%

(ICML25)

52.7

56.2

46.1

1505

75.1

62.2

68.2

51.8

50.1

51.1

23.3

57.5

84.6%

PDrop

(CVPR25)

47.5

58.8

50.5

1561

55.9

68.6

69.2

50.6

50.7

40.0

30.7

59.2

86.1%

VisionZip

(CVPR25)

55.1

60.1

55.3

1687

77.0

69.0

72.4

55.5

52.8

52.2

31.5

62.9

94.4%

FlowCut (Ours

)

55.6

60.8

55.4

1744

80.2

69.1

72.8

55.6

53.2

53.5

32.3

65.1

96.0%

3.3 Cumulative Flow Importance Tracking

Based on the observation that attention distribution dynamically and continually shift across layers, and the single-layer criterion is insufficient to capture a token's importance, we track cumulative importance by aggregating both historical and current criteria across layers, enabling a more accurate and stable assessment of each token's overall contribution. Specifically, we gain multi-criteria score S cur each layer and update cumulative score S cum using both the current and historical values:

We prune every two layers, relying on the cumulative score to ensure that the importance value takes into account both current and historical values, which implicitly filters out noise in flows.

4 Experiments

Experiment Setting. To demonstrate FlowCut's effectiveness, we conducted extensive experiments on diverse open-source LVLMs. For image understanding tasks, we implemented our designs on the widely-used LLaVA family-specifically LLaVA-1.5-7B [39] and the high-resolution LLaVA-Next7B [40](supporting up to 2880 image tokens)-evaluating across twelve benchmarks with varying reduction ratios. To further validate the universality of our method, we extended evaluation to the more advanced Qwen2-VL [55] model. For video understanding tasks, we employed Video-LLaVA [36] as a baseline and verified our method across three video benchmarks.

Implement Details Our method can be seamlessly integrated into models in a training-free manner, or training from scratch, resulting in substantial improvements to inference speed and training efficiency. For LLaVA-1.5, we complete the pruning at the penultimate layer of the vision encoder. For LLaVANeXT and Qwen2-VL, which contain a larger number of tokens, our pruning process spans from the vision encoder to the second layer of the LLM, which we prune tokens to twice the target number in the vision encoder, then further reduce to the final target number at the second layer of the LLM. All of our experiments are conducted on Nvidia A800-80G GPU.

4.1 Effectiveness of FlowCut in Inference

Image understanding tasks We apply FlowCut to LLaVA-1.5-7B during inference without additional training and compare it with other approaches. The results in Table 1 demonstrate that FlowCut achieves optimal results on almost all benchmarks, with its average performance significantly exceeding all existing methods. we conduct further experiments on LLaVA-NeXT7B. As shown in Table 2, FlowCut outperforming other methods by a significant margin. We even surpassing VisionZip by 4.3% while retaining only 160 tokens. Furthermore, as shown in Table 3, we extended the evaluation to Qwen2VLto validate that FlowCut can be applied seamlessly to different architectures.

Table 4: Performance of FlowCut on video understanding tasks. The original Video-LLaVA's video token number is 2048, while we only retain the 256 tokens.

Video understanding tasks To assess our method's effectiveness in video understanding, we applied it to Video-LLaVA, which encodes each video into 8 frames with 256 visual tokens per frame (totaling 2048 visual tokens). We compared FlowCut to other methods under the setting of retaining only 256 tokens in total (32 per frame). As shown in Table 4, FlowCut achieves 98.6% accuracy across three benchmarks, outperforming the previous state-of-the-art VisionZip by 4.2%. This notable gain highlights it's superior reasoning capability in handling complex multimodal video data.

Efficiency Analysis Our proposed FlowCut substantially improves inference efficiency in LVLMs while maintaining comparable performance. As shown in Table 5, we compare total inference time, prefilling time, and FLOPs on both LLaVA-1.5-7B and LLaVA-NeXT-7B against several existing methods. FlowCut consistently achieves the best efficiency gains. Notably, it delivers a 3.2 × speedup in prefilling and a 3.0 × speedup in the entire inference on LLaV A-NeXT.

4.2 Effectiveness of FlowCut in Training

Our method can also be seamlessly integrated into models during training to reduce the number of visual tokens, thereby saving memory and shortening training time. As shown in Table 6, we evaluate FlowCut with different token configurations across 10 benchmarks on LLaVA-1.5-7B. The results demonstrate that with 192 tokens, FlowCut achieves a 1.5 × training speedup while actually improving performance to 101.1% of the original model. Even with further reduction to 128 tokens, training speedup reaches 1.6 × while still retaining 99.8% of the original performance.

4.3 Ablation Studies

Effectiveness of each desigh To assess the contribution of each design in FlowCut, we conduct ablation studies on three benchmarks under a setting that retains only 128 visual tokens ( ↓ 77.8%), as shown in Table 7. Naively pruning at a single layer leads to significant performance drops. Introducing either the cumulative importance evaluation or the multi-criteria scoring strategy individually brings consistent gains across benchmarks, confirming their effectiveness. Moving from single-layer to multi-layer pruning yields notable improvements, yet applying a uniform prune ratio across layers remains suboptimal. Enabling adaptive prune ratio notably enhances performance (e.g., +1.8% on POPE). The best results are achieved when all components are integrated in FlowCut, yielding minimal performance degradation compared to the unpruned baseline. These results verify that each component contributes independently and synergistically to performance.

Adaptive layer-wise pruning outperforms single-layer pruning As shown in Figure 6, we measure information density across layers by the L1 norm of value vectors, where lower norms indicate sparser and less informative representations [22]. In single-layer pruning, the information density drops sharply after pruning, indicating substantial information loss. In contrast, FlowCut aligns pruning decisions with the token's dynamic inherent information flow, effectively preserving essential content.

4.4 Analysis and Discussion

FlowCut effectively preserves critical information. According to the attention computation, larger value vectors (V) contribute more to the output, indicating richer information content [17]. We use

0.85

-LLaVA-1.5

0.75

Prune atLayer23

FlowCut(Ours)

0.65

InformationDensityPlummets

0.55

0.45

0.35

0.25

0.15

0.20

0.15

0.10

0.05

口

0.00

2

8

6

1011121314

15

1718192021222324

101112131415

17

2021

23

24

box plots (Figure 7) to compare the distribution of value vectors before and after applying FlowCut. After pruning, the distribution shifts toward higher magnitudes, suggesting that although fewer tokens remain, each retains more information, effectively preserving overall content richness.

FlowCut guides model focus more effectively. Figure 8 compares the pruning results and VQA answers of FlowCut and the previous SoTA. VisionZip relies on single-layer attention scores, resulting in most tokens retained after pruning being concentrated in the center of the image. In contrast, FlowCut leverages multi-layer and multi-criteria evaluations, leading to a more uniform spatial distribution of retained tokens. Moreover, by effectively eliminating redundant and distracting tokens, FlowCut produces more accurate answers than even the non-pruned baseline in some cases.

5 Related Work

Visual Token Compression In LVLMs, visual tokens typically contain substantial redundant information. Various methods [5, 32, 55] have been proposed to address this inefficiency. FastV [6] leverages attention scores at the LLM's second layer for pruning, while SparseVLM [62] introduces text-guided pruning through cross-modal attention. VisionZip [59] employs CLS token attention at the vision encoder's final layer for compression, and PDrop [57] uses the last instruction token's attention map to drop visual tokens at several pre-selected layers according to manually designed fixed pruning ratios. However, these methods evaluate token importance using attention scores from a single layer, limiting their comprehensiveness. Our method, instead, accu-

Describe the colors of the outfit,

helmet,and snowboard in this image.

User

The outfit is redwith black and white

accents. The helmet is predominantly

blue with some decals or designs. The

GPT40

snowboard is mostly black.

The outfit is red，the helmet is black,

and the snowboard is white.

LLaVA

Correct

Incorrect

Partially Correct

retain 64 visual tokens

VisionZip

FlowCut

The outfit is red andwhite,the helmet

The outfit is red and white,the helmet

is black，and the snowboard is white.

is blue，and the snowboard is black.

mulates diverse criteria across multiple layers and performs progressive token reduction with adaptive pruning ratios, achieving efficient inference while preserving performance. (More in appendix A.2)

6 Conclusion

In this work, we provide a novel perspective (information flow) for understanding the emergence of redundant tokens, revealing redundancy as a natural consequence of asymmetric attention. Through comprehensive analysis, we demonstrate the limitations of single-layer, single-criterion metrics in identifying redundancy, and propose FlowCut, an information-flow-aware pruning framework that aligns pruning decisions with the inherent dynamic information flow of tokens. Extensive experiments show that FlowCut achieves efficient inference while preserving performance.

Acknowledgments

This work is supported by the National Natural Science Foundation of China under grants 62206102; the National Key Research and Development Program of China under grant 2024YFC3307900; the

National Natural Science Foundation of China under grants 62436003, 62376103 and 62302184; Major Science and Technology Project of Hubei Province under grant 2025BAB011 and 2024BAA008; Hubei Science and Technology Talent Service Project under grant 2024DJC078; and Ant Group through CCF-Ant Research Fund. The computation is completed in the HPC Platform of Huazhong University of Science and Technology.

References

Samira Abnar and Willem Zuidema. Quantifying attention flow in transformers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , pages 4190-4197, 2020.

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 , 2023.

Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.16609 , 2023.

Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966 , 1(2):3, 2023.

Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token merging: Your vit but faster. In ICLR , 2023.

Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang. An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models. In European Conference on Computer Vision , pages 19-35. Springer, 2024.

Yi Chen, Jian Xu, Xu-Yao Zhang, Wen-Zhuo Liu, Yang-Yang Liu, and Cheng-Lin Liu. Recoverable compression: A multimodal vision token recovery mechanism guided by text information. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 39, pages 2293-2301, 2025.

Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 24185-24198, 2024.

Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, march 2023. URL https://lmsys. org/blog/2023-03-30-vicuna , 3(5), 2023.

Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. arXiv preprint arXiv:2307.08691 , 2023.

Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances in neural information processing systems , 35:16344-16359, 2022.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 , 2020.

Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, et al. MME: A comprehensive evaluation benchmark for multimodal large language models. arXiv:2306.13394 , 2023.

Suyu Ge, Yunan Zhang, Liyuan Liu, Minjia Zhang, Jiawei Han, and Jianfeng Gao. Model tells you what to discard: Adaptive kv cache compression for llms. arXiv preprint arXiv:2310.01801 , 2023.

Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 6904-6913, 2017.

Zhiyu Guo, Hidetaka Kamigaito, and Taro Watanabe. Attention score is not all you need for token importance indicator in kv cache reduction: Value also matters. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , pages 21158-21166, 2024.

Ankit Gupta and Jonathan Berant. Value-aware approximate attention. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , pages 9567-9574, 2021.

Danna Gurari, Qing Li, Abigale J Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 3608-3617, 2018.

Chi Han, Qifan Wang, Wenhan Xiong, Yu Chen, Heng Ji, and Sinong Wang. Lm-infinite: Simple on-the-fly length generalization for large language models. 2023.

Drew A Hudson and Christopher D Manning. GQA: A new dataset for real-world visual reasoning and compositional question answering. Conference on Computer Vision and Pattern Recognition (CVPR) , 2019.

Yunseok Jang, Yale Song, Youngjae Yu, Youngjin Kim, and Gunhee Kim. Tgif-qa: Toward spatio-temporal reasoning in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 2758-2766, 2017.

Goro Kobayashi, Tatsuki Kuribayashi, Sho Yokoi, and Kentaro Inui. Attention is not only a weight: Analyzing transformers with vector norms. In 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020 , pages 7057-7075. Association for Computational Linguistics (ACL), 2020.

Dan Kondratyuk, Lijun Yu, Xiuye Gu, José Lezama, Jonathan Huang, Grant Schindler, Rachel Hornung, Vighnesh Birodkar, Jimmy Yan, Ming Chang Chiu, et al. Videopoet: A large language model for zero-shot video generation. Proceedings of Machine Learning Research , 235:25105-25124, 2024.

Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles , pages 611-626, 2023.

Bohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang, Rui Wang, Ruimao Zhang, and Ying Shan. Seed-bench: Benchmarking multimodal large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 13299-13308, 2024.

Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning , pages 19730-19742. PMLR, 2023.

Shiwei Li, Huifeng Guo, Lu Hou, Wei Zhang, Xing Tang, Ruiming Tang, Rui Zhang, and Ruixuan Li. Adaptive low-precision training for embeddings in click-through rate prediction. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 37, pages 4435-4443, 2023.

Shiwei Li, Xiandi Luo, Xing Tang, Haozhao Wang, Hao Chen, Weihong Luo, Yuhua Li, Xiuqiang He, and Ruixuan Li. Beyond zero initialization: Investigating the impact of non-zero initialization on lora fine-tuning dynamics. In Forty-second International Conference on Machine Learning , 2025.

Shiwei Li, Xiandi Luo, Haozhao Wang, Xing Tang, Ziqiang Cui, Dugang Liu, Yuhua Li, Xiuqiang He, and Ruixuan Li. Beyond higher rank: Token-wise input-output projections for efficient low-rank adaptation. In Advances in Neural Information Processing Systems , 2025.

Shiwei Li, Xiandi Luo, Haozhao Wang, Xing Tang, Ziqiang Cui, Dugang Liu, Yuhua Li, Xiuqiang He, and Ruixuan Li. Bora: Towards more expressive low-rank adaptation with block diversity. arXiv preprint arXiv:2508.06953 , 2025.

Shiwei Li, Wenchao Xu, Haozhao Wang, Xing Tang, Yining Qi, Shijie Xu, Weihong Luo, Yuhua Li, Xiuqiang He, and Ruixuan Li. Fedbat: communication-efficient federated learning via learnable binarization. In Proceedings of the 41st International Conference on Machine Learning , pages 29074-29095, 2024.

Yanwei Li, Chengyao Wang, and Jiaya Jia. Llama-vid: An image is worth 2 tokens in large language models. In European Conference on Computer Vision , pages 323-340. Springer, 2024.

Yanwei Li, Yuechen Zhang, Chengyao Wang, Zhisheng Zhong, Yixin Chen, Ruihang Chu, Shaoteng Liu, and Jiaya Jia. Mini-gemini: Mining the potential of multi-modality vision language models. arXiv:2403.18814 , 2024.

Yi Li, Hualiang Wang, Yiqun Duan, Jiheng Zhang, and Xiaomeng Li. A closer look at the explainability of contrastive language-image pre-training. Pattern Recognition , page 111409, 2025.

Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. arXiv preprint arXiv:2305.10355 , 2023.

Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , pages 5971-5984, 2024.

Zhihang Lin, Mingbao Lin, Luxi Lin, and Rongrong Ji. Boosting multimodal large language models with visual tokens withdrawal for rapid inference. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 39, pages 5334-5342, 2025.

Hao Liu, Matei Zaharia, and Pieter Abbeel. Ring attention with blockwise transformers for near-infinite context. arXiv preprint arXiv:2310.01889 , 2023.

Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 26296-26306, 2024.

Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024.

Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems , 36:34892-34916, 2023.

Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? In European conference on computer vision , pages 216-233. Springer, 2024.

Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems , 35:2507-2521, 2022.

Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. arXiv preprint arXiv:2306.05424 , 2023.

David Marr. Vision: A computational investigation into the human representation and processing of visual information . MIT press, 2010.

Yuzhang Shang, Mu Cai, Bingxin Xu, Yong Jae Lee, and Yan Yan. Llava-prumerge: Adaptive token reduction for efficient large multimodal models. arXiv preprint arXiv:2403.15388 , 2024.

Dachuan Shi, Chaofan Tao, Anyi Rao, Zhendong Yang, Chun Yuan, and Jiaqi Wang. Crossget: Crossguided ensemble of tokens for accelerating vision-language transformers. In International Conference on Machine Learning , pages 44960-44990. PMLR, 2024.

Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 8317-8326, 2019.

Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805 , 2023.

Jintao Tong, Ran Ma, Yixiong Zou, Guangyao Chen, Yuhua Li, and Ruixuan Li. Adapter naturally serves as decoupler for cross-domain few-shot semantic segmentation. arXiv preprint arXiv:2506.07376 , 2025.

Jintao Tong, Yixiong Zou, Guangyao Chen, Yuhua Li, and Ruixuan Li. Self-disentanglement and recomposition for cross-domain few-shot segmentation. arXiv preprint arXiv:2506.02677 , 2025.

Jintao Tong, Yixiong Zou, Yuhua Li, and Ruixuan Li. Lightweight frequency masker for cross-domain few-shot semantic segmentation. Advances in Neural Information Processing Systems , 37:96728-96749, 2024.

Peter Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Adithya Jairam Vedagiri IYER, Sai Charitha Akula, Shusheng Yang, Jihan Yang, Manoj Middepogu, Ziteng Wang, et al. Cambrian-1: A fully open, vision-centric exploration of multimodal llms. Advances in Neural Information Processing Systems , 37:87310-87356, 2024.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 , 2023.

Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. arXiv preprint arXiv:2409.12191 , 2024.

Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks, 2024. URL https://arxiv. org/abs/2309.17453 , page 1, 2024.

Long Xing, Qidong Huang, Xiaoyi Dong, Jiajie Lu, Pan Zhang, Yuhang Zang, Yuhang Cao, Conghui He, Jiaqi Wang, Feng Wu, et al. Pyramiddrop: Accelerating your large vision-language models via pyramid visual redundancy reduction. arXiv preprint arXiv:2410.17247 , 2024.

Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Yueting Zhuang. Video question answering via gradually refined attention over appearance and motion. In Proceedings of the 25th ACM international conference on Multimedia , pages 1645-1653, 2017.

Senqiao Yang, Yukang Chen, Zhuotao Tian, Chengyao Wang, Jingyao Li, Bei Yu, and Jiaya Jia. Visionzip: Longer is better but not necessary in vision language models. arXiv preprint arXiv:2412.04467 , 2024.

Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308.02490 , 2023.

Qizhe Zhang, Aosong Cheng, Ming Lu, Zhiyong Zhuo, Minqi Wang, Jiajun Cao, Shaobo Guo, Qi She, and Shanghang Zhang. [cls] attention is all you need for training-free visual token pruning: Make vlm inference faster. arXiv e-prints , pages arXiv-2412, 2024.

Yuan Zhang, Chun-Kai Fan, Junpeng Ma, Wenzhao Zheng, Tao Huang, Kuan Cheng, Denis Gudovskiy, Tomoyuki Okuno, Yohei Nakata, Kurt Keutzer, et al. Sparsevlm: Visual token sparsification for efficient vision-language model inference. arXiv preprint arXiv:2410.04417 , 2024.

Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, et al. H2o: Heavy-hitter oracle for efficient generative inference of large language models. Advances in Neural Information Processing Systems , 36:34661-34710, 2023.

Jiedong Zhuang, Lu Lu, Ming Dai, Rui Hu, Jian Chen, Qiang Liu, and Haoji Hu. St3: Accelerating multimodal large language model by spatial-temporal visual token trimming. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 39, pages 11049-11057, 2025.

Yixiong Zou, Shuai Yi, Yuhua Li, and Ruixuan Li. A closer look at the cls token for cross-domain few-shot learning. Advances in Neural Information Processing Systems , 37:85523-85545, 2024.

Yixiong Zou, Shanghang Zhang, Jianpeng Yu, Yonghong Tian, and José MF Moura. Revisiting mid-level patterns for cross-domain few-shot recognition. In Proceedings of the 29th ACM International Conference on Multimedia , pages 741-749, 2021.

A Technical Appendices and Supplementary Material

A.1 Dataset

We evaluated our method across fifteen benchmarks: twelve for image understanding tasks and three for video understanding tasks, each assessing different aspects of multimodal intelligence.

GQA [20] The GQA benchmark is structured around three parts: scene graphs, questions, and images. It enriches visual content with comprehensive spatial information and object-level attributes. Questions are specifically designed to evaluate models' ability to comprehend visual scenes and reason about diverse aspects of images.

MMBench [42] The MMBench Benchmark evaluates models through three hierarchical levels of abilities. The first level (L-1) focuses on fundamental perception and reasoning capabilities. The second level (L-2) expands into six distinct sub-abilities, while the third level (L-3) further refines these into 20 specific dimensions. This structure enables a granular and comprehensive assessment of a model's various capabilities. Additionally, MMB-CN is the Chinese version of the benchmark.

MME [13] MME comprehensively evaluates models' perceptual and cognitive abilities across 14 subtasks. By employing manually constructed instruction-answer pairs and concise instructions, it effectively minimizes data leakage and ensures fair assessment of model performance.

POPE [35] The POPE benchmark systematically evaluates object hallucination in models through a series of binary questions about object presence in images. Using accuracy, recall, precision, and F1 score as metrics, it precisely measures hallucination levels across different sampling strategies.

ScienceQA [43] ScienceQA encompasses a wide array of domains, including natural, language, and social sciences, with questions hierarchically organized into 26 topics, 127 categories, and 379 skills. Through this comprehensive structure, it offers diverse scientific questions that effectively evaluate multimodal understanding, multi-step reasoning capabilities, and interpretability.

VQA-V2 [15] VQA-V2 evaluates models' visual perception capabilities through open-ended questions about 265,016 real-world scence images. Each question contains 10 human-annotated ground truth answers, enabling thorough assessment of a model's ability to interpret and respond to visual queries.

TextVQA [48] TextVQA focuses on the integration of textual information within images. It evaluates a model's ability to interpret visual elements and embedded text in images through tasks, requiring both visual and textual comprehension to answer questions accurately.

VizWiz [18] The VizWiz benchmark contains over 31,000 visual questions created by blind individuals who photographed scenes with mobile phones while recording spoken questions. Each question includes 10 crowdsourced answers. The dataset presents distinct challenges: lower-quality images from visually impaired photographers, conversational spoken questions, and some questions that remain unanswerable due to content limitations.

SEEDBench [25] SEEDBench contains 19,000 human-annotated multiple-choice questions evaluating models across 12 distinct aspects. It comprehensively assesses capabilities in recognizing spatial and temporal patterns within both images and videos.

MMVet [60] MMVet defines six core abilities (recognition, OCR, knowledge, language generation, spatial awareness, and math) and evaluates models through 16 specific capability combinations. These integrations enable comprehensive assessment of models' performance across complex scenarios.

LLaVA-Bench [39] LLaVA-Bench categorizes questions into three types: conversational (simple QA), detailed description, and complex reasoning tasks, featuring 24 diverse images with 60 questions spanning indoor and outdoor scenes, memes, paintings, sketches, and more. Each image contains a manually curated detailed description and carefully selected questions, designed to assess model robustness across various prompts.

TGIF-QA [21] TGIF-QA extends the image question answering task to videos with 165,000 questionanswer pairs based on GIFs. It introduces four task types: three requiring spatio-temporal reasoning (repetition count, repeating action, state transition) and frame QA answerable from single frames.

MSVD-QA [58] The MSVD-QA benchmark, based on the MSVD dataset, features 1,970 video clips with approximately 50.5K question-answer pairs. It presents open-ended questions across five

categories (what, who, how, when, where) covering diverse video content aspects, serving both video question answering and captioning tasks.

MSRVTT-QA [58] MSRVTT-QA comprises 10K video clips with 243K question-answer pairs, challenging models to integrate visual and temporal elements of video content. Similar to MSVD-QA, it includes five question types, evaluating models' ability to comprehend complex video content.

A.2 Related Work

Large Vision-Language Models Recently, building on the success of large language models (LLMs) [2, 3, 9, 54], LVLMs [4, 8, 39, 41, 49, 53] have demonstrated impressive performance in multimodal reasoning by bridging visual and linguistic modalities. However, a critical challenge persists in processing images: the quadratic growth of visual tokens as image resolution increases, which leads to prohibitive computational costs during training and inference. High-resolution image models like LLaVA-NeXT [40] and mini-Gemini-HD [33] process thousands of tokens per image, while video models such as VideoLLaVA [36] and VideoPoet [23] demand even more tokens for multiple frames. This highlights the need for more efficient methods [28, 30, 29, 50] to extract information from visual tokens, instead of merely extending their length.

Efficient Large Language Models Efficient inference [11, 10, 24, 38] in LLMs is challenged by their autoregressive nature, where each token prediction depends on the full preceding context. Recently, a variety of token reduction strategies have been proposed to accelerate inference and compress the key-value (KV) cache [19]. For instance, StreamingLLM [56] retains only attention sinks and most recent tokens to shrink the KV cache, while FastGen [14] adaptively manages KV cache based on the behavior of attention head. Heavy-Hitter Oracle (H2O) [63] employs accumulated attention-based scoring to prune key-value pairs during the generation stage. These methods primarily target textual redundancy to improve inference efficiency [31, 27].

A.3 Evaluation under Additional Settings and Comparison with More Methods

Retain less visual tokens To further validate our method under highly constrained token budgets, we conduct experiments with only 32 visual tokens. As shown in Table 8, our method outperforms existing methods by a notable margin under aggressive token reduction, validating its effectiveness in preserving informative content despite severe compression.

Compare with more methods We have already compared with several representative methods in the main text. Here, we further include more prior works for comparison: LLaVA-PruMerge [46] clusters the visual tokens via k-nearest merging algorithm and merges them. ST 3 [64] prunes lowimportance tokens progressively across layers using token attention maps. VTW [37] removes unimportant visual tokens based on attention score at specific layers of the LLM. RCom [7] merges unimportant patch tokens based on their similarity to the CLS token and the text tokens. FasterVLM [61] prunes vision tokens based on CLS attention scores at vision encoder's output layer.

These methods all rely on single-layer, single-metric scores (e.g., attention or similarity) to assess token importance. In contrast, our method employs multiple metrics and aggregates scores across layers using an EMD-based strategy, which helps mitigate single-metric bias and layer bias [66, 51]. We also discuss our differences with CrossGET [47] here. CrossGET injects learnable cross tokens to capture cross-modal information and merges less important tokens based on cosine similarity. In contrast, our method leverages CLS attention, performs multi-layer and multi-metric evaluation, and applies dynamic layer-wise pruning. Moreover, CrossGET requires re-training due to its learnable cross tokens while our approach is entirely training-free.

As shown in Table 9, our method better preserves performance under similar FLOPs, achieving consistently stronger results across a wide range of benchmarks. Note that for ST 3 , as the code is not publicly available, we follow their benchmark protocol for fair comparison.

A.4 More Sensitivity Analyses of Hyperparameters

Cumulative importance mechanism's weighting coefficients We default to a setting of 0.5 for historical scores and 0.5 for current scores. As shown in Table 10, we further conduct a sensitivity analysis. When only the current score is used (i.e., without the cumulative strategy), the POPE performance is 83.8. The best performance is achieved at 0.5:0.5 and 0.6:0.4, while all other settings still outperform the non-cumulative baseline. This demonstrates that our method is robust to changes in the weighting coefficients and validates the effectiveness of the cumulative strategy.

Pruning frequency For pruning frequency, we default to pruning every two layers with dynamic prune ratios. As shown in Table 11: 1) Pruning every two layers yields the best performance. Performance decreases as the interval increases, and pruning every layer (n=1) performs slightly worse than n=2, which we attribute to the fact that the first pruning step under n=1 cannot leverage historical scores and must rely solely on the current layer. 2) Multi-layer pruning outperforms single-layer pruning (i.e., only pruning at the last layer), further demonstrating both robustness to this hyperparameter and the benefit of pruning redundancies as they emerge. 3) These results confirm that our method is not sensitive to the prune frequency, highlighting its robustness.

A.5 Pseudocode

Here, we provide the pseudocode of our method. And note that method is indeed compatible with FlashAttention because it does not require the full N × N attention map. Instead, it only needs the 1 × N attention value of a single CLS token or global token. When using FlashAttention, we can obtain the necessary attention values with a small, separate computation: 1) For models with a CLS token, we separately compute its 1 × N attention map with the patch tokens; 2) For models without a CLS token, we derive a global token (by averaging patch tokens) and then compute its 1 × N attention map. This targeted computation is highly efficient, with a complexity of just O ( n ) , adding virtually no overhead. This allows our method and FlashAttention to be used together effectively.

Algorithm 1 Pseudocode for FlowCut

A.6 Broader Impact

Our research rethink the essence of visual token reduction from a fundamental information flow perspective. Through systemical analysis of information flow patterns, we reveal that redundancy as a natural consequence of asymmetric attention. Through comprehensive analysis, we demonstrate the limitations of single-layer, single-criterion metrics in identifying redundancy, and propose FlowCut, an information-flow-aware pruning framework that aligns pruning decisions with the inherent dynamic information flow of tokens. In some cases, we observe that token pruning, by removing redundant and distracting tokens, produces more accurate answers than the non-pruned baseline. This suggests that token pruning may hold promise for mitigating hallucinations in multimodal models. Therefore, it is worth exploring in future work why token pruning helps reduce hallucinations, and how we can better leverage efficient techniques-such as token pruning and token merging-not only to accelerate inference but also to suppress hallucinations.

A.7 More Visualization Results

Layer 1

Layer 2

Layer 3

Layer 4

Layer 5

Layer 6

Layer 7

Layer 8

Layer 9

Layer 10

Layer 11

Layer 12

Layer 13

Layer 14

Layer 15

Layer 16

Layer 17

Layer 18

Layer 19

Layer 20

Layer 21

Layer 22

Layer 23

Layer 24

Layer 1

Layer 4

Layer 5

Layer 6

Layer 7

Layer 8

Layer 9

Layer 10

Layer 12

Layer 13

Layer 14

Layer 15

Layer 16

Layer 17

Layer 18

Layer 19

Layer 20

Layer 21

Layer 22

Layer 23

Layer 24

original image

192 tokens

128 tokens

64 tokens

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Our main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss the limitations of the work in the conclusion.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: We do not include theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We have reported the detailed settings and hyper-parameters. We will release our codes.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We have reported the detailed settings and hyper-parameters. We will release our codes.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We have reported the detailed settings and hyper-parameters. We will release our codes.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA]

Justification: Experiments are conducted based on the open-source LVLMs, and we follow the same random seed to conduct experiments and compare with other methods without the error bar.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We have reported the computer resources in the implementation details.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We conform, in every respect, with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We have discussed the broader impact in the appendix.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our work does not have the risk for misuse.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We have respected the license of assets used in this paper.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: We have respected the license of assets used in this paper.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This paper does not include human subjects information.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper does not include human subjects information.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

16. Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.

Answer: [NA]

Justification: We only used LLM for writing.

Guidelines:

The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

Please refer to our LLM policy ( https://neurips.cc/Conferences/2025/LLM ) for what should or should not be described.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper provide a novel perspective (information flow) for understanding the emergence of redundant tokens, revealing redundancy as a natural consequence of asymmetric attention. Based on authors' insights,  they propose a combined criteria to measure the importance of tokens. The method is tested on multiple benchmarks, demonstrating robustness.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n1. The figures are visually good.\n2. The experimental evaluations are conducted on several MLLMs and video benchmarks, the results are promising.\n3. The method is clear and easy to follow.\n\nWeaknesses\n\n1. The author proposed a new concept called information flow, but it is essentially just the attention in the model. This concept actually diminishes the readability and comprehensibility of the paper. \n2. The insights are neither novel nor appealing, as similar observations have been made in many previous articles on Vision Transformers.\n3. The proposed method is just attention-based and not aligned well with insights, what is the relationship between it and information flow?\n4. Some benchmarks are missed, see questions.\n5. Literature review is not comprehensive enough, some prior works should be discussed in related work or compared in experiments:\n\n[1] LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Models. Arxiv24\n\n[2] ST3: Accelerating Multimodal Large Language Model by Spatial-Temporal Visual Token Trimming. AAAI25\n\n[3] [CLS] Attention is All You Need for Training-Free Visual Token Pruning: Make VLM Inference Faster. Arxiv24\n\n[4] Recoverable Compression: A Multimodal Vision Token Recovery Mechanism Guided by Text Information. AAAI25\n\n[5] Boosting Multimodal Large Language Models with Visual Tokens Withdrawal for Rapid Inference. AAAI25\n\nQUESTIONS:\n1.What is the overhead of the proposed work？And how do authors reproduce the prior works in Tab. 5, using flash attention or not?\n\n2. In Eq. 6, the authors combine historical and current criteria by a equal weight 0.5, what about others weights? Moreover, the token importance in last layer is used in the current layer, does it make sense?\n\n3. Token pruning is implemented in every two layers，why？How about three or more layers?\n\n4.What about the performance on image caption task like coco caption, nocaps and flickr30k?\n\nLIMITATIONS:\nThe authors argue that they discuss the limitations of the work in the conclusion, but in fact not. For limitations, please refer to weaknesses and questions.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper targets the inference inefficiency of large-scale vision–language models (LVLMs) caused by redundant visual tokens by proposing an information-flow-based pruning method. The authors dynamically adjust pruning strength at each transformer layer according to the CLS token’s attention entropy, score tokens using attention weight, semantic similarity to CLS, and Value-vector norm, and aggregate importance across layers to avoid the instability of single-layer criteria. On multiple LVLM architectures and 12 image + 3 video benchmarks, FlowCut dramatically reduces token count—up to 94.4% fewer tokens—while preserving 91.9% inference accuracy. Under fixed final-token budgets, its layer-wise adaptive pruning achieves the greatest cumulative computation reduction compared to prior methods.\n\nSTRENGTHS AND WEAKNESSES:\nPros\n\n1. The framework is hierarchically structured and methodologically novel, achieving significant speed–accuracy gains with an intuitive combination of strategies.\n\n2. The Insights module rigorously analyzes visual token redundancy from a dynamic information-flow perspective, presenting core observations step by step with clear logic.\n\n3. Experiments are comprehensive across multiple LVLM architectures and 12 image + 3 video benchmarks, using diverse metrics to fully demonstrate the efficiency–effectiveness trade-off.\n\nCons\n\n1. The multi-criteria evaluation and cumulative flow tracking involve several hyperparameters (e.g., weighting coefficients) without ablation or sensitivity analyses to validate robustness.\n\n2. Evaluation is limited to the LLaVA family and Qwen2-VL; applicability to other vision encoders or CLS-free architectures remains unverified.\n\n3. Reliance solely on attention entropy and Value-vector L₁ norm to characterize information flow may misjudge redundancy in complex scenes (e.g., dense multi-object overlaps).\n\nQUESTIONS:\n1. The multi-criteria evaluation and cumulative flow tracking rely on several hyperparameters (e.g., weighting coefficients, pruning frequency). Could the authors include ablation studies or sensitivity analyses to demonstrate robustness across a reasonable range of these parameters?\n\n2. FlowCut currently uses attention entropy and the Value-vector L₁ norm to characterize information flow. How can it avoid mistakenly pruning critical tokens in dense multi-object or heavily occluded scenes?\n\n3. Some readers may not be familiar with Vision Transformer details; terms like \"CLS\" should be defined at first mention or briefly explained to improve clarity.\n\n4. Long video inputs dramatically increase token counts and introduce large inter-frame semantic shifts. Can FlowCut's layer-wise adaptive pruning maintain high-quality inference under these conditions?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper first analyzes the information propagation in Transformers from shallow to deep layers using attention scores. It finds that the CLS token functions as a relay, helping most patch tokens gradually expand their attention from neighboring positions in shallow layers to longer-range dependencies in deeper layers, eventually imitating the CLS token by focusing on a few specific tokens. This insight inspires a dynamic pruning strategy. By integrating attention scores, similarity to the CLS token, and the L1 norm, the authors propose a layer-wise importance metric, which is further accumulated across layers. This leads to an efficient and powerful dynamic pruning method that outperforms traditional static pruning approaches.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n1. The proposed method not only achieves faster computation but also delivers better performance.\n2. It is motivated by the perspective of information propagation, providing a certain theoretical foundation and offering potential for further improvement.\n3. The idea of dynamic pruning offers a more flexible and elegant solution.\n\n**Weaknesses**\n1. The paper does not provide a detailed rationale behind the design choices for combining multiple metrics.\n\nQUESTIONS:\n1. During forward propagation, is dynamic pruning performed simultaneously at each layer? If so, does it mean that pruning decisions made in earlier layers will influence the pruning results in later layers?\n2. Additionally, if that's the case, since the number of pruned tokens per layer is related to attention entropy, how is it ensured that the final number of tokens matches the target? Is the final pruning step forcibly adjusted to meet the target count?\n3. In Section 2 of the main paper, the analysis of information propagation is based on a tokenizer that includes a CLS token. If a tokenizer without a CLS token is used, and the result of global pooling is treated as the CLS token, would similar conclusions still hold?\n4. What is the design principle behind the combination of multiple metrics? Why are both addition and multiplication used?\n5. In Equation (4) of the main paper, what does $I^r$ represent? It is not mentioned earlier in the text—should it instead be $I^d$?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

comments: SUMMARY:
This paper introduces FlowCut, which is a token pruning method for reducing visual tokens in multimodal LLMs. While previous methods typically use attention scores from a single layer to identify redundant tokens, FlowCut proposes an information flow approach that identifies redundant tokens progressively through attention across multiple layers. Specifically, FlowCut suggests a layer-wise adaptive pruning ratio, a multi-criteria token scoring system, and a cumulative importance evaluation mechanism to aggregate importance over layers. Experiments are conducted on widely used multimodal LLMs such as LLaVA-1.5 and LLaVA-NeXT, showing that FlowCut outperforms some previous baselines.

STRENGTHS AND WEAKNESSES:
### Strengths 
* This paper is well-organized
* Extensive visualization is provided

### Weaknesses
* It seems that this method is incompatible with FlashAttention since it depends on the attention value. 

* For better clarity, I'd suggest the authors include pseudocode or an algorithm to describe their method precisely.

* It would be better to have experimental results under a very small token number, as there is too much redundancy in the visual domain.

* This paper shares some similar designs, such as the utilization of the CLS token with a prior work Crossget [1]. I’d suggest the authors provide a comparison or discussion.

[1] CrossGET: Cross-Guided Ensemble of Tokens for Accelerating Vision-Language Transformers.  (ICML 2024).

QUESTIONS:
See Weaknesses please

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 2
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-07-03