Title: Balanced Token Pruning: Accelerating Vision Language Models Beyond Local Optimization
Venue: NeurIPS 2025
Accepted: yes
Keywords: Large Vision Language Models; Efficiency; Token pruning
Abstract: Large Vision-Language Models (LVLMs) have shown impressive performance across multi-modal tasks by encoding images into thousands of tokens. However, the large number of image tokens results in significant computational overhead, and the use of dynamic high-resolution inputs further increases this burden. Previous approaches have attempted to reduce the number of image tokens through token pruning, typically by selecting tokens based on attention scores or image token diversity. Through empirical studies, we observe that existing methods often overlook the joint impact of pruning on both the current layer's output (local) and the outputs of subsequent layers (global), leading to suboptimal pruning decisions. To address this challenge, we propose Balanced Token Pruning (BTP), a plug-and-play method for pruning vision tokens. Specifically, our method utilizes a small calibration set to divide the pruning process into multiple stages. In the early stages, our method emphasizes the impact of pruning on subsequent layers, whereas in the deeper stages, the focus shifts toward preserving the consistency of local outputs. Extensive experiments across various LVLMs demonstrate the broad effectiveness of our approach on multiple benchmarks. Our method achieves a 78\% compression rate while preserving 96.7\% of the original models' performance on average. Our code is available at https://github.com/EmbodiedCity/NeurIPS2025-Balanced-Token-Pruning.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel Balanced Token Pruning (BTP) method to reduce the computational cost of large Vision-Language models. Through empirical analysis, the authors show that attention-based pruning mainly focuses on locality, while diversity-based pruning considers the impact on subsequent layers. BTP combines the strengths of both approaches by applying a diversity-based objective in the early layers to select semantically diverse tokens, and an attention-based objective in the later layers to locally optimize each layer and minimize output error. To support this framework, the paper introduces an attention rebalance module and a spatial initialization strategy. It also proposes a layer selection mechanism using a small calibration image set. Experiments applying the proposed method to various models (LLaVA-1.5-7B, LLaVA-1.5-13B, LLaVA-1.6-7B, and Qwen2.5-VL-7B) show that BTP achieves better performance not only compared to attention-based pruning, but also to diversity-based pruning methods. Ablation studies further validate the effectiveness of each component.\n\nSTRENGTHS AND WEAKNESSES:\n***Strenghts***\n\n1. The paper is well-motivated, supported by empirical analysis comparing attention-based and diversity-based pruning.\n2. The proposed method demonstrates superior performance over both attention-based and diversity-based pruning across multiple large-scale VL models.\n\n***Weaknesses***\n\n1. Some experimental results are missing or not fully detailed, limiting the understanding of the method’s generalizability.\n2. Certain components lack sufficient explanation.\n\nQUESTIONS:\n**1. Spatial Initialization**\n\nCould you explain why you chose Manhattan distance instead of Euclidean distance? For example, given a source token at (2, 2), both (6, 6) and (2, 10) have the same Manhattan distance, but different Euclidean distances. In many spatial contexts, Euclidean distance may provide a more accurate measure of proximity.\n\n\n**2. Pruning Layer Selection**\n\nIn Qwen2.5-VL, significant changes in image token representations are observed in layers 10 and 24, yet relatively fewer image tokens are selected in those layers. This appears inconsistent with the assumption that image tokens receive more attention in layers with significant feature changes. LLaVA-v1.5 shows a similar pattern—although layers 8–12 exhibit large representational shifts, more image tokens are selected in layers 17, 19, and 20. Further clarification would be helpful.\n\n**3. Attention Optimization**\n\nThe method uses the last token in the input prompt ($X_T$) to compute importance scores. It is unclear whether relying on a single token is sufficient to capture the full contextual relevance of the prompt. Further explanation of this choice would improve the understanding of its effectiveness.\n\n**4. $\\lambda_i$ Values**\n\nThe $\\lambda_i$ values are critical hyperparameters that influence the performance of the proposed framework. In Section 5.3, the authors apply smaller lambda values (0.5–0.7) to shallow layers and larger ones (0.8–1.0) to deeper layers. However, it is not clearly explained how the $\\lambda_i$ values are determined for each layer. It would be beneficial to provide a more concrete rule or conduct an ablation study on the impact of $\\lambda_i$ settings.\n\n**5. Inference Latency and Spatial Initialization**\nIn Table 4, when the attention rebalance (RA) module is used, adding spatial initialization significantly improves inference latency. However, in cases where RA is not used, the latency difference with or without spatial initialization becomes marginal. An explanation for this discrepancy would be appreciated.\n\n\n**6. Appendix A.1**\n\nIn Section 4.1, the authors present the assumption that when $V_l$ values are similar, selecting image tokens with high importance scores can effectively reduce the difference in outputs before and after pruning. However, the only supporting evidence is a figure, and no explanation is provided. This lack of explanation makes the assumption difficult to evaluate.\n\n\n**7. Equation (1)**\n\nThe equation for $\\text{Atten}^{(l)}$ is not explicitly defined in the paper. Including a clear mathematical formulation would help readers better understand how attention scores are computed at each layer.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Balanced Token Pruning (BTP), a plug-and-play method to reduce the computational cost of  LVLMs by more effectively pruning image tokens. Unlike prior approaches that rely solely on local attention or token diversity, BTP balances both local and global impacts of token pruning across model layers using a multi-stage strategy informed by a small calibration set. Experiments across several LVLMs show that BTP consistently improves efficiency with minimal performance loss.\n\nSTRENGTHS AND WEAKNESSES:\n- The paper gives a very interesting insight on how token pruning affects the subsequent blocks which is often overlooked in the existing approaches.\n- The approach is convincing, in that, a combination of local-global consideration for pruning would be more optimal than the exisiting pruning approaches.\n- BTP has been evalulated extensively and compared with many recent token pruning approaches in VLMs.\n- Some empirical observations, such as token attention patterns across layers, are counter-intuitive and insufficiently explained, with missing experimental details and unclear generality across the dataset.\n- The method’s use of input-dependent token pruning may affect batching efficiency and inference throughput, but this practical concern is not addressed.\n\nQUESTIONS:\n- Line 41 - 42: Is this phenomenon of different image tokens being attended by the text tokens consistently observed across the dataset? It seems counter-intuitive to me. At least in vision transformers, we do observe the same tokens being attended to by other tokens throughout the layers.\n- Lines 44 - 56: Why was the hidden state of only last word considered for this experiment? What dataset and model was used for this experiment? Overall, I think it would be insightful to add some more explanation (if any) on the observations shown in figure 1 and 2. But this idea of creating a new local-global objective is quite interesting.\n- Having different number of tokens pruned for different input sample breaks the regularity in processing batched input. Does it not deteriorate the throughput?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper aims to address the computational overhead in Large Vision-Language Models (LVLMs) caused by the large number of image tokens. The authors begin by empirically analyzing existing token pruning methods, categorizing them into attention-based and diversity-based approaches. They find that attention-based methods excel at optimizing the output of the current layer (a local objective) but neglect the impact on subsequent layers (a global objective). Conversely, diversity-based methods better preserve global information but sacrifice local output consistency.\n\nTo address this local-global trade-off, the authors propose Balanced Token Pruning (BTP), a plug-and-play method for visual token pruning. The core idea of BTP is to divide the pruning process into multiple stages and dynamically adjust the weight between local and global objectives. Specifically, in the early layers of the model, the method emphasizes a diversity-based criterion to preserve rich information that has a significant impact on subsequent layers. In the deeper layers, it shifts focus to an attention-based criterion to ensure the accuracy of the immediate layer's output. Additionally, the method incorporates several supporting techniques: 1) using a small calibration set to automatically select the pruning layers; 2) an attention rebalancing operation to correct for positional biases; and 3) a spatial initialization strategy to accelerate diversity calculations.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n1.  The authors clearly point out the inherent contradiction between \"local optimization\" and \"global impact\" through an in-depth analysis of existing methods. The visualization analysis of Figures 1 and 2 intuitively shows the difference in behavior of different pruning strategies at different depths of the model, providing a solid foundation and convincing motivation for the \"balance\" idea proposed in the paper.\n\n2. The core idea of ​​BTP - dynamically balancing local (attention) and global (diversity) goals according to the network depth where the pruning is located - is very novel and intuitive. Rather than simply mixing the two methods, it proposes a principled, staged strategy, that is, protecting information diversity at shallow levels to benefit the global, and focusing on task relevance at deep levels to ensure local. This idea is very elegant.\n\n3. Exhaustive Experimental Validation: The experimental part of the paper is very solid. The author tested on multiple models (LLaVA-v1.5/v1.6, Qwen2.5-VL) and multiple sizes (7B, 13B), covering a wide range of visual understanding benchmarks. The ablation study is very comprehensive and systematically verifies the effectiveness of each component such as balancing factor, rebalancing module, spatial initialization and layering strategy.\n\n**Weaknesses**\n1. The paper makes completely unclear statements when describing how the core balance factor $\\lambda$ changes with the depth of the network.\n- In Section 5.3, the author describes: \"In the early layers, we use a larger $\\lambda$ value to focus more on global information, while in the deeper layers, we use a smaller lambda to emphasize local details.\"\nHowever, in the title of Figure 6 on the same page, the description is: \"Therefore, we use smaller $\\lambda$ values ​​(0.5-0.7) in the shallow layers and larger values ​​(0.8-1.0) in the deeper layers.\" According to Formula 7, $\\lambda$ is the weight of the attention term, so a larger $\\lambda$ means more attention to local targets (attention). These two descriptions are completely contradictory. This serious contradiction makes me doubt the implementation details of the paper and the rigor of the author.\n\n2. Heuristic Treatment of \"Global Objective\": The paper formalizes the \"global objective\" as the sum of the output errors of all pruned layers (Formula 6), and proposes to use \"diversity\" as its proxy. This connection is intuitive, but lacks a formal proof. The final optimized objective function (Formula 7) is a weighted sum of attention and diversity scores, rather than a direct optimization of Formula 6. This jump from the true objective to the proxy objective is based on heuristics and lacks stronger theoretical support.\n\n3. Although the ablation experiment in Figure 6 shows the impact of a fixed $\\lambda$ value on performance, the paper does not detail how the $\\lambda$ schedule across different stages in the final experiment is determined. For example, what is the specific strategy for transitioning from 0.5/0.7 (shallow layers) to 0.8/1.0 (deep layers)? Is this strategy fixed or does it need to be adjusted for different models? This makes the reproducibility of the method and its sensitivity to hyperparameters unclear.\n\nQUESTIONS:\nRefer to weaknesses\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}

comments: SUMMARY:
This paper proposes Balanced Token Pruning (BTP), a novel plug-and-play method for accelerating large vision-language models (LVLMs) by reducing visual token redundancy. Unlike prior methods that focus on either local (layer-wise) or global (multi-layer) optimization, BTP introduces a local-global pruning objective that balances preserving immediate output fidelity with maintaining downstream feature diversity. BTP is extensively evaluated on several SOTA LVLMs (e.g., LLaVA-7B/13B, Qwen-VL), achieving significant reductions in token count, latency, and memory, while maintaining ~98% of original performance across multiple benchmarks.

STRENGTHS AND WEAKNESSES:
Strengths.

- The local-global pruning objective is conceptually well-motivated and addresses a clear limitation in previous token pruning strategies. The balance factor $\lambda$ and stage-wise optimization strategy are intuitive and theoretically grounded.
- Across multiple models and datasets (GQA, MME, POPE, SQA, MM-VeT), BTP consistently outperforms state-of-the-art approaches like FastV, DivPrune, and VTW in both performance retention and efficiency.
- The authors show that BTP is effective across different model architectures and scales (LLaVA-7B/13B, Qwen-VL), increasing its impact and generalization potential.

Weaknesses.

- The method relies on a calibration set to determine pruning layers and balance factors. While the authors state that the same calibration set is reused across experiments, further discussion is needed on how sensitive the method is to the size and content of this set.
- It is not clear how Figure 2 is depicted, e.g., which data and features are used to measure similarity? It would be good to provide more details.

QUESTIONS:
N/A

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 2
DATE: 2025-07-05