Title: Each Complexity Deserves a Pruning Policy
Venue: NeurIPS 2025
Accepted: yes
Keywords: Vision-Language Models

Pruning
Abstract: The established redundancy in visual tokens within large vision–language models (LVLMs) allows for pruning to effectively reduce their substantial computational demands. Empirical evidence from previous works indicates that visual tokens in later decoder stages receive less attention than shallow layers. Then, previous methods typically employ heuristics layer-specific pruning strategies where, although the number of tokens removed may differ across decoder layers, the overall pruning schedule is fixed and applied uniformly to all input samples and tasks, failing to align token elimination with the model’s holistic reasoning trajectory. Cognitive science indicates that human visual processing often begins with broad exploration to accumulate evidence before narrowing focus as the target becomes distinct. Our experiments reveal an analogous pattern in LVLMs. This observation strongly suggests that neither a fixed pruning schedule nor a heuristics layer-wise strategy can optimally accommodate the diverse complexities inherent in different inputs. To overcome this limitation, we introduce Complexity-Adaptive Pruning (AutoPrune), which is a training-free, plug-and-play framework that tailors pruning policies to varying sample and task complexities. Specifically, AutoPrune quantifies the mutual information between visual and textual tokens, and then projects this signal to a budget-constrained logistic retention curve. Each such logistic curve, defined by its unique shape, is shown to effectively correspond with the specific complexity of different tasks, and can easily guarantee adherence to a pre-defined computational constraints. We evaluate AutoPrune not only on standard vision-language tasks but also on Vision-Language-Action (VLA) models for autonomous driving. Notably, when applied to LLaVA-1.5-7B, our method prunes 89% of visual tokens and reduces inference FLOPs by 76.8%, but still retaining 96.7% of the original accuracy averaged over all tasks. This corresponds to a 9.1% improvement over the recent work PDrop (CVPR'2025), demonstrating the effectivenes.  Code is available at https://github.com/AutoLab-SAI-SJTU/AutoPrune.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nAdaPrune is a method for flexibly removing visual tokens in large vision–language models (LVLMs) based on input complexity. Unlike prior approaches that apply a fixed, layer-wise token removal schedule uniformly across all samples and tasks, AdaPrune measures the mutual information between visual and textual tokens and generates a logistic retention curve tailored to each sample’s complexity. This ensures that only the necessary tokens are retained under a given computational budget, drastically reducing computation.\n\nSTRENGTHS AND WEAKNESSES:\n**Strength**\n\n- Training-Free Framework: Offers a plug-and-play solution that requires no additional training.\n- Cognitively Inspired: Leverages visual–textual mutual information to quantify complexity, mimicking biological attention mechanisms.\n- Per-Sample Token Control: Employs individualized, budget-constrained logistic retention curves to dynamically prune tokens per decoder layer.\n\n---\n\n**Weaknesses**\n\n- Unclear writing: It makes the reader get easily confused with the terminology and the motivation of neuroscience. This writing has so many expert-level terminology for brain and redendunt detail explanation. These easily made me distract from the main point of what the authors want to say. Line 53-63 and Line 154-165 are most paragraphs that I can't understand much. Even, the rest of the writings is also somewhat hard to read smoothly. Unless really focusing on the writing, it is easy to miss some parts. I don't think the writing with hard and complex terminology is great paper at least for AI community due to rapid change and development. Please make the writing narrowed for the scope of vision language-perspective and pruning-perspective, although the authors want to say everything that you can come up with the expert concept.\n\n- Vague standard for simple and complex: As in figure 1, it represents layer-wise visual text interaction patterns where simeple task (object identification) and complex task (reasoning-intensive) feels some awkward. This is because according to question, I can recognize whether it is simple or comlpex task. However, in the perspective of feature visualization and saliency map, i don't think it converges on the salient region and remains stable, even when simple task is given. I think the problem setup should be more solid or more explored.\n\n- Applicability to more recent released models: Your models feel so out-dated because recently, there are so many VLMs such as Qwen2/2.5-VL or InternVL2.5/3-VL. Because none of these recent models, it may limits the evidecneof your solid experiments and applicability.\n\n- Lack of interesting phenomenon: Figure2 seems so interesting results but there are none of additional results for other benchmarks, because Figures2 is achieved on TextVQA only. I would highly recommend that kinda graphs for numerous benchmarks and have some analysis for them.\n\n- Challenging Evaluation Benchmarks: Your selected evaluation benchmarks are mainly from easy benchmarks including just selecting the multiplie choice or simple answer. However, there are so many challenging benchmarks including free-form answer (long context answer) like MM-Vet (or -v2) and really challening knowledge evlauation benchmark such as MMMU.\n\n- Overall, I think the motivation is somewhat lack and there are some redundent explanation beyond the main point that the authors should convey. This paper could be improved in terms of writing quality and clear standard to discriminate how we can acquire insight from salieny map because it is merely obvious phenomenon that many papers have reported. Furthermore, this paper must be more solid and having dense experimental resulsts for consistent phenomenon of numerous benchmarks and applicability to recent models.\n\nQUESTIONS:\nRefer to Weaknesses\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 1, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces AdaPrune, a training-free token pruning framework that, drawing inspiration from neuroscience, adaptively selects layer-wise token retention ratios subject to a global constraint on inference compute. The neuroscience background suggests that humans typically narrow their focus to constrained locations in visual scenes in early processing stages when the task is simple, while they conversely preserve several hypotheses for longer when the task is complex. This work shows an analogous pattern in Vision-Language Models (VLMs), where the attention maps may \"converge\" in early or late layers, depending on the textual query associated with it.  \n\nGiven this empirical evidence, AdaPrune maps a notion of query difficulty (computed via mutual information between visual and textual tokens) to a logistic curve that determines how many tokens to retain at each layer. Specifically, the logistic curve maps a layer index to a retention ratio, and the slope and inflection point of the curve are linear functions of the aforementioned mutual information. Such a curve is further normalised to ensure its integral (which reflects the total number of tokens processed end-to-end during inference) matches a given compute constraint.  \n\nSuccessful experiments are conducted on standard Vision-Language benchmarks with LLaVA-v1.5-7B and LLaVA-Next-7B, as well as with Senna, a Vision-Language-Action model employed for autonomous driving.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**.  \n1. The paper is well-written and nicely organised;  \n2. The underlying idea is simple, which I believe is a plus, is carefully designed (I particularly liked the overall reframing of attention scores into MI) and has a sound motivation; \n3. Experiments are comprehensive, and extending towards VLAs with fixed hyperparameters shows the practical applicability of the proposed framework beyond academic benchmarks; \n\n\n**Weaknesses**.  \nI think the major weakness of this paper is that some passages of the methodological section are unclear or partially omitted. For example:  \n1. In my understanding, the adaptive logistic curve provides a per-layer token retention ratio based on the overall scalar-valued mutual information, but I see no detail about which tokens are retained and which are discarded;  \n2. Other than the estimated mutual information, the logistic curve depends on some parameters ($k_0$, $\\gamma$, $x_o$, $\\beta$) which should be defined a priori. It's probably amiss for this paper not to report how these are selected;\n\nQUESTIONS:\nFor a productive rebuttal, I would probably start with the following questions (expanding on the weaknesses above).\n\n1. Is Mutual Information for a single visual token the criterion used to choose which tokens are retained after determining a retention ratio with the logistic curve? So, something like Eq. (3) of the paper, where the outer summation over visual tokens is unpacked into $N_v$ scalars determining the importance scores of each of the visual tokens? I have reviewed Section D of the supplementary materials, where an ad-hoc importance score is defined for the first layer. Line 40 therein states: *\"We assign each visual token an importance score based on the textual-visual attention weight.\"* It is, however, unclear to me if this holds for all layers or only for the first one. In general, could the authors please clarify this aspect about token selection?  \n\n2. Could the authors please clarify how the parameters of the logistic curve are selected, i.e., which data was used, how many data points, etc?\n\n3. In my understanding, the logistic curve computed from Mutual Information determines how tokens are progressively dropped in the stacked blocks of the LLM decoder, meeting the global compute constraint after normalisation, but at which layer is this Mutual Information computed? While reading, I assumed this computation happens immediately in the first block, and the derived scalar determines computation throughout the rest of the decoder, but could the authors please clarify this aspect as well?\n\nLIMITATIONS:\nThe final section of the paper is titled \"Conclusions and Limitations\", but it looks more like a \"Conclusions\" Section rather than a \"Limitations\" one. However, I do not see important unaddressed considerations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper concentrates on the vLLM's inference acceleration. In particular, previous methods typically employ heuristics layer-specific pruning strategies, and the overall pruning schedule is fixed and applied uniformly to all input samples and tasks, regardless of the samples, tasks or layers. In this work, inspired by the cognitive science, the authors introduce Complexity-Adaptive Pruning (AdaPrune), which is a training-free, plug-and-play framework that tailors pruning policies to varying sample and task complexities. The method is based the mutual information between visual and text inputs, and quantifies the compexity of a sample. Then, the pruning policy is designed based on the mutual information. Experiments are done on LLaVA and VQA benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\nThe strengths are as follows:  \n1.This work is well motivated. Recent training free acceleration methods mainly rely on hand-crafted metrics and the pruing policy is fixed regardless of the layers. This paper proposes a sample or task complexity adaptied pruing model.  \n2.The idea based on cognitive science and mutual information is interesting. Using mutual information between visual and text input as a proxy objective to measure the sample's comlexity and prune tokens is new to me. It sounds reasonable.   \n\nThe weakness, also my questions are as follows:  \n1.Why the Budget-Constrained Logistic Retention is designed as Eq.5. It is not easy to understand how this formulation is introduced and why it is good to achieve the target?  \n2.In Figure 2, Why simple samples mainly lie in retention rate between 0.2-0.4 in both low and high layers? For hard samples, it is easy to understand the retention rate is high in low layers, but how to understand the retention rate is low in high layers? Do we need more information for hard samples in both low and high layers?  \n3.The experiments could be improved.  \na).The test set of VQA task is very simple, only Yes or No, or answer selection. Are there any other more complex generation task for evaluation? Image caption or image generaion?  \nb).Some related works are not included and discussed. Recently, there are some other training-free pruning methods, such as VTW[1], Turbo[2] and Folder[3]. These methods should be compared and discussed.  \nc).In Table 3, the reported performance is  a relative improvement. What is the absolute performance of compared baseline model, it is worthwhile to take the absolute performance into cosideration to check the relative improvement.  \n\n[1]Boosting Multimodal Large Language Models with Visual Tokens Withdrawal for Rapid Inference AAAI'25  \n[2]Turbo: Informativity-Driven Acceleration Plug-In for Vision-Language Large Models, ECCV'24  \n[3]FOLDER: Accelerating Multi-modal Large Language Models with Enhanced Performance,ICCV'25\n\nQUESTIONS:\nSee above\n\nLIMITATIONS:\nSee above", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces AdaPrune, a training-free, plug-and-play framework for adaptive token pruning in Vision-Language Models. It measures cross-modal mutual information (MI) between early visual and textual tokens to gauge sample/task complexity, then maps MI to a budget-constrained logistic retention curve that dynamically allocates token budgets per decoder layer. Applied to LLaVA-1.5, LLaVA-NeXT, and a VLA model, AdaPrune prunes up to 89% of tokens while retaining >96% accuracy, outperforming prior methods.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\nThe manuscript is well written and easy to follow.\n\nAdaPrune uses a simple, training-free signal (mutual information between early visual and text tokens) to tailor pruning intensity for each input, avoiding manual schedule design or retraining.\n\nThe logistic retention curve formulation guarantees that any token or FLOPs budget is met exactly.\n\nThe authors conducted various experiments on LLaVA-1.5, LLaVA-NeXT, and a vision-language driving model.\n\n---\n\nWeaknesses:\n\nThe mapping from mutual-information scores to the curve’s slope and inflection point is described only at a high level, more detail on hyperparameter choices and token-selection criteria is needed.\n\nRelying solely on an early-layer MI estimate risks misjudging complexity for inputs where initial attention patterns are misleading, potentially pruning critical tokens too soon.\n\nEvaluation is limited to 7B-parameter models on static images. It is unclear how AdaPrune scales to much larger models, video inputs, or streaming scenarios. Table 4 in supp is helpful information, do you consider moving it to the main pages?\n\nQUESTIONS:\nHow exactly do the authors map the mutual information score to the logistic curve’s slope and midpoint? Did you apply any clamping or calibration on MI values?\n\nIt would be good to see AdaPrune on much larger VLMs or on streaming/video inputs. What challenges do the authors anticipate?\n\nHow sensitive is performance to noise in the MI estimate? Did you observe cases where mis-estimated MI led to under- or over-pruning?\n\nIt would be helpful to share more implementation details. For easier reimplementation, the authors can show more about how you compute MI from attention, your default hyperparameters, and how you pick which tokens to drop each layer?\n\nLIMITATIONS:\nThe paper omits an explicit “Limitations” section but combing it with the conclusion section. I think it is okay to do so.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}