Title: Accelerating Multimodal Large Language Models via Dynamic Visual-Token Exit and the Empirical Findings
Venue: NeurIPS 2025
Accepted: yes
Keywords: Multimodal Large Language Modal

Token Prune
Abstract: In this paper, we study the visual redundancy problem of multimodal large language models (MLLMs) from the perspective of attention behaviors. 
Via extensive empirical experiments, we observe and conclude three main inference stages of MLLMs:
(i) Early fusion between tokens is first accomplished quickly. 
(ii) Intra-modality modeling then comes to play. 
(iii) Multimodal reasoning resumes and lasts until the end of inference. 
In particular, we reveal that visual tokens will stop contributing to reasoning when the text tokens receive enough image information.
Based on this observation, we propose an effective method to improve the efficiency of MLLMs, termed dynamic visual-token exit (DyVTE), which is orthogonal but collaborative to previous token-wise visual compression methods.
To validate the efficacy of DyVTE, we apply it to a set of MLLMs, including LLaVA, VILA, EAGLE and InternVL.
The experimental results not only show the effectiveness of our DyVTE in improving MLLMs' efficiency, e.g., DyVTE reduces the computation overhead of LLaVA-1.5 by up to 45.7% without performance drop, but also reveal a general pattern across multiple MLLMs, well facilitating the in-depth analysis of MLLMs. 
Our code is anonymously released at https://anonymous.4open.science/r/AnonymousDyVTE-26AB/.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper \"Accelerating Multimodal Large Language Models via Token Reduction\" presents a method to enhance the efficiency of Multimodal Large Language Models (MLLMs) by reducing the number of visual tokens processed during inference. The core contribution is a token reduction framework that dynamically selects and preserves only the most informative visual tokens while discarding redundant ones, significantly improving inference speed without sacrificing accuracy.\n\nThe authors introduce a trainable token selection mechanism that leverages importance scores to prune the input token set. This method is evaluated across several MLLMs and vision-language benchmarks (e.g., VQA, image captioning), demonstrating that token reduction leads to substantial latency savings with minimal performance degradation. Additionally, the paper explores both plug-and-play and end-to-end training approaches for integrating the token reduction module into existing MLLM pipelines.\n\nOverall, the paper contributes a practical and generalizable framework for making MLLMs more computationally efficient, which is especially valuable for deployment in resource-constrained or real-time settings.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper presents a high-quality and timely contribution to the field of multimodal large language models (MLLMs), specifically addressing the challenge of computational inefficiency caused by long visual token sequences. The proposed token reduction framework is well-motivated, methodologically sound, and broadly applicable across multiple MLLM architectures. The authors introduce a token selection mechanism that is both trainable and compatible with existing models in a plug-and-play fashion, offering flexibility in deployment scenarios. Their extensive experiments across vision-language tasks—including VQA, image captioning, and visual grounding—demonstrate that substantial reductions in inference cost (e.g., 30–50% token reduction) can be achieved with minimal or negligible impact on task accuracy.\n\nIn terms of originality, while token reduction and pruning are not entirely new ideas in the vision or NLP domains, this work distinguishes itself by developing a general and efficient method tailored for MLLMs, a relatively underexplored area in terms of inference-time optimization. The clarity of presentation is strong; the paper is well-organized with clear explanations of the proposed modules, training objectives, and ablation studies. Diagrams and figures effectively illustrate the architecture and token retention behavior, aiding comprehension.\n\nHowever, a few limitations are worth noting. First, the paper could benefit from more discussion around the potential trade-offs introduced by aggressive token pruning in edge-case scenarios, such as images with dense or fine-grained visual detail. Second, the token scoring and selection mechanism, while effective, could be explained in more depth, especially regarding its training stability and generalization to different MLLMs. Lastly, the work does not explicitly address how token reduction may interact with downstream decoding strategies or cross-modal attention patterns—areas that could present challenges in real-world deployment.\n\nQUESTIONS:\n1. The proposed token reduction strategy shows strong results across standard benchmarks, but how does it perform in cases involving dense or fine-grained visual information (e.g., images with small, critical details)? Could you provide either qualitative examples or empirical results for such edge cases? Clarifying this would help assess whether the method consistently preserves semantically crucial content across varying data complexities. A strong showing on these examples would positively affect the significance and robustness assessment.\n2. Since token pruning alters the visual input sequence, how does this affect cross-modal attention alignment in models like Flamingo or BLIP-2 that heavily rely on interaction between visual and text tokens? Have the authors observed any degradation or shifts in attention patterns, and if so, are there mechanisms to preserve alignment fidelity? Addressing this woul\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper investigates the problem of visual redundancy in Multimodal Large Language Models (MLLMs) by analyzing their attention behaviors during inference. Through extensive empirical studies, the authors identify three primary inference stages in MLLMs: (i) early fusion, where visual and text tokens exchange information rapidly in shallow layers; (ii) intra-modality modeling, where tokens primarily interact within their respective modalities; and (iii) multimodal reasoning, where visual tokens resume propagating information to text tokens until a certain point, after which their contribution diminishes. A key finding is that visual tokens often become redundant in the multimodal reasoning stage once text tokens have received sufficient visual information. Based on this, the authors propose a novel method called Dynamic Visual-Token Exit (DyVTE), which uses lightweight hyper-networks to dynamically determine when to remove visual tokens during inference, thereby improving computational efficiency.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The paper presents a robust empirical analysis of MLLM attention behaviors, supported by quantitative experiments across multiple models.\n- The work addresses a critical challenge in MLLMs: the computational inefficiency caused by excessive visual tokens.\n\nWeaknesses:\n- The analysis and methods lack novelty. Analysis techniques based on attention score mechanisms have been applied in many previous works (e.g., FastV and most related studies). The conclusions derived from attention entropy and \"The impact of dropping all visual tokens\" have also been presented in some prior works (e.g., llava-mini). It seems that the authors should explore some specific analysis methods tailored to early-exit, which would make the work more compelling.\n- The connection between the analysis results in Section 3 and the proposed method is not strong. Clarifying why the method design can be inspired by these analysis results is crucial for establishing the method’s validity.\n\nQUESTIONS:\n- The paper mentions that DyVTE uses lightweight hyper-networks, but it does not quantify their computational or memory overhead. How significant is this overhead compared to the savings achieved by removing visual tokens?\n- The experiments focus on standard vision-language benchmarks (e.g., MMB, SQA, QQA). Have the authors tested DyVTE on more complex or real-world tasks, such as open-ended visual question answering or tasks requiring fine-grained visual understanding?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper shows that multimodal LLMs only need image tokens early on, once text tokens have fused image information, visual tokens can be removed. Exploiting this, the authors propose Dynamic Visual-Token Exit, which is a tiny hyper-network to decide when to discard all image tokens while letting text tokens keep processing.  Experimental results on five popular MLLMs show that DyVTE can reduce the inference cost without significant performance drop.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The paper is clearly written and easy to follow, effectively guiding readers through the key insights and contributions.\n2. The experiments conducted are extensive, covering multiple models and benchmarks, which provides robust empirical validation of the proposed method.\n\nWeaknesses:\n1. The issue of visual token redundancy has already been extensively studied in prior works, such as LLaVA-Mini and FastV. Therefore, the empirical findings presented in this paper regarding redundancy contribute limited novelty.\n2. The paper lacks novelty in terms of methodology. Specifically, important related early-exit approaches, like LLaVA-Mini—which discards all visual tokens after fusing visual information into text tokens within just four LLM layers—are not discussed or compared against. Including comparisons or at least a detailed discussion of such highly relevant work would significantly strengthen the paper.\n3. In the efficiency comparisons, the authors only evaluate theoretical FLops differences. However, in practical scenarios, MLLM inference typically involves optimization techniques like flash attention and KV cache. The reliability and impact of the proposed approach would be better assessed through actual inference latency rather than theoretical computational cost alone.\n\nQUESTIONS:\n1. At which specific layer is the early exit applied? Although line 164 mentions that DyVTE is applied at the k-th layer of an MLLM, the Implementation Details section lacks clarification about the exact layers or criteria used for selecting these layers in practice.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes DyVTE, an early-exit method for efficient MLLM inference. It first looks into the MLLM inference behavior by analyzing the attention scores, and categorizes the inference process into three stages. By examining the importance of visual tokens in the three stages, a learned hyper-network is used to remove the visual tokens adaptively. Extensive experiments over multiple benchmarks demonstrate the effectiveness of DyVTE.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\nThe in-depth analysis of token importance is interesting and novel.\nExtensive experiments are done, justifying the proposed method’s effectiveness.\nThe paper is well written, and the proposed findings are well explained and justified.\nWeaknesses:\nThere seems to be a weak connection between MLLM inference behavior and the proposed method, where though DyVTE is motivated by the findings on the changing importance of visual tokens, the categorization of MLLM inference did not lead to any technical contribution. It seems that it is only an empirical and rather intuitive finding, which weakens the overall contribution of the paper.\nThere is a lack of ablation studies on the computation cost of training the hyper-network. While many token pruning/merging techniques adopt a training-free scheme, it is unclear whether DyVTE can be more efficiently deployed.\nCan a universal DyVTE be used, i.e. a same DyVTE be shared by different MLLMs?\nDyVTE adopts an all-or-nothing visual token removal strategy which either retains all or discards all visual tokens at a chosen layer. While effective for efficiency, this coarse-grained approach lacks flexibility and may overlook intermediate solutions where only some visual tokens (e.g., those with low attention weights) can be removed. Fine-grained or per-region exit strategies may offer better trade-offs.\nMinor Weaknesses:\nL245 “it work very well with our DyVTE” should be “it works very well with our DyVTE”\nReferences [5] and [6] are duplicate citations of the same work “Token Merging”\nL225 “DyVTE can reduce 45.7% FLOPs of LLaVA” should be “DyVTE reduces the FLOPs of LLaVA by 45.7%”\n\nQUESTIONS:\n1. Please see the Weaknesses.\n2. In Table 5, combining DyVTE with FastV decreases performance, while in Table 3, improves accuracy. As there is only a change in LLaVa model size, what might be the reasons behind this?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-04"}