Title: ViSpec: Accelerating Vision-Language Models with Vision-Aware Speculative Decoding
Venue: NeurIPS 2025
Accepted: yes
Keywords: vision-language models

speculative decoding

synthetic dataset
Abstract: Speculative decoding is a widely adopted technique for accelerating inference in large language models (LLMs), yet its application to vision-language models (VLMs) remains underexplored, with existing methods achieving only modest speedups ($<1.5\times$). This gap is increasingly significant as multimodal capabilities become central to large-scale models. We hypothesize that large VLMs can effectively filter redundant image information layer by layer without compromising textual comprehension, whereas smaller draft models struggle to do so. To address this, we introduce Vision-Aware Speculative Decoding (ViSpec), a novel framework tailored for VLMs. ViSpec employs a lightweight vision adaptor module to compress image tokens into a compact representation, which is seamlessly integrated into the draft model's attention mechanism while preserving original image positional information. Additionally, we extract a global feature vector for each input image and augment all subsequent text tokens with this feature to enhance multimodal coherence. To overcome the scarcity of multimodal datasets with long assistant responses, we curate a specialized training dataset by repurposing existing datasets and generating extended outputs using the target VLM with modified prompts. Our training strategy mitigates the risk of the draft model exploiting direct access to the target model's hidden states, which could otherwise lead to shortcut learning when training solely on target model outputs. Extensive experiments validate ViSpec, achieving, to our knowledge, the first substantial speedup in VLM speculative decoding.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel framework to accelerate inference in VLMs. The core idea is that small draft models cannot efficiently process the highly redundant visual information that large target models can. ViSpec addresses this by using a lightweight vision adaptor to create a compact image representation for the draft model and a global feature augmentation technique to maintain multimodal coherence. The paper  also creates a new dataset with long-form responses to facilitate training. ViSpec achieves significant speedups on various benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths And Weaknesses\n\nStrengths:\n\n1. The proposed method is technically sound and well-motivated. The design of the vision adaptor is a simple but effective solution tailored for the speculative decoding for vision language models.\n\n2. The analysis that speculative decoding for LLM is suboptimal for VLMs is insightful.\n\n3. The experiment results are competitive. The method demonstrates advantages over previous SOTA methods on multiple VLM benchmarks.\n\n4. The implementation details and experiment settings are clearly described.\n\nWeaknesses:\n\n1. The paper lacks comparison with existing speculative decoding methods for VLMs, such as [1] and [2].\n\n2. The claim that the vision adaptor is \"lightweight\" is not substantiated with a quantitative analysis. A detailed breakdown of the associated overheads is necessary.\n\n3. The authors should investigate how the speedup varies with the length of the output, as it is critical for understanding the method's boundary.\n\n4. To demonstrate broader utility, how about the performance of the method on text-only inputs?  It is a frequent use case for VLMs.\n\n[1] Gagrani, Mukul, et al. \"On speculative decoding for multimodal large language models.\" \n\n[2] Lee, Minjae, et al. \"In-batch Ensemble Drafting: Toward Fast and Robust Speculative Decoding for Multimodal Language Models.\"\n\nQUESTIONS:\nSee the weaknesses above.\n\nLIMITATIONS:\nThe authors have addressed limitations and social impact of the work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-06-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces ViSpec, aiming to achieve acceleration for vision-language models (VLMs) through speculative decoding. ViSpec employs a lightweight vision adapter module to compress image tokens into a compact representation. It extracts a global feature vector for each input image and augments all subsequent text tokens with this feature to enhance multimodal coherence. Experiments show that the proposed method achieves significant speedup.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\nViSpec integrates compressed image embeddings, persistent global visual feature injection, and synthetic long-response dataset generation to address the limitations in processing multimodal sequences with shallow draft models. ViSpec achieves significant speedup compared to previous methods.\n\nWeaknesses\n\n1.\tThe paper only reports speedup metrics but not performance metrics. What is the performance of the model after acceleration?\n2.\tHigh-resolution images require more image tokens and thus necessitate acceleration. It would be beneficial to evaluate the acceleration performance on benchmarks containing high-resolution images, such as HR-Bench benchmark and MME-Realworld benchmark.\n3.\tIt lacks comparisons with relevant methods, such as FastV[1] which compresses using attention weights and MiniCPM-V[2] which employs learnable queries for compression.\n\n[1] Chen L, Zhao H, Liu T, et al. An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models[C]//European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2024: 19-35.\n\n[2] Yao Y, Yu T, Zhang A, et al. Minicpm-v: A gpt-4v level mllm on your phone[J]. arXiv preprint arXiv:2408.01800, 2024.\n\nQUESTIONS:\nWhy report only the speed-up metrics and not the corresponding accuracy?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n- This paper introduces Vision-Aware Speculative Decoding (ViSpec) to achieve acceleration for vision-language models (VLMs) through speculative decoding.\n\n- By integrating compressed image embeddings, persistent global visual feature injection, and synthetic long-response dataset generation, ViSpec addresses key limitations in processing multimodal sequences with shallow draft models.\n\n- The experiments demonstrate speedups across diverse VLMs and tasks.\n\n- This paper identifies two primary avenues for improvement: first, curating higher-quality multimodal training datasets with greater conversational depth to enhance the draft model’s predictive accuracy; second, optimizing vision encoder architectures, potentially via dynamic patch reduction or neural compression, to reduce visual processing overhead.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths \n\n- This paper hypothesizes that large VLMs can effectively filter redundant image information layer by layer without compromising textual comprehension, whereas smaller draft models struggle to do so.\n\n- This paper introduces Vision-Aware Speculative Decoding (ViSpec). ViSpec employs a lightweight vision adaptor module to compress image tokens into a compact representation.\n\n- This work extracts a global feature vector for each input image and augments all subsequent text tokens with this feature to enhance multimodal coherence.\n\n- This work curates a specialized training dataset by repurposing existing datasets and generating extended outputs using the target VLM with modified prompts.\n\nWeaknesses\n\n- The vision adaptor is heavily inspired by BLIP-2's Q-Former, and the global feature injection resembles EAGLE's target-aware mechanisms.\nThe incremental combination isn't sufficiently differentiated from prior work.\n\n- High speedups on COCO Captions but low speedup on TextVQA. The paper doesn't analyze the inconsistent gains.\n\n- Recent VLMs handle video and 3D input, but ViSpec is only evaluated on static images. Compression may fail for temporal data.\n\nQUESTIONS:\n- Most multimodal benchmarks only require a single-word response, such as 'yes' or 'no' for MME and 'A', 'B', 'C', 'D' for multi-choice QA.\nThus, it is unclear why there are more than 2x-3x speedups for ViSpec in those benchmarks.\n\n- Lack of experiments in high-concurrency settings. The throughput for large batch sizes is unknown.\n\nLIMITATIONS:\nyes.", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes the ViSpec framework to address the slow inference speed of vision-language models (VLMs). It accelerates inference through a specially designed vision-aware speculative decoding technique. \nViSpec employs a lightweight visual adapter to compress image information and injects global visual features to maintain multimodal coherence. This design thereby tackles the challenge of redundant visual inputs that small draft models struggle to handle. \nExperiments demonstrate that ViSpec achieves speedup on various VLMs without compromising much generation quality.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- This framework is specifically designed to address the unique challenges of VLMs. It handles redundant visual information through image embedding compression.\n\n- This paper is easy to follow. Experiments show that ViSpec consistently outperforms baseline methods such as Medusa and EAGLE-2 across all tested models and tasks. ViSpec demonstrates strong effectiveness on four mainstream VLMs and eight different multimodal benchmarks. \n\n- ViSpec introduces an innovative data generation approach to overcome the data limitation. \n\nWeaknesses\n- The \"experimental setup\" section is poorly written, with low information density and inappropriate sectioning.\n\n- The \"Global Visual Feature Integration\" is not novel; a similar idea already exists in \"ContextNet: Improving Convolutional Neural Networks for Automatic Speech Recognition with Global Context\". The difference exists in different backbones. \n\n- The paper claims that using only one compressed image embedding is sufficient to capture the necessary visual information but does not provide adequate justification. The compression might lead to the inability to recognize fine-grained small objects, but the paper did not test corresponding ability.\n\nQUESTIONS:\nPlease polish the writing and complemente more analysis and experiments.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}