ViSpec: Accelerating Vision-Language Models with Vision-Aware Speculative Decoding
Authors: Jialiang Kang
Han Shu
Wenshuo Li
Yingjie Zhai
Xinghao Chen
Venue: NeurIPS 2025 poster ()
Keywords: vision-language models
speculative decoding
synthetic dataset
Primary area: deep_learning
TL;DR: ViSpec accelerates vision-language model inference by integrating vision-aware speculative decoding with compressed image tokens and global feature injection, achieving up to 3.22× speedup.

Abstract: Speculative decoding is a widely adopted technique for accelerating inference in large language models (LLMs), yet its application to vision-language models (VLMs) remains underexplored, with existing methods achieving only modest speedups ($<1.5\times$). This gap is increasingly significant as multimodal capabilities become central to large-scale models. We hypothesize that large VLMs can effectively filter redundant image information layer by layer without compromising textual comprehension, whereas smaller draft models struggle to do so. To address this, we introduce Vision-Aware Speculative Decoding (ViSpec), a novel framework tailored for VLMs. ViSpec employs a lightweight vision adaptor module to compress image tokens into a compact representation, which is seamlessly integrated into the draft model's attention mechanism while preserving original image positional information. Additionally, we extract a global feature vector for each input image and augment all subsequent text tokens with this feature to enhance multimodal coherence. To overcome the scarcity of multimodal datasets with long assistant responses, we curate a specialized training dataset by repurposing existing datasets and generating extended outputs using the target VLM with modified prompts. Our training strategy mitigates the risk of the draft model exploiting direct access to the target model's hidden states, which could otherwise lead to shortcut learning when training solely on target model outputs. Extensive experiments validate ViSpec, achieving, to our knowledge, the first substantial speedup in VLM speculative decoding.

ViSpec: Accelerating Vision-Language Models with Vision-Aware Speculative Decoding
Jialiang Kangl.2
Han Shu?
Wenshuo Li² Yingjie Zhai?
Xinghao Chen
Peking University Huawei Noah's Ark Lab
jkangåstu, pku. edu. en
{han. shu, 1iin,liweshhu zhaiyingj ie, xinghao chen wwwwwhh..cccmmm com
Abstract
Speculative decoding widely adopted technique for accelerating inference in large language models (LLMs), yet its application vision-language models (VLMs) remains underexplored, with existing methods achieving only modest speedups (< 1.5x). This gap is increasingly significant multimodal capabil- ities become central to large-scale models. We hypothesize that large VLMs effectively filter redundant image information layer by layer without compro- mising textual comprehension, whereas smaller draft models struggle to do so. To address this, we introduce Vision-Aware Speculative Decoding (ViSpec), novel framework tailored for VLMs. ViSpec employs a lightweight vision adap- tor module to compress image tokens into compact representation, which seamlessly integrated into the draft model's attention mechanism while preserving original image positional information. Additionally, vector for each input image and augment all subsequent text tokens with this fea- ture enhance multimodal coherence. To overcome the scarcity multimodal datasets with long assistant responses, we curate a specialized training dataset by repurposing existing datasets and generating extended outputs using the tar- get VL.M with modified prompts. Our training strategy mitigates the risk of the draft model exploiting direct access to the target model's hidden states, which could otherwise lead to shortcut learning when training solely target model outputs. Extensive experiments validate ViSpec, achieving, to our knowledge, the first substantial speedup in VLM speculative decoding. Code is available at https///gthubbcoom/mamaaaaaaaaga
extract a global feature
Introduction
The success of large language models (LLMs) has spurred the development of vision-language models (VLMs) capable of processing and generating content from both visual and textual inputs. Recent VLMs, such LLaVA-NeXT (LLaVA-1.6) [15] and Qwen2.5-VL [I], demonstrate impressive performance in tasks including image captioning, visual question answering, and multimodal dialogue. However, VLMs increase in scale and complexity, their inference times grow substantially, posing significant challenges for practical deployment.
Speculative decoding 3 has proven effective in accelerating LLM inference by employing smaller, faster draft model to propose candidate token sequences, which the larger target model verifies in parallel. Correct predictions from the draft model enable the target model to skip costly autoregressive computations, resulting in significant speedups. While speculative decoding is well-established for
Work done during internship at Huawei Noah's Ark Lab.
Conresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Vanilla
Medusa
EAGLE-2
ViSpec
2.0
Speedup
1.5
1.0
0.5
LLa 7B
LLaVA-1.6 13B
Qwen2.5-VL
Qwen2.5-V6 7B
Figure Speedup ratios of various methods temperature O, evaluated on the GQA test set using four VLMs: w.w.--vl.--Vicnnaa. LLaVA-v1.6-Vicuaa113 -eee2..5-LL-B--nsttnue and
Qwen2.5-VL-7B-Instie
LLMs, its application to VLMs remains underexplored, with prior approaches B 12 achieving only marginal speedups. We attribute this limitation to fundamental differences between textual and visual data. Text, honed over centuries, is abstract and information-dense, whereas images, despite their visual richness, often contain considerable redundancy. Consequently, small draft models struggle to extract pertinent visual information while preserving textual coherence VLMs.
To address this challenge, we propose Vision-Aware Speculative Decoding (ViSpec), a novel speculative decoding framework designed specifically for VLMs. ViSpec incorporates lightweight vision adaptor module to compress numerous image tokens into compact, informative representation. These compressed tokens seamlessly integrated into the draft model's s attention layers, retaining the original image's s positional information. Furthermore, drawing inspiration from target-aware extract global feature vector for each input image and augment all subsequent text tokens with this feature until the next image is encountered. This mechanism equips the draft model with robust global visual context, enhancing prediction accuracy.
feature injection in EAGLE 19
18 20),
significant obstacle in developing speculative decoding for VLMs is the scarcity of large-scale, publicly available multimodal datasets with extended assistant responses. To overcome this, repurpose existing datasets by modifying prompts and leveraging the target VLM to generate long responses, thereby creating synthetic training data. Although the draft model could potentially exploit access to the target model's hidden states during training, the randomness in the target model's sampling strategy and our adoption of multi-token prediction, inspired by DeepSeek (21), effectively mitigate this risk.
Our experiments demonstrate that ViSpec significantly outperforms existing speculative decoding methods for VLMs, achieving substantial speedups without compromising generation quality. To knowledge, this work represents the first meaningful acceleration of VLM inference through speculative decoding.
Our main contributions are as follows:
We introduce Vision-Aware Speculative Decoding (ViSpec), speculative decoding frame- work tailored for VLMs.
We propose dual integration cattniimm-attettionnn integration and feature augmentation to enable a small draft model to efficiently incorporate visual context.
We develop a training strategy that extends existing vision-language datasets to include long- response tasks, leveraging multi-token prediction.
We empirically validate ViSpec on four popular VLMs, achieving notable speed improve- ments and establishing the first practical acceleration in this domain.

---

2 Related Work
2.1 Speculative Decoding
Speculative decoding [3] accelerates inference in LLMs by utilizing a smaller draft model to propose candidate token sequences, which the target model verifies in parallel. This method achieves speedups of 3-4x on text-only tasks while preserving output fidelity [13]. Subsequent advancements have refined this approach. Self-speculative decoding (36) derives the draft model from the target model, minimizing training overhead through shared parameters. On-the-fly adaptation methods, such SwiftDecode [33] dynamically adjust the draft model during inference adapt varying input distributions, enhancing robustness across tasks. Specifier (25) employs an ensemble of small models for parallel draft generation, increasing prediction diversity and speedup. Cascude Speculative Drafting [5] uses a sequence of draft models with increasing complexity to balance speed and accuracy, achieving up to 3.5x speedup on large-scale L.LMs. Medusa 2 integrates multiple decoding heads into the target model's 's architecture, eliminating the need for a separate draft model while maintaining comparable performance. The EAGLE series [99 18 20 improves draft predictions by injecting larget-aware hidden states, aligning the draft model closely with the target model's output distribution. A concurrent work, EAGLE-3 [20], adopts a multi-token prediction strategy termed training-time test, though its performance gains depend heavily on scaling training data, with limited improvements when dataset size is fixed. Additionally, SpecTr (29] optimizes token acceptance rates using optimal transport, while REST [10) enhances draft predictions with retrieval-bee based external knowledge, excelling in knowledge-intensive tasks. Recent surveys (34) offer detailed analyses of speculative decoding techniques, discussing their trade-offss and open challenges.
2.2 Vision-Language Models
Vision-language models (VLMs) integrate visual and textual inputs to address tasks such as image captioning, visual question answering, and multimodal dialogue. Recent advancements have led to state-of-the-ar models like LeaV [T3] and Qwen2.5-VL. M. which combine powerful vision encoders, such as CLIP (26). with large-scale LLMs to achieve superior performance across diverse applications. Models like BLIP-- [16] and MiniGPT-4 (38 leverage pretrained vision and language components with learnable interfaces to bridge modality gaps, enabling efficient multimodal processing. However, the computational complexity of processing high-dimensionall image inputs, coupled with the autoregressive nature of text generation, results in significant inference latency, posing challenges for real-time deployment. Efforts to address these issues include efficient vision encoders, such as those proposed in EVA-CLIP (28), and optimized training strategies that reduce memory overhead. Despite these advances, inference efficiency remains a critical bottleneck, motivating the exploration of speculative decoding for VLMs.
2.3 Speculative Decoding for Vision-Language Models
The application of speculative decoding to VLMs an emerging area with limited prior work. The only notable effort, by [8], applied speculative decoding to LLaVA-7B using a small language-only draft model, achieving up 1.5x x speedup. Their experiments with small VLM draft model incorporating an image encoder yielded only marginal gains, highlighting the challenge of effectively processing visual information in the draft model due to the high redundancy and computational complexity of image inputs. These limitations highlight the need for specialized frameworks that can effectively integrate visual and textual information draft models while maintaining high prediction accuracy. Our proposed ViSpec framework addresses these challenges by introducing vision-aware mechanisms to enhance the draft model's s ability to process multimodal inputs efficiently.
Preliminaries
3.1 Speculative Decoding
29 is a lossless acceleration technique for LLMS that alternates between a drafting stage and a verification stage to expedite autoregressive decoding. Let 4; denote the i-th token sequence, and let Tas = (la, latt....... (b) represent token sequence. Given prefix Tiy. speculative decoding proceeds as follows: in the drafting stage, lightweight draft
Speculative decoding 3
3

---

model autoregressively generates sequence of lo tokens, T along with their probabilities ()-((1+ts) for each token Ez+e- In the verification stage, the target model evaluates T Ti+1:ttk- computing its own probabilities ps+i((z+e)- Each draft token lj+s is accepted with probability min (1, (itile1))) If token is rejected, new token is sampled from the normalized distribution norm (max P)++((()++)+ - -)++
and subsequent draft tokens are discarded. This process ensures that the generated sequence maintains the same probability distribution as that produced by the target model without acceleration, preserving distributional consistency. We enhance this approach by adopting the context-awars dynamic draft tree from EAGLE-2 [18), improvement over the draft tree in [25). which enables the draft model to generate multiple candidate tokens per position, facilitating more efficient exploration of the token space.
3.2 Vision-Language Models
typically extend a base large language model (LLM) by incorporating visual information through vision encoder. Formally, given an input image 7, vision encoder & maps it to a sequence of visual embeddings 1 = (T) E Rrxd, where r denotes the number of visual embeddings and d is the embedding dimension. Let & represent the text embedding layer of the LLM. For a multimodal input sequence comprising both visual and textual tokens, the joint input representation is constructed as H ((Ti:) @ Vir + ((T++tis)) where denotes sequence concatenation. The VLM processes this hybrid sequence autoregressively. Notably, the LLM architecture remains unchanged; the only modification is the inclusion of visual embeddings VEr within the input sequence. Since the output space remains the text token vocabulary and the autoregressive generation mechanism is preserved, speculative decoding methods designed for LLMs can, in principle, be directly applied to VLMs by treating visual embeddings as part of the input context. Formally, for any prefix containing visual embeddings Var and text tokens Tiyj, the speculative decoding procedure outlined in Sec. 3.1 remains valid, with probabilities py+s and P5+s implicitly conditioned on Vi.r.
Modern VLMs 15 16
4 Method
4.1
Overcoming Redundancy: Image Embedding Compression
In speculative decoding, the draft model is typically smaller, shallower version of the target model. We demonstrate that single-layer Transformer-based draft model is fundamentally limited in processing long, redundant sequences, particularly when redundant image patches (e.g., uniform color blocks) dominate the input.
Consider sequence of R + 1 image and text embeddings e1, E1...., ER+1 E Rd, where R embeddings are identical, i.e., Er₁ ... erg s, and unique token position has embedding e, The Transformer has single self-attention layer with weight matrices W,, We, W, E Rdxd Ignoring positional encoding and attention scaling for simplicity, the output at position is:
exp (ack.]
y:
where
0gj
(1)
R+1 k=1
exp (1.
with 0 Wyes, ky Wkej, and Uy = Wyey. For the R redundant tokens, we have:
g.k. (Wie)) Tyess
which is identical across all redundant tokens. As R increases, the attention weight to the unique token becomes:
exp (B)
Oii Rexp (A) + exp (B)
(3)

---

dog
LM Head
@ LM Head
69 LM Head
@: Decoder Layers
Decoder Layer
Decoder Layer
MLP
MLP
Eras
8
g,
g
Rx
&
g.
Sin..
fis
Viiion Encoder
8: Embodding
Vision Adaptor
Cx
This
dog.
Target Model
Draft Model
Figure 2: Overview of the ViSpec framework. Given input image and text prompt, ViSpec compresses image tokens using a lightweight vision adaptor to produce a small set of visual tokens. These tokens are prepended the text input and fed into the draft model's attention mechanism. A global visual feature vector, extracted from the compressed image tokens, injected into the draft model's text generation process. The figure illustrates two decoding steps of the draft model, where denotes the target model's last-layer hidden state, I' the draft model's last-layer hidden state, v visual embeddings, text emheddings, compressed image tokens, and g the global visual feature vector.
where Wsess W.T. is the score for redundant tokens, and B Wget WT is the score for the unique token. As R - o0, the denominator is dominated by R Rexp (A), causing Oiaa + 0. Mcanwhile, Oir n2 for each redundant token, the output approximates:
R
Urm mm
R3
Wos,
R
m=1
effectively averaging over the redundant tokens and neglecting the unique token. Furthermore, has been proven theoretically that K + layer network is required to handle nesting complexity of K [30], indicating that shallow draft models struggle to extract useful information from long, redundant image embeddings, thus constraining their effectiveness in speculative decoding for VI.Ms.
The limitations of shallow draft models in processing multimodal sequences necessitate specialized approach for speculative decoding in VLMs. Drawing on insights from [22), which emphasize the critical role of draft token generation speed in achieving end-to-end speedup, propose lightweight Q-Former-inspired [16] vision adaptor (see Fig. 33. This module utilizes a lightweight Transformer encoder with fixed set of learnable query vectors. The visual features extracted from the input image serve as key and value inputs to the Transformer's attention layers, while the learnable query vectors function as queries. Through this attention mechanism, each query vector selectively attends to relevant portions of the visual features, condensing them into small set of compact feature vectors. These vectors, significantly fewer than the original embeddings, act as compressed visual 's attention mechanism, preserving the positional information the original image by maintaining relative spatial locations. By splitting wewessedd-mae--pluee sequence, improve the
embeddings. They are seamlessly integrated into the draft model's
the input into concise image sequence and draft model's efficiency in handling long multimodal sequences.
4.2 Addressing Lost-in-the-Middil Global Visual Feature Integration
While image embedding compression can condense visual tokens into a compact sequence amidst series of tokens, this approach poses challenges speculative decoding, which prioritizes long assistant responses. The lost in the middle effect 223 particularly pronounced in shallow models such as our draft model, causes performance degradation when critical visual information situated in the middle of long contexts, leading to a U-shaped performance curve.
5

---

Jaa
Jmay.
g
Pi
Joss
Pyee
Pyea
Attention
Draft
Draft Model
Drafl
Model
Model
Linear Projection
fsso
Boo
Pun
Pexo
Sean
Emay
Eaay
(a)
(b)
(c)
Figure Architecture the vision adap- lor module. A compact Transformer en- coder with fixed learnable query vectors q processes input visual embeddings v through an attention layer, yielding a small set of compressed image tokens e and a single global visual feature g-
Figure 4: Comparison of training procedures: (a) EA- GLE training, (b) training with greedy target model responses without multi-token prediction, and (c) Vi- Spec training. Here, denotes input embeddings, represents target model hidden states, f indicates EA- GLE draft model hidden states, f' denotes ViSpec draft model hidden states, and p signifies token probabilities.
Although compressed visual embeddings provide compact representation of images for the draft model, they may not fully capture the holistic visual context. As discussed in Sec. 441 simply shallow draft models lack the capacity to
increasing the number of image tokens is suboptimal, effectively attend to lengthy, redundant sequences, Moreover, as generated text sequences lengthen, image tokens become increasingly obscured within the text, exacerbating the lost in the middle effect [23] and undermining the draft model's ability to maintain consistent visual grounding. This often results in reduced coherence between the generated text and the input image. To address these challenges, we propose extracting a global feature vector from the input image and integrating it into each subsequent text token, ensuring persistent access to global visual context throughout the text generation process.
We derive the global feature vector from the final output of the vision adaptor module. This vector is transformed and incorporated into the hidden states of all subsequent text tokens in the draft model. Formally, at each text position i, we compute the augmented hidden state faug as:
aug fe + Wgg.
(5)
where ft original hidden state, 9 denotes the global visual feature vector, and W is a learned projection matrix. This architectural enhancement equips the draft model with continuous visual context, enhancing its ability to generate accurate speculative tokens that maintain strong alignment with the imput image across extended generation sequences.
4.3 Dataset Generation and Training
Training an effective draft model for speculative decoding requires a large, diverse dataset of high- quality target model outputs. For VLMs, this necessitates multimodal datasets with extended assistant responses, which are scarce in the public domain. To address this, we propose a novel data generation strategy that repurposes existing multimodal datasets, even those lacking long responses. We modify prompts datasets such as visual question answering or image captioning to elicit longer, more descriptive responses from the target VLM. For instance, in visual question answering, we rephrase simple questions to request detailed explanations or reasoning. Similarly, for image captioning, we prompt the VLM to produce elaborate descriptions. This approach yields robust synthetic training dataset without requiring manual annotation.
A potential concern is that the draft model might overfit to the target model's 's outputs by exploiting its hidden states during training. However, the randomness in the target model's sampling strategy, driven by a temperature parameter, and the adoption of multi-token prediction,
proposed by DeepSeek (21) mitigate this risk. We illustrate this in Fig. 4 (a) EAGLE's text-only training, the target model is likely to generate "may" instead of "can" when the ground truth is "can." Here, the target model's 's hidden state SHow acts as noisy augmentation, enabling the draft model to learn corrective behavior, feem is conditioned on both HHow and the embedding Ccan In (b) training with

---

greedy target model responses without multi-token prediction, the draft model's inputs (e.g., fHow emay) exhibit a one-to-one correspondence, and the supervision fmay is conditioned solely on fHow (ignoring previous inputs, as I already contains most of the information), effectively reducing to single-step Medusa [2. In (c) ViSpec's 's training procedure, we avoid this issue by using sampling to disrupt one-to-one correspondences between hidden states f and embeddings e. Additionally, we incorporate the draft model's s own hidden states f' as input, which serve a similar corrective role as in (a) when the target model deviates from the ground truth, allowing the draft model to learn self-correction without manually crafted datasets. We optimize the loss:
CrossEntropy(p. Ps).
(6)
where P; and P. denote the target model's and draft model's probabilities for the i-th token, respec- tively.
Experiments
5.1 Experimental Setup
Hardware. All experiments are conducted on a single GPU. Draft models are trained using 8x GPUs.
Models. We evaluate our proposed Vision-Aware Speculative Decoding (ViSpec) framework on four open-source lasoon-anguuae models: LLaVA-v1.66 V..Vivval.- (I5). LLaVA-v1. Vieuna- 13B [[S], Owen2.5-VL--3B-Inste Instruct [1. and Qwen2.5-VL. 11-22---VL-B--Isttuue [1] We use their official weights and
configurations from the Hugging Face Transformers library (32]
Baselines. We compare ViSpec against two established speculative decoding frameworks originally designed for language models: Medusa ( and EAGLE-2 [T8]. To adapt these frameworks for VLMs, we modify their imput pipelines to process image patch embeddings from the VLM's original vision encoder, enabling the draft models to generate speculative tokens conditioned on both visual and textual contexts. This adaptation is feasible as both Medusa and EAGLE-2 rely on general token prediction mechanisms that are theoretically compatible with multimodal sequences, provided the draft model can handle visual inputs.
Training Datasets. We train the draft models for both baselines and ViSpec using two-stage process. Initially, all draft models are trained on the ShareGPT dataset, comprising 68,000 dialogue iterations, establish robust text-based foundation. For multimodal training, we fine-tune the baseline draft models (Medusa and EAGLE-2) on 68,000 samples randomly selected from the LLaVA Visual Instruct Pretrain LCS dataset [15] enabling them process visual inputs. For ViSpec, we augment this dataset with synthetic long assistant responses generated using the target VLM, as described in Sec.
Tasks. We evaluate performance on eight diverse multimodal benchmarks: ScienceQA (SQA) 24 MM- Vet (35, MME 771, TextVQA (27) COCO Captions (COCO Caps) 4 VizWiz 99. GQA
and BEED-Bench [1]. These datasets cover tasks such as visual question answering, image caption- ing, and multimodal evaluation. To ensure generalizability, we use consistent model weights across all tasks without task-specific fine-tuning. Following
[
we design prompts to elicit long, detailed
responses from the models.
Metrics. As ViSpec employs strict speculative decoding to preserve the target model's generation quality, quality evaluation is unnecessary. We focus on acceleration performance, measured using the following metrics:
Average Acceptance Length :: The average number of tokens accepted from the draft model per trafting-verificatin cycle.
Speedup Ratio: The ratio of inference time for standard autoregressive decoding to that for different speculative decoding methods.
5.2 Comparison with Baselines
Table [ and Figure I present a comprehensive evaluation of ViSpec's acceleration performance
compared to Medusa 2 and EAGLE-2 [18 across multiple vision-language models and tasks. The
7

---

Table I: Speedup ratios and average acceptance lengths for different methods. For fair comparison, we do not relax the draft token acceptance condition of Medusa under non -nneddy settings as proposed in the original paper, instead, adopt the same acceptance condition EAGLE-2. Speedup ratios are computed based on the average time required to generate each token.
SQA
Ve
TexVQA
MME
COCO Caps
VizWiz
GQA
BeED.Bench
Avg.
Model
Method
Speedup
Speedup r Speedup Speedup 7 Speedup Speedup
Speedup
Speedup
Speedup
Temperature=0
Medusa 0.72 1.4xx 0.73
1.42x 0.77 1.46x 0.70 1.41x 0.66
1.6xx 0.76
1,38x
0.73
1.29k
0.72
1,38x
0.72
1.42x
LLaVA-16
2.44
2.14x 0.63
1,48x
0.63
1.25x
1.25
1688
1.24
1880x
1,15
1.40%
1.74
1.64x
1.40
1.59x
1,31
1.62x
7B
ViSpec 2,86
2.52x
2,95
2.90x
2.55x
3.30
3.22x
3,16
2,88
2.22x
3.03
2.22x
2.98
2.58x
Medusa 0.84 1.61x 0.80
1.47x
0.89
1,51x
0.79
1.47x 0.75
1.48x
0.81
1.45%
0.85
1.45%
0.82
1.40x
0.82
1.48%
LLaVA-16
EAGLE-22.02
2.12x 1.64
1.59
1.71
1.91x
181
1.85x
1.83
2,01x
1.98
1.90s
2.10
1.82x
2.03
1.66x
1.89
1.86x
13B
ViSpec 2.76 2.57x
2.73
2.34x
2,78
2.43x
2.78
2.36x
3.18
2.82x
2.93
2.26x
2.95
2.12x
3.04
2.16x
2.89
2.38x
Medusa
0.57 1.07x
1.12x
0.66
1,88x
0.62
1.21x
0.65
1.21%
0.61
1.15k
0.61
1.14x
Qwen2.5- EAGLE-2 1.18 1.41x 1.03
1,30x
0.98
1.26X
1.07
1.38%
1,40
1.60x
1.11
1.32x
1.39
1.52x
1.11
1.32x
1.16
1.39x
3B
ViSpec 1.99 188% 2,13
1.81x
2.15
1.85%
1.96
1.82x
2.37
2.15%
2.22
1.71x
2.28
2.01x
2.37
1.78x
2.19
1.87%
Medusa
0.60 1.13x 0.59
1.06x
0.58
1.05x
0.59
1.19
0.61
1.11x 0.59
1.0%
0.64
1.19k
0.62
1.08x
0.60
1.11x
Quen2.5 RAGLE22 1.40 1.49x
1.19
1,36x
1.14
1.23x
1.29
1.54x
1.46
1.50x
1.27
1.20x
1,53
1.54x
1.42
1,32x
1.34
1.40x
7B
ViSpec 2.19 1.84x
1.74x
2.21
1.72x
2.27
199x
2.30
1.918
2.34
1.55x
2.24
1.80x
Temperature=:
Medusa 0.58 1,36x 0.58
1.37x 0.57
1,32x 0.56
1.35x 0.58
1.67x 0.57
1.29%
0.60
1.19%
0.59
1,32x
0.58
1,36x
LLaVA-16 EAGLE-2
1.78 2,17x
0.51
1.34x
0.41
1.11x
1.02
1.53x
1,03
1.78x
0.77
1.32%
1.33
1.47%
0.98
1.57x
0.98
1.54x
7B
ViSpec 2,06
2.20x
1.9%
1.78
1,93x
2.36
3.05X
2.32
1.83%
2.16
1,94k
2.09
2.14x
Medusa
0.68 1.41x 0.67
1.44x 0.66
1.42x
0.66
1.40x
0.67
1.40x 0.64
1.37x
0.70
1.37x
0.68
1.37x
0.67
1.40s
LLaVA-16
EAGLE-2 1.51 1.98x 1.29
1.73x
1.26
1.72x
1.45
1.78x
1.54
1.83x
1.46
1.72x
1.64
1.73x
1.60
1,79x
1.47
1.79x
13B
ViSpec 2.02 2.25x 1.98
2.15x
1.90
2,08x
2.07
2,08x
2.43
2.39%
2.04
2.01x
2.19
2.03x
2.22
2.07x
2.11
2.13x
Medusa 0.52 1.02x.
1.02s
0.46
0.99x
0.51
L.03x
0.46
0.55
1.13%
0.49
1.03K
0.49
1.03x
Qwen2.5- EEALE-20099 0.92 1.25x 0.70
1.19x
0.70
1.06x
0.84
1.26x
0.97
1.28%
0.84
1.19k
1.02
1.31x
0.86
1.16x
0.86
1.21x
3B
ViSpec 1.49 1.49% 1.23
1.39%
1.32
1.38%
1.45
1.58x
1.42
1.50%
1.39
1.43%
1.49
1.59x
1.55
1.42x
1.42
1.47%
Medusa 0.56 1.05x. 0.51
0.95x
0.49
0.96x
0.51
1.02x
0.52
1.00x
0.50
1.02x
0.53
1.02x
0.53
1.02x
0.52
1.01x
Quen2 BAGLE-2
1.19 1,52x 0,92
1.19%
0.88
1.08x
1.00
1.23x
1.08
1.22x
0.94
1.13x
1,11
1.32x
1.04
1.19x
1.02
1.18x
7B
ViSpec 1822
166x
1.47x
1.51
1.37x
1.63
1.50x
1.61
1.56x
1.70
1.38X
1.66
1.49x
table reports the average acceptance length r and speedup ratios, calculated the ratio of the average time required for standard autoregressive decoding to that of each method per token, under two temperature settings (0 and 1).
ViSpec consistently outperforms both Medusa and EAGLE-2 across all evaluated tasks and models, achieving the highest speedup ratios and values. For instance, at temperature 0 with LLaVA-v Vicuna-7B, ViSpec achieves a speedup of 2.90x on TextVQA, surpassing EAGLE-22 (1.25 )) and Medusa (1.46x) by wide margin. Similarly, with LLaVA- Vicuna- at temperature O, ViSpec delivers a speedup of 2.57x on ScienceQÁ, compared to EAGLE-2 (2.12x) and Medusa (1.61 x). At temperature ViSpec maintains its advantage, achieving speedup of 2.25x x on ScienceQA with LaaVA-vl.--Vicunaa 13B, outperforning EAGLE-2 (1.98x) ) and Medusa (1.41 x). These results underscore ViSpec's superior acceleration capabilities, with speedup ratios ranging from 1.37x to 3.22x, compared to EAGLE-2 (1.06x to 2. 17) ) and Medusa (0.95x to 1.67x).
ViSpec demonstrates robust performance and generalizability across diverse set of tasks, with notably high acceptance lengths on TextVQA, VizWiz, SEED-Bench, and COCO Captions. This suggests that its vision-aware approach effectively handles the varied sequential patterns inherent in these tasks. In contrast, the performance of EAGLE-2 and Medusa is more task-dependent. While they perform adequately on tasks like ScienceQA, they struggle on others, such as TextVQA and MM-Vet. Vet, particularly when compared to ViSpec. This indicates that their general purpose draft mechanisms may not adapt effectively to the complexities of visual-linguistic sequences.
Performance also varies across model architectures. V.L. models generally achieve higher speedup ratios and acceptance lengths compared to QW models. Such differences can be
8

---

attributed to the significantly larger vocabulary sizes of Qwen models, potentially increasing the complexity of token prediction.
5.3
Ablation Studies
Impact of Compressed Image Embedding Count. We evaluate the effect of varying the number of compressed image embeddings from to 64 on ViSpec's ' performance, with results shown in Tab. When the number remains significantly smaller than the original thousands of image embeddings, increasing the count has minimal impact on the average acceptance length т. However, reduces the speedup ratio due to the increased computational load the draft model during token generation. A single compressed image embedding adequately captures essential visual information, prompting us adopt one compressed embedding in our final implementation.
Table Impact of varying the number of compressed image embeddings on ViSpec's performance across three datasets, measured by average acceptance length and speedup ratio.
COCO Captions Speedup
GQA
MME
Image Embeddings
Speedup
Speedup
3.30
3.22x
2.88
2.22x
2.84
2.55x
4
3.24
3.24x
2.84
2.24x
2.74
2.35x
16
3.23
3.21x
2.84
2.20x
2.76
2.38x
64
3.25
2.71x
2.86
1.91x
2.76
2.42x
Table Ablation study on the effectiveness of ViSpec's components across three datasets, measured by average acceptance length and speedup ratio, with EAGLE-2 as the baseline.
COCO Captions Speedup
GQA
MME
Components
T
Speedup
Speedup
bascline
1.24
1.80x
1.74
1.64x
1.25
1.68x
+image embedding compression
2.04
2.37x
2.15
1.92x
2.04
1.83x
+global visual injection
2.14
2.42x
2.25
2.03x
2.14
1.95x
+dataset generation
3.30
3.22x
2.88
2.22x
2.84
2.55x
Effectiveness of Each Component. We conduct an ablation study to assess the contribution of ViSpec's core components: image embedding compression, global visual feature injection, and dataset generation. Using EAGLE-2 [8] as the baseline, we report the average acceptance length and speedup ratio across the COCO Captions, GQA, and MME datasets, shown in Tab. 3 Adding image embedding compression increases the speedup ratio by up to 30%, enabling the draft model to efficiently process visual information. Incorporating global visual feature injection further improves speedup by 7%, underscoring its role in maintaining persistent visual context and enhancing multi- modal coherence. The inclusion of dataset generation yields additional 30% speedup, equipping the draft model to handle extended multimodal sequences effectively. Together, these components synergistically enhance ViSpec's acceleration performance while ensuring robust performance across diverse tasks.
Vision Adaptor Overheads. While the vision adaptor increases the draft model's parameter count, theoretically reduces the prefill computation by processing fewer visual tokens. However, as draft models are small and efficient, observe no statistically significant change in prefill latency (Tab, 4 The minor variations recorded are attributed to measurement noise.
Output Length vs. Speedup. Table 5 illustrates the relationship between the average output length and the achieved end-to-end speedup across various datasets. As expected, longer generation sequences generally yield higher speedup ratios, since they offer more opportunities for successful draft model predictions. Despite this trend, our method demonstrates robust performance, providing significant acceleration even datasets characterized by shorter responses.

---

Table 4: Analysis of vision adaptor overheads during the prefill stage on the COCO Captions dataset.
Table 5: Relationship between
output length and speedup ratio.
Params (M)
GFLOPs
Latency (s)
Dataset
Tokens
Speedup
Model
Base
+Adap
Base
+Adap
Base
+Adap
GQA
46.25
2.22x
SEED-Bench
57.66
2.22x
LLaVA-1.6 7B
367
451
956
179
0.227
0.231
SQA
74.07
2.37x
LLaVA-1.6 13B
534
665
1460
279
0.334
0.334
VizWiz
105.91
2.26x
Qwen2.5-VL 3B
404
425
57.3
18.3
0.002
0.004
MME
115.01
2.55x
Qwen2.5-VI.7B
826
890
172
55.5
0.018
0.016
Vet
171.13
2.52x
COCO Caps
236.04
3.21x
Tex:VQA
353.58
2.90x
6
Conclusion
We introduce Vision-Aware Speculative Decoding (ViSpec), the first framework to achieve signifi- cant acceleration for vision-language models (VLMs) through speculative decoding. By integrating compressed image emheddings, persistent global visual feature injection, and synthetic long-response dataset generation, ViSpec addresses key limitations in processing multimodal sequences with shallow draft models. Our experiments demonstrate speedups of up to 3.22 across diverse VLMs and tasks, establishing ViSpec as a pioneering solution for multimodal inference acceleration. Despite this breakthrough, ViSpec's absolute speedup trails state-of-the primary avenues for improvement: first, curating nigher-quality multimodal training datasets with greater conversational depth to enhance the draft model's predictive accuracy; second, optimizing vision encoder architectures, potentially via dynamic patch reduction or neural compression, to reduce visual processing overhead. These advancements, coupled with hardware-awwe aware kernel optimizations, could bridge the performance gap between multimodal and text-only speculative decoding, enabling real-time deployment of advanced VL.ss.
text-only methods, We identify two
References
[1] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang. Shijie Wang, Jun Tang, et al. Qwen2.5-vl technical report. arXiv preprint arXiv:2502.13923, 2025.
[2] Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D Lee, Deming Chen, and Tri Dao. Medusa: Simple Ilm inference acceleration framework with multiple decoding heads, In International Conference on Machine Learning, pages 5209-5235. PMLR, 2024.
[3] Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, -apttitt Lespiau, Laurent Sifre, and John Jumper. Accelerating large language model decoding with speculative sampling. arXiv preprint w.Xii:23022.0333, 2023.
[4] Xinlei Chen, Hao Fang, Tsung- Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv: 55.5.00.03555 2015.
[5] Ziyi Chen, Xiaocong Yang, Jiacheng Lin, Chenkai Sun, Kevin Chang, and Jie Huang. Cascade speculative drafting for even faster Ilm inference. Advances in Neural Information Processing Systems, 37:86226-86244 2024.
[6] Muhammad Iqbal Hasan Chowdhury, Kien Nguyen, Sridha Sridharan, and Clinton Fookes. Hierarchical relational attention for video question answering. 2018 25th IEEE International Conference on Image Processing (ICIP), pages 599-603. IEEE, 2018.
[7] Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, et al. Mme: A comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306 13394, 2023.
[8] Mukul Gagrani, Raghavv Goel, Wonseok Jeon, Junyoung Park, Mingu Lee, and Christopher Lott. On speculative decoding for multimodal large language models. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages $285-8288 2024.
10

---

[9] Danna Gurari, Qing Li, Abigale J Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In Proceedings of the IEEE conference computer vision and puttern recognition, pages 3608 3617, 2018.
[10] Zhenyu He, Zexuan Zhong, Tianle Cai, Jason Lee, and Di He. Rest: Retrieval-based speculative decoding. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies (Volume I: Long Papers), pages 1582-1595, 2024.
[11] Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 6700-6709, 2019.
[12] Minjae Lee, Wonjun Kang, Minghao Yan, Christian Classen, Hyung Koo, and Kangwook Lee. In-batch ensemble drafting: Toward fast and robust speculative decoding for multimodal language models. OpenReview.ner, 2024.
[13] Yaniv Leviathan, Matan Kalman, and Yossi Matias, Fast inference from transformers via speculative decoding. International Conference on Machine Learning, pages 19274-19286. PMLR, 2023.
[14] Bohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang, Rui Wang, Ruimao Zhang, and Ying Shan. Seed-bench: Benchmarking multimodal large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13299-13308, 2024.
[15] Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan Li. eentteeet-interleaaee:: Tackling multi-image, video, and 3d in large multimodal models. CoRR, 2024.
[16] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19730-19742. PMLR, 2023.
[17] Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang. Jilan Xu, Guo Chen, Ping Luo, al. Mybench: A comprehensive multi-modal video understanding benchmark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22195-222000 2024.
[18] Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. Eagle-2: Faster inference of language models with dynamic draft trees. In Proceedings of the 2024 Conference Empirical Methods in Natural Language Processing, pages 7421-7432, 2024.
[19] Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. Eagle: speculative sampling requires rethinking feature uncertainty. Proceedings of the 41st International Conference on Machine Learning, pages 29948, 2024.
[20] Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. Eagle-3: Scaling up inference acceleration of large language models via training-time time test. arXiv preprint arXiv:2503.01840. 2025.
[21] Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseck-v3 technical report. arXiv preprins arXiv:2412.19437, 2024.
[22] Fangcheng Liu, Yehui Tang, Zhenhua Liu, Yunsheng Ni, Kai Han, and Yunhe Wang. Kangaroo: Lossless self-speculative decoding via double early exiting. arXiv preprint arXiv:2404. 18911, 2024.
[23] Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12, 2024.
11

---

[24] Pan Lu, Swaroop Mishra, Tony Xia. Liang Qiu, Kai- Wei Chang, Song- Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In The 36th Conference on Neural Information Process- ing Systems (NeurIPS), 2022.
[25] Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao Cheng, Zeyu Wang, Zhengxin Zhang, Rae Ying Yee Wong, Alan Zhu, Lijie Yang, Xiaoxiang Shi, et al. Specinfer: Accelerating large language model serving with tree-based speculative inference and verification. In Proceedings of the 29th ACM International Conference Architectural Support for Programming Languages and Operating Systems, Volume 3. pages 332-949, 2024.
[26] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8763. PmLR, 2021.
[27] Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings of the IEFE/CVF conference on computer vision and pattern recognition, pages 8317-8326, 2019.
[28] Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang. and Yue Cao. Eva-clip: Improved training techniques for clip at scale. arXiv preprint arXiv:2303.15388 2023.
[29] Ziteng Sun, Ananda Theertha Suresh, Jae Hun Ro, Ahmad Beirami, Himanshu Jain, and Felix
Yu. Spectr: Fast speculative decoding via optimal transport. Advances in Neural Information Processing Systems, 36:30222-30242. 2023.
[30] Mingze Wang and E Weinan. Understanding the expressive power and mechanisms of trans- former for sequence modeling. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
[31] Wenbin Wang. Liang Ding, Minyan Zeng, Xiabin Zhou, Li Shen, Yong Luo, Wei Yu, and Dacheng Tao. Divide, conquer and combine: A training-free free framework for high-resolution image perception in multimodal large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 7907-7915, 2025.
[32] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the- art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online, October 2020. Association for Computational Linguistics.
[33] Heming Xia, Yongqi Li, Jun Zhang, Cunxiao Du, and Wenjie Li. Swift: On-the-fly fly self- speculative decoding for Ilm inference acceleration. arXiv preprint arXiv:2410.06916 2024.
[34] Heming Xia, Zhe Yang, Qingxiu Dong, Pciyi Wang, Yongqi Li. Tao Ge, Tianyu Liu, Wenjie Li, and Zhifang Sui. Unlocking efficiency in large language model inference: A comprehensive survey of speculative decoding. In ACL (Findings), 2024.
[35] Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. In International conference on machine learning. PMLR, 2024.
[36] Jun Zhang, Jue Wang, Huan Li, Lidan Shou, Ke Chen, Gang Chen, and Sharad Mehrotra. Draft & verify: Lossless large language model acceleration via self-speculative decoding. CoRR, 2023.
[37] YiFan Zhang, Huanyu Zhang, Haochen Tian, Chaoyou Fu, Shuangqing Zhang, Junfei Wu, Feng Li, Kun Wang, Qingsong Wen, Zhang Zhang, et al. Mme-realworld: Could your multimodal 11m challenge high-resolution real-world scenarios that are difficult for humans? In The Thirteenth International Conference Learning Representations, 2025.
12

---

[38] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En- hancing vision-language understanding with advanced large language models, In The Twelfth International Conference on Learning Representations, 2024.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction outline the proposed ViSpec framework, its contributions (visio--aware speculative decoding, image token compression, and dataset generation), and its scope (accelerating VLM inference).
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
See Sec. i
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-specfiiaatinn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
14

---

Answer: [Yes]
Justification: See Sec. 4.1
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: See Sec. 5.1
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
15

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: Code will be made available upon acceptance, and we use only publicly available datasets.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
Polooyoooolyyoooooooll for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips.ccpublle pwwpip/guuuiieeeeep Sudi:///uuiieeeeeeeuuuubiiii
(CodeSubiisss wwtplubmmmssiiiii ssionpolccy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: See Sec.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[NA]
Justification:
We fix the random seed for all experiments, so the results are deterministic.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for
example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
16

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: See Sec. 5.1
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make it into the paper).
9. Code ethics
Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Question:
Answer: [Yes]
Justification: The research does not involve human subjects, crowdsourcing, or sensitive data. There is no indication of violating the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our work presents technical advancement for accelerating inference in VLMs without modifying their outputs capabilities. The acceleration method itself does introduce new societal risks beyond those already present in the underlying models.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
17

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We do not release any new models or datasets.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We use only publicly available datasets, and properly credit the original authors.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
18

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssit
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The answer NA means that the paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
19

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [Yes]
Justification: The paper uses VLMs (LLaVA, Qwen2.5-VL) to generate synthetic training
data with long responses (Sec. 4.3 This usage clearly described.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM for what should or should not be described.
20

---

A Implementation Details
Vanilla. We utilize models from the Hugging Face Transformers library with the PyTorch backend and pre-allocated KV cache. All other methods build upon these models.
Medusa. We implement I-layer, 3-head Medusa model, adhering to its default configuration. For training on both text-only and vision-language datasets, we use learning rate of 3e-5, a batch size of 8, and the A optimizer. The model is trained for 20 epochs with a 1-epoch warmup and linear learning rate decay. We set a maximum sequence length of 2048 for both dataset types. For inference, adopt EAGLE-2's draft tree structure, configuring total of 30 draft tokens, tree depth of 3, and selecting nodes during the expansion phase across all models and tasks.
EAGLE-2. We employ 1-layer EAGLE-2 model, following its default settings. Training on text-only and vision-language datasets uses a learning rate of 3e-5, a batch size of 8, and the Adam optimizer. The model is trained for 20 epochs with a 1-epoch warmup and linear learning rate decay, with a maximum sequence length of 2048 for both dataset types. For inference, we use EAGLE-2's draft tree with 30 draft tokens, tree depth of 3, and 8 nodes selected during expansion, applied uniformly across models and tasks.
ViSpec. We implement single-layer draft model that mirrors a decoder layer of the target model. For training on text-only and vision-language datasets, we use a learning rate of 3e-6, batch size of 8, and the AdamW W optimizer. The model is trained for 20 epochs with a 1-epoch warmup and linear learning rate decay, supporting a maximum sequence length of 2048 for both dataset types. During inference, we adopt EAGLE-2's draft tree structure, configuring 30 draft tokens, tree depth of 3, and selecting 8 nodes during expansion, applied consistently across all models and tasks.
Generation Prompts. For training dataset generation, we append the prompt "Please answer with at least 1000 words." " to each sample to elicit long responses. For inference, we use task-specific prompts to encourage detailed responses. For visual question answering (VQA) tasks, the prompt is: "Please answer with an explanation." " For optical character recognition (OCR) tasks, the prompt is: "Perform an OCR task on the provided image. Extract the text accurately and provide detailed explanation of the process. Ensure the response comprehensive and well-structured"" For captioning tasks, the prompt is: "Provide detailed description of the given image." For ScienceQA, we use its official chain-of-thought prompt to generate the answer, followed by the lecture and explanation (QCM-ALE).
B Additional Experiments
B.1
Experiments on High- Resh-ueeool Datasets
We conducted experiments on high-resolution datasets 31 37]. where ViSpec continues to demon- strate strong performance. Table 6 compares ViSpec against the EAGLE-2 baseline using both LLaVA-1.6 7B and Qwen2.5-VI. 7B.
Table 6: Performance on high-resolution datasets, comparing average acceptance length and speedup ratio.
Model
Dataset
Method
Speedup
EAGLE-2
1.43
1.52x
HR-Bench 4K
ViSpec
2.86
1.93x
LLa 7B
EAGLE-2
1.42
1.75x
MME-RealWorld
ViSpec
2.85
2.35x
EAGLE-2
0.34
0.90x
HR-Bench 4K
ViSpec
2.16
1.29x
Qwen2.5-VL 7B
EAGLE-2
0.52
0.95x
MMF-ReellWrld
ViSpec
2.11
1.37x
Notably, Qwen- VL does not cap its input image token count, resulting a longer prefill time for high- resolution datascts. Since speculative decoding accelerates only the decoding stage, this extended
21

---

prefill duration reduces the overall speedup ratio. However, ViSpec's robust average acceptance length 7 indicates that the decoding phase itself still effectively accelerated.
B.2 Experiments with Temporal Data
In principle, ViSpec could be more effective for video inputs, as videos contain temporal redundancy in addition the spatial redundancy found in static images. From input processing standpoint, this task is not fundamentally different from handling image patches, as video inputs are typically processed sequence of frame embeddings. To test this hypothesis, we apply our draft model, which was trained exclusively on static image data, directly to video tasks without fine-tuning.
For this preliminary experiment, compress each video frame into single embedding, average their global features, and evaluate the Qwen2.5-VL. 7B model on the MSVD-QA (6 and MVBench [17] datasets. MSVD-QA is video question-answering task, while MVBench is benchmark evaluating temporal understanding across 20 different tasks. We limit the input frames, as processing more would lengthen the prefill time, thereby reducing the speedup gained from the accelerated decoding stage. The results are presented in Tab.
Table Performance of Qwen2.5-VL 7B on video datasets, comparing average acceptance length and speedup ratio.
Dataset
Method
Speedup
EAGLE-2
1.10
1.22x
MSVD-QA
ViSpec
2.16
1,46x
EAGLE-2
0.83
0.83x
MVBench
ViSpec
2.09
1.32x
The results demonstrate that ViSpec achieves notable speedup even without video-specific training. Developing dedicated framework optimized for video data remains a promising direction for future work.
22


Reviews:
[{"review_id": "qjztfjG19A", "reviewer": "Reviewer_3b1n", "summary": "This paper proposes the ViSpec framework to address the slow inference speed of vision-language models (VLMs). It accelerates inference through a specially designed vision-aware speculative decoding technique. \nViSpec employs a lightweight visual adapter to compress image information and injects global visual features to maintain multimodal coherence. This design thereby tackles the challenge of redundant visual inputs that small draft models struggle to handle. \nExperiments demonstrate that ViSpec achieves speedup on various VLMs without compromising much generation quality.", "questions": "Please polish the writing and complemente more analysis and experiments.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n- This framework is specifically designed to address the unique challenges of VLMs. It handles redundant visual information through image embedding compression.\n\n- This paper is easy to follow. Experiments show that ViSpec consistently outperforms baseline methods such as Medusa and EAGLE-2 across all tested models and tasks. ViSpec demonstrates strong effectiveness on four mainstream VLMs and eight different multimodal benchmarks. \n\n- ViSpec introduces an innovative data generation approach to overcome the data limitation. \n\nWeaknesses\n- The \"experimental setup\" section is poorly written, with low information density and inappropriate sectioning.\n\n- The \"Global Visual Feature Integration\" is not novel; a similar idea already exists in \"ContextNet: Improving Convolutional Neural Networks for Automatic Speech Recognition with Global Context\". The difference exists in different backbones. \n\n- The paper claims that using only one compressed image embedding is sufficient to capture the necessary visual information but does not provide adequate justification. The compression might lead to the inability to recognize fine-grained small objects, but the paper did not test corresponding ability.", "was_revised": true, "final_justification": "The authors have clarified my concerns, and the reviewer is willing to raise the score to 4.", "rebuttal": "We sincerely thank you for your detailed feedback and constructive suggestions. We address your points below.\n\n### 1. Writing and sectioning issues\n\nWe will refine the writing and consolidate these subsections in the final version. If you have more specific suggestions for improvement, we would be very grateful to hear them.\n\n### 2. Architecturally analogous “Global Visual Feature Integration”\n\nThe concept of a \"global feature\" is widely applied in deep learning. Its core idea is not only adopted in *ContextNet* [1] but also well-established in numerous other influential works, such as *Squeeze-and-Excitation Networks* for channel-wise feature recalibration [2], the `[CLS]` token in *BERT* for sentence-level representation [3], and the class token in *Vision Transformers* for image-level classification [4]. However, the mere presence of a global feature is not where the novelty lies. The crucial aspects are the specific **motivation** for its use and more importantly the **method** by which it is generated and integrated. In this regard, our Global Visual Feature Integration differs significantly from ContextNet.\n\n* **Motivation**:\n    ContextNet introduces the global context vector to address the limited receptive field of CNNs, which struggle with long-range dependencies in speech signals. Authors of ContextNet note that this is *NOT* considered a problem for architectures like RNNs or Transformers. In contrast, as we argue theoretically in Section 4.1, the core issue we address is the inherent deficiency of *shallow* Transformers (draft models) in processing long and redundant sequences, particularly a large number of image tokens. To the best of our knowledge, we are the first to identify this specific challenge. This insight leads to our proposal to compress image tokens. Consequently, the reduced number of image tokens relative to text tokens introduces the risk of the draft model \"forgetting\" the visual context. The Global Visual Feature Integration is therefore needed to mitigate this risk.\n\n* **Method**:\n    ContextNet generates a global context vector from a convolutional block's output using a Squeeze-and-Excitation (SE) module. This vector is then used to perform feature recalibration by multiplicatively rescaling the local feature maps. The global feature, therefore, modulates the local information. We derive the global visual feature vector directly from our vision adaptor module, which uses learnable queries to summarize the image. We then integrate this feature by additively injecting it into the hidden state of every subsequent text token via a learned projection. This ensures that a constant, global representation of the image is available at every step of the generation, acting as a continuous anchor to the visual input.\n\nIn summary, while both approaches use a \"global feature,\" they do so for different reasons (CNN limitations vs. draft model *forgetting*) and with different mechanisms (SE module vs. vision adaptor) tailored to their unique problems. Moreover, as we demonstrate in our ablation study in Section 5.3, our framework significantly outperforms prior work even **without** Global Visual Feature Integration, highlighting the contribution of our vision-aware token compression strategy.\n\n### 3. Recognizing fine-grained objects with compressed embeddings\n\nA key principle of speculative decoding is that the performance of the draft model affects only the **acceleration speed**, *NOT* the **final output quality**. If the draft model fails to recognize a fine-grained object, its prediction is simply rejected and corrected by the target model. The output distribution is guaranteed to be identical to that of the target model decoding alone.\n\nIn Table 2 of our main paper, we have already presented experiments on three diverse datasets (COCO Captions, GQA, and MME), varying the number of compressed image embeddings from 1 to 64. These results consistently show that increasing the number of embeddings beyond 1 does not yield meaningful gains in speedup, which demonstrates that a single compressed image embedding is often sufficient for answering common visual questions, including relatively fine-grained ones such as, “What is the short person holding?” (an example from GQA) .\n\nTo further address the reviewer’s concern about fine-grained tasks, we conduct additional experiments on the more fine-grained OCR dataset *SynthDoG EN* [5]. The results below show the performance of LLaVA-1.6 7B with greedy decoding as we vary the number of compressed image embeddings.\n\n| Compressed Image Embeddings | Mean Acceptance Length | Speedup Ratio |\n| :-------------------------: | :--------------------: | :-----------: |\n|              1              |          1.98          |     1.97x     |\n|              4              |          2.02          |     1.93x     |\n|             16              |          2.00          |     1.92x     |\n|             64              |          1.99          |     1.90x     |\n|              -              |          1.05          |     1.45x     |\n\nWe observe a performance drop on this benchmark compared to less fine-grained tasks like GQA (from 2.22x down to 1.97x), but our method still significantly outperforms the EAGLE-2 baseline, which uses no compression (last row). While increasing the number of image embeddings from 1 to 4 slightly increases the mean acceptance length (τ), this gain is offset by the additional computational cost, resulting in a decreased end-to-end speedup.\n\nWe believe it is counterproductive to force a shallow draft model to capture every fine-grained detail. Its purpose is to generate **plausible** token sequences for **the majority of cases** to maximize acceleration. The more difficult cases are efficiently handled by the powerful target model, which is the essence of speculative decoding.\n\n\n[1] W. Han, et al. \"ContextNet: Improving Convolutional Neural Networks for Automatic Speech Recognition with Global Context.\" Proc. Interspeech, 2020.\n\n[2] J. Hu, L. Shen, and G. Sun. \"Squeeze-and-Excitation Networks.\" CVPR, 2018.\n\n[3] J. Devlin, M.W. Chang, K. Lee, and K. Toutanova. \"BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.\" NAACL, 2019.\n\n[4] A. Dosovitskiy, et al. \"An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale.\" ICLR, 2021.\n\n[5] G. Kim, et al. \" Donut: OCR-Free Document Understanding Transformer.\" ECCV, 2022."}, {"review_id": "4WZwydhfRp", "reviewer": "Reviewer_DJLc", "summary": "- This paper introduces Vision-Aware Speculative Decoding (ViSpec) to achieve acceleration for vision-language models (VLMs) through speculative decoding.\n\n- By integrating compressed image embeddings, persistent global visual feature injection, and synthetic long-response dataset generation, ViSpec addresses key limitations in processing multimodal sequences with shallow draft models.\n\n- The experiments demonstrate speedups across diverse VLMs and tasks.\n\n- This paper identifies two primary avenues for improvement: first, curating higher-quality multimodal training datasets with greater conversational depth to enhance the draft model’s predictive accuracy; second, optimizing vision encoder architectures, potentially via dynamic patch reduction or neural compression, to reduce visual processing overhead.", "questions": "- Most multimodal benchmarks only require a single-word response, such as 'yes' or 'no' for MME and 'A', 'B', 'C', 'D' for multi-choice QA.\nThus, it is unclear why there are more than 2x-3x speedups for ViSpec in those benchmarks.\n\n- Lack of experiments in high-concurrency settings. The throughput for large batch sizes is unknown.", "limitations": "yes.", "rating": 2, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths \n\n- This paper hypothesizes that large VLMs can effectively filter redundant image information layer by layer without compromising textual comprehension, whereas smaller draft models struggle to do so.\n\n- This paper introduces Vision-Aware Speculative Decoding (ViSpec). ViSpec employs a lightweight vision adaptor module to compress image tokens into a compact representation.\n\n- This work extracts a global feature vector for each input image and augments all subsequent text tokens with this feature to enhance multimodal coherence.\n\n- This work curates a specialized training dataset by repurposing existing datasets and generating extended outputs using the target VLM with modified prompts.\n\nWeaknesses\n\n- The vision adaptor is heavily inspired by BLIP-2's Q-Former, and the global feature injection resembles EAGLE's target-aware mechanisms.\nThe incremental combination isn't sufficiently differentiated from prior work.\n\n- High speedups on COCO Captions but low speedup on TextVQA. The paper doesn't analyze the inconsistent gains.\n\n- Recent VLMs handle video and 3D input, but ViSpec is only evaluated on static images. Compression may fail for temporal data.", "was_revised": true, "final_justification": "Thanks for your response. I have read the rebuttal and other reviews. While I appreciate your response, I would like to keep my scores.", "rebuttal": "We appreciate you taking the time to review our paper and providing this valuable feedback. We address your specific concerns below.\n\n### 1. Novelty of this work against BLIP-2 and EAGLE\n\nOur primary contribution is the identification and analysis of a fundamental bottleneck in **applying speculative decoding to Vision-Language Models**, an issue *unaddressed* by prior work like **BLIP-2 and EAGLE**. We subsequently introduce a comprehensive solution to tackle this challenge: a lightweight Vision Adapter to compress redundant visual information and a Global Feature Injection mechanism to combat the *lost-in-the-middle* effect. These are complemented by novel dataset generation and training strategies designed to produce high-quality, long-form training data and to train a robust, effective draft model.\n\nIn **Section 4.1**, we provide a theoretical analysis from the unique perspective of speculative decoding. We demonstrate how the performance of a **shallow draft model** degrades when processing the long and highly redundant sequences typical of image patch embeddings. Our analysis reveals that as the number of redundant visual tokens increases, a shallow Transformer's attention mechanism becomes saturated, effectively averaging over the redundant inputs while ignoring unique and critical information. This leads to our central argument: **for speculative decoding to be effective in a multimodal context, this redundant visual information should be compressed before it is processed by the draft model**. This core insight directly motivates our structural solution: the vision adaptor for compression and the global feature injection for maintaining context.\n\n### 2. Performance gap between COCO Captions and TextVQA\n\nThe difference in speedup between COCO Captions (3.22x) and TextVQA (2.90x) likely stems from the differing granularity of these two tasks. COCO Captions require a holistic, overall description of the image, a task for which our compressed visual representation is highly effective for the draft model. In contrast, TextVQA is a fine-grained task that demands focusing on specific text within the image, token compression may reduce the draft model's ability to predict these fine-grained details, lowering the acceptance rate. This does not affect the final output quality, as the target model makes corrections, but it does logically result in a lower speedup ratio.\n\nAnother factor that can contribute to varying speedups across different benchmarks is the length of the generated output. This occurs because speculative decoding accelerates only the **decoding** stage, not the **prefilling** stage. Since the overall speedup calculation includes the fixed prefilling time, tasks that produce longer sequences see a greater impact from acceleration. For instance, on the GQA dataset, where the average response is short (46 tokens), we observe a 2.22x speedup. In contrast, on MME, with longer responses (115 tokens), the speedup increases to 2.55x. For longer outputs, the accelerated decoding phase constitutes a larger proportion of the total inference time, naturally leading to a higher end-to-end speedup ratio.\n\n### 3. Experiment with Temporal Data\n\nIn principle, our method could be even *more* effective for video inputs, as video data contains additional temporal redundancy compared with static images. Since video inputs are typically processed as a sequence of embeddings, the problem is not fundamentally different from handling image patches. To test this hypothesis, we apply our draft model, which was trained purely on static image data, directly to video tasks. For a preliminary experiment, we compress each video frame into a single embedding, average their corresponding global features, and evaluate the Qwen2.5-VL 7B model on the MSVD-QA [1] and MVBench [2] datasets. The MSVD-QA dataset is a video question-answering task where the model must answer natural language questions based on the content of a video. MVBench is a benchmark specifically designed to evaluate temporal understanding in videos, featuring 20 different tasks that require reasoning about actions, their sequence, and temporal relationships. We limit the maximum number of frames to 32, as processing more frames would only lengthen the prefilling time, diminishing the speedup gained from accelerating the decoding stage. Results are presented below:\n\n| Benchmark | Method  | Mean Acceptance Length | Speedup Ratio |\n| :-------: | :-----: | :--------------------: | :-----------: |\n| MSVD-QA   | ViSpec  | 2.16                   | 1.46x         |\n|           | EAGLE-2 | 1.10                   | 1.22x         |\n| MVBench   | ViSpec  | 2.09                   | 1.32x         |\n|           | EAGLE-2 | 0.83                   | 0.83x         |\n\n\nThe results demonstrate that our ViSpec method achieves a notable speedup even without any video-specific training, whereas the baseline method actually **decelerates** inference on MVBench (0.83x speed). Developing a dedicated framework optimized for video remains a promising direction for future work.\n\n### 4. Speedup on multimodal benchmarks such as MME\n\nAs we have described in **Appendix A, \"Implementation Details,\"** following [3], we use modified prompts to instruct the model to generate long-form, explanatory answers, rather than just single-word responses. For example, we ask the model to \"*Provide a detailed description of the given image.*\" This allows us to rigorously evaluate the decoding acceleration on these benchmarks.\n\n### 5. Experiment in high-concurrency settings\n\nWe conduct experiments on the LLaVA-1.6 7B model using the SQA dataset, with the batch size varying from 1 to 8. We compare ViSpec to the standard batched decoding inference in the Hugging Face Transformers library. The results are as follows:\n\n| Batch Size | Speedup Ratio |\n| :--------: | :-----------: |\n| 1          | 2.37x         |\n| 2          | 2.00x         |\n| 4          | 1.75x         |\n| 8          | 1.56x         |\n\n\nWe still observe a substantial gain when using batched decoding. The principle of speculative decoding is to lower the number of memory access cycles to accelerate memory-bound decoding. Therefore, it is expected that the *relative* speedup over standard decoding will decrease as the batch size increases, since higher concurrency better utilizes the computation units. Nevertheless, our method maintains a significant speedup across all tested batch sizes. Implementation optimizations could further boost this performance.\n\n\n[1] D. Xu, et al. \"Video question answering via gradually refined attention over appearance and motion.\" ACM-MM, 2017.\n\n[2] K. Li, et al. \"MVBench: A comprehensive multi-modal video understanding benchmark.\" CVPR, 2024.\n\n[3] Gagrani, Mukul, et al. \"On speculative decoding for multimodal large language models.\" 2024."}, {"review_id": "2UT6shjP7W", "reviewer": "Reviewer_mKSj", "summary": "This paper introduces ViSpec, aiming to achieve acceleration for vision-language models (VLMs) through speculative decoding. ViSpec employs a lightweight vision adapter module to compress image tokens into a compact representation. It extracts a global feature vector for each input image and augments all subsequent text tokens with this feature to enhance multimodal coherence. Experiments show that the proposed method achieves significant speedup.", "questions": "Why report only the speed-up metrics and not the corresponding accuracy?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\nViSpec integrates compressed image embeddings, persistent global visual feature injection, and synthetic long-response dataset generation to address the limitations in processing multimodal sequences with shallow draft models. ViSpec achieves significant speedup compared to previous methods.\n\nWeaknesses\n\n1.\tThe paper only reports speedup metrics but not performance metrics. What is the performance of the model after acceleration?\n2.\tHigh-resolution images require more image tokens and thus necessitate acceleration. It would be beneficial to evaluate the acceleration performance on benchmarks containing high-resolution images, such as HR-Bench benchmark and MME-Realworld benchmark.\n3.\tIt lacks comparisons with relevant methods, such as FastV[1] which compresses using attention weights and MiniCPM-V[2] which employs learnable queries for compression.\n\n[1] Chen L, Zhao H, Liu T, et al. An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models[C]//European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2024: 19-35.\n\n[2] Yao Y, Yu T, Zhang A, et al. Minicpm-v: A gpt-4v level mllm on your phone[J]. arXiv preprint arXiv:2408.01800, 2024.", "was_revised": true, "final_justification": "Most of my concerns have been addressed. Therefore, I would like to raise my scores.", "rebuttal": "We sincerely thank you for your insightful feedback and the opportunity to clarify these important aspects of our work.\n\n### 1. Only reporting speedup metrics\n\nOur work, ViSpec, is based on the principle of **speculative decoding**. The token compression and drafting process *ONLY* involves a small, auxiliary **draft model**. The final output is always verified by the original, unmodified **target model**. The speculative decoding process has been mathematically proven to be **lossless**, meaning it produces an output distribution that is identical to that of standard autoregressive decoding from the target model alone [1, 2].\n\nBecause the final output is provably identical, the model's accuracy and any other performance metrics remain unchanged. Consequently, it is standard practice in the speculative decoding literature to report only the acceleration (i.e., speedup), as performance metrics are not affected. Our paper follows this established convention.\n\n### 2. Experiment on benchmarks containing high-resolution images\n\nWe conduct experiments on the suggested benchmarks, and our method continues to demonstrate strong performance. The results are presented below, comparing ViSpec against the EAGLE-2 baseline for both LLaVA-1.6 7B and Qwen2.5-VL 7B.\n\n| Model         | Benchmark     | Method  | Mean Acceptance Length | Speedup Ratio |\n| :-----------: | :-----------: | :-----: | :--------------------: | :-----------: |\n| LLaVA-1.6 7B  | HR-Bench      | ViSpec  | 2.86                   | 1.93x         |\n|               |               | EAGLE-2 | 1.43                   | 1.52x         |\n|               | MME-Realworld | ViSpec  | 2.85                   | 2.35x         |\n|               |               | EAGLE-2 | 1.42                   | 1.75x         |\n| Qwen2.5-VL 7B | HR-Bench      | ViSpec  | 2.16                   | 1.29x         |\n|               |               | EAGLE-2 | 0.34                   | 0.90x         |\n|               | MME-Realworld | ViSpec  | 2.11                   | 1.37x         |\n|               |               | EAGLE-2 | 0.52                   | 0.95x         |\n\n\nNotably, Qwen-VL does not limit its input image token count, leading to a significantly longer prefilling time on high-resolution benchmarks. Since speculative decoding accelerates only the decoding stage, this extended prefilling time results in a lower overall speedup ratio. However, ViSpec’s mean acceptance length remains robust, indicating that the acceleration of the decoding phase itself is effective. Interestingly, the baseline method actually decelerates the Qwen-VL model on these benchmarks (i.e., speedup < 1.0x). This finding further validates our core argument from Section 4.1: a shallow draft model’s performance inherently degrades when processing the long and redundant token sequences from high-resolution images, highlighting the critical need for the compression strategy that ViSpec introduces. We will include this analysis in the final version of our paper.\n\n### 3. Comparison with FastV and MiniCPM-V\n\nOur method is fundamentally different from approaches like FastV and MiniCPM-V. The core distinction lies in the nature of the acceleration:\n\n* **FastV & MiniCPM-V (Lossy Compression & Prefill Focus)**: These methods are **lossy model compression** techniques. Their main goal is to accelerate the computationally intensive **prefilling stage** by processing fewer visual tokens. For instance, FastV reduces the number of tokens fed into the transformer, which cuts down the FLOPs for prefilling. However, this approach offers minimal benefit for the **decoding stage**. Autoregressive decoding is typically **memory-bound**, where the bottleneck is loading model weights from memory for each forward pass, not the computation itself. Since these methods still require the full model weights to be loaded for every single token generation, they do not alleviate this memory bandwidth bottleneck, and the actual wall-clock speedup during decoding is often negligible.\n\n* **ViSpec (Lossless Acceleration & Decode Focus)**: Our method is a speculative decoding framework that accelerates an existing, unmodified target model. It is mathematically guaranteed to be **lossless**, meaning it does not alter the model's final output quality in any way. Our focus is purely on reducing the **decoding latency**. In stark contrast to FastV & MiniCPM-V, ViSpec is designed specifically to accelerate the memory-bound decoding phase by reducing the number of sequential steps required to generate the full output.\n\nOur method is, in fact, **orthogonal** to approaches like FastV and MiniCPM-V. FastV or MiniCPM-V can be used to compress the target model itself, accelerating the computationally-intensive **prefill stage**. Following that, our ViSpec framework can be applied to this already-optimized model to accelerate its memory-bound **decoding stage**. Therefore, these methods are not in conflict; combining them could lead to a cumulative effect, further enhancing overall inference efficiency by optimizing both prefill and decoding stages.\n\n[1] Y. Leviathan, et al. \"Fast Inference from Transformers via Speculative Decoding.\" ICML Oral, 2023.\n\n[2] C. Chen, et al. \"Accelerating Large Language Model Decoding with Speculative Sampling.\" DeepMind, 2023."}, {"review_id": "3H5v2epbVw", "reviewer": "Reviewer_3Mw6", "summary": "This paper introduces a novel framework to accelerate inference in VLMs. The core idea is that small draft models cannot efficiently process the highly redundant visual information that large target models can. ViSpec addresses this by using a lightweight vision adaptor to create a compact image representation for the draft model and a global feature augmentation technique to maintain multimodal coherence. The paper  also creates a new dataset with long-form responses to facilitate training. ViSpec achieves significant speedups on various benchmarks.", "questions": "See the weaknesses above.", "limitations": "The authors have addressed limitations and social impact of the work.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strengths And Weaknesses\n\nStrengths:\n\n1. The proposed method is technically sound and well-motivated. The design of the vision adaptor is a simple but effective solution tailored for the speculative decoding for vision language models.\n\n2. The analysis that speculative decoding for LLM is suboptimal for VLMs is insightful.\n\n3. The experiment results are competitive. The method demonstrates advantages over previous SOTA methods on multiple VLM benchmarks.\n\n4. The implementation details and experiment settings are clearly described.\n\nWeaknesses:\n\n1. The paper lacks comparison with existing speculative decoding methods for VLMs, such as [1] and [2].\n\n2. The claim that the vision adaptor is \"lightweight\" is not substantiated with a quantitative analysis. A detailed breakdown of the associated overheads is necessary.\n\n3. The authors should investigate how the speedup varies with the length of the output, as it is critical for understanding the method's boundary.\n\n4. To demonstrate broader utility, how about the performance of the method on text-only inputs?  It is a frequent use case for VLMs.\n\n[1] Gagrani, Mukul, et al. \"On speculative decoding for multimodal large language models.\" \n\n[2] Lee, Minjae, et al. \"In-batch Ensemble Drafting: Toward Fast and Robust Speculative Decoding for Multimodal Language Models.\"", "was_revised": true, "final_justification": "The rebuttal has addressed my concerns about the potential overheads and generality of the proposed method, and it demonstrates superior performance compared to previous work. I agree with the other reviewers that the work is designed specifically to address the unique challenges of applying speculative decoding for VLMs through thorough theoretical analysis and valid experiments, which provide an interesting perspective for accelerating the decoding phase of VLMs. Overall, I would like to raise my evaluation score.", "rebuttal": "We are grateful for your insightful questions and suggestions for improvement.\n\n### 1. Comparison with existing speculative decoding methods for VLMs\n\nWe acknowledge and have cited these valuable prior works in our paper. However, as their source code is not publicly available, we encountered difficulties in reproducing their results for a direct, controlled comparison. Therefore, we directly cite the performance data reported in their original papers and apply our method to the same LLaVA-1.5 7B model and datasets they used to ensure a fair comparison. The results are as follows:\n\n|       Method       |  SQA  | COCO Caption | TextVQA | VQAv2 |\n| :----------------: | :---: | :----------: | :-----: | :---: |\n| Gagrani et al. [1] | 1.46x |    1.37x     |    -    |   -   |\n|   Lee et al. [2]   |   -   |      -       |  1.61x  | 1.72x |\n|   ViSpec (ours)    | 2.48x |    3.25x     |  2.67x  | 2.53x |\n\n\nAs shown in the table, our method significantly outperforms the previous state-of-the-art works across all common benchmarks.\n\n### 2. Detailed breakdown of vision adaptor overheads\n\nTheoretically, the image adaptor increases the parameter count of the draft model but reduces its computational load during prefilling by processing fewer tokens. Since the draft model is already very small and efficient, we observed no statistically significant impact on its overall speed. The analysis of the prefilling stage on the COCO Captions dataset is presented below:\n\n|     Model     | Parameters (M) | Parameters (M) |   GFLOPS    |   GFLOPS   | Prefill Latency (s) | Prefill Latency (s) |\n| :-----------: | :------------: | :------------: | :---------: | :--------: | :-----------------: | :-----------------: |\n|               |  w/o Adaptor   |   w/ Adaptor   | w/o Adaptor | w/ Adaptor |     w/o Adaptor     |     w/ Adaptor      |\n| LLaVA-1.6 7B  |      367       |      451       |     956     |    179     |        0.227        |        0.231        |\n| LLaVA-1.6 13B |      534       |      665       |    1460     |    279     |        0.334        |        0.334        |\n|  Qwen-VL 3B   |      404       |      425       |    57.3     |    18.3    |        0.002        |        0.004        |\n|  Qwen-VL 7B   |      826       |      890       |     172     |    55.5    |        0.018        |        0.016        |\n\n\nAnd the token rate for the decoding stage:\n\n|     Model     |     ms/token      |     ms/token     |\n| :-----------: | :---------------: | :--------------: |\n|               | w/o Image Adaptor | w/ Image Adaptor |\n| LLaVA-1.6 7B  |       1.077       |      1.185       |\n| LLaVA-1.6 13B |       1.225       |      1.192       |\n|  Qwen-VL 3B   |       1.081       |      1.211       |\n|  Qwen-VL 7B   |       1.402       |      1.539       |\n\n\nWe attribute these minor speed variations primarily to measurement noise.\n\n### 3. Relationship between output length and speedup ratio\n\nThe VLMs we tested tend to generate text of varying lengths across different datasets. We present the relationship between the average number of generated tokens and the speedup below:\n\n|  Dataset   | Average New Tokens | Speedup Ratio |\n| :--------: | :----------------: | :-----------: |\n|    GQA     |       46.25        |     2.22x     |\n| SEED-Bench |       57.66        |     2.22x     |\n|    SQA     |       74.07        |     2.37x     |\n|   VizWiz   |       105.91       |     2.26x     |\n|    MME     |       115.01       |     2.55x     |\n|   MM-Vet   |       171.13       |     2.52x     |\n| COCO Caps  |       236.04       |     3.21x     |\n|  TextVQA   |       353.58       |     2.90x     |\n\n\nGenerally, longer text outputs tend to yield a higher speedup ratio, as there are more opportunities for the draft model to make correct predictions. Nevertheless, our method maintains stable and robust performance across all datasets, even those with shorter response lengths.\n\n### 4. Performance on text-only inputs\n\nWe evaluate the speedup on the text-only **MT-Bench** [3] dataset using the LLaVA-1.6 7B model. We compare the performance of the original text-only draft model with our draft model that has been fine-tuned on multimodal data. The speedup after our multimodal fine-tuning is **0.96x** of the speedup achieved before fine-tuning. This demonstrates that our proposed method has a negligible impact on performance for text-only tasks, preserving the model's utility as a general-purpose assistant.\n\n[1] M. Gagrani, et al. \"On speculative decoding for multimodal large language models.\" 2024.\n\n[2] M. Lee, et al. \"In-batch Ensemble Drafting: Toward Fast and Robust Speculative Decoding for Multimodal Language Models.\" 2024.\n\n[3] L. Zheng, et al. \"Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena.\" NeurIPS, 2023."}]

Author rebuttal:

Meta-review: This paper presents ViSpec, a framework designed to accelerate inference in vision-language models (VLMs) without sacrificing generation quality. The key idea is that draft models struggle with redundant visual tokens, slowing speculative decoding. To address this, ViSpec introduces a lightweight vision adapter that compresses image inputs into compact representations. In addition, it extracts a global feature vector per image and augments text tokens with this feature to preserve multimodal coherence. This dual strategy enables efficient handling of visual information while ensuring alignment between modalities. The authors also propose a new dataset with long-form responses to better support training and evaluation. Experiments demonstrate that ViSpec delivers significant inference speedups across multiple VLM benchmarks.    

Reviewers were divided on this submission. They asked many questions about the method and its evaluation.  
The detailed answers and additional experiments provided, particularly with video and high-resolution images, as well as the statistics on overheads and speed-up, totally convinced the active reviewers.  

The final discussions between the AC and the active reviewers clearly revealed a very positive consensus, and active reviewers  increased or confirmed their positive ratings.  

As a result, we are convinced that this submission deserves to be published, and we strongly encourage the authors to incorporate all the suggestions that came out of the rebuttal and discussions into the final version.
Decision: Accept (poster)