DREAM: Drafting with Refined Target Features and Entropy-Adaptive Cross-Attention Fusion for Multimodal Speculative Decoding
Authors: Yunhai Hu
Tianhua Xia
Zining Liu
Rahul Raman
Xingyu Liu
BO BAO
Eric Sather
Vithursan Thangarasa
Sai Qian Zhang
Venue: NeurIPS 2025 poster ()
Keywords: Speculative Decoding
Vision language model
Primary area: deep_learning

Abstract: Speculative decoding (SD) has emerged as a powerful method for accelerating autoregressive generation in large language models (LLMs), yet its integration into vision-language models (VLMs) remains underexplored. We introduce DREAM, a novel speculative decoding framework tailored for VLMs that combines three key innovations: (1) a cross-attention-based mechanism to inject intermediate features from the target model into the draft model for improved alignment, (2) adaptive intermediate feature selection based on attention entropy to guide efficient draft model training, and (3) visual token compression to reduce draft model latency. DREAM enables efficient, accurate, and parallel multimodal decoding with significant throughput improvement. Experiments across a diverse set of recent popular VLMs, including LLaVA, Pixtral, SmolVLM and Gemma3, demonstrate up to 3.6x speedup over conventional decoding and significantly outperform prior SD baselines in both inference throughput and speculative draft acceptance length across a broad range of multimodal benchmarks.

DREAM: Drafting with Refined Target Features and Patropy-Adaptive Cross-Attention Fusion for Multimodal Speculative Decoding
Yunhai Hu'
Tianhua X
Zining Liu Rahul Raman'
Xingyu Liu?
Bo Bao
Eric Sather Vithursan Thangarasa? Sai Qian Zhang
Courant Institute of Mathematical Sciences, New York University
Tandon School of Engineering, New York University $Cerebras Systems Inc. University of Pennsylvania
{yh5961, tx856, rr4549, x15444, sai (ahang))nny edu edu zliu00seas upenn. edu [bo. bao, sather, vithu) net
Abstract
Speculative decoding (SD) has emerged a powerful method for accelerating autoregressive generation in large language models (LLMs), yet its integration into vision-language models (VLMs) remains underexplored. We introduce DREAM, a novel speculative decoding framework tailored for VLMs that combines three key innovations: (1) a cross-attention-aasee mechanism to inject intermediate features from the target model into the draft model for improved alignment, (2) adaptive intermediate feature selection based on attention entropy to guide efficient draft model training, and (3) visual token compression to reduce draft model latency. DREAM enables efficient, accurate, and parallel multimodal decoding with significant throughput improvement. Experiments across diverse set of recent popular VLMs, including LLaVA, Pixtral, SmolVLM and Gemma3, demonstrate up to 3.6x speedup over conventional decoding and significantly outperform prior SD baselines both inference throughput and speculative draft acceptance length across broad range of multimodal benchmarks. The code is publicly available at: battps://gitaee cow.SAII Lab- BAAA--LLbNYUUDREAAA git
Introduction
Large language models (LLMM)) have shown impressive performance across diverse tasks, but their inference speed remains limited due to the standard autoregressive process, which includes both prefilling and decoding stages. To overcome this limitation, speculative decoding (SD) [48] [9 accelerates the autoregressive process by splitting it into a low-cost drafting stage and a parallel verification stage, enabling multiple drafted tokens be validated in single pass through the target LLM. This approach accelerates the decoding stage while preserving the output quality of the target model through a mechanism of acceptance and rejection.
While these techniques have heen extensively developed to accelerate inference in text-only LLMs 25 26
24
4
2
55
50 [55, there has been limited work integrating SD into multimodal LMs 43 especially vision-language models (VLM) [10]. VLMs differ from their text-only
1
40
(MLLM) 20 counterparts by requiring seamless integration of visual and textual information. This integration process typically occurs in two stages: first, extracting meaningful representations from images, and second, applying language reasoning capabilities to generate appropriate responses. For example, in
Authors contributed equally; the order of authorship was assigned randomly.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

vision encoder transforms images using 24 layers of self-attention and feed-forward networks. These features are then projected into the text space and fused with text embeddings within the language model's s backhone. The effectiveness of this process heavily depends on the model's s ability to maintain coherent representations across modalities, which presents unique challenges for acceleration techniques.
VL 129].
In this paper, we propose Drafting with Refined Target Features and Entropy-Adaptive Cress-Attention Fusion for Multimodal Speculative Decoding (DREAM). Specifically, DREAM employs a specialized cross-attention mechanism that enhances the interaction between visual and textual features, ensuring that key information in the target model is adequately captured even in the draft generation stage. Another key innovation of DREAM lies its selective use of intermediate-laye -laeer representations, which encapsulate the most informative features from both modalities to effectively supervise the draft model with high accuracy. Finally, DREAM introduces a visual input compression scheme for the draft model, guided by the intermediate features from the target model, which substantially reduces processing latency without compromising accuracy.
We evaluate DREAM on a diverse set of popular VLMs, including I. 6-Vicuna-7BB13BB 22 SmolVLM-2B 38 Pixtral-12B M. and Gemma3-12B (3) across multiple multimodal tasks. Our extensive experiments demonstrate that DREAM substantially outperforms well-established SD methods, while consistently achieving high acceptance rates across various multimodal applications. such as table structure recognition, interactive segmentation, animal keypoint detection, and chart question answering. Our main contributions are summarized follows:
DREAM incorporates a cross-attention mechanism to leverage intermediate outputs from the target model, facilitating more effective knowledge transfer to the draft model and leading to significant performance gains.
During DREAM training phase, intermediate features from the middle layers of the target model are dynamically selected based on their entropy and used to supervise the draft model. This enhances the draft model's predictive accuracy and increases token acceptance lengths.
We investigate the relative importance of visual input and show that visual input in the draft model can be effectively compressed using a scheme guided by intermediate target features, reducing draft processing time and overall speculative decoding latency.
Evaluation results show that these three strategies enable DREAM to achieve up to a 3.6x reduction in latency compared to conventional decoding methods across a range of VLMs and tasks, outperforming existing speculative decoding approaches.
2 Background and Related Work
2.1 Speculative Decoding
Speculative decoding (SD) 488 has proven effective in mitigating the se- quential bottleneck in language model inference. It operates in two stages: lightweight draft model rapidly gen- erates a sequence of candidate tokens, which are then verified in parallel by a more accurate target model. Specifi- cally, the draft model generates tokens di, d2. dy, where is predefined number of tokens. These candidates are verified concurrently by the target model but accepted sequentially. Specifically, draft token d, is accepted with probabil- ity min(1, (d.)) (d.)) where Paran and Purger are the probabilities as- signed by the draft and target models, respectively. Otherwise, the token is rejected. This acceptance rule is compatible with various sampling temperatures. If rejection occurs at d; all subsequent
Taken Concatenation
"An of two golden retrfevers"
What IIw Ersrd of car? Text
Language model
Projectin Visual Encodor
Embedding
Visual encoder
Draft model
Embedding
Target model
Language Model
"Describe the image"
Loss funciion
(a)
(b)
Figure I: (a) Standard VLM. (b) DREAM overview.

---

draft tokens from de onward are discarded, and the target model's 's token iq is used by draft model to continue generation. If all tokens are accepted, the draft model proceeds generate next batch.
Building on this foundation, researchers have proposed a range of techniques to enhance the efficiency of the draft-verify process. For drafting, various approaches have been developed eliminate or improve draft models. Medusa [4] introduces lightweight decoding heads on the target model itself, eliminating the need for separate drafter. Self-speculative techniques include layer skipping [14
61
8 28
32 (56, which accelerates draft generation by selectively processing fewer transformer
layers. Other strategies leverage token distillation [64], N-gram prediction 42 49
and retrieval-
augmented drafting [57 13 54] to improve draft quality while minimizing computational overhead.
Tree-based verification methods 44
[52 6 47 support parallel exploration of multiple completion
paths, substantially improving throughput over traditional linear verification. EAGLE (25) introduces
feature-based uncertainty estimation for tree construction, while EAGLE-2 24 further improved
this with dynamic context-aware trees. Numerical optimization-inppree approaches include Jacobi iterations (45) and Lookahead decoding (9. which reformulate autoregressive generation as parallel optimization problems. For production deployment, various frameworks enhance system-level efficiency. TRIFORCE [50] employs hierarchical speculative decoding combined with sparse KV cache to support ultra-long sequences exceeding 100,000 tokens. Parallel scheduling approaches AI 33 enable draft generation to run concurrently with target verification, while heterogeneous compute solutions like Dovetail [62] optimally distribute models across CPU/GPU resources.
While the aforementioned techniques focus on SD for language-only models, there has been limited exploration of SD in MLL.Ms. In speech synthesis, VADUSA (20) applies SI) to accelerate inference in text-to-ppeech systems, simultaneously improving synthesis quality. Building on the principles of SD, the authors of 43 propose multi-token prediction mechanism that significantly boosts inference efficiency for speech generation. In the VLM context, [0] applies speculative decoding to the LLaVA- model, demonstrating up to 2.37x speedup by utilizing lightweight, language-only draft model under memory constraints. IbED [8] proposes an In-batch Ensemble Drufting strategy that employs ensemble techniques at the batch level without introducing additional model parameters. In contrast, DREAM introduces a novel cross-attention mechanism and adaptive intermediate feature selection to enhance draft model training, resulting in substantial latency reductions.
2.2 Vision Language Model and Computation Profiling
Vision- Models (VLMs) are designed to jointly pro- cess visual and textual inputs, allowing machines to interpret and generate content that combines both modalities. As illus- (a),
15
10.3
Txt
TFLOPs
trated in Figure (a), a typical VLM consists of a visual encoder and a language model. The input image is first processed by the visual encoder to generate visual tokens, which are then con- catenated with the textual tokens. These combined tokens are passed to the language model, which generates the final textual have been developed. Recent works like LLaVA 31. InstructBLIP 7 and Pixtral ① aim to enhance the zero-shot capabilities of VLMS by better aligning them with human preferences. While the large model sizes have resulted in significant performance improvements, their computational complexity and storage requirements limit their deployment on resource-constrainet devices. Lightweight VL.Ms, such as TinyGPT-V (59] and TinyL.La (3), explore the potential of small-scale models and focus on developing efficient VLM architectures. The recent development of SmolVLM 38) introduces a family of compact VLMs with parameters ranging from 256M to 2B, achieving exceptional performance while maintaining smaller model sizes.
10
Img Txt
5.8
5.1
2.6
Llava Llava Pixtral
Smol Gemma
7B
138
12B VLM 3 12B
outputs. Several VLMs 23 (2)
Figure 2: Computational cost of VLMS processing text only (Txt) and multi-modal (Img+Txt) inputs.
To quantify the computational cost introduced by visual inputs processing in VLMs, we profile the floating point operations (FLOPs) required by various models, including LLv Vicuna- 7B [3I], LLa 1-- V--Vicnna-7B. Pixtral-12B [I]. SmolVLM-2B [38], and Gemma3-128 [1] over the ScienceQA dataset. We select typical sample which contains 480 x 300 image, a prompt with 166 tokens, and use the model to generate 500 tokens, representing an average case across the dataset. FL.OPs are measured using Py PyTorch Profiler. Figure comppres the TFLOPs for each model when processing text-only (Txt) versus multimodal (Img+Txt) inputs, revealing average 2. X increase
3

---

CKL.
Inputioulput tokens
Features target models
Fealures the draft
Head
Sear
Het of
largel decoding
Target Model
loyers
Cross-
Draft Model Mis
Larget decocing
leyars
Layyers
Draft Model Mas
Head
Target Model M
Cross
Decodder Layers
Adsefive Inferrieddaa
Model Mis
Feature Selection
B
Middle
Layara
Ses Buy
Layars
Embeddling
Embadding
Visual
Eibbading
Visual ensoder
Selection
Text
Image
Text
Image
(a)
(b)
(c)
Figure 3: (a) illustrates the training paradigm of DREAM, while (b) and (c) depict its inference
workflow. For simplicity, the tree decoding is not shown in (b) and (c).
in computation with the inclusion of visual data and highlighting the need for more efficient visual processing methods.
2.3 Intermediate Feature Distillation
Intermediate Feature Distillation (IFD) enhances traditional knowledge distillation by aligning student models not only with the teacher's final outputs but also with its intermediate representations. FitNets [44] introduced the idea of using intermediate hints, showing that projecting selected teacher
layers through lightweight adapters can improve compact student models. Later methods refine
this by identifying high-similarity layers (60, using attention-based weighting (27), dynamically
selecting task-relevant features, as in TED [27] CoDIR (51) further introduces contrastive losses for tighter feature alignment. In multimodal contexts, OLA-VLM [66 and VLsI [77] adapt these ideas to distill visual embeddings into language representations. Recently, Skean et al. [46] proposed evaluating per-layer representations using information compression, geometric separability, and robustness, showing that mid-layer features often outperform final-layer ones across tasks, revealing a non-monotonic trade-off between information richness and task relevance.
3 Methodology
Figure (bb provides an overview of DREAM, where the draft model receives the textual input alongside subsampled visual input accelerate output generation. Additionally, the intermediate features are adaptively selected to better guide the training of the draft model. Figure (3) presents the detailed training and inference scheme of DREAM. Specifically, let Mi and Ma. represent the target and draft models, respectively, with a total of L and M transformer blocks, respectively. let In and d, denote the n-th token produced by Mr. and Man- For Mr.. assume the textual prompt consists of y tokens and the visual input comprises tokens, respectively. We use s and e to denote the n-th intermediate feature token layer for Mta and Maa, respectively.
(b)
Figure (b) highlights the DREAM inference process. Let r denote the subsampling factor, and q and represent the number of prompt and visual tokens, respectively. As illustrated in Figure B (b), the target model processes all q + v input feature vectors (sq...., s) "0.......00.... In contrast, with our subsampling method applied, the draft model processes only q + [v/r] tokens, substantially lowering the computational load and improving the decoding speed of the draft model, motivated in Section 2.2 Next, we depict the cross-attention architecture of DREAM (Section 3.T). followed by the adaptive intermediate feature selection strategy for training the draft model (Section 3.2 Finally. we introduce the visual token compression method used to accelerate the draft model in Section 3.3
3.1 Cross-attention Mechanism for Efficient Knowledge Injection
To more effectively transfer knowledge from the target model and enhance the performance of the draft model, we introduce cross-attention mechanism to fuse features from the target model

---

to the draft model. This mechanism naturally aligns with the dual-model setup, using the draft's newly generated token embeddings as queries to efficiently retrieve cached target features. This lightweight attention layer integrates long-range, multimodal context into the decoding process. The attention weights act as soft gating, enabling adaptive selection of the refined target visual and textual cues. Crucially, unlike the EAGLE family of methods [25] 24], which concatenate draft token embeddings with precomputed target features and process them jointly, DREAM adopts a more structured fusion strategy. While EAGLE's concatenation approach is effective for purely textual tasks, it tends to weaken structured visual representations by treating visual and textual features simple concatenated sequence, potentially disrupting the spatial relationships learned by the vision encoder. In contrast, our cross-attention mechanism adaptively retrieves relevant visual and textual cues, preserving the integrity of multimodal information. This leads to more effective knowledge transfer and improved draft model performance, as demonstrated in our evaluation results in Section 4.2
As depicted in (i) in the beginning decoding stage, both textual and visual prompt tokens t1,..., bn are initially fed into the target model, which then begins generating the next token In+1. where n = q + v. The draft model Más subsequently predicts the (n + 2)tthh token, denoted as dn+2. To better help the draft token generation, the cached last-layer features from the target model M... denoted as Sh = (sf sy...... sh). and the intermediate features from the draft model Máa denoted .g....,,,,, (,)), where n' = q 1 + 1, are integrated using a cross-attention mechanism. In this setup. E' is used as the query, while 5 provides the keys and values. With denoting the dimensionality of queries and keys, the cross-attention then computed as follows:
as E' = (c) ........................
F = softmax (OK*
- Wo K K- S' V S'
(I)
where Wo, WK. Wy denote the weight matrices in the draft model. The fused features I' replace the original first-layer features and are propagated through the subsequent decoder block. For simplicity, assume single attention head in this illustration. After generating the first token dn+2, the draft model leverages cross-attentin over both the previously verified features from the target model and its own final-layer features. Specifically, we concatenate the target features S eM) with the latest draft feature e.M to form the key and value set S = (+f, s4 :-... S e ) for generating the next draft token d3,+3, as described in 3
s/................. ..... s/)
Moreover, we adopt the top-k reordering logic and masking strategy introduced EAGLE-2 224. Specifically, we use the newly generated first-layer draft tag features and perform tree-based width-k expansion to construct the candidate feature set E =
e . +.). The tree mask is then
applied to filter out irrelevant tags, resulting in the final representations Ftree = (e t1)...... + + which are used to simultancously generate k new draft tokens. This approach enables efficient parallel decoding, and the same tree mask is also leveraged during verification to reduce computation overhead. More details are provided in the supplementary materials on tree.
3.2 Adaptive Intermediate Feature Selection for Draft Training
In this section, we detail the training strategy for the draft model. To help the draft model effectively learn the behavior of the target model Mea, utilize intermediate-layee features from the target model as supervision signals. For these features to effectively guide the training of the draft model Máa. they must satisfy two key criteria: they can provide key information and capture rich semantic content, and they should be essential, exhibiting low variability across tokens, to enable faster and more stable learning in the draft model. As discussed in Section 2.3] intermediate features with low token-lever attention entropy are well-suited for distillation, as they focus on salient content and offer stable guidance. DREAM implements a simple yet effective mechanism to select such features from each layer of the target model to guide the efficient training of the draft model, which is shown in Figure
3 (a). For the I-th decoder block, its input tokens and output tokens are denoted as se-1 (s/ 82 ......st sf-1) and St = (sf, 22. .2,......, sy),, respectively. Let the attention matrix Ag
associated with /-th layer as Ag- = softmax( Qxk=) where Qr = -te-WW and K₁ = - k. the
Vz
average attention entropy (AE) is calculated as:
1
AE(() =
Ae..,, log A
(2)
n
5

---

where A denotes the (i.j)-th element of Aq. In practice with multiple heads, AE(() is also averaged over all attention heads. During each decoding step n. we dynamically select the layer ! with the lowest average entropy, defined as E = arg min [ (E)). We then distill the information from "f into the initial decoder block of the draft model using a smooth & loss, guiding the draft model to align with the most informative intermediate representation of the target model. This adaptive feature distillation strategy leads to improved performance by minimizing the E1 loss between the feature vector of the m-th layer of the draft model E (ef" ... e and St = (sf" ..... s namely:
Lintermed = smoothLi (E" )²
(3)
1
where smoothL1 v) equals 2 y)² if [ yl < 1, and |xr yl otherwise 12
3.2.1 Loss Functions
As shown in Figure (a), the loss function of DREAM comprises three components. First, encourage the output features of the draft model to closely match those of the target model by minimizing the difference between the final-layer features Edd (et ..... and 5 = (of...., % This will improve the acceptance rate of the draft tokens. Second, following the adaptive feature selection strategy described in Section 3.2, we minimize the difference between an early-layer feature ETH and the selected intermediate target feature S For DREAM, we set m = 1, yielding the loss: Lintermed = smoothL1
(ef',..., e from the draft model from the target model, using a smooth L1 loss: Lfeat smoothL1 s*).
S ) Finally, to ensure the token outputs from the draft model Mde align with those from the target model Mia. apply a KL divergence loss between their softmax outputs: LKL = KL (softmax( D), softmax(T)), where D = (di.....dd and = (f1...., fn) the predicted token logits from the draft and target models, respectively. Finally, the overall loss function can be described as:
Lisnal = Afeat Lfeat + Xintermed Lintermed + AKL LKL
where feat, Xintermed and A denotes the relative importance between the loss functions.
3.3 Visual Token Compression
Given the high computational cost of visual tokens, as discussed in Section 2.2 DREAM adopts subsampling strategy to reduce this overhead. A naive solution would be to uniformly subsample the visual tokens. However, this approach risks discarding crucial visual information necessary for accurate draft model predictions, while retaining redundant regions that could be aggressively compressed removed. To address this, DREAM incorporates simple yet effective token selection mechanism, as illustrated in the visual token selection block of Figure 3 (b). Specifically, it computes importance scores for the final-layer features (sf ...., s st+ by summing each token's 's attention weights across all other tokens, as derived from the attention matrix. These scores reflect the relative importance of each token in the last layer with respect to the final model accuracy.
Among the attention scores, we isolate those corresponding to the visual input and sort this subset by magnitude. We then record the indices of the top scores based on the subsampling ratio r. These indices are used to subsample the visual tokens they are considered critical to the final output. This strategy significantly reduces the number of tokens while retaining the most important visual information. Although visual token subsampling may cause slight decrease in the draft model's 's accuracy, our evaluation in Section 4 shows that can effectively reduce speculative decoding latency.
with the top high scores, where
4 Empirical Results
We conduct experiments on five VLMs representing a range of parameter scales, including LLaVA-
v1.6 VieVinu (7B, 13B) (30], Pixtral (12B) M. SmolVLM (2B) (38,, and Gemma3 (12B) [33
DREAM evaluated across eight diverse benchmarks: MMT- Bench (58, SEED-Bench-2 21]. ScienceQA (37. OCRBench 35 ChartQA RН and Math Vista [6). All evaluations are performed
under two softmax temperature settings: Temp and Temp- = 1. We report two key metries:
(1) Speedup ratio over standard autoregressive generation, defined as tAR / method. where tAR is

---

the average wall-clock time per token for standard decoding, and Emethod is the corresponding time for each evaluated method. A larger speedup directly corresponds to lower end-to-end latency in real-world use. (2) Average token acceptance length т, representing the number of consecutive draft tokens accepted by the verification model. larger T implies fewer verification steps and higher effective decoding throughput. We adapt six recent SD baselines, originally developed for text-only LLMs, for use with VLMs, including SPD [10], Kangaroo [28). Medusa [4], Hydra (2), and EAGLE
1 and 2 125 24).
We freeze the target VLMs and train only the draft model using the LLaVA mix665k dataset, with 55,000 training samples and 1. 1,000 training samples from each corresponding evaluation benchmark dataset. For benchmarks without predefined train -eest splits (MMT-Bench, SEED-Bench Vista, and OCRBench), we applied stratified random sampling using train_ Baain_test__tt past_spiit (with random_ random_state=42 state=42 for reproducibility) to select 1, 1,000 training samples and up to 3,000 test samples per dataset. For ChartQA and ScienceQA, which already provide official splits, we randomly sampled 1,000 training examples from their respective training sets. All selected training samples were then processed through the target model to generate ground-truth responses in conversational form.
Math-
Training is performed for 68,000 iterations using AdamW optimizer (3, 0.9, B2 0.95), learning rate of 3 x 10-5, and gradient clipping set to 0.5. Each draft model consists of an initial decoder block, an intermediate cross-attention block, and a final decoder block. The weights for the loss terms Lleat, Lintermed, and LKL are set to 0.2, 0.2, and 1.0. The parameter sizes are 0,65B for LLaVA-v1.6- Vicuna- 7B. 0.98B for LLaVA-v1.6-Vicu Vicuna- 13B. 13B, 0.9B for Pixtral- 12B. 0.28B for SmolVLMM2BB, and 0.9B for Gemma3- 12B. Training is conducted on two NVIDIA A100 80 GB GPUs with batch size set to Our training procedure adopts a two-stage pipeline: (1) offline calibration for feature selection, and (2) draft model training. During the calibration stage, we compute the attention entropy for each sample once to identify the optimal intermediate layer e", and cache these selections for subsequent training.
During the DREAM evaluation, we use speculative sampling with a batch size of 1, following prior work. The tree structure is configured with k 4 child nodes, depth of 6, and maximum draft length of 32 tokens. Note that the average accepted token length (T) reported in Table may exceed a depth of 6, as the target model generates an additional token once all drafts are verihed, resulting in up to 7 tokens per round 19]. 75% of the visual tokens are retained. These values were chosen based on preliminary experiments to balance aggressive speculation against verification overhead. A full sensitivity analysis and ablation study on these hyperparameters can be found in supplementary materials. All models are evaluated on single NVIDIA A100 80GB GPU.
All models are based on their official Hugging Face implementations. Baseline methods, including Kangaroo (28). Medusa (4. EAGLE (25) and EAGLE-2 (24). are executed using their publicly released code and default configurations, with minimal modifications to support VLM inputs. During inference, all models are run with KV caching enabled to ensure efficient autoregressive decoding. To ensure a fair comparison, all methods are trained and evaluated using the same dataset, number of training epochs, learning rate, batch size, and hardware environment. We fix random seeds across all runs to reduce performance variance and report the average over three runs.
4.1 Evaluation Results
Table presents the speedup ratios and average acceptance lengths 7 for DREAM compared to baseline methods. Across all tasks and target models, DREAM consistently achieves the highest speedup and longest acceptance lengths. Notably, DREAM delivers 1.5x 3.6x speedup over standard autoregressive decoding with the target model, and achieves a 20% to 40% improvement over EAGLE-2. Particularly, we observe that larger models gain greater benefits from DREAM. At T= O. DREAM achieves an average speedup of 3.06x on LLa 1.6 Vicvmic and 2.65 on Pixtral-12B, whereas the speedups are lower for smaller models, with 2.27x x on SmoIVLM-2B and 2.23x on LLaVA-v1..6 16-VVicna--7B This is because larger models experience more severe decoding bottlenecks, allowing the draft model to more effectively substitute the costly decoding process of them. Furthermore, models like LLaVA and Pixtral embed visual features directly into the language decoder, offering clearer multimodal cues. This allows DREAM to achieve higher acceptance lengths, for example, - = 5.51 on LL V---vl.--VV Vicuna--BB In contrast, models such as Gemma3-120 which handle cross- modal information through more complex processing pathways, reach lower acceptance rates, with 2.36.
7

---

Table Evaluation of SD methods through speedup ratio (S) and average accepted token length (7).
SEED
SciencoQA
OCRBench
CharQQ
MathVista
Average
Models
Methods
8
S
S
S
S
S
S
Temperature =
SPD 1D
1.10
0.81
1.17
1.08
1.25
1.06
1.76
Kangaroo
28
1.32
2.11
1.33
2.12
1.31
2.09
1.17
1,89
1.18
1.98
1.15
1.86
1.24
2.01
Medusa
1.58
3.01
1.44
2.33
1.25
1.22
2.34
1.38
LLaVA-v16
Hydra
1.78
3.86
1.72
3.88
1.68
3.79
1.41
3.21
1.35
3.11
1.42
3.25
1.56
3.52
Vicuna-7B
EAGLE 25
2.10
5.01
1.98
1.72
4.13
1.56
1.78
4.25
1.87
EAGLE-2 24
2.31
5.48
2.31
5.61
2.15
5.22
1.92
4,88
1.77
4.22
1.87
4.67
2.05
5,01
DREAM
2.52
6.20
2.05
4.88
5.32
SPD
1.07
1.78
1.06
1.79
1.09
1.88
0.86
1.12
0.89
1.25
0.87
1.22
1.00
1.58
Kangaroo
1.43
1.77
1.51
1.87
1.22
1.55
1,21
1.54
1.27
1.61
1.53
2.01
1.36
1.72
Medusa
1.99
2.67
1.96
2.76
1.93
2.77
1,40
2.92
1.51
2.82
1.51
2.62
1.72
2.76
LLaVA-v1.6
Hydra
2.12
2.87
2,08
2.99
2.21
3.12
1,49
3.07
1.65
3.03
1.66
2.87
1.87
2.99
Vicuna-13B
EAGLE
2.45
3.56
2.19
3.24
2.63
3.98
1.65
3,31
1.85
3.27
1.8
3.09
2.10
3.41
EAGLE-2
2.89
4.05
3.18
4.33
3.09
4.97
2,20
4.12
2.41
4.15
2.39
3.76
2.69
4,23
DREAM
3.68
5.58
3.51
5.34
3,36
5.29
2.69
4.64
2,59
4.20
2.53
4,18
3.06
4.87
SPD
1.08
1.51
1.03
1.47
1.05
1.49
1.05
1.49
1.04
1.43
1.04
1.46
1.05
1.47
Kangaroo
1.26
1.54
1.09
1.39
1.14
1.16
1.52
1.12
1.13
1.49
1.15
Medusa
1.37
1.81
1.37
1.81
1.35
1.87
1.24
1.69
1.22
1.68
1.16
1.47
1.28
1.72
Pixtral-12B
Hydra
1.58
2.24
1.47
2.04
1.53
2.06
1,38
1.81
1.34
1.79
1.36
1.78
1.44
1.95
EAGLE
2.38
3.47
1.97
2.53
2.31
3.64
1.69
2.73
1.78
2.84
1.64
2.47
1.96
2.95
EAGLE-2
2.81
3.95
2.31
3.07
2.64
4.03
2.12
3,25
2.14
3.17
1.81
2.73
2.31
3.37
DREAM
2.93
4.52
2.61
3.67
2.98
4.33
2.38
3.55
2.35
3.49
2.36
3.42
2.65
3.78
1.02
1.41
1.06
1.42
1.07
1.46
1.34
1.04
Kangaroo
1.28
1.48
1.08
1.18
1.03
1.17
1,06
1.22
1.04
1.14
1.08
1,23
1.10
1.24
Medusa
2.12
2.00
1.72
1.61
1.15
1.75
1.51
SmolVLM
RAGLE
Hydra
2.33
2.57
3.07
1.62
2.08
2.56
2.16
1.98
2.66
1.32
1.88
1.74
1.22
1.34
1.58
1.51
1.65
2.22
1.98
1.66
1.83
2.19
EAGLE-2
2.96
3.89
2.12
2.93
2.39
3.21
1.65
2.11
1.51
2.13
1.81
2.63
2.07
2,82
DREAM
3.05
3.18
2.56
1.62
2.01
2.88
Kangaroo
1.37
1.66
1.47
1.79
1.52
1.57
3.17
2.28
2.28
1.85
1.18
1.64
1.83
1.80
EAGLE
1.73
1.98
1.69
2.52
1.72
1.97
4.26
2.42
3.40
1.99
1.42
1.89
2.37
2.13
Gemma3-12B
EAGLE-2
2.92
1.99
1.74
2.79
1.92
1.98
4.68
2.57
3.48
2.23
1.52
1.91
2.71
2.25
DREAM
2.99
2.13
3.53
2.84
2.60
2.05
4.81
2.58
3.68
2.56
1.98
1.99
3.27
2.36
Temperature
SPD
0.83
1.19
0.81
1.15
0.85
1.18
0.75
1.06
0.72
1.08
0.92
1.48
0.81
1.19
LLaVA-v1.6
Kangaroo
1.20
1.97
2.03
1.23
2.01
1.09
1.80
1.11
1.89
1.07
1.77
1.16
1.91
7B
EAGLE-2
2.19
5.37
2.20
5.48
2.04
5.12
1.82
4.77
1.68
4.13
1.76
4.56
1.95
4.91
DREAM
2.39
6.29
2.35
6.07
2.25
4.88
1.84
2.02
5.23
2.14
5.43
SPD
0.88
1.22
0.84
1,25
0.84
1.32
0.79
1.18
0.81
1.14
0.88
1.24
0.84
1.22
v1.6
Kangaron
1.23
1.57
1.17
1.53
1.07
1.44
1,01
1.24
1.07
1.34
1,21
1.67
1.13
1.46
Vicuna-13B
EAGLE-2
2.35
3.75
3.02
4.30
3.03
4.67
2.03
3.87
2,18
3.83
2.18
3.41
2.46
3.97
DREAM
3.34
5.38
3.32
5.06
3.20
5.98
2.23
3.89
2.43
4.04
2.29
4.03
2,80
4.73
0.81
1.11
0.80
1.13
0.77
1.09
Pixtral-12B
Kangaroo
1.18
1.41
1.08
1.35
1.03
1.36
1.19
1.48
1.14
1.45
1.09
1.41
1.12
1.41
EAGLE-2
2.76
3.01
2.76
2.23
3.24
1.79
2.69
2.30
DREAM
2.90
4.02
2.47
3.57
2.93
3.94
2.29
3.46
2,21
3.21
2.16
3.27
2.49
3.58
SPD
1.07
1.47
1.01
1.33
1.07
1.46
0.97
1.26
1.06
1.44
0.85
1.20
1.00
1.36
Smo/VLM-2B
Kangaroo
1.37
1.59
1.12
1.24
1.22
1.41
1.12
1.29
1.18
1.36
1.28
1.42
1.22
1.39
EAGLE-2
2.62
3.60
1.92
2.67
2.24
3.11
1.41
1.77
1.60
2.18
1.77
2.49
1.93
2.64
DREAM
2.88
3.66
2.25
3.33
2.91
3.74
1.54
2.12
1.77
2.51
1.97
2.70
2.22
3.01
Kangaroo
1.83
1.66
1.23
2.61
1.56
2.29
3.34
2.27
2.23
1.86
1.16
1.65
1.89
2.06
Gemma3-
EAGLE
2.23
1.96
1.60
2.52
2.16
3.74
2.65
3,30
2.03
1.59
2.44
2.16
EAGLE-2
2.73
1.94
2.13
2.79
2.21
2.07
4.67
2.47
3.35
2.23
1.65
1.89
2.79
2,23
DREAM
2.88
3.49
2.84
2.39
4.79
2.56
3.61
1.96
1.91
3.19
2.32
Second, the task-level analysis reveals that long-form QA tasks such as MMT-Bench and ScienceQA benefit most from speculative decoding. These tasks require generating structured answers grounded in high-level visual semantics, which DREAM captures efficiently. For example, on ScienceQA with LLaVA-vI. Viv.nnaa. DREAM achieves speedup of 3.36x with 5.32, outperforming Kangaroo (1 (1.22x) ) and EAGLE-2 (3.09x). x) In contrast, OCR-style datasets such as Math Vista pose greater challenges due to their reliance on fine-ge fine-grained character-level recognition, which current draft models struggle to replicate. As a result, all methods show lower т and speedup on this benchmark.
Finally, temperature settings critically impact performance. At low temperature (T = 0), determinis- tic decoding leads to higher token alignment and longer acceptance spans. For instance, DREAM achieves 4.52 on MMT-Bench with fixtral-12B. When sampling is enabled (T = 1), perfor-
8

---

S No Mid
5-255
S Chain Tree
S
5 5//4
0.05
50.1
5-50%
S.775%%
1/2
1//4
0.4
Dyn. Ent.
Accept Length
Speedup
1.0
2:00
0.99
1.0
0.5
0.8
0.8
MMT
SEED
Sci. QA
LV
LV
PX
PX
MMT
SEED Sci. QA
MMT
SEED
Sci. QA
MMT
SQ
MMT S0
(a)
(b)
(c)
(d)
Figure 4: Normalized speedup S and normalized accepted token length across (a) intermediate feature selection strategies. (b) chain-based and tree-based decoding. (c) visual token compression ratios, where and 3/4 denote 100% and 75% of the visual tokens are retained, respectively. (d) loss weight settings, where the number is the value for feat and Xintermed- ÄKL is fixed to 1.
mance moderately degrades due to increased token variance, yet DREAM's tree-based architecture maintains robust performance. On MMT-Bench with Pixtral-12B DREAM still delivers a speedup 2.90x and т 4.02, outperforming all baselines under the same temperature.
4.2 Ablation Study
Impact of Draft Model Archi- tecture In this section, eval- uate the impact of draft model ar- chitecture by testing DREAM V--icuna-7BB across multiple datasets, with VTC dis- abled for all. We design six vari- ants of the DREAM draft model to assess architectural contribu- tions. In the first three baseline configurations, we remove one component at a time: the initial decoder block, denoted w/o Initial, the cross-attention block (w/o CA), the final decoder block (w/o Final). Additionally, we include a variant with an extra cross-attention block to evaluate the effect of deeper feature fusion, denoted as 2 CA. We further evaluate EAGLE-2 by varying its draft model depth, using one, three, and four decoder blocks, denoted as E2- (default setting in EAGLE-2), E2-3B, and E2-4B, respectively. Table summarizes the results. Notably, the removal of the cross-attention block causes the most substantial performance drop. showing its essential role in the draft model architecture. Adding an extra cross-attention block increases the accepted sequence length, indicating improved draft quality. However, the larger draft model also incurs additional latency, which offsets the overall speedup gains. Similarly, increasing the number of blocks in the EAGLE-2 draft model improves the average acceptance length, but speedup diminishes once the depth reaches four blocks. Overall, DREAM outperforms EAGLE-2 in both speedup and acceptance length, highlighting the importance of the cross-attention block in DREAM.
Table 2: Comparison over different model architectures. Numbers are normalized to DREAM results.
MMT-Bench
SEED-Bend Bench
ScienceQA
Baseline
Speedup
Speedup
Speedup
DREAM
1
w/o Initial
0.60
0.59
0.60
0.58
0.64
0.61
w/o CA
0.48
0.44
0.46
0.41
0.41
0.32
w/o Final
0.73
0.63
0.71
0.65
0.67
0.71
2 CA
0.99
1.09
1.02
1.13
0.79
1.10
E2-1B
0.82
0.71
0.81
0.77
0.80
0.75
E2-3B
0.88
0.86
0.89
0.88
0.91
0.88
E2-4B
0.63
0.92
0.63
0.92
0.68
0.93
Impact of Intermediate Feature Selection In this section, we evaluate DREAM's intermediate feature selection strategy for draft model training, referred to as Dynamic Entropy (Dyn. Ent.) against four baseline methods on the LLaVA-v1.C wtw.v....Vvvuna---.
model across the MMT-Bench, SEED-Bench, and ScienceQA datasets. The first baseline, No Mid Tuning (No Mid), trains the draft model without using any intermediate features. The second baseline, Static-25% (S-25%), uses intermediate features from the 25% depth of the target model (the 10th layer in LLaVA-v1. VA-v1.6.6666 .66 Vicuna-7B) for training. Similarly, Statie-50% (S-50% and Static-75% (5%7%% use features from the 50% and 75% depths, corresponding to the 20th and 30th layers, respectively. Figure 4 (a) presents the results. The draft model trained without intermediate features achieves limited speedup, as fewer tokens are accepted. Static-25% and Static-50% yield comparable speedup, while Static-75% performs best among the static approaches, suggesting that deeper intermediate layers provide more informative guidance. Dynamic Entropy applied by DREAM outperforms all basclines with the highest speedup.

---

Impact of Tree Structure As shown by prior work. stresstrctturd drafts improve the accepted sequence length compared to chain- structured drafts [25] 24 o Figure 4 (b) shows the performance of DREAM with both chain- and tree-structured drafts on the LLaVA-v1. 66 V-Vicuna-7B (IV) and Pixtral-12B (PX) models across the MMT-Bench (MMT) and ScienceQA (SQ) datasets. Tree decoding provides an average of 1.32x speedup on top of our proposed methods described in Section 3 It is important to note that all the baseline algorithms we compared except Kangaroo utilize tree-structured decoding. Even without the help of tree decoding, compared with Kangaroo, DREAM achieves an average of 56% and 75% speedup on LLaVA and Pixtral, respectively.
Impact of Visual Token Compression Rate To improve decoding efficiency, DREAM subsamples and retains only a fraction of visual tokens from the ViT encoder, as detailed in Section 3.3 In Figure 4 (c). we evaluate the effect of different visual token compression ratios, where fraction of indicates no compression, and 3/4 means 75% of the tokens are kept. Experiments on the LLaVA- v1.6- 16-Vicuna-7B 7B model across MMT-Bench, SEED-Bench, and ScienceQA show that retaining 3/4 of the tokens yields 7% speedup with only a minor reduction in acceptance length. However, when only 1/2 and 1/4 of the tokens are retained, the reduced visual input leads to noticeable information loss, lowering acceptance length and diminishing the speedup.
Specifically, we examine how the visual token compression (VTC) rate affects both inference speed and draft token accuracy in DREAM. We test four compres- sion levels (100%, 75%, 50%, and 25%) and evaluate their ef- fects using two key metrics: the speedup ratio (S) and the average accepted token length (T) across three benchmarks: MMTT-ench, SEED-Bench, and ScienceQA.
Table 3: Effect of Visual Token Compression Rate on Speedup and Token Acceptance
Models
VTC Rate
MMT-Bench
SEED-Bench
ScienceQA
S
S
S
100%
3.53
5.67
3.32
5.40
3.18
5.33
LLa
75%
3.68
5.58
3.51
5.34
3.36
5.29
Vicuna-13B
50%
3.19
4.96
3,13
5.04
3.11
4.98
25%
2.44
4.78
2.31
4.38
2.31
4.36
100%
2.88
4.48
2.55
3.87
2.91
4.52
Pixtral-12B
75%
2.93
4.52
2.61
3.67
2.98
4.03
50%
1.51
2.68
1.58
2.83
1.53
2.77
As presented in Table 3 the re- sults show that a 75% VTC rate generally offers the best balance between speed and accuracy. For LLaVA-13B and Pixtral- this setting achieves the highest or second- -ighhsst speedups (3.68x and 2.93x on MMT-Bench, respectively) while keeping T within about 2% of the full-token baseline. In comparison, reducing the visual tokens to 50% or 25% yields only minor speed improvements but causes т to drop by 10-25%, indicating that excessive visual compression leads more draft token rejections due to reduced contextual information.
25%
1.24
2.18
1.51
2.35
1.33
2.64
Impact of Lambda Settings The weights Afeat, Aintermed- and λ balance different supervisory signals. Since L and Lintermed are smooth LI losses of similar scale and play comparable roles in guiding the model by promoting feature alignment, tuning. The KL loss is typically smaller in magnitude and is fixed at λKι. = 1 to maintain consistent influence. As shown in Figure 4 (d), on V.aVA-v..6-Vvene improves speedup and average accepted token length. However, at 0.4, both metrics drop, indicating that excessive feature supervision may harm generalization.
set X Aintermed simplify
7B, increasing Afeat from 0.05 to 0.2
5 Conclusion and Limitation
We present DREAM, speculative decoding framework optimized for VLMs. By integrating visual token compression, cross-attention feature fusion, and adaptive intermediate distillation, DREAM achieves up to 3.6x speedup over standard decoding while maintaining high accuracy, offering scalable and efficient solution for fast multimodal inference.
Although DREAM shows strong performance, its evaluation is limited to NVIDIA GPUs. As future work, it is important to assess its effectiveness across a broader range of hardware platforms, datasets, and VLMs. Moreover, although DREAM improves the efficiency of VLM, enabling faster and more accessible deployment in real-world applications such assistive technologies and interactive agents. However, care should be taken to prevent misuse generating harmful multimodal content.
10

---

References
[1] Pravesh Agrawal, Szymon Antoniak, Emma Bou Hanna, Baptiste Bout, Devendra Chaplot, Jessica Chudnovsky, Diogo Costa, Baudouia De Monicault, Saurabh Garg. Theophile Gervet, et al. Pixtral 12b. arXiv preprint arXiv:2410.07075 2024.
[2] Zachary Ankner, Rishab Parthasarathy, Aniruddha Nrusimha, Christopher Rinard, Jonathan Ragan-Kelley and William Brandon. Hydra: Scquentially-depeddent draft heads for medusa decoding, arXiv preprint 200.v2240.005// 2024.
[3] Lucas Beyer, Andreas Steiner, André Susano Pinto, Alexander Kolesnikov, Xiao Wang, Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, Michael Tschannen, Emanuele Bugliarello, et al. Paligemma: versatile 3b vim for transfer. arXiv preprint arXiv:2407.0777 077.07776 2024.
[4] Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D Lee, Deming Chen, and Tri Dao. Medusa: Simple Ilm inference acceleration framework with multiple decoding heads. arXiv preprint arXiv:2401.1072 2024.
[5] Charlic Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptin Baptiste Lespiau, Laurent Sifre, and John Jumper Accelerating large language model decoding with speculative sampling. arXiv preprint 2rX1022302.01318 2023.
[6] Zhuoming Chen, Avner May. Ruslan Svirschevski, Yuhsun Huang, Max Ryabinin, Zhihao Jia, and Beidi Chen. Sequoia: Scalable, robust, and bardware-aware speculative decoding. arXiv preprint arXiv:2402.12377 12374. 2024.
[7] Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang. Boyang general-purpose
Li, Pascale Fung, and Steven Hoi. Instructblip: towards general purpose vision Lainunla models with instruction tuning. In Proceediggs of the 37th International Conference on Newral Information Processing Systems, NIPS 23, Red Hook, NY, USA, 2023. Curran Associates Inc.
[8] Mostafa Elhoushi, Akshat Shrivastava, Diana Liskovich, Basil Hosmer, Bram Wasti, Liangzhen Lai, Anas Mahmoud, Bilge Acun, Saurabh Agarwal, Ahmed Roman, et al. Layer skip: Enabling early exit inference and self specuuaeee decoding. arXiv preprint arXiv:2404. 16710, 2024.
[9] Yichao Fu, Peter Bailis, Ion Stoica, and Hao Zhang. Break the sequential dependency of Ilm inference using lookahead decoding. arXiv preprint ar 240.00240, 2024.
[10] Mukul Gagrani, Raghavv Goel, Wonseok Jeon, Junyoung Park, Mingu Lee, and Christopher Lott. On speculative decoding for multimodal large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8285 8289, 2024.
[11] Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, Louis Rouillard, Thomas Mesnard, Geoffrey Cidcron, Jean bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon, Etienne Pot, Ivo Penchev, Gaël Liu, Francesco Visin, Kathleen Kenealy, Lucas Beyer, Xiaohai Zhai, Anton Tsitsulin, Robert Busa-Fekete, Alex Feng, Noveen Sachdeva, Benjamin Coleman, Yi Gao, Basil Mustafa, Iain Barr, Emilio Parisotto. David Tian, Matan Eyal, Colin Cherry, Jan-Thorsten Peter, Danila Sinopalnikov. Surya Bhupatiraju, Rishabh Agarwal, Mchran Kazemi, Dan Malkin, Ravin Kumar, David Vilar, Idan Brusilovsky, Jiaming Luo, Audreas Steiner, Abe Friesen, Abhanshu Sharma, Abheesht Sharma, Adi Mayrav Gilady, Adrian Goedeckemeyer, Alaa Saade, Alex Feng. Alexander Kolesnikov, Alexei Bendebury, Alvin Abdagic, Amit Vadi, András György, André Susano Pinto, Anil Das, Ankur Bapna, Antoine Miech, Antoine Yang, Antonia Paterson, Ashish Shenoy, Ayan Chakrabarti, Bilal Piot, Bo Wu, Bobak Shahriari, Bryce Petrini, Charlic Chen, Charline Le Lan, Christopher A. Choquette-Choo Chho, CJ Carey, Cormac Brick, Daniel Deutsch, Danielle Eisenbud. Dee Cattle, Derek Cheng, Dimitris Paparas. Divyashree Shivakumar Sreepathihalli. Doug Reid, Dustin Tran, Dustin Zelle, Eric Noland, Erwin Huizenga. Eugene Kharitonov, Frederick Liu, Gagik Amirkhanyan, Glenn Cameron, Hadi Hashemi, Hanna Climezak-Plucifska. Harman Singh, Harsh Mehta, Harshal Tushar Lehri, Hussein Hazimeh, Ian Ballantyne, Idan Szpektor, Ivan Nardini, Jcan Pouget-Abadic, Jetha Chan, Joe Stanton, John Wicting, Jonathan Lai, Jordi Orbay, Joseph Fernandez, Josh Newlan, Ju yeong Ji, Jyotinder Singh. Kat Black, Kathy Yu, Kevin Hui, Kiran Vodrahalli, Klaus Linhai Qiu, Marcella Valentine, Marina Coelho, Marvin Ritter, Matt Hoffmann Matthew Watson, Mayank Chaturvedi, Michael Moynihan, Min Ma, Nabila Babar, Natasha Noy, Nathan Byrd, Nick Roy, Nikola Momchev, Nilay Chauhan, Noveen Sachdeva, Oskar Bunyan, Pankil Botarda, Paul Caron, Paul Kishan Rubenstein, Phil Culliton, Philipp Schmid, Pier Giuseppe Sessa. Pingmei Xu. Piotr Stanezyk, Pouya Tatti, Rakesh Shivanna. Renjie Wu, Renke Pan, Reza Rokni, Rob Willoughby, Rohith Vallu, Ryan Mullins, Sammy Jerome, Sara Smoot, Sertan Girgin, Shariq Iqbal. Shashir Reddy, Shruti Sheth, Siim Pöder. Sijal Bhatnagar, Sindhu Raghuram Panyam, Sivan Eiger, Susan Zhang, Tiangi Liu, Trevor Yacovone, Tyler
Greff,
11

---

Liechty. Uday Kalra, Uiku Evci. Vedant Misra, Vincent Roseberry. Vlad Feinberg. Vlad Kolesnikov, Woohyun Han, Woosuk Kwon, Xi Chen, Yinlam Chow, Yuvein Zhu, Zichuan Wei, Zoltan Egyed, Victor Cotruta, Minh Giang, Phoebe Kirk, Anand Rao, Kat Black, Nabila Babar, Jessica Lo, Erica Moreira. Luiz Gustavo Martins, Omar Sanseviero, Lucas Gonzalez, Zach Gleicher, Tris Warkentin, Vahab Mirrokni, Evan Senter, Eli Collins, Joelle Barral, Zoubin Ghahramani, Raia Hadsell, Yossi Matias, D. Sculley, Slav Petrov, Noah Fiedel, Noam Shazeer, Oriol Vinyals, Jeff Dean, Demis Hassabis, Koray Kavukcuoglu, Clement Farabet, Elena Buchatskaya, Jean-Baptiste Alayrac, Rohan Anil, Dmitry, Lepikhin, Sehastian Borgeaud, Olivier Bachem, Armand Joulin, Alek Andreev, Cassidy Hardin, Robert Dadashi, and Léonard Hussemot. Gemma 3 technical report, 2025.
[12] Ross Girshick. Fast r-cnn. In Proceedings of the IEEE international conference on computer vision, pages 1440-1448, 2015.
[13] Zhenyu He. Zexuan Zhong, Tianle Cai. Jason Lee, and Di He. Rest: Retrieval- Retrieval-based speculative decoding. arXiv preprint arXiv:231 2023.
[14] Coleman Hooper, Schoon Kim, Hiva Mohammadzadeh, Hasan Gene, Kurt Keutzer, Amir Gholami. and Sophia Shao. Speed: Speculative pipelined execution for efficient decoding. arXiv preprint arXiv:2310. 12072, 2023.
[15] Yunhai Hu, Zining Liu, Zhenyuan Dong, Tianfan Peng. Bradley McDanel, and Sai Qian Zhang. Speculative decoding and beyond: An in-depth survey of techniques. arXiv preprint arXiv:2502. 19732, 2025.
[16] Jitesh Jain, Zhengyuan Yang, Humphrey Shi, Jianfeng Gao, and Jianwei Yang. Ola-vim: Elevating visual perception in multimodal Ilms with auxiliary embedding distillation. arXiv preprint arXiv:2412.095555 2024.
[17] Byung- Kwan Lee, Ryo Hachiuma, Yu-Chi Chiang Frank Wang. Yong Man Ro, and Yueh-Hua Wu. Vlsi: Ver- balized layers-to- ivees-to-interaction from large to small vision language models. arXiv preprint 2r1.::241..01822. 2024.
[18] Minjae Lee, Wonjun Kang, Minghao Yan, Christian Classen, Hyung Il Koo, and Kangwook Lee. In-batch ensemble drafting: Toward fast and robust speculative decoding for multimodal language models.
[19] Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning, pages 19274-19286. PMLR, 2023.
[20] Bohan Li, Hankun Wang. Situo Zhang, Yiwei Guo, and Kai Yu. Fast and high- auto auto-regressive speech synthesis via speculative decoding. arXiv preprint arXiv:2410.21951, 2024.
[21] Bohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang, Rui Wang. Ruimao Zhang, and Ying Shan. Seed-bench- 2: Benchmarking multimodal large language models. arXiv preprint Xxxx:3311.17099 2023.
[22] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping Inguage-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19742. PMLR, 2023.
[23] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified language understanding and generation. In International conference on machine learning, pages 12888 12900. PMLR, 2022.
[24] Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. Eagle-2: Faster inference of language models with dynamic draft trees. arXiv preprint arXiv:2406. 16858, 2024.
[25] Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. Eagle: Speculative sampling requires rethinking feature uncertainty. arXiv preprint 2xXiv:2401.1507, 2024.
[26] Yuhui Ii, Fangyun Wei, Chao Zhang, and Hongyang Zhang. Eagle-3: Scaling up inference acceleration of large language models via training-time test. arXiv preprint 25Xiv:2503.01844 2025.
[27] Chen Liang, Simiao Zuo Qingru Zhang, Pengcheng He, Weizhu Chen, and Tuo Zhao. Less is more: task-aware layer-wise distillation for language model compression. In Proceedings of the 40th International Conference on Machine Learning, pages 20852-20867, 2023.
[28] Fangcheng Liu, Yehui Tang, Zhenhua Liu, Yunsheng Ni. Kai Han, and Yunhe Wang. Kangaroo: Lossless self-speculative decoding via double early exiting. arXiv preprint 24X4:24404.19)) 2024.
[29] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning, 2023.
12

---

[30] Haotian Liu. Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang. Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024.
[31] Haotian Liu, Chunyuan 1i, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information procesing systems, 36:34892-34916, 2023.
[32] Jiahao Liu, Qifan Wang. Jingang Wang, and Xunliang Cai. Speculative decoding via carly-exiting for faster Ilm inference with thompson sampling contral mechanism. arXiv preprint arXiv:2406..33833 2024.
[33] Tianyu Liu, Yun Li, Qitan Lv, Kai Liu, Jianchen Zhu, and Winston Hu. Parallel speculative decoding with adaptive draft length. arXiv preprint arXiv:2408. 1850, 2024.
[34] Xukun Liu, Bowen Lei, Ruqi Zhang, and Dongkuan Xu. Adaptive draft-verification for efficient large language model decoding. arXiv preprint arXiv:2407. 12022 2024.
[35] Yuliang Liu, Zhang Li, Mingxin Huang, Biao Yang, Wenwen Yu, Chunyuan Li, Xu-Cheng Yin, Cheng-I.in Lin Liu, Lianwen Jin, and Xiang Bai. Ocrbench: on the hidden mystery of oer in large multimodal models. Science China Information Sciences, 67(1),, December 2024.
[36] Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannanch Hajishirzi, Hao Cheng, Kei-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. In International Conference on Learning Representations (ICL.), 2024.
[37] Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In The 36th Conference on Neural Information Processing Systems (NeurlPS). 2022.
[38] Andrés Marafioti, Orr Zohar, Miquel Farré, Merve Noyan, Elie Bakouch, Pedro Cuenca. Cyril Zakka, Loubna Ben Allal, Anton Lozhkov, Nouamane Tazi, Vaibhav Srivastav, Joshua Lochner, Hugo Larcher, Mathicu Morlon, Lewis Tunstall, Leandro von Werra, and Thomas Wolf. Smolvim: Redefining small and efficient multimodal models. arXiv preprint arXiv:2504.05299 2025.
[39] Ahmed Masry, Do Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. ChartQA: A benchmark for question answering about charts with visual and logical reasoning. In Findings of the Association for Computational Linguistics: ACL 2022, pages 2263-222 2269, Dublin, Ireland, May 2022. Association for Computational Linguisties.
[40] Xupeng Miao, Gabriele Oliaro, Zhihao Zhang. Xinhao Cheng, Zeyu Wang. Zhengxin Zhang, Rae Ying Yee Wong, Alan Zhu, Lijie Yang, Xiaoxiang Shi, et al. Specinfer: Accelerating generative large language model serving with tree-based speculative inference and verification. arXiv preprint 095701.335 2023.
[41] Giovanni Monea. Armand Joulin, and Edouard Grave Pass: Parallel speculative sampling, arXiv preprint arXiv:2311.1358/ 2023.
[42] Jie Ou, Yueming Chen, and Wenhong Tian. Lossless acceleration of large language model via adaptive n-gram parallel decoding. arXiv preprint arXiv:2404.08690 2024.
[43] Desh Raj, Gil Keren, Junteng Jia, Jay Mahadeokar, and Ozlem Kalinli. Faster speech-llame inference with multi-token prediction. arXiv preprint vrXiv:2409.08144 2024.
[44] Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets, 2015.
[45] Andrea Santilli, Silvio Severino, Emilian Postolache, Valentino Maiorca, Michele Mancusi, Riccardo Marin, and Emanuele Rodolà. Accelerating transformer inference for translation via parallel decoding. arXiv preprint arXiv:2305.104: 2022.
[46] Oscar Skean, Md Rifat Arefin, Dan Zhao, Niket Patel, Jalal Naghiyev, Yann LeCun, and Ravid Shwartz-Ziv. Layer by layer: Uncovering hidden representations in language models, 2025.
[47] Benjamin Spector and Chris Re. Accelcrating Ilm inference with staged speculative decoding, arXiv preprint arXiv:2308. 04623, 2023.
[48] Mitchell Stem, Noam Shazcer, and Jakob Uszkoreit, Blockwise parallel decoding for deep autoregressive models. Advances in Neural Information Processing Systems, 31, 2018
[49] Lawrence Stewart, Matthew Trager, Sujan Kumar Gonugondla, and Stefano Soatto. The n-grammys: Accel- erating autoregressive inference with learning-fre free batched speculation. arXiv preprint arXi::241..0388. 2024.
13

---

[50] Hanshi Sun, Zhuoming Chen, Xinyu Yang, Yuandong Tian, and Beidi Chen. Triforce: Lossless acceleration of long sequence generation with hierarchical speculative decoding. arXiv preprint 11X1:22404.11112. 2024.
[51] Siqi Sun, Zhe Gan, Yuwei Fang. Yu Cheng. Shuohang Wang. and Jingjing Liu. Contrastive distillation on intermediate representations for language model compression. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 498-508 2020.
[52] Ziteng Sun, Ananda Theertha Suresh, Jae Hun Ro, Ahmad Beirami, Himanshu Jain, and Felix Yu. Spectr: Fast speculative decoding via optimal transport. Advances in Neural Information Processing Systems, 36, 2024.
[53] Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej. Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, et al. Gemma 3 technical report. arXiv preprint arXiv: 19786, 2025.
[54] Zilong Wang, Zifeng Wang, Long Le, Huaixiu Steven Zheng, Swaroop Mishra, Vincent Perot, Yuwei Zhang, Anush Mattapalli, Ankur Taly, Jingho Shang, al. Speculative rag: Enhancing retrieval augmented generation through drafting. arXiv preprint arXiv:2407.0822.; 2024.
[55] Heming Xia. Tao Ge. Peiyi Wang, Si- Si-Qing Chen, Furu Wei, and Zhifang Sui. Speculative decoding: Exploiting speculative execution for accelerating seq2seq generation. Findings of the Association for Computational Linguistics: EMNLP 2023, pages 3909-3925, 2023.
[56] Heming Xia, Yongqi Li, Jun Zhang, Cunxiao Du, and Wenjic Li. Swift: On-the-fly self-speculative decoding for Ilm inference accelcration. arXiv preprint arXiv:2410.06916, 2024.
[57] Nan Yang. Tao Ge, Liang Wang. Binxing Jiao, Daxin Jiang. Linjun Yang, Rangan Majumder, and Furu Wei. Inference with reference: Lossless acceleration of large language models. arXiv preprint ar arXiv:2304..4445. 2023.
[58] Kaining Ying. Fanqing Meng. Jin Wang, Zhiqian Li, Han Lin, Yue Yang, Hao Zhang, Wenbo Zhang, Yuqi Lin, Shuo Liu, Jiayi Lei, Quanfeng Lu,, Runjian Chen, Peng Xu, Renrui Zhang, Haozhe Zhang, Peng Gao, Yali Wang, Yu Qiao, Ping Luo, Kaipeng Zhang, and Wenqi Shao. Mmt-bench: A comprehensive multimodal benchmark for evaluating large vision-language models towards multitask agi. 2024.
[59] Zhengqing Yuan, Zhaoxu Li, Weiran Huang, Yanfang Ye, and Lichao Sun. Tinygpt-v: Efficient multimodal large language model via small backbones. arXiv preprint 2XXv:2332..16622 2023.
[60] Sergey Zagoruyko and Nikos Komodakis. Paying more attention to attention: Improving the perfor- mance of convolutional neural networks via attention transfer. In International Conference on Learning Representations, 2017.
[61] Jun Zhang, Jue Wang, Huan Li, Lidan Shou, Ke Chen, Gang Chen, and Sharad Mehrotra. Draft & verify: Lossless large language model acceleration via self-speculative decoding. arXiv preprint arXiv:2309.08 2023.
[62] Libo Zhang, Zhaoning Zhang, Baizhou Xu, Songzhu Mei, and Dongsheng Li. Dovetail: A epu/gpu beterogeneous speculative decoding for Ilm inference. arXiv preprint arXiv:2412.18934 2024.
[63] Baichuan Zhou, Ying Hu, Xi Weng, Junlong Jia, Jie Luo, Xien Liu, Ji Wu, and Lei Huang. Tinyllava: framework of small-scale large multimodal models. arXiv preprint arXiv:2402. 14289, 2024.
[64] Yongchao Zhou, Kaifeng Lyu, Ankit Singh Rawat, Aditya Krishna Menon, Afshin Rostamizadeh, Sanjiv Kumar, Jean- -ranFFannoos Kagy, and Rishabh Agarwal. Distillspec: Improving speculative decoding via knowledge distillation, arXiv preprint arXiv: 10.00866,, 2023.
14

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes].
Justification: The abstract and introduction accurately capture the paper's main contributions and scope. They clearly outline the proposed DREAM framework, including cross-attention fusion, entropy-based feature selection, and visual token compression, all of which are thoroughly developed and validated in the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes].
Justification: Yes, the limitation is discussed in the conclusion section.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "I "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and prool's
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
15

---

Answer:
[No]
Justification:
This paper does not involve theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proof's provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Yes, the paper provides sufficient information to reproduce the main exper- imental results to the extent that supports its core claims and conclusions. It details the model architectures, training procedures, datasets used (e.g.. LLaVA mix665k), loss func- tions, hyperparameters. evaluation benchmarks, and metrics. We will release the code upon acceptance of the work.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
16

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
We have provided the code with the supplementary materials.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. ec/ gublic/guidess CooeeSumissiionooo
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips ccpps.cc/ppp wttppppllllllppiee http://ooooooooasssoooooooeeeeee
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Yes, the paper specifies the key training and testing details necessary to understand the results, as depicted in the beginning of Section 4
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Yes, the paper specifies the error bars and their calculation details in the appendix, or as supplemental material.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
17

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We describe the hardware resources information in the beginning of Section
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: //te:://nnu.p..ee hww.rrpp..cppubli//Batee... cccpppulii//thicccccullllllllll
Answer: [Yes]
Justification:
The paper conforms with the NeurIPS Code of Ethics, providing transparency
in methodology and acknowledging limitations.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. if there special consid- eration due to laws or regulations in their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss it in the conclusion section of the paper.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
18

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We do not introduce any new models or datasets in this work.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Yes, the paper properly credits the original creators of all external assets used, including models, datasets, and baseline methods. Citations are provided for works such as LLaVA, Pixtral, and various speculative decoding methods.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
19

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the datasets.paperssiii paperswi thcode. www.com//aaaa datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
20

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does not involve LLMS as any important, original, non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy hatt:p///earrrppccc/ooooeereeee/ /ct/////oeeeeeccc//22////.. for what should or should not be described.
21


Reviews:
[{"review_id": "qqMHweyFlW", "reviewer": "Reviewer_5Pd8", "summary": "This paper present DREAM, a novel speculative decoding framework designed to accelerate vision-language models (VLMs) by incorporating cross-attention-based feature injection, adaptive feature selection, and visual token compression. It improves inference throughput and draft acceptance length, achieving up to 3.6× speedup over conventional decoding across various VLMs like LLaVA and Gemma3.", "questions": "1. **Performance Impact**: Since the final tokens are primarily generated by the draft model, could this lead to a performance drop compared to the original VLM?  \n\n2. **Adaptive Feature Selection**: In the adaptive intermediate feature selection method, \\( l^* \\) changes dynamically while \\( m = 1 \\) remains fixed. Given that different layers in the target model may contain distinct information, could this design introduce conflicting or ambiguous guidance?  \n\n3. **Tree Width (k) Sensitivity**: How does the choice of \\( k \\) (tree width) affect performance?  \n\n4. **Typo Correction**: Line 252 contains a duplicate \"are\"", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strengths**\nAccelerating large language models (LLMs) is crucial for enhancing their practical utility. This paper introduces a lightweight draft model to expedite long-token generation while leveraging supervision from the original LLM to maintain output quality. The writing is clear, the technical design is well-structured, and the experimental results effectively demonstrate the speed-up achieved by the proposed DREAM method.\n\n**Weaknesses**\nMy primary concern revolves around the loss design for adaptive intermediate feature selection, as well as the performance gap observed before and after applying the proposed speed-up method. Please refer to the question part for details.", "was_revised": true, "final_justification": "The author has addressed my concerns. \nGiven the practicality and innovation of the proposal of a lightweight draft model to expedite long-token generation, I tend to accept.", "rebuttal": "**Dear Reviewer 5Pd8**\n\nThank you for your insightful review. We are grateful that you recognized the practical importance of accelerating large language models and appreciated how DREAM addresses this crucial need through our lightweight draft model approach. We are particularly pleased that you highlighted (i) our method's ability to expedite long-token generation while maintaining output quality through target model supervision, (ii) the clarity of our writing and well-structured technical design, and (iii) the effectiveness of our experimental results in demonstrating DREAM's substantial speed-up achievements. Your acknowledgment of these core contributions validates our goal of developing a practical and deployable solution for VLM acceleration. We have carefully considered your questions and concerns, and provide detailed responses below to further clarify our design choices and strengthen the paper.\n\n**Q1:** Performance Impact: Since the final tokens are primarily generated by the draft model, could this lead to a performance drop compared to the original VLM?\n\n**A:** In our method, the draft model proposes candidate tokens, but crucially, each token in the final output must pass the acceptance test: a token $x$ is accepted with probability min(1, p(x)/q(x)), where p(x) and q(x) are the target and draft model probabilities respectively. This rejection sampling mechanism, proven in [1], mathematically guarantees that the output distribution is exactly p(x), which is identical to running the target model alone. The draft model has zero influence on the final distribution; it mainly enables parallel verification of multiple tokens by the target model. Therefore, DREAM achieves its $3.6\\times$ speedup while maintaining identical output quality to the original VLM.\n\n[1] Leviathan, Yaniv, Matan Kalman, and Yossi Matias. \"Fast inference from transformers via speculative decoding.\" In ICLR 2023.\n\n**Q2:** Adaptive Feature Selection: In the adaptive intermediate feature selection method, (l^*) changes dynamically while (m = 1) remains fixed. Given that different layers in the target model may contain distinct information, could this design introduce conflicting or ambiguous guidance?\n\n**A:** Our dynamic selection of $\\ell^*$ actually prevents conflicting guidance by ensuring the draft model always receives the most stable and informative features at each training step. The key insight is that minimizing attention entropy (Eq. 2) identifies layers where the model has converged to stable, focused representations; precisely the features that provide consistent supervisory signals across different contexts.\n\nThis adaptive mechanism offers significant advantages over fixed-layer selection. As shown in Figure 4(a), our entropy-based selection (Dyn. Ent.) achieves 1.41x speedup on MMT-Bench, outperforming all static baselines including S-75% (1.33x), which uses the deepest fixed layer. This improvement occurs because different tokens and contexts have varying optimal supervision layers, where some benefit from mid-layer semantic features while others need deeper syntactic representations.\n\nRegarding m=1, our ablation study definitively validates this design choice. As shown in Table 2, removing the initial decoder block (\"w/o Initial\") reduces $\\tau$ by 41% on MMT-Bench and 42% on SEED-Bench, while removing cross-attention (\"w/o CA\") causes a catastrophic 56% drop. The fixed m=1 position provides the optimal balance, because it is early enough to influence subsequent draft generation while late enough to benefit from input processing. This architectural choice, combined with dynamic $\\ell^*$ selection, enables DREAM to achieve consistent improvements across all benchmarks. We will clarify this point further in the revised version of the paper to ensure a comprehensive understanding.\n\n**Q3:** Tree Width (k) Sensitivity: How does the choice of ( k ) (tree width) affect performance?\n\n**A:** We appreciate your question as it raises an insightful point regarding the choice of tree width. The tree width k controls the breadth of parallel exploration during speculative decoding. We evaluated DREAM's sensitivity to k on Pixtral-12B across MMT-Bench and SEED-Bench. Our results shown in the Table below demonstrate that k=4 provides optimal performance, achieving speedup/accepted length (\\tau) of 2.88/4.48 on MMT-Bench, compared to 2.19/3.51 (k=1), 2.75/4.12 (k=2), and 1.89/4.58 (k=8). Beyond k=4, performance degrades, for example, at k=8, speedup drops to 1.89 despite $\\tau$ reaching 4.58. This occurs because larger trees increase verification latency, offsetting the benefits of higher acceptance rates. Similar patterns appear on SEED-Bench, where k=4 achieves 2.59/3.67 versus 2.42/3.32 (k=2) and 1.94/3.95 (k=8). Based on these findings, we use k=4 as our default, balancing exploration breadth with computational efficiency. We will include these additional ablations in the final manuscript.\n\n|  |  | MMT-Bench |  | SEED |  |\n| --- | --- | --- | --- | --- | --- |\n| Pixtral-12B | topk | Speedup | average acceptance lengths | Speedup | average acceptance lengths |\n|  | 1 | 2.19 | 3.51 | 2.02 | 2.94 |\n|  | 2 | 2.75 | 4.12 | 2.42 | 3.32 |\n|  | **4(DREAM)** | **2.93** | **4.42** | **2.61** | **3.67** |\n|  | 8 | 1.89 | 4.58 | 1.94 | 3.95 |\n\n**Q4:** Typo Correction: Line 252 contains a duplicate \"are\"\n\n**A:** Thank you for pointing that out. We will correct the typo on Line 252 by removing the duplicate \"are\" in the next revision of the paper."}, {"review_id": "oN0WckkjiV", "reviewer": "Reviewer_6Joc", "summary": "DREAM's key contributions include:\n\n(1) A cross-attention mechanism that injects intermediate features from the target model into the draft model, improving alignment and knowledge transfer. \n\n(2) Adaptive intermediate feature selection based on attention entropy, which dynamically guides draft model training for higher accuracy and longer token acceptance.\n\n(3) Visual token compression to reduce draft latency by subsampling critical visual inputs, leveraging target model features.", "questions": "Benchmarking for video MLLMs would make this paper more credible.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n​Significant Technical Innovation​:\n\n1. Cross-attention injection of target model features into the draft model, enhancing alignment.\n\n2. Entropy-adaptive intermediate feature selection (), improving draft accuracy.\n\n3. Visual token compression (75% retention) reducing latency without major accuracy loss ().\n\n​Rigorous and Comprehensive Evaluation​:\n\n1. Tests across 5 VLMs (LLaVA-7B/13B, Pixtral-12B, SmolVLM-2B, Gemma3-12B) and 8 diverse benchmarks (ScienceQA, OCRBench, etc.).\n\n2. Outperforms 6 strong baselines (SPD, Medusa, EAGLE-2) with ​up to 3.6× speedup​ and ​56% higher acceptance length​ than prior SOTA.\n\n3. ​Strong Ablation Studies to systematically validates each contribution.\n\n**Weakness**\n\nBenchmark Limitations​:\n\n1. MathVista/OCRBench show lower gains (fine-grained tasks), suggesting unresolved challenges for token-level precision.\n\n2. No testing on video VLMs.", "was_revised": false, "final_justification": "", "rebuttal": "**Dear Reviewer 6Joc**\n\nThank you for your thorough and constructive review. We are delighted that you recognized DREAM's significant technical innovations, particularly (i) our novel cross-attention injection mechanism that enhances target-draft model alignment, (ii) the entropy-adaptive intermediate feature selection that improves draft accuracy, and (iii) the visual token compression achieving 75% retention while maintaining performance. We especially appreciate your acknowledgment of our rigorous evaluation methodology, including (iv) comprehensive testing across 5 diverse VLMs and 8 benchmarks, (v) demonstrated superiority over 6 strong baselines with up to 3.6x speedup and 56% higher acceptance length than prior SOTA, and (vi) systematic ablation studies validating each contribution. Your recognition of both our technical advances and empirical rigor confirms that DREAM makes meaningful contributions to accelerating vision-language models. We have carefully addressed your questions below to further strengthen the paper and clarify the scope of our contributions.\n\n**Q1:** MathVista/OCRBench show lower gains (fine-grained tasks), suggesting unresolved challenges for token-level precision.\n\n**A:** Thank you for raising this insightful question. MathVista and OCRBench indeed present unique challenges due to their reliance on fine-grained visual details. MathVista [1] requires precise interpretation of mathematical diagrams, plots, and geometric shapes where small visual distortions can fundamentally alter meaning. Similarly, OCRBench [2] demands pixel-level accuracy for recognizing text in diverse contexts (documents, receipts, scene text) where character-level precision is critical.\n\nDespite these challenges, DREAM still achieves meaningful improvements on both benchmarks. As shown in Table 1, on MathVista with LLaVA-v1.6-Vicuna-7B (T=0), DREAM delivers 2.11x speedup with $\\tau$=5.32, compared to EAGLE-2's 1.87x/4.67. On OCRBench, while the gains are more modest (2.05x/4.88 vs EAGLE-2's 1.92x/4.88), DREAM maintains its lead. This performance gap on fine-grained tasks primarily stems from our visual token compression, which trades some pixel-level detail for efficiency. We acknowledge this limitation in Section 4.1 and view task-adaptive compression strategies as promising future work.\n\n[1] Lu et al. \"MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts.\" ICLR 2024.\n\n[2] Liu et al. \"OCRBench: On the Hidden Mystery of OCR in Large Multimodal Models.\" arXiv 2024.\n\n**Q2:** No testing on video VLMs, benchmarking for video MLLMs would make this paper more credible.\n\n**A:** We appreciate this suggestion and have conducted preliminary experiments on video VLMs during the rebuttal period. We trained DREAM on LLaVA-Video-7B-Qwen2 using 10k samples from LLaVA-Mix665K for 5 epochs. Given the limited rebuttal timeframe, this represents initial validation rather than full optimization.\n\n| LLaVA-Video-7B-Qwen2 | MMT-Bench |  | Seed-Bench |  |\n| --- | --- | --- | --- | --- |\n|  | Speedup | average acceptance lengths | Speedup | Accept Length |\n| **DREAM** | **1.68** | **2.18** | **1.77** | **2.28** |\n| EAGLE | 1.67 | 2.13 | 1.73 | 2.22 |\n\nThe results demonstrate that DREAM successfully extends to video VLMs, achieving 1.68x speedup on MMT-Bench and 1.77x on SEED-Bench, outperforming EAGLE's 1.67x/1.73x under identical training conditions. The lower speedups compared to image VLMs reflect video's additional temporal complexity and our limited training budget. We expect significant improvements with extended training and video-specific datasets. Video VLMs were initially out of scope due to their distinct architectural requirements (temporal modeling, frame aggregation) and computational demands. However, these promising results suggest DREAM's principles generalize well to video domains, and we will include this as an important future direction in our final version."}, {"review_id": "eSoYikzwE0", "reviewer": "Reviewer_Gi3S", "summary": "This paper presents a speculative method for vision-language model, which utilizes a cross-attention architecture to inject intermediate information from the target model into the draft model. To train the draft model efficiently, the authors propose to select the intermediate features adaptively. To reduce the latency of the draft model, the visual tokens are compressed during the training. The authors conduct experiments across several VLMs, including LLaVA, Pixtral, SmolVLm and Gemma3, somehow verifies the effectiveness of the proposed method.", "questions": "The description of speed-up ration in Line.302 is inconsistent with the results in Table.1", "limitations": "Y", "rating": 2, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 1, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths: \n\nThe paper is well organized and easy to follow. \n\nThe investigation of compressed visual input guided by intermediate target features sounds interesting, which reduces the latency of draft model. \n\nThe authors conduct plentiful experiments ablation studies. \n\nWeaknesses: \n\nThe design of draft model is simple and the analysis between the size of draft model and speed-up of spd is expected. \n\nThe dynamic feature selection may increase the training cost largely and there is no analysis about it in the manuscript. \n\nThe depth is set to 6 as mentioned in Line.279, however, the accepted token length is 6.4 in Table.1, which decreases the reliability of the experiments. \n\nAs mentioned in Line.269, the training dataset contains training samples from the evaluation benchmark, which is unreasonable. The experiments without the evaluation benchmark in the training dataset should be included. \n\nThe experimental results in Table.2 and Figure.4 are relative numbers, the absolute number would make the manuscript more legible.", "was_revised": true, "final_justification": "Thanks for the authors' responses. However, I recommend to reject this paper for the current time.\n\n- As shown in the code (see details from the discussion with the authors), the training and validation datasets are overlapped due to the mistakes and the shuffle operation, which I think is unacceptable.\n\n- The method need to be verified with the training dataset not containing the evaluation benchmarks, just as Eagle-2 does in the spd of LLM, which will set up a reproductive and more fair comparison for the following research in this field. From this aspect, this paper need to be major revised. Besides, the additional experimental results shows that the advantage of DREAM decreased largely in this situation(0.97 vs 0.82).\n\n- There are some critical mistakes such as the draft model size etc..\n\nI hope these discussions and comments will revise this paper in the future.", "rebuttal": "**Dear Reviewer Gi3S**\n\nThank you for your insightful comments. We’re glad you appreciated the paper’s clarity, the practical value of our target-feature-guided visual compression, and the thoroughness of our experiments. We have summarized your questions below and provided detailed responses to each.\n\n**Q1:** The design of draft model is simple and the analysis between the size of draft model and speed-up of spd is expected.\n\n**A:** DREAM's core contributions lie in three novel mechanisms for efficient knowledge transfer: cross-attention injection for better target-draft alignment, entropy-based adaptive feature selection for optimal supervision, and visual token compression for reduced latency. These innovations are architectural-agnostic and strategically paired with a simple draft model for optimal performance. In speculative decoding, model complexity directly reduces speedup through increased latency. **Table 1** validates this: our DREAM achieves 2.52× speedup ($\\tau$= 6.40), while adding one more layer drops to 2.49x despite marginal $\\tau$ gains. This design philosophy aligns with established practices where EAGLE and Medusa achieve SOTA with minimal architectures. The field consensus is clear: effective draft models prioritize speed over complexity, making our simple design essential for maximizing DREAM's practical impact.\n\nAn ablation study illustrating the relationship between draft model size and speedup ratio is presented in Table 2. Specifically, we vary the draft model size by removing individual layers, resulting in three baselines (denoted as w/o Initial, w/o CA, and w/o Final). We also evaluate a larger draft model variant (denoted as 2 CA). The results show that neither simplifying nor increasing the depth of the draft model yields better speedup compared to our default configuration. For example, removing layers like cross-attention (w/o CA) or the initial module (w/o Initial) indeed reduces model size, but leads to large drops in accepted token quality. On the other hand, increasing model depth (*2 CA*) marginally improves token acceptance length, but the larger draft leads to longer draft decoding time, resulting in no speedup or even degradation.\n\n**Q2:** The dynamic feature selection may increase the training cost largely and there is no analysis about it in the manuscript.\n\n**A:** In our design, the training pipeline is explicitly divided into two stages: (1) offline calibration and (2) draft model training. During the offline calibration stage, we perform dynamic feature selection by computing the attention entropy at each layer and selecting the optimal intermediate layer $\\ell^*$ for each token, as defined in Eq. 2. This process **does not introduce additional passes or significant computational overhead** during the draft model training stage. A similar two‑stage training regime is used by Medusa, EAGLE1 and EAGLE2: first generate data (the corresponding code can be found in the Medusa/data_generation folder of the Medusa GitHub repository and the eagle/gen_data folder in the EAGLE1 repository), then train the draft model. Moreover, this design allows training without loading the full target model, keeping only the draft components in memory and thereby reducing GPU memory, compute, and overall training cost.\n\nTo quantify the cost of dynamic intermediate feature selection, we measured the time required to compute $\\ell^\\*$ over a calibration set of 50,000 training samples on a single A100 80G GPU. The total computation time was approximately 500 seconds, averaging 10 milliseconds per sample. This overhead is negligible compared to the draft model training process, which can take up to 6 hours on a single GPU.\n\n**Q3:** The depth is set to 6 as mentioned in Line.279, however, the accepted token length is 6.4 in Table.1, which decreases the reliability of the experiments.\n\n**A:** We would like to clarify that this is actually a **fundamental feature of speculative decoding**, **not an error**. Although the draft model in DREAM generates up to 6 tokens per speculative decoding step (with depth equal to 6), if all tokens are successfully verified, the target model naturally returns the next candidate token as well. This results in a theoretical maximum of depth plus 1, meaning up to seven tokens can be accepted in a single round. Therefore, the maximum number of accepted tokens per round is **7**. The observed average of 6.4 indicates excellent draft quality.\n\nThis mechanism guarantees progress by ensuring that, even with a fixed draft depth, the system advances by at least one token in each verification pass[1]. A similar strategy is implemented in the EAGLE codebase (see the file *utils.py*, line 448).\n\n[1] Leviathan, et al. \"Fast inference from transformers via speculative decoding.\" PMLR, 2023.\n\n**Q4:** As mentioned in Line.269, the training dataset contains training samples from the evaluation benchmark, which is unreasonable. The experiments without the evaluation benchmark in the training dataset should be included.\n\n**A:** We appreciate the reviewer’s comment and would like to provide clarification. For DREAM training and evaluation, the dataset was strictly partitioned into **two disjoint subsets**: a training split used exclusively for training the draft model, and an evaluation split reserved solely for performance measurement. No samples from the evaluation set were used during training or hyperparameter tuning. We will add this detailed description to the final version of the paper.\n\nTo verify the exact implementation, we encourage the reviewer to refer to the code provided in the supplementary material. Specifically, the data‑generation script can be found at  DREAM_Code/dream/ge_data/ge_data_all_llava_vicuna_scienceQa.py (line 24).\n\nOnly loads the training partition; the test set is never used during training and is reserved solely for evaluation.\n\n**Q5:** The experimental results in Table.2 and Figure.4 are relative numbers, the absolute number would make the manuscript more legible.\n\n**A:** We thank the reviewer for the valuable suggestion. In Table 2 and Figure 4, we report DREAM’s performance across different settings using speedup, a standard metric in prior works such as Medusa, Kangaroo, and Eagle. To provide a more comprehensive view, we now include the absolute throughput (tokens per second) alongside the previously reported relative results in the table below. Due to space limitations, we retain the most representative results.\n\n|  | MMT-Bench |  |  | ScienceQA |  |  |\n| --- | --- | --- | --- | --- | --- | --- |\n| Llava-vicuna-7B | Speedup | Tokens Per Second | average acceptance lengths | Speedup | Tokens Per Second | average acceptance lengths |\n| w/o first attention layer | 1.51 | 30.55 | 3.65 | 1.50 | 30.30 | 3.53 |\n| w/o mid cross attention layer | 1.19 | 24.09 | 2.72 | 0.95 | 19.25 | 1.89 |\n| w/o last attention layer | 1.82 | 36.72 | 3.92 | 1.57 | 31.74 | 4.11 |\n| 1-layers eagle2 | 2.05 | 41.45 | 4.46 | 1.87 | 37.77 | 4.35 |\n| 3-layers eagle2 | 2.22 | 44.75 | 5.35 | 2.12 | 42.80 | 5.10 |\n| 4-layers eagle2 | 1.59 | 32.13 | 5.74 | 1.59 | 32.17 | 5.42 |\n| 2 cross attention | 2.49 | 50.34 | **6.97** | 1.84 | 37.20 | **6.42** |\n| **DREAM** | **2.52** | **50.63** | 6.40 | **2.33** | **47.11** | 5.82 |\n\n|  | MMT-Bench |  |  | SEED-Bench |  |  |\n| --- | --- | --- | --- | --- | --- | --- |\n|  | Speedup | Tokens Per Second | average acceptance lengths | Speedup | Tokens Per Second | average acceptance lengths |\n| No mid tuning | 1.63 | 33.38 | 4.27 | 1.65 | 33.72 | 4.47 |\n| first 25% layer | 2.05 | 42.01 | 4.86 | 2.09 | 42.76 | 5.66 |\n| Static 50% layer | 2.01 | 41.16 | 4.74 | 2.08 | 42.62 | 5.60 |\n| Static 75% layer | 2.29 | 46.96 | 5.02 | 2.27 | 46.49 | 5.89 |\n| Dynamic Entropy-Based Select | **2.52** | **50.63** | **6.40** | **2.48** | **50.09** | **6.20** |\n\n|  | Lllava-vicuna-7B |  |  |  |  |  | Pixtral |  |  |  |  |  |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n|  | Chain |  |  | **Tree Based** |  |  | Chain |  |  | **Tree Based** |  |  |\n|  | Speedup | Tokens Per Second | average acceptance lengths | Speedup | Tokens Per Second | average acceptance lengths | Speedup | Tokens Per Second | average acceptance lengths | Speedup | Tokens Per Second | average acceptance lengths |\n| MMT-Bench | 1.77 | 36.24 | 3.88 | **2.54** | **50.63** | **6.40** | 1.86 | 38.16 | 3.16 | **2.93** | **53.73** | **4.52** |\n| Science QA | 1.94 | 39.71 | 4.42 | **2.48** | **50.09** | **6.20** | 1.97 | 40.43 | 3.55 | **2.61** | **54.29** | **3.67** |\n\n| ViT Compression | MMT-Bench |  |  | SEED-Bench |  |  |\n| --- | --- | --- | --- | --- | --- | --- |\n| Percent of weight remaining | Speedup | Tokens Per Second | average acceptance lengths | Speedup | Tokens Per Second | average acceptance lengths |\n| 1 | 2.39 | 49.08 | 5.67 | 2.32 | 47.64 | 5.41 |\n| **¾(DREAM)** | **2.52** | **50.63** | **6.40** | **2.48** | **50.09** | **6.20** |\n| 1/2 | 2.23 | 45.78 | 4.96 | 2.19 | 44.92 | 5.04 |\n| 1/4 | 1.71 | 35.01 | 4.78 | 1.62 | 33.15 | 4.38 |\n\n| lambda | MMT-Bench |  |  | SEED-Bench |  |  |\n| --- | --- | --- | --- | --- | --- | --- |\n|  | Speedup | Tokens Per Second | average acceptance lengths | Speedup | Tokens Per Second | average acceptance lengths |\n| 0.05,0.05,1 | 2.44 | 48.02 | 6.21 | 2.33 | 47.77 | 6.01 |\n| 0.1,0.1,1 | 2.51 | 49.40 | 6.38 | 2.48 | 50.09 | 6.20 |\n| **0.2,0.2,1(DREAM)** | **2.52** | **50.63** | **6.40** | **2.48** | **50.09** | **6.20** |\n| 0.4,0.4,1 | 2.33 | 45.85 | 5.89 | 2.17 | 44.49 | 5.78 |\n\n**Q6:** The description of speed-up ration in Line.302 is inconsistent with the results in Table.1\n\n**A:** Thank you for pointing this out. The value mentioned in Line 302 was mistakenly attributed to the 13B model instead of the 7B model. We will correct this in the next revision of the paper."}, {"review_id": "9HCRKhAIyv", "reviewer": "Reviewer_o6oe", "summary": "The paper introduces DREAM, a novel speculative decoding framework for VLMs. It combines three key innovations: cross-attention injection for better feature alignment, entropy-based feature selection for efficient training, and visual token compression to cut draft model latency. These help DREAM achieve efficient, accurate, and parallel multimodal decoding. Experiments on popular VLMs show DREAM speeds up decoding by up to 3.6x and outperforms prior SD baselines in inference throughput and acceptance length across multimodal benchmarks.", "questions": "1.In the \"Adaptive Intermediate Feature Selection for Draft Training\" section, how does the average attention entropy (AE) help in selecting features that satisfy the two key criteria you mentioned—\"they can provide key information and capture rich semantic content, and they should be essential, exhibiting low variability across tokens\"? Could you provide a brief explanation of the underlying mechanism? Are there previous studies that have employed similar ideas? Additionally, is AE only applicable to Vision-Language Models or can it also be used for general Large Language Models?\n\n2.Notably, in Table 1, DREAM employs the same hyperparameters (e.g., Visual Token Compression Rate and Lambda Settings) for training and inference across different base models as those used for LLaVA-v1.6-Vicuna-7B. However, do different base models necessitate distinct optimal hyperparameter configurations? Could there be room for further hyperparameter tuning to better adapt to specific base models?\n\n3.To date, official implementations of Speculative Decoding (SD) methods such as Medusa and EAGLE do not natively support Vision-Language Models. To handle image inputs, modifications to their original architectures—such as adding a visual encoder—may be necessary. When reproducing these baselines in Table 1, did the authors make appropriate modifications to adapt these SD methods to VLMs? I would appreciate it if you could share some details of the reproduction process, particularly regarding how these baseline methods were adapted for VLM compatibility.", "limitations": "The authors have listed some major limitations in Section 5 \"Conclusion and Limitation\".", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1.DREAM adapt Speculative Decoding method to the scenario of VLMs by making targeted modifications such as the Cross-Attention Mechanism to align with the distinct characteristics of VLMs compared to LLMs, ensuring the method is tailored to their specific requirements.\n\n2.Unlike traditional methods that concatenate draft token embeddings with precomputed target features, the Cross-Attention Mechanism proposed by DREAM is novel and holds the potential to inspire subsequent research.\n\n3.Detailed experiments are conducted to validate the effective acceleration performance of DREAM across different base models and diverse tasks.\n\nWeaknesses:\n\n1.Although the authors compared multiple Speculative Decoding (SD) methods in the experimental section, including Medusa, EAGLE, etc., these SD methods are inherently designed for Large Language Models (LLMs). Notably, there remains a lack of comparison with other SD methods specifically tailored for Vision-Language Models (VLMs), such as the work referenced in the text titled \"On Speculative Decoding for Multimodal Large Language Models.\"\n\n2.During the training and inference of the draft model, the authors employ several techniques to enhance the acceleration effect. However, these techniques introduce additional hyperparameters that require tuning, and adapting to different base models may demand extensive optimization to achieve optimal performance, resulting in the method’s generalizability being slightly constrained.", "was_revised": true, "final_justification": "The authors have well addressed my concerns. \nStrengths:\nIntroduces a novel draft–target fusion mechanism that advances beyond simple concatenation.\nComprehensive experiments confirm consistent speed-ups across models and tasks.", "rebuttal": "**Dear Reviewer O6oe**\n\nThank you for your insightful comments. We are delighted that you highlighted (i) our targeted adaptation of speculative decoding to VLMs, most notably the cross‑attention, based alignment that addresses multimodal specifics beyond LLM settings, (ii) the novelty of our Cross‑Attention Mechanism compared with prior concatenation‑based approaches, which we hope will inspire follow‑up research on draft–target interaction, and (iii) the breadth of our experiments demonstrating effective acceleration across diverse base models and tasks. These were core goals of DREAM, and your recognition is much appreciated.\n\nWe have summarized your questions below and provided detailed responses to each.\n\n**Q1:** Although the authors compared multiple Speculative Decoding (SD) methods in the experimental section, including Medusa, EAGLE, etc., these SD methods are inherently designed for Large Language Models (LLMs). Notably, there remains a lack of comparison with other SD methods specifically tailored for Vision-Language Models (VLMs), such as the work referenced in the text titled \"On Speculative Decoding for Multimodal Large Language Models.\"\n\n**A:** We appreciate the reviewer's concern and would like to clarify that we did include the VLM-specific baseline mentioned. The work \"On Speculative Decoding for Multimodal Large Language Models\" appears as SPD [10] throughout Table 1, evaluated across all VLMs and benchmarks. Our results show DREAM significantly outperforms SPD, achieving an average speedup of 2.23x compared to SPD's 0.97x (a 130% improvement) and an average acceptance length of $\\tau$ = 5.51 versus SPD's $\\tau$ = 1.53 (a 260% improvement). In addition, our implementation of EAGLE in the baseline is consistent with another contemporaneous SD method for VLMs **MSD**[1]. The other methods (Medusa, Kangaroo) were originally LLM-focused and adapted by us for fair comparison. We will clarify this in the revision of this paper to avoid confusion.\n\n[1] Lin, Luxi, et al. \"Speculative Decoding Reimagined for Multimodal Large Language Models.\" arXiv (2025).\n\n**Q2:** In the \"Adaptive Intermediate Feature Selection for Draft Training\" section, how does the average attention entropy (AE) help in selecting features that satisfy the two key criteria you mentioned—\"they can provide key information and capture rich semantic content, and they should be essential, exhibiting low variability across tokens\"? Could you provide a brief explanation of the underlying mechanism? Are there previous studies that have employed similar ideas? Additionally, is AE only applicable to Vision-Language Models or can it also be used for general Large Language Models?\n\n**A:** As shown in Eq. 2, we select the layer $\\ell^\\*$ that minimizes the average attention entropy (AE). A lower AE indicates sharper and more focused attention, suggesting that the features encode clearer and more essential information with less noise, which better guides the training of the draft model. At the same time, to ensure effective information propagation within the target model, the selected features retain sufficient semantic richness, thereby satisfying the requirement for both key information and rich contextual representation. The selected layer $\\ell^\\*$ is then used to guide the draft model training, ensuring it consistently learns from the clearest and most stable representation for a given input. As shown in the ablation study (Line 336), this leads to higher verifier acceptance rates and improved speedup ratios.\n\nUsing attention entropy to select features from different layers has also appeared in prior work[1] which shows that mid-layer representations often outperform final-layer ones. Similar approaches have been used in layer importance analysis [2, 3] .These studies support the rationale behind our dynamic feature selection for the draft model. Different from previous work, we use the attention entropy to dynamically select intermediate layers of the target model to train the draft model, which provides stronger guidance during training, thus improving the overall performance. We appreciate your insightful questions and will clarify these points further in the final version of the paper to ensure a comprehensive understanding.\n\n[1] Skean et al. \"Layer by Layer: Uncovering Hidden Representations in Language Models.\" ICML 2025.\n\n[2] Cheng, Emily, et al. \"Emergence of a high-dimensional abstraction phase in language transformers.\" arXiv 2024.\n\n[3] Valeriani, Lucrezia, et al. \"The geometry of hidden representations of large transformer models.\" In NeurIPS 2024.\n\n**Q3:** Notably, in Table 1, DREAM employs the same hyperparameters (e.g., Visual Token Compression Rate and Lambda Settings) for training and inference across different base models as those used for LLaVA-v1.6-Vicuna-7B. However, do different base models necessitate distinct optimal hyperparameter configurations? Could there be room for further hyperparameter tuning to better adapt to specific base models?\n\n**A:** In Table 1, we fix the compression rate at 3/4, which yields great performance for most models.\n\nTuning hyperparameters for each model can further enhance performance. To demonstrate this, we conducted an additional analysis on Pixtral and SmolVLM, evaluating their performance under varying visual token pruning ratios. As shown in the table below, Pixtral and SmolVLM perform best when 3/4 and 7/8 token are kept, respectively. Enabling dedicated hyperparameter settings for each base model can further improve DREAM's performance by better aligning with the characteristics and capacities of individual models.\n\nAlthough dedicated hyperparameter settings could further improve performance, DREAM adopts a shared configuration across all VLMs, retaining three-fourths of the visual tokens. We will include these ablation results in the final version to guide practitioners on model-specific optimization.\n\n| Model | ViT Compression | MMT-Bench |  | SEED-Bench |  |\n| --- | --- | --- | --- | --- | --- |\n|  | Percent of weight remaining | Speedup | average acceptance lengths | Speedup | average acceptance lengths |\n| SMOLVLM | 1 | 2.88 | 4.08 | 2.25 | 3.33 |\n|  | **7/8** | **3.09** | **4.01** | **2.26** | **3.22** |\n|  | ¾(DREAM) | 3.05 | 3.97 | 2.24 | 3.18 |\n|  | 1/2 | 1.76 | 2.12 | 1.45 | 1.88 |\n| Pixtral | 1 | 2.85 | 4.56 | 2.55 | 3.87 |\n|  | 7/8 | 2.87 | 4.55 | 2.58 | 3.78 |\n|  | **¾(DREAM)** | **2.93** | **4.52** | **2.61** | **3.67** |\n|  | 1/2 | 2.15 | 3.67 | 1.98 | 2.76 |\n\n**Q4:** To date, official implementations of Speculative Decoding (SD) methods such as Medusa and EAGLE do not natively support Vision-Language Models. To handle image inputs, modifications to their original architectures—such as adding a visual encoder—may be necessary. When reproducing these baselines in Table 1, did the authors make appropriate modifications to adapt these SD methods to VLMs? I would appreciate it if you could share some details of the reproduction process, particularly regarding how these baseline methods were adapted for VLM compatibility.\n\n**A:** We made minimal, principled modifications to preserve the integrity of baseline comparisons and all speculative decoding logic while adding VLM input handling. VLMs such as LLaVA and Pixtral use a built‑in processor that first inserts special \\<image\\> token placeholders into the input text, then runs a visual encoder to convert each image into a sequence of embeddings, replaces those \\<image\\> tokens with the corresponding embedding sequence, and finally feeds the combined stream into the decoder layers for autoregressive generation. To Medusa and EAGLE over VLM, we did not change any of their core speculative‑decoding logic. Instead, we simply patched their prefill routines to invoke the VLM’s processor so that images are handled exactly as in standard VLM inference. After that single multimodal prefill pass, all candidate generation, verifier calls, and KV‑cache reuse run unmodified, just as in the original text‑only implementations. We have included the implementation in the supplementary materials (DREAM-main/dream/model/utils.py line 293), and we will publicly release the complete codebase upon acceptance of the paper."}]

Author rebuttal:

Meta-review: The paper presents DREAM, a speculative decoding framework for vision-language models that combines cross-attention-based draft–target alignment, entropy-adaptive feature selection, and visual token compression. The method yields up to 3.6x speedup and improved acceptance lengths, consistently outperforming baselines across multiple models and benchmarks. The work is technically novel, well-motivated, and supported by thorough experiments and ablations.

Three reviewers support acceptance, citing innovation, rigorous evaluation, and broad applicability. One reviewer maintains rejection due to concerns about potential issues with evaluation, though the authors provided detailed clarifications. Given the strong contributions, comprehensive validation, and positive consensus from most reviewers, I recommend acceptance.
Decision: Accept (poster)