Venue: NAACL 2025

# Speculative Diffusion Decoding:

Accelerating Language Generation through Diffusion

 Jacob K. Christopher

University of Virginia

csk4sr@virginia.edu &Brian R. Bartoldson

Lawrence Livermore National Laboratory

bartoldson1@llnl.gov &Tal Ben-Nun

Lawrence Livermore National Laboratory

talbn@llnl.gov &Michael Cardei

University of Virginia

ntr2rm@virginia.edu &Bhavya Kailkhurra

Lawrence Livermore National Laboratory

kailkhurla1@llnl.gov &Ferdinando Fioretto

University of Virginia

fioretto@virginia.edu

###### Abstract

Speculative decoding has emerged as a widely adopted method to accelerate large language model inference without sacrificing the quality of the model outputs. While this technique has facilitated notable speed improvements by enabling parallel sequence verification, its efficiency remains inherently limited by the reliance on incremental token generation in existing draft models. To overcome this limitation, this paper proposes an adaptation of speculative decoding which uses discrete diffusion models to generate draft sequences. This allows parallelization of both the drafting and verification steps, providing significant speedups to the inference process. Our proposed approach, _Specialative Diffusion Decoding (SpecDiff)_, is validated on standard language generation benchmarks and empirically demonstrated to provide up to 7.2x speedups over standard generation processes and up to 1.75x speedups over existing speculative decoding approaches.

## 1 Introduction

As autoregressive language modeling with transformers Vaswani et al. (2017) is scaled to larger compute levels, performance improves and new capabilities emerge Kaplan et al. (2020); Brown et al. (2020). Indeed, scaling has been shown to to improve the performance of large language models (LLMs) for a diverse array of tasks, including code generation, question answering, summarization, and many other use cases Achiam et al. (2023); Gemini Team (2023); Llama Team (2024). For instance, models such as LLMaMA 3.2 90B Llama Team (2024), ChatGPT OpenAI et al. (2024), Cohere 52B Ruis et al. (2023), Google's Gemini-ULTRA Team et al. (2024) exemplify the ongoing trend of deploying and releasing increasingly large models, enabling broader access and application across various domains.

However, while these desired capability arise, running LLMs in inference mode for millions of users produces burdensome electricity, time, and monetary demands. Many methods exist to mitigate these costs - including sparsity, quantization, and distillation - but they often introduce new tradeoffs, e.g., their application can degrade the performance of the model Hong et al. (2024).

Unlike other methods for accelerating LLM inference, _speculative decoding_Xia et al. (2023); Leviathan et al. (2023) can improve LLM efficiency by \(2\text{--}3\times\) with _no degradation in the quality of the model outputs_. In Leviathan et al. (2023), speculative decoding achieves this by sequentially generating multiple tokens with a small, efficient draft model, then running the target, large, LLM in parallel on all of the drafted tokens, simultaneously evaluating their consistency with the target LLM's output token probabilities. Provided that the drafting model's tokens are frequently accepted by the target model and that the drafting model operates substantially faster than the target model, speculative decoding can directly match the sampling output from the target model while significantly reducing runtime Leviathan et al. (2023). This functioning is shown in Figure 1 (left).

Notably, since both the drafting model's speed and its alignment relative to the target model are critical to the success of speculative decoding, simultaneous improvements in each of these areas are necessary to ensure speculative decoding's relevance to future, more capable target models. For instance, a small GPT-2 Radford et al. (2019) drafting model could produce drafts that are often re

jected by GPT-4 (Achiam et al., 2023), and simply scaling the drafting model to address its weaker generations risks diminishing the speed advantage necessary to speculative decoding's success. To address this challenge, previous efforts have focused on introducing additional parallelization techniques that incorporate prediction trees and branching to refine the drafting process Fu et al. (2024); Miao et al. (2023b); Svirschevski et al. (2024). However, the gain in generation efficiency are at the expense of much increased number of operations and/or memory for each generation.

This paper proposes a fundamentally different approach to improve speculative decoding: _It proposes to replace the auto-regressive drafter with recently introduced discrete diffusion models_Lou et al. (2024); Sahoo et al. (2024). These models offer several key advantages when used as drafters: Firstly, they provide a smooth trade off between the compute cost of generation and the quality of generation (via the number of reverse diffusion steps). Second, while they have historically struggled relative to traditional language models, recent diffusion models have been shown to require \(32\times\) fewer function evaluations than autoregressive models to produce text with comparable perplexity Lou et al. (2024), with more recent works reporting even further speedups Sahoo et al. (2024). This is a trend which is anticipated to continue. Finally, future advances in diffusion model generation quality are highly aligned with their ability to perform strongly as speculative drafts: as drafted tokens are accepted by the target model at a higher rate, a larger number of proposed drafted tokens becomes optimal from an efficiency/speed point of view, and (unlike sequential drafters) diffusion models can easily accommodate generation of many more tokens since they are able to generate entire sequences in a single step.

ContributionsMore specifically, this paper makes the following contributions:

1. It introduces a novel integration of generative diffusion language models with speculative decoding, schematically illustrated in Figure 1 (right).
2. It empirically demonstrate the hybrid model's ability to significantly accelerate inference times while maintaining the same high-quality outputs of the original, target large language model.
3. The proposed method ensures that all generations from the diffusion language model, which are empirically shown to produce outputs with significantly higher perplexity than current state-of-the-art autoregressive models Sahoo et al. (2024); Lou et al. (2024); Austin et al. (2021); Gloeckle et al. (2024), align with the outputs generated by larger, more computationally demanding models.
4. Finally, the paper sets a new benchmark for speed in language completion tasks on the CNN/DM Nallapati et al. (2016)and OpenWebText datasets Gokaslan et al. (2019).

Figure 1: Illustration of classical speculative decoding (left) and speculative diffusion decoding (right).

 

Related Work

While autoregressive language models provide state-of-the-art performance on language generation tasks, the incremental decoding used by these architectures results in significant overhead at inference time Miao et al. (2023). This is largely a result of the inability to parallelize the sequential process of generating tokens in the output sequence as each token generation is dependent upon the preceding tokens in the sequence; consequentially, scaling the compute associated with the inference cannot directly reduce this overhead when using standard decoding schemes. In recent literature studying how to accelerate large language model generation, two primary approaches have been explored: (1) advanced decoding implementations that better parallelize token generation and (2) non-autoregressive language models allowing full sequences to be generated simultaneously.

Speculative decoding.Speculative decoding accelerates autoregressive generation by leveraging a smaller autoregressive models of the same architecture (the drafter model) to predict candidate sequences, which the original model (the target model) then verifies Leviathan et al. (2023); Chen et al. (2023). Notably, the earliest literature on speculative diffusion adapted a non-autoregressive model to act as the drafter model Xia et al. (2023), using a masked language model with a bidirectional decoder Ghazvininejad et al. (2019). However, the integration of non-autoregressive draft models has not received much attention due to the difficulty introduced by the necessary additional training in existing approaches and the modest speedups that were previously reported using these methods (less than 2x speedup over vanilla decoding schemes).

Thus, recent advancements in speculative decoding have focused on overcoming memory-related constraints, with improvements achieved through various approaches: drafting directly with the target model Cai et al. (2024); Zhang et al. (2024), enhancing draft algorithms Sun et al. (2024), and introducing additional parallelization techniques that incorporate branching to refine the drafting process Fu et al. (2024); Miao et al. (2023); Svirschevski et al. (2024).

Non-autoregressive language models.Models which stray from the autoregressive paradigm have been shown to speedup generation by generating blocks or even entire sequences simultaneously. Gloeckle et al. (2024) propose a method of adapting traditional autoregressive models to sample blocks of tokens, improving inference time over similarly scaled models. In a similar vein, diffusion language models have been recognized for their efficiency in generating extended token sequences concurrently, offering even greater speed enhancements. These models recast language generation as a diffusion process either across the embedding space Austin et al. (2021) or, more recently, through the probability distributions of generated tokens Sahoo et al. (2024); Lou et al. (2024). Recent models report up to a 32x speedup over similarly sized GPT-2 models Lou et al. (2024), and current state-of-the-art further improves runtime speed Sahoo et al. (2024). However, despite they have been shown to dramatically accelerate the inference time for language generation, diffusion models typically perform less effectively than state-of-the-art autoregressive models in terms of standard language metrics, often exhibiting significantly higher perplexity scores Zheng et al. (2024). In the following section, _we will demonstrate, for the first time, how the speed of these models can be leveraged without being subject to this critical limitation._

## 3 Preliminaries and Settings

For open-ended language generation, we focus on the task of token generation, where given a sequence of tokens \(x_{1},x_{2},\ldots,x_{i}\), denoted here with shorthand notation \(x_{1:i}\), the goal is to generate the next \(n\) tokens \(x_{i+1},\dots,x_{i+n}\) from the conditional distributions \(p(x_{i+1}|x_{1:i}),\ldots,p(x_{i+n}|x_{1:i+n-1})\) or more succinctly \(p_{i+1},\ldots,p_{i+n}\).

Speculative decoding.Speculative decoding leverages two LLMs, \(M_{p}\) and \(M_{q}\), to parallelize token generation:

* \(M_{p}\) is the original, _target_, model whose output probability distributions for the tokens are \(p_{i+1},\ldots,p_{i+n}\).
* \(M_{q}\) is a smaller and more efficient _drafter_ model, used to generate approximations of the distribution of \(M_{p}\) as \(q_{i+1},\ldots,q_{i+n}\).

This process follows a _draft-then-verify_ approach Stern et al. (2018), where \(M_{q}\) efficiently computes a candidate sequence of tokens, which \(M_{p}\) then verifies in parallel.

During each speculative decoding iteration, \(M_{q}\) generates a subset of the total \(n\) tokens that are required for the generation task. The size of this 

subset is denoted as \(\gamma\). As shown in Figure 1 (left), the tokens \(x_{i+1:i+\gamma}\) sampled from \(M_{q}\) are then used by \(M_{p}\) to generate the corresponding probability distributions \(p_{i+1},\ldots,p_{i+\gamma}\). The distributions \(q_{i+1},\ldots,q_{i+\gamma}\) from \(M_{q}\) are stored for evaluating acceptance in subsequent steps. Critically, the target model's inference over \(p_{i+1},\ldots,p_{i+\gamma}\) can now be run in parallel as the model has access to tokens \(x_{i+1:i+\gamma}\), alleviating the sequential dependency for generation with \(M_{p}\).

To ensure high-quality outputs despite potential discrepancies between \(M_{p}\) and \(M_{q}\), tokens are subjected to an acceptance criterion. For each token \(x_{j}\) with \(j\in[i+1,i+\gamma]\), if \(q(x_{j})\leq p(x_{j})\), the token is accepted. If \(q(x_{j})>p(x_{k})\), the token is rejected with a probability of \(1-\frac{p(x_{j})}{q(x_{j})}\). This criterion is applied sequentially from left to right; rejection of any token results in the discard of all subsequent tokens. Hence, the token acceptance is maximized when the output distributions of \(M_{q}\) and \(M_{p}\) are closely aligned.

Previous literature quantifies the likelihood of token acceptance, denoted \(\alpha\), and theoretically demonstrate that \(\alpha=1-\mathbb{E}(D_{LK}(p,q))\) where \(D_{LK}\) represents a divergence measure between distributions Leviathan et al. (2023). This has led to the prevalent use of drafters taken from the same series as the target models, _a paradigm that we challenge in this paper_.

## 4 Speculative Diffusion Models

Speculative decoding has provided state-of-the-art results for improving language generation inference time but requires meticulous tuning of the associated hyperparameters to achieve optimal results. Particularly \(\gamma\), the sequence length generated by the drafter model, needs to be appropriately calibrated not only to maximize potential speedup but to even outperform standard autoregressive decoding. This is an important consideration when using current autoregressive draft models, provided that the inference time to generate \(M_{q}(x)\), the draft logits, is directly scaled by the size of \(\gamma\). Increasing this value too high reduces the number of operations that are conducted in parallel, _potentially leading to speculative decoding increasing inference time,_ while reducing this value too low results in speculative decoding "missing out" on token generations that could have been handled by the draft model.

Leviathan et al. (2023) has conducted theoretical analysis on how to best optimize the value of \(\gamma\), however, it has been contingent upon accurately estimating the percentage of tokens in a the sequence that will be accepted by the target model. By their own acknowledgment, it would be necessary to predict this value _for each draft_ and numerically solve for the optimal value of \(\gamma\) to fully realize the potential speedup of speculative decoding. Thus, a significant portion of the residual suboptimality in current implementations can be attributed directly to the sensitivity of this hyperparameter.

Diffusion language models are juxtaposed to conventional language models in that they do not sample token sequences in a sequential manner, rather generating entire sequences in parallel. This has resulted in significant speedup over similarly sized autoregressive models when generating extended sequences Lou et al. (2024). This can particularly be observed in longer sequence generations as scaling the draft length \(\gamma\) results in minimal overhead due to the ability to directly parallelize token generation.

### SpecDiff: Formulation

Diffusion models generally operate on continuous data spaces by progressively adding Gaussian noise to the data in a forward diffusion process and then learning to reverse this process to generate new samples Sohl-Dickstein et al. (2015); Ho et al. (2020); Song and Ermon (2019). This framework is well-suited for data types such as images, where pixel values can be treated as continuous and thus can naturally accommodate additive Gaussian noise. However, when dealing with discrete data like natural language (tokens), the assumption of continuous noise addition does not hold, as it would result in non-integer values that do not correspond to valid tokens.

Discrete diffusion models address this limitation by redefining the forward processes to transition between discrete token states, such as replacing tokens according to a transition matrix or introducing randomness through categorical distributions Austin et al. (2021); Hoogeboom et al. (2021); Sohl-Dickstein et al. (2015). This process enables the generation of coherent and meaningful sequences in natural language processing tasks. However, unlike continuous diffusion models, traditional score-matching techniques cannot be directly applied to learn discrete diffusion models. Instead, various surrogate losses have been proposed for training 

In particular, Sahoo et al. introduce the Masked Diffusion Language Model (MDLM), which gradually masks and then reconstructs tokens within a text sequence, enabling efficient text generation. MDLM optimizes a continuous-time Negative ELBO (NELBO) objective to minimize the negative log-likelihood over a continuous-time diffusion process, which can be formulated as follows:

\[L_{\infty}^{\text{NELBO}}=\mathbb{E}_{q}\left[\int_{t=0}^{t=1}\frac{(1-\beta_{ t})^{\prime}}{\beta_{t}}\sum_{i}\log(x_{\theta}(z_{i}^{t}),x_{i})\,dt\right] \tag{1}\]

where \(x_{\theta}(z_{i}^{t})\) represents the model's estimate of the original token \(x_{i}\) at time \(t\) given the current noisy state \(z_{i}^{t}\), and \(\beta_{t}\) denotes the noise schedule controlling the diffusion process. Here, \(\tilde{q}\) is the forward noising process of in the masked diffusion, defining the distribution over the noisy latent variable \(z_{i}^{t}\), and can be related to the NELBO as described in Equation (8) of Sahoo et al., 2024. The expectation \(\mathbb{E}_{\tilde{q}}\) signifies averaging over the possible outcomes of \(\tilde{q}\) allowing the model to account for all possible variations of \(z_{t}\). The Noise Schedule Derivative term, \(\frac{(1-\beta_{t})^{\prime}}{\beta_{t}}\), represents the rate of change in the noise schedule \(\beta_{t}\) over time, and \(t\) is a continuous timestep between 0 and 1.

This loss is directly used for pretraining our draft model. As this process learns to denoise over the probability mass vectors, the output of the draft model is a matrix of \(\mathbb{R}^{n\times m}\) where \(n=\gamma\) and \(m\) is the size of the vocabulary. The candidate sequence can then be generated using standard decoding methods over these probability mass vectors, the logits of which are stored to be used when determining whether to accept each draft token.

Now, the draft logits produced by the output matrix of the discrete diffusion drafter directly substitute the autoregressive drafter used to generate \(M_{q}([x_{0:i}]\circ[x_{i+1},\dots,x_{i+\gamma}])\), where \(\circ\) is a list concatenation operator. This substitutes the draft step taken by Leviathan et al. and Chen et al. and is the primary difference between SpecDiff and standard speculative decoding approaches. The subsequent steps of verifying this draft with the target model follow the previously proposed decoding algorithm, thus our proposal requires minimal modifications to existing speculative decoding code bases.

A complete overview of the SpecDiff decoding is provided in Algorithm 1 (adapted from Leviathan et al.).

```
\(\triangleright\) Take \(T\) diffusion steps to generate the draft. \(q_{t+1,...,t+\gamma}^{t}\sim\mathcal{N}(\mathbf{0},\sigma_{T}\mathbf{I})\)for\(t=T\)to ldo \(q_{t+1,...,t+\gamma}^{t-1}(x)\gets M_{q}([x_{0:}]\circ[q_{t+1,...,t+ \gamma}^{t}(x)],t)\) \(x_{i+1,...,t+\gamma}\sim q^{0}\) \(\triangleright\) Run \(M_{p}\) in parallel. \(p_{i}(x),\ldots,p_{i+\gamma+1}(x)\leftarrow\\ M_{p}(x_{0:i}),\ldots,M_{p}(x_{0:i+\gamma})\) \(\triangleright\) Determine the number of accepted guesses \(n\). \(r_{i}\sim U(0,1),\ldots,r_{i+\gamma}\sim U(0,1)\) \(n\leftarrow\min(\{j-1\mid i\leq j\leq i+\gamma,r_{j}>\frac{p_{j}(x)}{q_{j}(x)}\} \cup\{\gamma\})\) \(\triangleright\) Adjust the distribution from \(M_{p}\) if needed. \(p^{\prime}(x)\gets p_{n+1}(x)\)if\(n<i+\gamma\)then \(p^{\prime}(x)\gets norm(max(0,p_{n+1}(x)-q_{n+1}(x)))\) \(\triangleright\) Return one token from \(M_{p}\), \(n\) tokens from \(M_{q}\). \(t\sim p^{\prime}(x)\)return\(x_{1},\ldots,x_{n},t\)
```

**Algorithm 1**SpecDiff Decoding

Drafter's evaluations.Next, we highlight an important difference between standard speculative decoding and our approach. While in standard speculative decoding the number of evaluations by the drafter model is dictated by the value of \(\gamma\) (used in the first loop for Algorithm 1), in speculative diffusion it is dictated by the number of diffusion steps, \(T\). This allows SpecDiff to scale \(\gamma\) to higher values, as discussed further in Section 6.2.

Instead, the value of \(T\) is selected to optimize the trade-off between draft quality and computational overhead. While analysis by Lou et al. shows that lower values of \(T\) lead to higher perplexity in the generated sequence, this only impacts SpecDiff with respect to its effect on the percentage of tokens from the draft which are accepted (as can be noted in the analysis reported in Figure 2 and discussed in details Section 6.2).

Sequence initialization.An important consideration in implementing SpecDiff is the initial alignment between the diffusion draft model and the target model's data distribution since these models architectures are fundamentally different one another. A key strength of SpecDiff lies in its ability to leverage the alignment between the prefixes used in discrete diffusion and the target model's data distribution. Specifically, the better the prefixes align with the target distribution, the more effectively the diffusion drafter can generate longer, coherent sequences matching the target distribution, resulting in progressively higher acceptance rates.

This however, also means that when the diffusion draft model has not been finetuned, its output distribution may initially differ from that of the tar 

get model, potentially leading to lower acceptance rates at the beginning of the generation process. To address this, we employ standard speculative decoding for the initial few tokens, thereby optimizing SpecDiff's performance from the outset. This strategy ensures that SpecDiff achieves speedups that are empirically at least as significant as those observed with standard speculative decoding and can realize substantial improvements, as we show in Tables 2 and 4. Moreover, once the diffusion draft model is finetuned to better match the target distribution, this initial speculative decoding becomes unnecessary, as the drafter effectively aligns with the target model from the beginning.

## 5 Comparative Analysis of Speculative Decoders

Table 1 compares the computational aspects of SpecDiff with state-of-the-art baselines in speculative decoding.

According to the Work-Depth parallel computation model Blumoffe and Leiserson (1999), an algorithm can be represented by a DAG, in which each node is an operation and each edge is a dependency. The longest shortest path in this computational DAG (i.e., the longest dependency chain) is called the _depth_. It follows that the average parallelism of a computation is the total number of nodes divided by the depth, and that higher depth means fewer opportunities for parallelism. We represent the operations by the FLOPs of the model.

The table reasons about the approaches through three parameters for \(M_{p}\) and \(M_{q}\): \(C\) (memory consumption), \(F\) (arithmetic/floating-point operations, or FLOPs), and \(D\) (depth). Due to the quadratic memory and computation requirements of transformers, and for simplicity, \(C_{\{p,q\}}\) and \(F_{\{p,q\}}\) are represented as functions of the number of tokens after the initial prompt, i.e., \(C_{p}(k)\) is the memory cost for generating \(P+k\) tokens, where \(P\) is the number of prompt tokens. The depth \(D_{\{p,q\}}\) is generally independent of the number of tokens computed and is thus represented by a scalar.

In Medusa, the prediction tree is set by a number of fixed hyperparameters Cai et al. (2024), which we aggregate by using the number of nodes and the depth of the tree \(\mathcal{T}_{n}\) and \(\mathcal{T}_{d}\). Medusa-2 exhibits the same parallel properties as Medusa-1, but requires a joint fine-tuning of \(M_{p}\) along with \(M_{q}\). Hydra is an adaptation of Medusa, in which the heads are applied in sequence. It thus only affects the depth of the computations.

EAGLE uses a drafter model that contains the target embedding layer, one autoregressive layer, and the target LM head. The fixed tree size is 5 levels deep and contains 25 nodes Li et al. (2024b). EAGLE-2 Li et al. (2024a) dispenses with the fixed tree defined in EAGLE, and the induced dynamic tree used in generation is represented with \(\mathcal{T}\).

Notice that in our method, the depth of the algorithm is dependent on \(T\), the number of diffusion steps, rather than \(\gamma\). Empirically, \(T\) is smaller than \(\gamma\), which enables more parallelism in the computation w.r.t. the number of generated tokens. Combined with diffusion models enabling longer prediction horizons, this allows SpecDiff to produce more speculative predictions faster.

Practically, we observe that this enables SpecDiff to produce near state-of-the-art inference acceleration while requiring significantly less computational overhead and reduced memory requirements.

## 6 Evaluation

### Experimental Setup

Settings.To empirically evaluate the improvements provided by using SpecDiff, the paper provides an empirical analysis on three standard natural language processing tasks: **(1)** text summarization using the CNN/DM dataset Nallapati et al. (2016), **(2)** text generation on the OpenWebText OWT dataset Gokaslan et al. (2019), and **(3)** text generation using MT Bench Zheng et al. (2023). Additionally, we assess the performance of our method against the current state-of-the-art using _SpecBench_, a unified evaluation platform for speculative decoding techniques Xia et al. (2024).

In each setting, the model is queried for 1024 tokens using a greedy decoding scheme (temperature = 0). For the experiments, we evaluate the pre-trained target models \(M_{p}\) GPT-2 XL (1.5B), GPT-NEO (2.7B), and Vicuna (33B). We use Masked Diffusion Language Model (110M) as our drafter model \(M_{q}\), which is a comparable size to the baseline drafter GPT-2 (86M) employed for our standard speculative decoding baseline Sahoo et al. (2024).

All evaluation is conducted on two NVIDIA A100 series GPUs (80GB) using CUDA 12.2. Additionally, FlashAttention Dao et al. (2022) is used to optimize the performance in all experiments.

Evaluation metrics.Our method is assessed empirically by walltime speedup, acceptance rate \(\alpha\) 

and total floating point operations (FLOPs). The reported results are compared to recognized baselines of vanilla autoregressive decoding, standard speculative decoding (SpS) implementations as proposed by Leviathan et al.; Chen et al., which our method most closely resembles, and Eagle-2 Li et al. (2024a), the state-of-the-art for speculative decoding methods. Additionally, we provide comprehensive comparison to these method, as well as the current fastest-to-date speculative decoding approaches, detailing the improvements SpecDiff provides in reduction of FLOPs and memory footprint in Table 1.

### Results and Discussion

Empirically we highlight the comparison between our approach, other speculative decoding methods, and vanilla autoregressive generation. Across the tested settings and target model architectures, SpecDiff significantly outperforms standard speculative decoding, achieving speedups of up to 7.2x compared to the target models and **increasing the efficiency of standard speculative decoding by more than 1.75x**. Furthermore, SpecDiff rivals the performance of current state-of-the-art methods while **reducing the FLOPs/draft by over 33\(\%\) and reducing memory consumption (shown in Table 1).**

Table 2First, we compare SpecDiff to the performance of standard speculative decoding. Despite SpS generally reporting slightly higher acceptance rates, the **improved parallelization provided by SpecDiff results in greater speedups ranging from \(1.45\text{--}1.75\times\) improvement over the baseline.** As we have not finetuned the target model or draft model, we utilize sequence initialization with SpS to enhance generation speed by sampling the first 100 tokens of the sequence. We ablate the performance of SpecDiff without this enhancement in the Table 4.

Table 3To provide comparison to other state-of-the-art speculative decoding methods, evaluation is conducted on generation prompts from MT Bench Zheng et al. (2023) using Vicuna 33B as the tar

\begin{table}
\begin{tabular}{l c r r r r} \hline \hline
**Method** & \multicolumn{1}{c}{**Total Memory**} & \multicolumn{1}{c}{**FLOPs**} & \multicolumn{1}{c}{**Depth**} & \multicolumn{1}{c}{**Max**} & \multicolumn{1}{c}{**Extra Training**} \\  & \multicolumn{1}{c}{**Consumption**} & & & \multicolumn{1}{c}{**Tokens**} & \multicolumn{1}{c}{**Requirements**} \\ \hline Autoregressive & \(C_{p}(1)\) & \(F_{p}(1)\) & \(D_{p}\) & 1 & N/A \\ \hline SpS Leviathan et al. (2023)1  & \(C_{p}(\gamma)+C_{q}(\gamma)\) & \(F_{p}(\gamma)+F_{q}(\gamma)\) & \(D_{p}+\gamma D_{q}\) & \(1+\gamma\) & \(M_{q}\) \\ PLD Saxena (2023) & \(C_{p}(\gamma)\) & \(F_{p}(\gamma)+\mathcal{O}(Pn)\) & \(D_{p}+\mathcal{O}(n)\) & \(1+\gamma\) & None \\ Lookahead Fu et al. (2024)1  & \(C_{p}(2\gamma(n-1))\) & \(F_{p}\left(2\gamma(n-1)\right)+F_{p}(1)\) & \(D_{p}+\mathcal{O}(1)\) & \(n-1\) & None \\ Medusa-1 Cai et al. (2024) & \(C_{p}(\mathcal{T}_{n})+KC_{q}(\gamma)\) & \(F_{p}(\mathcal{T}_{n})+KF_{q}(\gamma)\) & \(D_{p}+D_{q}\) & \(1+\mathcal{T}_{d}\) & \(M_{q}\) \\ Medusa-2 Cai et al. (2024)2  & — & — & — & — & \(M_{p}\) \\ Hydra Ankner et al. (2024)3  & — & — & \(D_{p}+KD_{q}\) & — & \(M_{q}\) \\ EAGLE Li et al. (2024b) & \(C_{p}(25)+C_{q}(25)\) & \(F_{p}(25)+5F_{q}(5)\) & \(D_{p}+5D_{q}\) & 6 & \(M_{q}\) \\ EAGLE-2 Li et al. (2024a)4  & \(C_{p}(\mathcal{T}_{n})+C_{q}(\mathcal{T}_{n})\) & \(F_{p}(\mathcal{T}_{n})+\mathcal{T}_{d}F_{q}(\mathcal{T}_{n}^{*})\) & \(D_{p}+\mathcal{T}_{d}D_{q}\) & \(1+\mathcal{T}_{d}\) & \(M_{q}\) \\ \hline SpecDiff Ours & \(C_{p}(\gamma)+C_{q}(\gamma)\) & \(F_{p}(\gamma)+TF_{q}(\gamma)\) & \(D_{p}+TD_{q}\) & \(1+\gamma\) & \(M_{q}\) \\ \hline \end{tabular}
\end{table}
Table 1: Comparison of different speculative decoding strategies and their parallel properties. The quantities represent the cost of one additional decoding step (i.e., following the initial prompt). Note that \(M_{q}\) (and related quantities such as \(F_{q}\)) are not constant across methods; e.g., EAGLE uses a different draft model architecture than SpecDiff. \(T\) is the number of diffusion steps, \(\mathcal{T}\) is a sparse draft tree with \(\mathcal{T}_{n}\) nodes and depth \(\mathcal{T}_{d}\), \(K\) is the number of heads in a multi-head speculative decoder, \(P\) is the number of prompt tokens, \(n\) is an n-gram length, and \(\gamma\) is the number of proposed tokens. \({}^{1}~{}\mathcal{O}(1)\) represents a database lookup. \({}^{2}\) Medusa-1 with target fine-tuning for Medusa-2. \({}^{3}\) Medusa with sequential draft heads for Hydra. \({}^{4}\) EAGLE with dynamic draft trees for EAGLE-2.

\begin{table}
\begin{tabular}{|r l l l l l l|} \hline  & \multirow{2}{*}{\(M_{p}\)} & \multirow{2}{*}{\(M_{q}\)} & \multirow{2}{*}{\(M_{q}\)} & \multirow{2}{*}{\(\gamma\)} & \multirow{2}{*}{\(\alpha\)} & Speedup \\ \cline{5-8} \cline{9-8}  & & GPT-2 XL & GPT-2 & 8 & 0.92 & 3.58x \\  & & GPT-NEO & GPT-2 & 9 & 0.95 & 5.45x \\ \cline{2-8}  & & GPT-2 XL & MDLM & 15 & 0.87 & **4.80x** \\  & & GPT NEO & MDLM & 15 & 0.88 & **6.63x** \\ \hline \multirow{4}{*}{
\begin{tabular}{} \end{tabular} } & GPT-2 XL & GPT-2 & 8 & 0.93 & 3.66x \\  & & GPT-NEO & GPT-2 & 7 & 0.85 & 4.12x \\ \cline{1-1} \cline{2-8}  & & GPT-2 XL & MDLM & 15 & 0.89 & **5.38x** \\ \cline{1-1} \cline{2-8}  & & GPT NEO & MDLM & 20 & 0.88 & **7.23x** \\ \hline \hline \end{tabular}
\end{table}
Table 2: Evaluation of walltime speedup over autoregressive decoding using SpecDiff (Ours) compared to standard speculative decoding (SpS). The best result for each setting and target model is displayed in **bold**.

 

get model. Generation lengths are reduced to 512 tokens to better align with the task. For this setting, MDLM (141M) is pretrained with the Llama 1 tokenizer for compatibility with the target model used. As Vicuna is instruction tuned, we utilize a teacher-student instruction tuning approach resembling the knowledge distillation approach proposed by Zhou et al. to align the output distribution of MDLM with the target model. This approach is similar to the additional instruction tuning conducted to align Eagle and Eagle-2 on ShareGPT instruction tuning dataset Li et al. (2024b, a). Because of this finetuning step, it is not necessary to use sequence initialization as with pretrained models. Notably, SpecDiff provides a speedup \(70\text{--}85\%\) of the state-of-the-art methods, Eagle and Eagle-2, while **requiring 33\(\%\) fewer floating point operations per draft generation.** Thorough comparison to Eagle-2 and other speculative decoding strategies is highlighted in Table 1.

Table 4While we observe a slight decrease in SpecDiff's speed when standard speculative decoding is not used to initialize the generation, the change in \(\alpha\) is most noticeable when comparing to Table 2. This reflects the poor acceptance rates at the beginning of the generation. While SpecDiff does outperform SpS in this setting, _providing up to 1.5x speedups over SpS_, this highlights the benefit of adjointly using SpecDiff and other speculative decoding methods. Such hybrid approaches can be particularly effective for shorter generations.

While previous implementations of speculative decoding rely on a common architecture between the drafter and target models Leviathan et al. (2023); Chen et al. (2023), using smaller versions of the same architecture to generate draft sequences, these experiments demonstrate a robustness to using a completely different architecture for sequence

Figure 2: Evaluation of SpecDiff’s sensitivity to \(\gamma\) and number of diffusion steps when optimizing speed (left) and accepted tokens per draft (right) as reported on the OpenWebText task using GPT-2 NEO as the target model. Average token acceptance increases with \(T\) (x-axis), as examined theoretically in Appendix A. Despite this, the additional compute required as \(T\) is scaled results in reduced speedup.

 

drafting. _This is particularly significant given the absence of finetuning in the reported results (Tables 2 and 4)._ Pretrained diffusion models can be directly purposed as draft models requiring no additional training.

The much larger values of \(\gamma\) used for SpecDiff should particularly be highlighted. This is a key discrepancy between diffusion language models, which generate entire sequences in parallel, and autoregressive models. Hence, there is minimal overhead to increasing the sequence length generated by the diffusion-based drafter, and \(\gamma\) can be significantly increased without incurring significant cost.

The hyperparameters used in the reported results have been optimized empirically. We highlight that while in standard speculative diffusion the performance is highly sensitive to \(\gamma\), SpecDiff is robust to a range of values for \(\gamma\) making it unnecessary to precisely tune this hyperparameter (in our experiments we found between 10 and 20 worked well). Rather, SpecDiff's performance is much more sensitive to the number of diffusion steps selected. Similar to the role of \(\gamma\) in an autoregressive model, the number of diffusion steps \(T\) dictates the number of network evaluations during a single drafting step. As reported in the Figure 2, while increasing this hyperparameter arbitrarily results in higher values of \(\alpha\), SpecDiff performs best when this is optimized to balance the objectives of maximizing the number of accepted tokens and minimizing the drafter's overhead.

## 7 Conclusion

Motivated by the costly inference time of current large language models, this paper has proposed the novel integration of discrete diffusion models with autoregressive language models. The proposed method, Speculative Diffusion Decoding, alters existing speculative decoding schemes to integrate a non-autoregressive diffusion model as the draft model. As shown by the empirical evaluation on standard language generation benchmarks, the proposed method leverages the dramatic runtime advantages of discrete diffusion models while also maintaining the dramatically higher generation quality of autoregressive target models. The reported results demonstrate the utility of this approach in effectively accelerating runtime, outperforming vanilla decoding by over 7x and speculative decoding methods by over 1.75x.

## 8 Limitations

While our proposed Speculative Diffusion Decoding method represents a significant step forward in accelerating large language model inference, several limitations warrant discussion. Addressing these limitations highlights promising avenues for future research that could further enhance the efficiency and applicability of SpecDiff.

Calibration of discrete diffusion models.A primary limitation of our approach lies in the challenge of using different architectures for the drafter and verifier models. Specifically, the adopted discrete diffusion models do not output well-calibrated probability distributions that align with those of the target autoregressive models, particularly when the sampling temperature (\(T\)) is greater than zero. The diffusion models tend to produce over-confident predictions, often assigning near-certain probability to the top-1 token while assigning negligible probabilities to all other tokens. This results in deterministic sampling regardless of the temperature setting, challenging the applicability of SpecDiff in scenarios where diversity and stochasticity in generation are desired.

Thus, the development of techniques to better align the output probabilities of diffusion drafts with the target models is an key area of future work. Achieving proper calibration would enable effective use of SpecDiff at higher temperatures, as well as unlocking massive further speedups beyond what we have reported, as the acceptance rates are predicted to increase dramatically.

Limited tokenization availability.We note that available discrete diffusion models are based on the GPT-2 tokenizer, which would restrict one immediate compatibility with target models using different tokenization schemes. For our experiments, we indeed trained a new discrete diffusion model from scratch with a different tokenizer, a process that demanded substantial computational resources and time. All our models will be released on HuggingFace and thus be directly used by the community.

Performance on shorter generation tasks.We observe that SpecDiff exhibits optimal performance on longer sequence generations. In shorter generation tasks, the benefits of parallelization are less pronounced, and finetuning the diffusion drafter may be necessary to achieve comparable efficiency gains. Without finetuning, the drafter may not effec 

tively capture the target model's token distributions for shorter sequences.

An interesting outcome of these observations is that tailoring the drafter to better model shorter sequences, could improve its alignment with the target model, thereby maintaining speedups even in less extensive generation tasks. We believe that this adjustment may broaden SpecDiff's applicability.

Stochastic sampling.Note that our experiments are conducted with the sampling temperature set to zero, resulting in deterministic token generation. While this setting simplifies the verification process it limits the exploration of the model's capabilities in generating diverse outputs. In the future, we plan to exploring SpecDiff's performance at non-zero temperatures. As highlighted earlier, addressing the calibration issue of the diffusion drafter would enable effective stochastic sampling.

Despite these challenges, our work lays the foundational framework for integrating discrete diffusion models with autoregressive models in speculative decoding. Each limitation discussed highlights a specific area where further research could yield significant benefits for the performance of the proposed SpecDiff. We are optimistic that overcoming these challenges will not only reinforce the strengths of SpecDiff but also unlock new possibilities for accelerating language model inference.

## 9 Acknowledgements

This work is partially supported by NSF awards 2143706, 2232054, and 2242931. This work was performed under the auspices of the U.S. Department of Energy by Lawrence Livermore National Laboratory under Contract DE-AC52-07NA27344 and was supported by the LLNL-LDRD Program under Project No. 24-ERD-010 (LLNL-CONF-2000386). This research used resources of the National Energy Research Scientific Computing Center (NERSC), a Department of Energy Office of Science User Facility using NERSC award ASCR-ERCAP0027427. The views and conclusions provided in this paper reflect those of the authors only.

## References

* Achiam et al. (2023) Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. _arXiv preprint arXiv:2303.08774_.
* Ankner et al. (2024) Zachary Ankner, Rishab Parthasarathy, Aniruddha Nrusimha, Christopher Rinard, Jonathan Ragan-Kelley, and William Brandon. 2024. Hydra: Sequentially-dependent draft heads for medusa decoding. _Preprint_, arXiv:2402.05109.
* Austin et al. (2021) John Austin, David D. Johnson, Jonathan Ho, David Tarlow, and Remco van den Berg. 2021. Structured denoising diffusion models in discrete state-spaces. In _Advances in Neural Information Processing Systems_, volume 34, pages 17981-17993.
* Blumofe and Leiserson (1999) Robert D. Blumofe and Charles E. Leiserson. 1999. Scheduling multithreaded computations by work stealing. _J. ACM_, 46(5):720-748.
* Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. _Advances in neural information processing systems_, 33:1877-1901.
* Cai et al. (2024) Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D. Lee, Deming Chen, and Tri Tao. 2024. Medusa: Simple llm inference acceleration framework with multiple decoding heads. _Preprint_, arXiv:2401.10774.
* Chen et al. (2023) Charlie Chen, Sebastian Borgesaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, and John Junper. 2023. Accelerating large language model decoding with speculative sampling. _arXiv preprint arXiv:2302.01318_.
* Dao et al. (2022) Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Re. 2022. Flashattention: Fast and memory-efficient exact attention with io-awareness. _Advances in Neural Information Processing Systems_, 35:16344-16359.
* Fu et al. (2024) Yichao Fu, Peter Bailis, Ion Stoica, and Hao Zhang. 2024. Break the sequential dependency of llvm inference using lookahead decoding. _arXiv preprint arXiv:2402.02057_.
* Team (2023) Gemini Team. 2023. Gemini: a family of highly capable multimodal models. _arXiv preprint arXiv:2312.11805_.
* Ghazvininejad et al. (2019) Marjan Ghazvininejad, Omer Levy, Yinhan Liu, and Luke Zettlemoyer. 2019. Mask-predict: Parallel decoding of conditional masked language models. _arXiv preprint arXiv:1904.09324_.
* Gloeckle et al. (2024) Fabian Gloeckle, Badr Youbi Idrissi, Baptiste Roziere, David Lopez-Paz, and Gabriel Synnaeve. 2024. Better & faster large language models via multi-token prediction. _Preprint_, arXiv:2404.19737.
* Gokaslan et al. (2019) Aaron Gokaslan, Vanya Cohen, Ellie Pavlick, and Stefanie Tellex. 2019. Openwebtext corpus.
* Ho et al. (2020) Jonathan Ho, Ajay Jain, and Pieter Abbeel. 2020. Denoising diffusion probabilistic models. _Advances in neural information processing systems_, 33:6840-6851 

* Hong et al. (2024) Junyuan Hong, Jinhao Duan, Chenhui Zhang, LI Zhangheng, Chulin Xie, Kelsey Lieberman, James Diffenderfer, Brian R Bartdolson, AJAY KUMAR JAISWAL, Kaidi Xu, et al. 2024. Decoding compressed trust: Scrutinizing trustworthiness of efficient llms under compression. In _Forty-first International Conference on Machine Learning_.
* Hoogeboom et al. (2021) Erik Hoogeboom, Niels Bonnier, Tal Schuster Cohen, and Max Welling. 2021. Argmax flows and multinomial diffusion: Learning categorical distributions. In _Advances in Neural Information Processing Systems_, volume 34, pages 12454-12465.
* Kaplan et al. (2020) Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. _arXiv preprint arXiv:2001.08361_.
* Leviathan et al. (2023) Yaniv Leviathan, Matan Kalman, and Yossi Matias. 2023. Fast inference from transformers via speculative decoding. In _International Conference on Machine Learning_, pages 19274-19286. PMLR.
* Li et al. (2023) Gen Li, Yuting Wei, Yunxin Chen, and Yuejie Chi. 2023. Towards faster non-asymptotic convergence for diffusion-based generative models. _arXiv preprint arXiv:2306.09251_.
* Li et al. (2024a) Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. 2024a. EAGLE-2: Faster inference of language models with dynamic draft trees. In _Empirical Methods in Natural Language Processing_.
* Li et al. (2024b) Yuhui Li, Fangyu Wei, Chao Zhang, and Hongyang Zhang. 2024b. EAGLE: Speculative sampling requires rethinking feature uncertainty. In _International Conference on Machine Learning_.
* Team (2024) Llama Team. 2024. The llama 3 herd of models.
* Lou et al. (2024) Aaron Lou, Chenlin Meng, and Stefano Ermon. 2024. Discrete diffusion modeling by estimating the ratios of the data distribution. In _Fordy-first International Conference on Machine Learning_.
* Miao et al. (2023a) Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao Cheng, Hongyi Jin, Tianqi Chen, and Zhihao Jia. 2023a. Towards efficient generative large language model serving: A survey from algorithms to systems. _arXiv preprint arXiv:2312.15234_.
* Miao et al. (2023b) Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao Cheng, Zeyu Wang, Zhengxin Zhang, Rae Ying Yee Wong, Alan Zhu, Lijie Yang, Xiaoxiang Shi, et al. 2023b. Specimer: Accelerating generative large language model serving with tree-based speculative inference and verification. _arXiv preprint arXiv:2305.09781_.
* Nallapati et al. (2016) Ramesh Nallapati, Bowen Zhou, Caglar Gulcehre, Bing Xiang, et al. 2016. Abstractive text summarization using sequence-to-sequence rnns and beyond. _arXiv preprint arXiv:1602.06023_.
* OpenAI et al. (2016) OpenAI, Josh Achim, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Alleman, Disgo Almeida, Janko Atlemschirolt, San Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Beltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Jakhe Berdine, Gabriel Bernadett-Shapiro, Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madeline Boyd, Anna-Luisa Brakman, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Britany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decataux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoflet, Attyle Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Simon Posada Fishman, Juston Forte, Isabella Fullford, Leo Gao, Elie Georges, Christian Gibson, Vik Goel, Taurin Gogineni, Gabriel Goh, Rapha Gontijo-Lopes, Jonathan Gordon, Morgan Garstein, Scott Gray, Ryan Greene, Joshua Gross, Shixing Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidelberg, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizina, Shantanu Jain, Shawn Jaain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhu Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tumer Kartan, Lukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongik Kim, Jan Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Lukasz Konradcivalt, Andrew Kohrdrich, Aris Konstantinidis, Kyke Kosic, Gretench Krueger, Vishal Kuo, Michael Lamp, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lune, Anna Makanju, Kim Malfacini, Sam Manning, Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murat, Oleg Murk, David Melby, Ashvin Nair, Reichiro Nakano, Rajeev Nayak, Arvind Neelakantian, Richard Ngo, Heyonvoho Noh, Long Ouyang, Cuillen O'Keefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emy Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng, Adam Perellaman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Pokorny, Michelle Polkrass, Vitchyr H. Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Prochl, Raul Puri, Alec Radford, Jack Bae, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussee, Nick Ry 

der, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Seslam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shocker, Pranav Shyam, Syzmon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Alama, Ian Sohl, Benjamin Sokolowsky, Yang Song, Natalie Staudacher, Felipe Petroski Such, Natale Summers, Ilya Sutskever, Jie Tang, Nikolas Tezak, Madeleine B. Thompson, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Preston Tuggle, Nick Turley, Jerry Tworek, Juan Fepipe Ceron Uribe, Andrea Vallone, Arun Vijayergiya, Chelsea Voss, Carroll Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang, Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojciech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. 2024. Gpt-4 technical report. _Preprint_, arXiv:2303.08774.
* Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. _OpenAI blog_, 1(8):9.
* Ruis et al. (2023) Laura Ruis, Akbir Khan, Stella Biderman, Sara Hooker, Tim Rocktaschel, and Edward Greenstette. 2023. The goldilocks of pragmatic understanding: Fine-tuning strategy matters for implicative resolution by llms. _Preprint_, arXiv:2210.14986.
* Sahoo et al. (2024) Subham Sekhar Sahoo, Marianne Arriola, Yair Schiff, Aaron Gokaslan, Edgar Marroquin, Justin T Chiu, Alexander Rush, and Volodymyr Keleshov. 2024. Simple and effective masked diffusion language models. _arXiv preprint arXiv:2406.07524_.
* Saxena (2023) Apoorv Saxena. 2023. Prompt lookup decoding.
* Sohl-Dickstein et al. (2015a) Jascha Sohl-Dickstein, Eric Weiss, Nikhil Maheswaranathan, and Surya Ganguli. 2015a. Deep unsupervised learning using nonequilibrium thermodynamics. In _International Conference on Machine Learning_, pages 2256-2265.
* Sohl-Dickstein et al. (2015b) Jascha Sohl-Dickstein, Eric Weiss, Niru Mahneswaranathan, and Surya Ganguli. 2015b. Deep unsupervised learning using nonequilibrium thermodynamics. In _International conference on machine learning_, pages 2256-2265. PMLR.
* Song and Ermon (2019) Yang Song and Stefano Ermon. 2019. Generative modeling by estimating gradients of the data distribution. _Advances in neural information processing systems_, 32.
* Stern et al. (2018) Mitchell Stern, Noam Shazeer, and Jakob Uszkoreit. 2018. Blockwise parallel decoding for deep autoregressive models. _Advances in Neural Information Processing Systems_, 31.
* Sun et al. (2024) Ziteng Sun, Ananda Theertha Suresh, Jae Hun Ro, Ahmad Beirami, Himanshu Jain, and Felix Yu. 2024. Spectr: Fast speculative decoding via optimal transport. _Preprint_, arXiv:2310.15141.
* Svirchevski et al. (2024) Ruslan Svirchevski, Avner May, Zhuoming Chen, Beidi Chen, Zhihao Jia, and Max Ryabinin. 2024. Specexec: Massively parallel speculative decoding for interactive llvm inference on consumer devices. _arXiv preprint arXiv:2406.02532_.
* Team et al. (2021) Gemini Team, Rohan Anil, Sebastian Borgeaud, JeanBaptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schlakwyk, Andrew M. Dai, Anja Hauth, Katie Millican, David Silver, Melvin Johnson, Ioannis Antonoglou, Julian Schrittweiser, Amelia Glaese, Jilin Chen, Emily Pitter, Timothy Lillicrape, Angelik Lazaridou, Orhan Firat, James Molloy, Michael Isard, Paul R. Barham, Tom Hennigan, Benjamin Lee, Fabio Viola, Malcolm Reynolds, Yuanzhong Xu, Ryan Doherty, Eli Collins, Clemens Meyer, Eliza Rutherford, Erica Moreira, Kraerem Ayoub, Megha Goel, Jack Krawczyk, Cosmo Du, Ed Chi, Heng-Tze Cheng, Eric Ni, Purvi Shah, Patrick Kane, Betty Chan, Manal Farquii, Aliakesei Severyn, Hanzhao Lin, YaGuang Li, Yong Cheng, Abe Ittycherirah, Mahdis Mahdieh, Mia Chen, Pei Sun, Dustin Tran, Sumit Bagri, Balaji Lakshminarayanan, Jeremiah Liu, Andras Orban, Fabian Gura, Hao Zhou, Xinying Song, Aurelein Boffy, Harish Ganopathy, Steven Zheng, HyumJoung Choe, Agoston Weisz, Tao Zhu, Yifeng Lu, Siddharth Gopal, Jarrod Kahn, Macieje Klula, Jeff Pitman, Rushin Shah, Emanuel Taropa, Majd Al Marey, Martin Baeuml, Zhifeng Chen, Laurent El Shafey, Yujing Zhang, Olcan Sercinoglu, George Tucker, Enrique Piqueras, Maxim Kriktun, Iain Barr, Nikolay Savinov, Ivo Danihelka, Becca Roelofs, Anais White, Anders Andreassen, Tamara von Glehn, Lakshman Yagati, Mehran Kazemi, Lucas Gonzalez, Misha Khalman, Jakub Sygnowski, Alexandre Frechette, Charlotte Smith, Laura Culp, Lev Proleev, Yi Luan, Xi Chen, James Lottes, Nathan Schucher, Federico Lebron, Alban Frustemi, Natale Clay, Phil Crone, Tomas Kocisky, Jeffrey Zhao, Bartek Perz, Dian Yu, Heidi Howard, Adam Bloniarz, Jack W. Rae, Han Luurent Siffe, Marcello Magioni, Fred Alcober, Dan Garrente, Megan Barnes, Shantanu Thakoor, Jacob Austin, Gabriel Barth-Maron, William Wong, Rishabb Joshi, Rahma Chaabouni, Deeni Faiitha, Arun Ahuja, Gaurav Singh Tomar, Evvan Sentert, Martin Chadwick, Ilya Kornakov, Nithya Attaluri, Iizaki Iwtrate, Ruibo Liu, Yunxuan Li, Sarah Cogan, Jeremy Chen, Chao Jia, Chenjie Gu, Qiao Zhang, Jordan Grimstad, Ale Jakse Hartman, Xavier Garcia, Thanulayasan Sankaranayana Pillai, Jacob Devlin, Michael Laskin, Diego de Las Casas, Dasha Valter, Connie Tao, Lorenzo Blanco, Adria Puigdomeneda Badia, David Reitter, Minanna Chen, Jenny Brennan, Clara Rivera, Sergey Brin, Shariq Iqbal, Gabirela Surita, Jane Labanowski, Abhi Rao, Stephanie Winkler, Emilio Parisotto, Timing Gu, Kate Olszewska, Ravi Addanki, Antoine Miech, Annie Louis, Denis Teplyashin, Geoff Brown, Elliot Catt, Jan Balaguer, Jackie Xiang, Pidong Wang, 

Zoe Ashwood, Anton Briukhov, Albert Webson, Sanjay Ganapathy, Smit Sanghavi, Ajay Kannan, Ming-Wei Chang, Axel Sjergren, Josib Djolonga, Yuting Sun, Ankur Bapna, Matthew Aitchison, Pedram Pejman, Henryk Michalewski, Tianhe Yu, Cindy Wang, Juliette Love, Junwhan Ahn, Dawn Boxwich, Kehang Han, Peter Humphreys, Thibault Sellam, James Bradbury, Varun Godbole, Sina Samangooei, Bogdan Damoc, Alex Kaskasoli, Sebastien M. R. Arnold, Vijay Vasudevan, Shubham Agrawal, Jason Riesa, Dmitry Lepikhin, Richard Tauburn, Srivatsen Srinivasan, Hyeontaek Lim, Sarah Hodkinson, Pranav Shyam, Johan Ferret, Steven Hand, Ankush Garg, Tom Le Paine, Jian Li, Yuja Li, Minh Giang, Alexander Neitz, Zaheer Abbas, Sarah York, Machel Reid, Elizabeth Cole, Aakanksha Chowdhery, Dipjanjan Das, Dominika Rogozinska, Vitaliy Nikolaev, Pablo Sprechmann, Zachary Nado, Lukas Zilka, Flavien Prost, Luheng He, Marianne Monteiro, Gaura Mishra, Chris Welty, Josh Newman, Dawei Jia, Militadis Allamanis, Clara Huiyi Hu, Raoul de Liedekerke, Justin Gilmer, Carl Saroufim, Shruti Rijhwani, Shaobo Hou, Disha Shrivastava, Anirudh Baddepudi, Alex Goldin, Adnan Ozturel, Albin Cassirer, Yunhan Xu, Daniel Sohn, Devendra Sachan, Reinald Kim Amplayo, Craig Swanson, Desie Petrova, Shashi Narayan, Arthur Guez, Siddhartha Brahma, Jessica Landon, Miteyan Patel, Ruizhe Zhao, Kevin Villela, Luyu Wang, Wenhao Jia, Matthew Rahtz, Mai Gimenez, Legg Yeung, James Keeling, Petko Georgiev, Diana Micu, Bovic Wu, Salem Haykal, Rachel Saputto, Kirau Vodrahalli, James Qin, Zeynep Cankar, Abhanshu Sharma, Nick Fernando, Will Hawkins, Behnam Neyshabur, Solomon Kim, Adrian Hutter, Priyanka Agrawal, Alex Castro-Ros, George van den Driessche, Tao Wang, Fan Yang, Shov jin Chang, Paul Komarek, Ross McIlroy, Mario Lucic, Guodong Zhang, Wael Farhan, Michael Sharman, Paul Natsev, Paul Michel, Yamini Bansal, Siyuan Qiao, Kris Cao, Siamak Shakereri, Christina Butterfield, Justin Chung, Paul Kishaneh, Rubenstein, Shirani Agrawal, Arthur Mensch, Kodar Soparkar, Karel Lenc, Timothy Chuang, Acadne Pope, Loren Maggiore, Jackie Kay, Priya Jhakra, Shibo Wang, Joshua Mayzner, Mary Phuong, Taylor Tobin, Andrea Tacchetti, Maja Trebacz, Kevin Robinson, Yash Katariya, Sebastian Riedel, Paige Bailey, Kerfan Xiao, Nimesh Gheleni, Lora Aroyo, Ambrose Slone, Neil Houlsby, Xuehan Xiong, Zhen Yang, Elena Gribovskaya, Jonas Adler, Mateo Wirth, Lisa Lee, Music Li, Thais Kagohara, Jay Payagadhi, Sophie Bridgers, Anna Bortsova, Sanjay Ghemawat, Zarafail Ahmed, Tianqi Liu, Richard Powell, Vijay Bo lonia, Mariko Iinuma, Polina Zablotskaia, James Besley, Da-Woon Chung, Timothy Dozat, Ramona Comanescu, Xiance Si, Jeremy Greer, Guolong Su, Martin Polacek, Raphael Lopez Kaufman, Simon Tokumine, Hexiang Hu, Elena Buchatskaya, Yingjiae Miao, Mohamed Elhawaty, Adivis Siddhart, Nandel Tomasev, Jinwei Xing, Christina Greer, Helen Miller, Shereen Ashraf, Aurko Roy, Zixhao Zhang, Ada Ma, Angelos Filios, Milos Besta, Rory Blevins, Ted Klimenko, Chih-Kuan Yeh, Soravit Changpinyo, Jiaqi Mu, Oscar Chang, Mantas Pajarskas, Carrie Muir, Vered Cohen, Charline Le Lan, Krishna Haridasan, Amit Marathe, Steven Hansen, Sholfo Douglas, Rajkumar Samuel, Mingqiu Wang, Sophia Austin, Chang Lan, Jiepu Jiang, Justin Chiu, Jaime Alonso Lorenzo, Lars Lowe Sjosund, Sebastien Cevey, Zach Gleicher, Thi Avrahami, Anudhyan Boral, Hansa Srinivasan, Vittorio Selo, Rhys May, Konstantinos Aisopos, Leonard Husenoft, Livio Baldini Soares, Kate Baumli, Michael B. Chang, Adria Recasens, Ben Caine, Alexander Pritzel, Filip Pavetic, Fabio Pardo, Anita Gergely, Justin Frye, Vinay Ramasesh, Dan Horgan, Kartikeya Badola, Nora Kassner, Subhrajit Roy, Ethan Dyer, Victor Campos Campos, Alex Tomala, Yunhao Tang, Dalia El Badawy, Elspeth White, Basil Mustafa, Oran Lang, Abhishek Jindal, Sharald Vikram, Zhitao Gong, Sergi Caelles, Ross Hensley, Gregory Thornton, Fangxiaouy Feng, Weojciech Stokowice,ce Zheng, Phoechecker, Caglar Unlu, Zhishnuai Zhang, Mohammad Saleh, James Svensson, Max Bileschi, Piyush Patil, Ankesh Anand, Roman Ring, Katerina Tsihlas, Arpi Vezer, Marco Selvi, Toby Shevlane, Mikel Rodriguez, Tom Kwiatkowski, Samira Daruki, Kearan Rong, Allan Dafee, Nicholas FitzGerald, Keren Gu-Lemberg, Mina Khan, Lisa Anne Hendricks, Marie Pellat, Vladimir Feinberg, James Cobon-Kerr, Tara Sainath, Maribeth Rauh, Sayed Hadi Hashemi, Richard Ives, Yana Hasson, Eric Noland, Yuan Cao, Nathan Byrd, Le Hou, Qingze Wang, Thibault Sottiaux, Michela Paganimi, Jean-Baptiste Lespiau, Alexandre Mourfex, Samer Hassan, Kaushik Shrikakumar, Joost van Amersoftort, Amol Mandhane, Pratik Joshi, Anirudh Goyal, Matthew Tung, Andrew Brock, Hannah Sheahan, Vedant Misra, Cheng Li, Nemanja Rakicevic, Mostafa Dehghani, Fanygu Liu, Sid Mittal, Junhyuk Oh, Seb Noury, Eren Sezener, Fantine Huot, Matthew Lamm, Nicola De Cao, Charlie Chen, Sidharth Mudgal, Romina Stella, Kevin Brooks, Gauntan Vasudevan, Chenai Liu, Mainachain, Nievidta Melinkeri, Aaron Cohen, Venus Wang, Krstie Seymore, Sergey Zubov, Rahul Goel, Summer Yue, Kai Krishnakumaran, Brian Albert, Nate Hurley, Motoki Sano, Anhad Mohananey, Jonah Joughin, Egor Finonov, Tomasz Kepa, Yoma Eldawy, Jiavern Lim, Rahul Rishi, Shirin Badezidegan, Taylor Bos, Jerry Chang, Samil Jain, Sri Gayatri Sundara Padmanabhan, Subha Puttaguna, Kalpesh Krishna, Leslie Baker, Norbert Kalb, Vamsi Bedapudi, Adam Kurzrob, Shunting Lei, Anthony Yu, Oren Litvin, Xiang Zhou, Zhichun Wu, Sam Sobell, Andrea Siciliano, Alan Papir, Robby Neale, Jonas Bragaagnolo, Tejo Toor, Tina Chen, Valentin Anklin, Feliran Wang, Richic Feng, Mild Cholami, Kevin Ling, Lijun Liu, Justle Walter, Hamid Moghaddam, Arun Kishore, Jakub Adamek, Tyler Mercado, Jonathan Mallinson, Siddhinita Wandekar, Stephen Cagle, Eran Ofek, Guillermo Garrido, Clemens Lombriser, Maksim Mukha, Botu Sun, Hafeezahl Rahman Mohammad, Josip Mutak, Yadi Qian, Vikas Peswani, Pawel Janus, Quan Yuan, Leif Schelin, Onma David, Ankur Garg, Yifan He, Oleksii Duzbi, Anton Algmyr, Timothe Lottaz, Qi Li, Vikas Yadav, Luyao Xu, Alex Chinien, Rakesh Shivanna, Aleksandr Chuklin, Josie Li, Carrie Spadine, Travis Wolfe, Kareom Mohamed, 

Subhabrata Das, Zihang Dai, Kyle He, Daniel von Dincklage, Shyam Upadhyay, Akanksha Maurya, Luyan Chi, Sebastian Krause, Khalid Salama, Pam G Rabinovitch, Pavan Kumar Reddy M, Aarush Selvan, Mikhail Dektiarev, Golonz Miasi, Erdem Guven, Himanshu Gupta, Boyi Liu, Deepak Sharma, Idan Heimlich Shatcher, Shachi Paul, Oscar Akerlund, Francois-Xavier Aubet, Terry Huang, Chen Zhu, Eric Zhu, Elico Teixeira, Matthew Fritze, Francesco Bertolini, Liana-Eleomor Marinescu, Martin Bolle, Dominik Paulus, Khyatti Gupta, Tejasi Latkar, Max Chang, Jason Sanders, Roopa Wilson, Xuewei Wu, Yi-Xuan Tan, Lam Nguyen Thiet, Tulse Doshi, Sid Lal, Swaroop Mishra, Wanming Chen, Thang Luong, Seth Benjamin, Jasmine Lee, Ewa Andrejczuk, Dominik Rabiej, Vipul Ranjan, Krzysztof Stryc, Pengecheng Yin, Jon Simon, Malcolm Rose Harriott, Mudit Banal, Alexei Robsky, Geoff Bacon, David Greene, Daniil Mirylenka, Chen Zhou, Obaid Sarvana, Abhimanyu Goyal, Samuel Andermatt, Patrick Siegler, Ben Horn, Assaf Israel, Francesco Pongetit, Chih-Wei "Louis" Chen, Marco Selevatici, Pedro Silva, Kathite Wang, Jackson Tollins, Kelvin Guo, Rocy Yogev, Xiaochen Cai, Alessandro Agostini, Maulk Shah, Hung Nguyen, Noah O Donnaile, Sebasnstein Pereira, Linda Friso, Adam Stambler, Adam Kurzrok, Chenkai Kuang, Yan Romanikhin, Mark Geller, ZJ Yan, Kane Jang, Cheng-Chun Lee, Wojciech Fica, Eric Malmi, Qijun Tan, Dan Banica, Daniel Balke, Ryan Pham, Yanping Huang, Diana Avram, Hongzhi Shi, Jasjo Singh, Chris Hidey, Nihrakia Ahuja, Pranab Saxena, Dan Dooley, Srividya Prunaraju, Eileen O'Neill, Anand Gokulchandran, Ryan Foley, Kai Zhao, Mike Dusenberry, Yuan Liu, Pulkit Mehta, Ragha Kotikalapudi, Chalence Safraneek-Shrader, Andrew Goodman, Joshua Kessinger, Eran Globben, Prateek Kollar, Chris Gorgolewski, Ali Ibrahim, Yang Song, Ali Eichenkoum, Thomas Brovelli, Sahitya Potluri, Preethi Lahoti, Cip Baetu, Ali Ghorbani, Charles Chen, Andy Crawford, Shalin Pal Mukund Sridhar, Petru Gurita, Asier Mujika, Igor Petrovski, Pierre-Louis Cedoz, Chemeii Li, Shiyuan Chen, Niccolo Dal Santo, Siddharth Goyal, Jitesh Punjabi, Karthik Kapagnathu, Chester Kwak, Pallavi LV, Sarmishta Velny, Himadi Choudhary, Jamie Hall, Premal Shah, Ricardo Figueira, Matt Thomas, Minjie Lu, Ting Zhou, Chintut Kumar, Thomas Jurdi, Sharat Chhikkerur, Yenai Ma, Adams Yu, Soo Kwak, Victor Abdel, Sujeveau Rajayogam, Travis Choma, Fei Liu, Aditya Barua, Colin Ji, Ji Ho Park, Vincent Hellendoorn, Alex Bailey, Taylan Bilal, Huanjie Zhou, Mehrdad Rahitar, Charles Sutton, Wejocich Rezdkowski, Fiona Macintosh, Konstantin Shagin, Paul Medina, Chen Liang, Jinjing Zhou, Pararth Shah, Yingying Bi, Attila Dankoviacs, Shipra Banga, Sabine Lehmann, Marissa Predesen, Zifan Lin, John Eric Hoffmann, Jonathan Lai, Raynald Chung, Kai Yang, Nihal Balani, Arthur Brazilianskas, Andrei Sozanscchi, Matthew Hayes,ictor Fernandez Alcalde, Peter Makarov, Will Chen, Antonio Stella, Liselotte Snijders, Michael Mandl, Ante Kirrman, Pawel Nowak, Xinyi Wu, Alex Dyck, Krishnan Vaidyanathan, Raghavender R, Jessica Mallet, Mittic Rudominer, Eric Johnston, Susibil Mittal, Akhl Udatu, Janara Christensen, Vishan Verma, Zach Irving, Andreas Santucci, Gamaledlin Elsayed, Elnaz Davoodi, Marin Georgiev, Ian Tenney, Nan Hua, Geoffrey Cideron, Edouard Leurent, Mahmud Almahawfi, Ionut Georgescu, Nan Wei, Ivy Zheng, Dylan Scandinarov, Heinrich Jiang, Jasper Snoek, Mukund Sundararajan, Xuezhi Wang, Zak Ontiveros, Itay Kato, Jeremy Cole, Vinu Rajasekhar, Lara Tumph, Eyal Ben-David, Risbluh Jain, Jonathan USato, Romina Datta, Oskar Bunyan, Shimu Wu, John Zhang, Piotr Stanczyk, Ye Zhang, David Steiner, Subhajit Naskar, Michael Arzam, Matthew Johnson, Adam Paszke, Chung-Cheng Chiu, Jaume Sanchez Elias, Afroz Mohiuddin, Faizan Muhammad, Jin Miao, Andrew Lee, Nano Vieillard, Jane Park, Jiaegeng Zhang, Jeff Stanway, Drew Garmon, Abhibir Kamarkarar, Zhe Dong, Jong Lee, Aviral Kumar, Luowei Zhou, Jonathan Evens, William Isaac, Geoffrey Irving, Edward Loper, Michael Fink, Isha Arkatkar, Nanxin Chen, Izhak Shafran, Ivan Petrychenko, Zhe Chen, Johnson Jia, Ansell Levskaya, Zhenkai Zhu, Peter Grabowski, Yu Mao, Alberto Magni, Kaisheng Yao, Javier Snaidert, Norman Casagrande, Evan Palmer, Paul Suganthan, Alfonso Castano, Irene Giannoulis, Wooyeql Kim, Mikolaj Rybitski, Ashwin Sreevatsa, Jennifer Prendoki, David Soergel, Adrian Goedeckemeyer, Willi Gierke, Mohsen Jafari, Meenn Gaba, Jeremy Wiiesner, Diana Gage Wright, Yawen Wei, Harsha Vashight, Yana Kulizshkaya, Jay Hoover, Maigo Leo, Lu Li, Chimezie Iwunayanwu, Lu Liu, Kevin Ramirez, Andrey Knordini, Albert Cui, Tian LIN, Marcus Wu, Ricardo Aguilar, Keith Pallo, Abhishek Chakladar, Ginger Perng, Elena AliceAbellan, Mingyang Zhang, Ishita Dasgupta, Nate Kushman, Ivo Penchev, Alena Repina, Xihui Wu, Tom van der Weide, Priya Ponnampalli, Caroline Kaplan, Jiri Simsa, Shuangfeng Li, Olivier Dousse, Fan Yang, Jeff Piper, Nathan le, Rama Pssumarthi, Nathan Linnitz, Anitha Vijayakumar, Daniel Andor, Pedro Valenzuela, Minnie Lui, Cosmin Padvararu, Daayi Peng, Katherine Lee, Shuyuan Zhang, Somere Greene, Duc Dung Nguyen, Paula Kurylowicz, Cassidy Hardin, Lucas Dixon, Lilj Janzer, Kiam Choo, Ziqiang Feng, Biao Zhang, Achunya Singhal, Dayou Du, Dan McKinnon, Natash Anthropova, Tolga Bolukhasi, Orpad Keller, David Reid, Daniel Finchelstein, Maria Abi Raad, Remi Crocker, Peter Hawkins, Robert Dadashi, Colin Gaffney, Ken Franko, Anna Bulanova, Remi Leblond, Shirley Chung, Harry Kashham, Luis C. Cobo, Kelvin Xu, Felix Fischer, Jun Xu, Christina Sorokin, Chris Alberti, Chu-Cheng Lin, Colin Evans, Alek Dimitirev, Hannah Forbes, Dylan Banarse, Zora Tung, Mark Omernick, Colton Bishop, Rachel Sterneck, Rohan Jain, Jiawei Xia, Ehsan Amid, Francesco Piccinino, Xingyu Wang, Praseen Banzal, Daniel J. Mankowitz, Alex Polozov, Victoria Krakovana, Sasha Brown, MohammadHossein Bateni, Dennis Duan, Vlad Firotiu, Meghana Thotakuri, Tom Natan, Matthieu Geist, Ser tan Girgin, Hui Li, Jiayu Ye, Ofir Royal, Reiko Tojo, Michael Kwong, James Lee-Thorp, Christopher Yew, Danila Sinopalnikov, Saela Ramos, John 

Mellor, Abhishek Sharma, Kathy Wu, David Miller, Nicolas Sonnerat, Denis Vnuukov, Rory Greige, Jennifer Beattie, Emily Caveness, Libin Bai, Julian Eisenschlos, Alex Korchemny, Tony Tsai, Mimi Jasarevic, Weize Kong, Phuong Dao, Zeyu Zheng, Frederick Liu, Fan Yang, Rui Zhu, Tian Huey Teh, Jason Sanmiya, Evgeny Gladchenko, Nejc Trdin, Daniel Toyama, Evan Rosen, Sasan Tavakkol, Linting Xue, Chen Elkind, Oliver Woodman, John Carpenter, George Papamakarios, Rupert Kemp, Sushant Kafle, Tanya Grunina, Rishika Sinha, Alice Talbert, Diane Wu, Denese Owusu-Afriyiye, Cosmo Du, Chloe Thornton, Jordi Pont-Tuset, Pradyumna Narayana, Jing Li, Saaber Fatehi, John Wieting, Omar Ajmeri, Benigno Uria, Yeongil Ko, Laura Knight, Amelie Heliou, Ning Niu, Shane Gu, Chenxi Pang, Yeqing Li, Nir Levine, Ariel Stolovich,beca Santamaria-Fernandez, Sonnam Goenka, Wenny Yusstalim, Robin Strudel, Ali Elqursh, Charlie Deck, Hyco Lee, Zonglin Li, Kyle Levin, Raphael Hoffmann, Dan Holtmann-Rice, Olivier Bachem, Sho Arora, Christy Koh, Soheil Hassas Yeganeh, Siim Poder, Mukaram Tarig, Yanhua Sun, Lucian Ionita, Mojtaba Seyedhosseini, Pouya Tafti, Zhiyu Lin, Amp Gulati, Jasmine Liu, Xinyu Ye, Bart Chrzaszcz, Lilly Wang, Nikhil Sethi, Tiarrun Li, Ben Brown, Shreya Singh, Wei Fan, Aaron Parisi, Joe Stanton, Vinod Koverkathu, Christopher A. Choquette-Choo, Yunjie Li, Tl Ju, Abe Ittycheirah, Prakash Shroff, Mani Varadarajan, Sanaz Bahargam, Rob Willoughby, David Gaddy, Guillaume Desjardins, Marco Cornero, Bronne Rohenke, Bhavishya Mittal, Ben Albrecht, Ashish Shenoy, Fedor Moiseev, Henrik Jacobsson, Alireza Ghaffarkhah, Morgane Riviere, Alanna Walton, Clement Creppy, Alicia Parrish, Zongwei Zhou, Clement Farabet, Carey Radebaugh, Praveen Srinivasan, Claudia van der Salm, Andreas Fidjeland, Salvatore Scellato, Eri Lactremitomo, Hanna Klimczak-Plucinski, David Bridson, Dario de Cesare, Tom Hudson, Pierramia Mendolicchio, Lexi Walker, Alex Morris, Matthew Mauger, Alexey Guseynov, Alison Reid, Seth Odoom, Lucia Loher, Victor Cotruta, Madhavi Yenugula, Dominik Grewe, Anastasia Petrushkina, Tom Duerig, Antonio Sanchez, Steve Yadowsky, Amy Shen, Amir Globerson, Lynette Webb, Sahill Dua, Dong Li, Surya Bhuquatriraju, Dan Hut, Haroon Qureshi, Ananth Agarwal, Tumer Shanai, Matan Eyal, Anuj Khare, Shreyas Rammohan Belle, Lei Wang, Chetan Tekur, Mihir Sanjay Kale, Jinliang Wei, Ruoxin Sang, Brennan Saeta, Tyler Lichetly, Yi Sun, Yao Zhao, Stephan Lee, Pandu Navab, Doug Fritz, Mansi Reddy Vuyuru, John Aslanides, Nichi Vyas, Martin Wicke, Xiao Ma, Evgenii Eltsyhev, Nina Martin, Hardie Cate, James Manyika, Keyvan Amiri, Yelin Kim, Xi Xiong, Kai Kang, Florian Luisier, Nilesh Tripuraneni, David Madras, Mandy Guo, Austin Waters, Oliver Wang, Joshua Ainslie, Jason Baldridge, Han Zhang, Garima Pruthi, Jakob Bauer, Feng Yang, Riham Mansour, Jason Gelman, Yang Xu, George Polyots, Ji Liu, Honglong Cai, Warren Chen, XiangHai Sheng, Emily Xue, Sherjil Ozair, Christof Angermueller, Xiaowei Li, Anooip Sinha, Weiren Wang, Julia Wiesinger, Emmanouil Koukoumidis, Yuan Tian, Anand Iyer, Madhu Gurmurthy, Mark Goldenson, Paraskar Shah, MK Blake, Hongkun Yu, Anthony Urbanowicz, Jennarina Palomaki, Chrisantha Fernando, Ken Durden, Harsha Mehta, Nikolac Momchev, Elaine Rahimtoroghi, Maria Georgski, Amit Raul, Sebastian Ruder, Morgan Redshaw, Injhyuk Lee, Denny Zhou, Komal Jalan, Dinghua Li, Blake Hechtman, Parker Schuh, Milad Nasr, Kiearan Milan, Vladimir Mikulik, Juliana Franco, Tim Green, Nam Nguyen, Joe Kelley, Aroma Mahendru, Andrea Hu, Joshua Howland, Ben Varges, Jeffrey Hui, Kchitijan Basl, Vikram Rao, Rakesh Gihiya, Emma Wang, Ke Ye, Jean Michel Sarr, Melanie Moranski Preston, Madeleine Elish, Steve Li, Aaksah Kaku, Jiger Gupta, Ice Pasupat, Da-Cheng Juan, Milan Someswar, Teiyi M., Xinjun Chen, Aida Amini, Alex Fabrikant, Eric Chu, Xuanyi Dong, Amrata Mutual, Senaka Butbitiya, Sarthak Jaubari, Nan Hua, Urvashi Khanadjeval, Ayal Hirton, Jie Ren, Larissa Rinalidi, Shahar Drath, Avigail Dabush, Nan-Jiang Jiang, Harshal Godhia, Uli Sachs, Anthony Chen, Yicheng Fan, Hagai Taitelbaum, Hila Noga, Zhuyun Dai, James Wang, Chen Liang, Jenry Hamer, Chun-Sung Ferng, Chenel Elkhid, Aviel Atias, Paulina Lee, Vit Listk, Mathias Carlen, Jan van de Kerkhof, Marcin Pikus, Krumoslav Zaher, Paul Muller, Sasha Zykova, Richard Stefanec, Vitaly Gatsko, Christoph Hirnschall, Ashwin Sethi, Xingyu Federico Xu, Chetan Ahuja, Beth Tsai, Anca Stefanoiu, Bo Feng, Keshav Dhandhannia, Manish Katyal, Akshay Gupta, Atharya Parulekar, Divya Pitta, Jing Zhao, Vivanan Bhatia, Yashodha Bhavanani, Olmar Alhadlaq, Xiaolin Li, Peter Danenberg, Dennis Tu, Alex Pine, Vera Filipprova, Abbingo Ghosh, Ben Limonchik, Bharagava Urala, Chaitanya Krishna Lanka, Derik Clive, Yi Sun, Edward Li, Hao Wu, Kevin Hongtongsak, Inna Li, Kalind Thakkar, Kuanysh Omarov, Kushal Majumdar, Michael Alverson, Michael Kubarski, Mohak Patel, Mudiu Jain, Maksim Zabelin, Paolo Pelagatti, Rohan Kohli, Saurabb Kumar, Joseph Kim, Swetha Sankar, Vineet Shah, Lakshmi Ramachandran, Xiangkai Zeng, Ben Briach, Laura Weidinger, Tu Vu, Alek Andreev, Antoine He, Kevin Hui, Sheleken Kashem, Amar Subramanya, Sixisie Hsiao, Demiss Hassabis, Koray Kavukcuoglu, Adam Sadovsky, Quoc Le Trevor Strohman, Yonghui Wu, Slav Petrov, Jeffrey Dean, and Oriol Vinyals. 2024. Gemini: A family of highly capable multimodal models. _Preprint_, arXiv:2312.11805.
* Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. _Advances in neural information processing systems_, 30.
* Xia et al. (2023) Heming Xia, Tao Ge, Peiyi Wang, Si-Qing Chen, Furu Wei, and Zhifang Sui. 2023. Speculative decoding: Exploiting speculative execution for accelerating seq2seq generation. In _Findings of the Association for Computational Linguistics: EMNLP 2023_, pages 3909-3925.
* Xia et al. (2016) Heming Xia, Zhe Yang, Qingxiu Dong, Peiyi Wang, Yongqi Li, Tao Ge, Tianyu Liu, Wenjie Li, and Zhi 

fang Sui. 2024. Unlocking efficiency in large language model inference: A comprehensive survey of speculative decoding. In _Findings of the Association for Computational Linguistics ACL 2024_, pages 7655-7671, Bangkok, Thailand and virtual meeting. Association for Computational Linguistics.
* Zhang et al. (2024) Aonan Zhang, Chong Wang, Yi Wang, Xuanyu Zhang, and Yunfei Cheng. 2024. Recurrent drafter for fast speculative decoding in large language models. _Preprint_, arXiv:2403.09919.
* Zheng et al. (2024) Kaiwen Zheng, Yongxin Chen, Hanzi Mao, Ming-Yu Liu, Jun Zhu, and Qinsheng Zhang. 2024. Masked diffusion models are secretly time-agnostic masked models and exploit inaccurate categorical sampling. _arXiv preprint arXiv:2409.02908_.
* Zheng et al. (2023) Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhubuan Li, Dacheng Li, Eric Xing, et al. 2023. Judging IIm-as-a-judge with mt-bench and chatbot arena. _Advances in Neural Information Processing Systems_, 36:46595-46623.
* Zhou et al. (2024) Yongchao Zhou, Kaifeng Lyu, Ankit Singh Rawat, Aditya Krishna Menon, Afshin Rostamizadeh, Sanjiv Kumar, Jean-Francois Kagy, and Rishabh Agarwal. 2024. Distillspec: Improving speculative decoding via knowledge distillation. _Preprint_, arXiv:2310.08461.

## Appendix A Draffer Convergence Analysis

In previous speculative decoding approaches the computational overhead of the drafting stage has been proportional to \(\gamma\), as this parameter dictates the number of network evaluations during the draft phase. When using a discrete diffusion draft model, the number of network evaluations is dictated by the number of diffusion steps \(T\). The following reports an analysis of the possible speedups that can be achieved by our approach, as a function of the diffusion steps \(T\).

First, note that the expected number of tokens per draft can be derived from \(\alpha\) and \(\gamma\):

\[\mathbb{E}(\#tokens)=\frac{1-\alpha^{\gamma+1}}{1-\alpha}. \tag{2}\]

In prior studies, theoretical results have focused on determining the optimal \(\gamma\) to maximize the throughput of speculative decoding methods Leviathan et al. (2023). For SpecDiff, extending \(\gamma\) introduces minimal overhead and becomes less important to consider. The number of sequential diffusion operations, \(T\), instead impacts \(\alpha\) as increasing this number improves the convergence of \(M_{q}(x)\) to the learned distribution, aims to closely approximate the distribution of \(M_{p}(x)\). Hence, an implicit dependency arises between \(T\) and \(\alpha\), which is reflected in Figure 2 (right).

First, note that the computation overhead of a single network evaluation of \(M_{q}(x)\) and \(M_{p}(x)\) is constant. \(M_{q}(x)\) is scaled by the number of diffusion steps, whereas all evaluations of \(M_{p}(x)\) are conducted in parallel. Now, consider that, provided Equation 2:

\[\mathbb{E}(\#tokens/second)=\frac{(\frac{1-\alpha^{\gamma+1}}{1-\alpha})}{Tc_{ 1}+c_{2}} \tag{3}\]

where \(c_{1}\) is the computation overheads of a single network evaluation of \(M_{q}(x)\) and \(c_{2}\) is the computation overheads of a single network evaluation of \(M_{p}(x)\).

Next, consider that the convergence of \(q(x)\) to the original data distribution, which we will denote as \(\hat{q}(x)\), is proportional to the \(1/T\).

**Theorem A.1** (Li et al., 2023).: _Under standard assumptions, the convergence rate of samplers based on the probability flow Ordinary Differential Equation (ODE), converge at the rate_

\[TV(q_{1},\hat{q}_{1})\leq c_{3} \frac{d^{2}\log^{4}T}{T}+c_{3}\frac{d^{6}\log^{6}T}{T^{2}}\] \[+c_{3}\sqrt{d\log^{3}T\epsilon_{sconv}}+c_{3}d(\log T)\epsilon_{ snculi}\] 

_where \(d\) is the dimensions of the sample, \(\epsilon_{score}\) is the error in the score function estimation, \(\epsilon_{Jacobi}\) is the error in the Jacobian matrices, and universal constant \(c_{3}>0\)._

Given that \(q_{1}\approx q_{0}\), this result provides a practical upper bound on the distance between \(q(x)\) and \(\hat{q}(x)\). As Theorem A.1 provides an explicit relation to \(T\), this can be used to determine an upper bound on the distance between \(q(x)\) and \(p(x)\) for a given number of diffusion steps. By the triangle inequality:

\[TV(p(x),q(x))\leq TV(p(x),\hat{q}(x))+TV(\hat{q}(x),q(x)) \tag{4}\]

Now, the remaining step to find the upper bound on the distance from \(q(x)\) to \(p(x)\) is to determine \(TV(p(x),\hat{q}(x))\). First, consider that this relation can be expressed as follows:

**Definition A.2** (Leviathan et al., 2023).: \(D_{LK}(p,q)=\sum_{x}\|p(x)-M(x)\|=\sum_{x}\|q(x)-M(x)\|\) where \(M(x)=\frac{p(x)+q(x)}{2}\)__

**Corollary A.3** (Leviathan et al., 2023).: \(\alpha=1-\mathbb{E}(D_{LK}(p,q))=\mathbb{E}(\min(p,q))\)__

Provided Corollary A.3, \(D_{LK}(p(x),\hat{q}(x))\) can be computed empirically by setting \(T\) arbitrarily high and evaluating \(\alpha\); subsequently, we will refer to this distance as \(c_{4}\). We note that the \(D_{LK}(p(x),\hat{q}(x))\) captures any error in the draft model's learned distribution that is introduced in Theorem A.1 as \(\epsilon_{\text{score}}\) and \(\epsilon_{\text{Jacobi}}\), so we will set these to zero when applying the theorem.

Note that the metric \(D_{LK}\) is equivalent to the discretized total variation:

Proof.: \[D_{LK}(p,q)= \sum_{x}\|p(x)-M(x)\|\] \[= \frac{1}{2}\sum_{x}\|p(x)-q(x)\|\] \[\approx \frac{1}{2}\int\|p(x)-q(x)\|dx\] \[= TV(p,q)\]

For practical applications such as this, the discrete total variation is used, and for the purpose of this analysis we will consider the metrics equivalent. Now, we are ready to compute a lower bound for \(\alpha\) that is dependent on \(T\), applying Equation 4:

\[\begin{split}&\alpha=1-\mathbb{E}(D_{LK}(p,q))=1-\mathbb{E}(TV( p,q))\\ &\geq 1-(c_{4}+TV(\hat{q}(x),q(x)))\end{split} \tag{5}\]

Equation 5 provides a lower bound on \(\alpha\) that is dependant on \(T\). While in practice this remains computationally intractable, given that \(c_{3}\) is unknown, this can be approximated using a surrogate network to predict this constant; this is not dissimilar from the suggestion by Leviathan et al. to optimize runtime using such an approach to predict \(\alpha\).

We are now ready to connect this to Equation 3:

\[\mathbb{E}\left(\#tokens/second\right)\geq\tfrac{\nicefrac{{1-(1-c_{4}-c_{3} \frac{c^{2}b^{4}T}{T}-c_{3}\frac{c^{6}b^{7}}{T^{2}})}^{n+1}}{{c_{4}+c_{3}\frac {c^{2}b^{4}T}{T^{2}}}}\times\tfrac{1}{{r_{3}}!{c_{4}}!}}{\nicefrac{{c_{4}}!{ c_{3}}!}{{c_{4}}!}+c_{3}\frac{a^{2}b^{4}c^{2}}{T^{2}}}}\times\tfrac{1}{{r_{3}}!{c_{4}}!}\]

This equation can now be solved analytically to optimize the lower bound. Practically, in the presence of a surrogate network, this can be simplified further, given convergence of the diffusion model is proportional to \(1/T\).

\[\mathbb{E}(\#tokens/second)\geq\tfrac{\nicefrac{{1-(1-c_{4}-c_{5}\frac{1}{T} )\gamma+1}}{{c_{4}+c_{5}\frac{1}{T}}}}{\nicefrac{{1}}{{c_{1}}!{c_{5}}!}}\times \tfrac{1}{{r_{3}}!{c_{1}}!{c_{2}}} \tag{6}\]

Hence, the dependency between \(T\) and \(\alpha\) can be exploited to estimate the optimal number of diffusion steps. In our experiments we find that the optimal value of \(T\leq 5\).

## Appendix B Accepted Draft Lengths

We notice a starch constrast in the lengths of accepted drafts between these two settings. While the distribution of accepted draft lengths in Figure 4 is what would be anticipated given the lower values of \(\alpha\), the longer generations on OpenWebText (Figure 3) speak to the parallelism that can be realized when the distribution is effectively aligned to the target model.

Figure 3: Accepted draft lengths for OpenWebText evaluation.

 



## Appendix C Implementation Details

For all evaluation we utilize the following hyperparameter setups. If not explicitly noted here, parameters are consistent with those specified by the authors of MDLM Sahoo et al. (2024). For reported results on CNN/DM and OWT, 200 iterations are conducted on samples randomly selected from the datasets. For evaluation on MT bench, 160 iterations are conducted.

## Appendix D Choice of Diffusion Model

The selection of a discrete diffusion model as the drafter plays a critical role in optimizing the overall framework's speedup performance. The models explored, MDLM and SEDD, represent the current state-of-the-art in discrete diffusion, achieving near auto-regressive perplexity results with comparably sized models. We observe significant speedups over SpS when using MDLM as our drafter, as MDLM demonstrates superior generation speeds overall. These gains are not observed in SEDD for two primary reasons: first, SEDD exhibits lower perplexity compared to MDLM, resulting in a lower acceptance rate and second, MDLM's generation speed surpasses that of SEDD.

## Appendix E Temperature Impact on Acceptance Rate

As discussed in Section 8, overconfidence exhibited by discrete diffusion models, a result of poor model calibration, results in the top-1 token having a probability close to 1 for the vast majority of generations. For these outputs, the temperature is effectively zero. In the speculative decoding framework, this overconfidence is reflected as higher \(q(x)\) values, which has a significant impact on the token acceptance rate when sampling stochastically. Specifically, high \(q(x)\) values lead to a decrease in the number of accepted tokens because it increases the frequency of \(q(x)>p(x)\). Increasing the temperature has a smoothing effect on \(p(x)\) leading to misaligned distributions between \(p(x)\) and \(q(x)\). Consequently, tokens are more frequently rejected with a probability of \(1-\frac{p(x)}{q(x)}\), where a larger \(q(x)\) further increasing the likelihood of rejections.

This obstacle is avoided when sampling deterministically (temperature=0) as in the results reported. This challenge motivates future study of discrete diffusion model calibration.

Figure 4: Accepted draft lengths for MT Bench evaluation.

\begin{table}
\begin{tabular}{|c c l r r r r r r|} \hline  & & \multicolumn{3}{l|}{\(M_{p}\)} & \multicolumn{1}{r}{\(M_{q}\)} & Strategy & \(\gamma\) & \(T\) & Gen Length & Precision \\ \cline{5-9} \multirow{3}{*}{**Ours**} & & GPT-2 XL (1.5B) & MDLM (110M) & _ddpm cache_ & 15 & 2 & 1024 & 32 bit \\  & & GPT NEO (2.7B) & MDLM (110M) & _ddpm cache_ & 15 & 2 & 1024 & 32 bit \\ \cline{2-9} \multirow{3}{*}{**Ours**} & & GPT-2 XL (1.5B) & MDLM (110M) & _ddpm cache_ & 15 & 2 & 1024 & 32 bit \\  & & GPT NEO (2.7B) & MDLM (110M) & _ddpm cache_ & 20 & 2 & 1024 & 32 bit \\ \cline{1-1} \cline{2-9} \multirow{3}{*}{**Ours**} & & Vicuna (33B) & MDLM (141M) & _ddpm cache_ & 15 & 2 & 512 & 16 bit \\ \hline \end{tabular}
\end{table}
Table 5: Additional details on parametric setups for reported results.