Self-Speculative Decoding Accelerates Lossless Inference in Any-Order and Any-Subset Autoregressive Models
Authors: Gabe Guo
Stefano Ermon
Venue: ICLR 2026 Poster (2026)
Keywords: speculative decoding
any-order autoregressive models
diffusion language models
Primary area: generative models
TL;DR: We present an algorithm that provably accelerates inference in any-order autoregressive models, without loss of quality.

Abstract: In arbitrary-order language models, it is an open question how to sample tokens
in parallel from the correct joint distribution. With discrete diffusion models, the
more tokens they generate in parallel, the less their predicted distributions adhere
to the originally learned data distribution, as they rely on a conditional independence 
assumption that only works with infinitesimally small timesteps. We find
that a different class of models, any-subset autoregressive models (AS-ARMs),
holds the solution. As implied by the name, AS-ARMs can generate tokens in any
order, and in parallel. Moreover, AS-ARMs support parallelized joint probability
density estimation, allowing them to correct their own parallel-generated token
distributions, via our Any-Subset Speculative Decoding (ASSD) algorithm. ASSD
provably enables generation of tokens from the correct joint distribution, with the
number of neural network calls upper bounded by the number of tokens predicted
– notably, previous speculative decoding algorithms lack our efficiency guarantee. 
We empirically verify that ASSD speeds up language generation, without
sacrificing quality. Furthermore, we provide a mathematically justified scheme for
training AS-ARMs for generation, and show that AS-ARMs achieve state-of-the-art
performance among sub-200M parameter models on infilling benchmark tasks,
and nearly match the performance of models 50X larger on code generation. Our
theoretical and empirical results indicate that the once-forgotten AS-ARMs are a
promising direction of language modeling.

Published as a conference paper at ICLR 2026
SELF-SPECULATIVE DECODING ACCELERATES Loss- LESS INFERENCE IN ANY-ORDER AND ANY-SUBSET AUTOREGRESSIVE MODELS
Gabe Guo
Stefano Ermon
Department of Computer Science Stanford University
Department of Computer Science
Stanford University
gabeguo@stanford.ccm
ermonêcs stanford....ooo. edu
ABSTRACT
In arbitrary-order language models, is an open question how to sample tokens in parallel from the correct joint distribution. With discrete diffusion models, the more tokens they generate in parallel, the less their predicted distributions adhere to the originally learned data distribution, as they rely conditional indepen- dence assumption that only works with infinitesimally small timesteps. We find that different class of models, any-subset autoregressive models (AS-ARMs), holds the solution. As implied by the name, AS-ARMs can generate tokens in any order, and in parallel. Moreover, AS-ARMs support parallelized joint probability density estimation, allowing them to correct their own parallel-generated distributions, via our Any-Subser Speculative Decoding (ASSD) algorithm. ASSD provably enables generation of tokens from the correct joint distribution, with the number of neural network calls upper bounded by the number of tokens predicted notably, previous speculative decoding algorithms lack our efficiency guaran- tee. We empirically verify that ASSD speeds up language generation, without sacrificing quality. Furthermore, we provide mathematically justified scheme for training AS-ARMs for generation, and show that AS-ARMs achieve state-of-the performance among sub-200M parameter models infilling benchmark tasks, and nearly match the performance of models 50X larger on code generation. Our theoretical and empirical results indicate that the once-forgotten AS-ARMs promising direction language modeling.
generated token
INTRODUCTION
Almost all the SoTA LLMs (Achiam al. 2023 Touvron al. 2023 Liu et al. 2024a Team et al.)
2023) are autoregressive, i.e., they only support left-to-right token generation. As a result, they suffer from two major problems: (1) they must generate tokens one-by-one, which limits their speed; (2) they cannot infill sequences in orders besides left-to-right (unless specialized training strategies are adopted (Bavarian al. 2022: Fried al. 2022 Roziere et al. (023). but these are heuristic and not guaranteed to output the correct structure).
Regarding models that inherently support infilling, there discrete diffusion models (Austin et al. Sahoo et al. (024) and any-order autoregressive models
2021 Campbell et al. 2022 Lou et al. 2023
(A0-ARMs) YYngg 2019: Shih al.
2022 p. Discrete diffusion models have the benefit of parallel
sampling of multiple tokens at time, which potentially speeds up generation, but at the cost of
fidelity to the learned data distribution (Lou al. 2023). On the other hand, fast sampling schemes have generally not been explored for AO-ARMs.
We explore the problem of fast parallel sampling from AO-ARMs, without any degradation in output quality. Particularly, we are inspired by speculative decoding techniques, which have accelerated generation in standard autoregressive models by using draft model to quickly generate multiple tokens. Then, tokens are accepted rejected based their fidelity to the joint distribution as evaluated by the oracle model. Interestingly, the outputted distribution is provably the same as would have been obtained from sampling only from the expensive oracle model, while usually (but not

---

Published as a conference paper at ICLR 2026
always) using fewer neural network forward passes (Chen et al. 2023 Leviathan al. 2023 Crucial to speculative decoding are (1) density estimation from the oracle, and (2) a fast draft model. Indeed, AO-ARMs, by design, estimate joint probability density of sequences YYng 2019). Furthermore,
AO-ARMs can even act as their own draft models; due to their architectura design and training objective, they can generate tokens in any order and in parallel (Yang) 2019 Shhih et al., 2022).
As such, we propose Any-Subset Speculative Decoding (ASSD), an algorithm that combines specula- tive decoding with AO-ARMs. ASSD provably generates sequences from the true joint distribution learned by the oracle model. Our method's chief advantages over previous speculative decoding algorithms include: (1) It is mathematically guaranteed to never increase the number of function evaluations (i.e., language model calls): in practice, this means that it speeds up generation without losing quality (as we empirically verify). In contrast, vanilla speculative decoding has no such guar- antee on NFEs, and could slow down runtime in some cases. (2) ASSD can handle O(2 ) possible infilling tasks, exponentially more than the O(N) allowed by traditional speculative decoding. (3) We get the speculator "for free", without the need to train auxiliary draft model.
We also provide a mathematically justified training scheme for AO-ARMs. Finally, we show that appropriately trained AO-ARMs achieve otate-of-the-art performance among sub-200M parameter models (diffusion and autoregressive) on infilling benchmark tasks, and nearly match the performance of models 50X larger code generation, while needing fewer finetuning tokens.
BACKGROUND
Autoregressive Models: Given a text sequence x ~ D, where 'D is a data distribution, autoregressive (AR) models learn
x|-1
p(x)=
II p( (...
The product rule factorization means that AR model only needs to learn conditional distributions with support of size OSS), where S is the size of the vocabulary. This factorization also admits straightforward generation strategy, where token æ; is sampled from p(-]zo.... T4-1). 1). i.e. the previous tokens are used to produce the next (Achiam et al. (023). This also means that the prompt for an AR model must always be the prefix, i.e., Xo:i arbitrarily-
prompts are not supported.
Infilling: We consider infilling tasks, where the prompt is not necessarily the prefix, but can be arbirarily-located. Examples of this are code generation, story completion, and scientific data imputation. Mathematically, this can be formulated sampling from a joint conditional probability distribution with discrete state space. That is, we wish to sample from p( (2x) (<m))) where (i) is the (zero-indexed) positional index of the i-th ordered item in the sequence of length N. In other words, a a permutation of 10 N 1}. and is the generation order. So, (x<m)) represents the prompt tokens, and X²(2m) represents the tokens whose distributions we want to predict. In
general, we can have any (0,...,N N 10-1 -} - (O...., N 1}. so long as is bijection. As previously established, regular AR models cannot address this task general, except when (i) = i.
Any-Order Autoregressive Models: Any-order autoregressive models (AO-ARMs) (Shih et al.;
2022, Hoogeboom et al.) 2021 Yang 2019) can be seen as collections of N! joint distributions
indexed by the factorization order o:
log (x
Σ log (o(en))))
That is. the distribution for each token is calculated one at time, in order of increasing i, and used as conditioning for the next token to be predicted. Concretely, each evaluation of the network produces probability distribution (for single token) with support of size O(S), where S is the size of the vocabulary. The probability of each member of the support is explicitly calculated and stored in memory, incurring O(S) memory cost per token.
2

---

Published as a conference paper at ICLR 2026
If trained to optimality, all the joint distributions should be equal. However, except with infinite data and capacity, given different ordering functions a and o (Shih et al. 2022).
N *
log (() Xo(e));
log p( (aao))))
(3)
Any-Subset Autoregressive Models: Any-subset autoregressive models (AS-ARMs) are a subclass of AO-ARMs that reduce the number of joint distributions (o) learned from N! to 2 This puts less training burden on the finite model capacity, while keeping strictly the same expressivity as relates to conditional joint distributions of the form in Equation 2 To achieve this, AS-ARMs adopt the recursive binary lattice mask decomposition protocol from (Shih et al.) 2022). The idea underlying this protocol is that we can split every generation task into two parts: the prompt (denoted by x²(<m)) and the tokens that need to be generated (denoted by (2 Mathematicallyy we want to estimate ((Xo(2m) (<m)), where m is the number of tokens in the prompt (assuming O-indexing). Now, make two observations.
Firstly, since we never need to evaluate the density of the conditioning Xo(<m), we have every token attend to every other token within Xz(cm). Secondly, within X²(2m)) there is no need to learn all the possible factorization paths, as long as we can get the joint conditional probability p(X)(2m) (c(m)) Taking inspiration from vanilla autoregressive models, we enforce
V m,j 2 m aii) a(j) + +i>j-
That is, we simply process the masked tokens left to right. Thus, given the location of the prompt, we only have to learn one path to calculate the joint probability of the generation. This also solves the inconsistency problem in Equation 3 (As later shown, this is crucial to Algorithm I correctness.)
This reduces the number of queries learned by the model from N! (all possible permutations) to 2d (all possible mask location selections, times one ordering per mask selection). This makes optimization casier, as noted by (Shih et al. 2022) and verified in our ablations (Figure 3 In summary, AS-ARMs are subclass of AO-ARMs incorporating Equation disambiguated ordering strategy. The architecture design is typically the same, but the way we query is different.
3 SAMPLING STRATEGIES FOR JOINT DISTRIBUTIONS
We want to accurately and efficiently sample from the joint conditional distribution Equation We assume that have access AS-ARMs that explicitly predict single-variabe marginals, i.e., next-token prediction under some ordering.
2
One-Step Sampling from the Joint: Sampling directly from this joint distribution in a single step is typically infeasible, because its support has size N-NN- where is the number of tokens in the vocabulary. If we wanted to explicitly calculate the probability of every tuple in the support, the exponential space cost to just store the distribution would quickly exceed memory capacities.
Sequential Sampling via Factorization: We can sample the log p( (x)())))) one-by-one, using each generated token as the conditioning for the next one. Following Equation 2 we can get samples
from the joint conditional distribution by doing this N m times, once for each i E m,, N). In (any-order, any-subset) autoregressive models (Achiam et al. [2033] Touvron et al. [0233] Yang 2019) Shih al. 2022). there is typically OSS) time cost per-token, so the overall time cost would be O * (N m) Indeed, this is the dominant generation strategy for autoregressive models like GPT (Achiam al. 2023 where (i) =.
Parallel Sampling via Independence Assumption: At another extreme, we can independently sample multiple single-variable marginals parallel. That is, we predict log p( (x())) for all i e [m,N) NN. Since N m generations are done in parallel, computational time cost is only O(S). which is effectively constant with respect to the number of tokens. The limitation is that
log ( (x())) * log ((X)(2m) Xo(c)));
(5)
ZE(m.NN
except in the unlikely scenario of true independence, ie., log (((oo)))X = log ((
3

---

Published as a conference paper at ICLR 2026
This is analogous to how discrete diffusion models take large discretized timesteps in the reverse CTMC to predict tokens in parallel, even though the predictions at each position are actually generated with conditional independence assumption that only holds for infinitesimllly small timestep (i.e.,
one-by-one generation) (Lou et al. 2023 Sahoo et al. 2024).
Best of Both Worlds: Any-Subset Speculative Decoding: We seek way to combine the runtime benefits of parallel sampling with the fidelity of sequential sampling. That is, can we achieve OSS) time complexity for arbitrary-subset generation, while recovering true samples from log (2x) {x)))))?? ? The key insight here is that if we had an oracle model that could evaluate the joint density of sequence with a singular function evaluation, we could leverage the quick speed of independent parallel generation and use these samples log p(*o(t)) ((Zo())(X(((m)) (x((s))) as estimates for the true log p( {=)()))) ) Then, via some rejection sampling scheme which uses as an oracle the joint density evaluation of this newly generated sequence, we could keep only the samples that adhere to
log p(X)(2m)) (o(s))). In principle, this could take best-case o{S) time (parallelized), if the oracle accepts all the samples, and worst-case O(S * (N- m)) time.
Speculative decoding is one such method that guarantees fidelity to the true distribution. It is mathematccally proven to generate samples from the target distribution, and the empirical results show a stark decrease in the number of function evaluations required (Leviathan al. 2023 Chen et al. 2023). However, prior to our work, speculative decoding has only been shown to work for left-to-right (Achiam al. 2023 autoregressive models.
4 ARCHITECTURAL DESIGN OF AS-ARMS
Towards the goal of fast, principled parallel sampling in any order, we seek any-subset autoregressive models (AS-ARMs) that can support our Any-Subset Speculative Decoding (ASSD) algorithm (which fully described in Section 55. To recap, the criteria are: (1) generates arbiteerily-ordered tokens in parallel; (2) evaluates joint density with only one forward pass. We now describe how AS-ARMs/AO- ARM architectures should be designed to fulfill these criteria.
Parallel Sampling via Arbitrary Positional Queries: With one function evaluation, we should be able to simultancously predict in parallel (conditionally independent) distributions for all the masked tokens Xz(2m), conditioned on the prompt Xx()m). This allows the network to act quick "draft" model. Concretely, we should be able to pass in arbitrary positional queries a2 m) of not-yet-predicted tokens to condition on the visible prompt tokens Xx()m). As implied by the "any-order" moniker, there is constraint which positions we query: we could query the leftmost, rightmost, or even a randomly selected unfilled position. Furthermore, the positional queries all attend to the same prompt tokens, but the prompt tokens cannot attend to the positional queries. So, no matter how many positions we query in parallel, each query cannot change the representations of the prompt tokens, and therefore cannot change the outcomes of the other simultaneous queries. See Figure la
Density Estimation via Causal-Like Attention Masking: Another crucial ingredient of speculative decoding is density estimation - this allows the "oracle" model to correct the mistakes of the draft model. As such, discrete diffusion models trained with ELBO (Lou et al. 2023 Sahoo et al., 2024 Deschenaux & Gulcehre 2024) are not readily adaptable to this scheme. In contrast, the training Yang 2019) teaches them evaluate joint
objective of DO-ARM//AS-ARA (Shih al., 2022
densities of sequences.
Care must be taken, however, to pick an architecture that evaluates the joint density of a sequence in one function evaluation, i.e., O(S) time. Some architectures (Shih et al.) 2022 Hoogeboom et al. 2021) take (SS * N) steps, as only logits of masked tokens are predicted at each function evaluation they are unable to predict the logits of visible tokens. To get architecture that can calculate the logits for all visible and masked tokens in a single function evaluation with (SS) time, we can design the attention masks for each token to only allow attention (Le., conditioning) to the preceding tokens in the ordering. Then, we can construct factorization as in Equation [ with this attention mask giving us the desired O(S) time. Mathematically,
We process tokens in parallel
4

---

Published as a conference paper at ICLR 2026
Key/Value
Key/Value
Thi
Thu
loaming
the
1hk
Query
Query
leariing
(a) Parallel Sampling Attention Mask
(b) Density Estimation Attention Mask
Figure 1: Attention Masks in AS-ARMs: Red means that attention is not allowed, while green means that attention is allowed from the query to the key/value. The numbers on each token represent the generation order, where lower-numbered tokens come first. Panel shows how, by attending to the same conditioning ("This", "is"), the tokens "machine" and "learning" can be generated in parallel. They do attend to themselves nor each other. Panel [b shows how we can conduct one-step density estimation on a sequence, with a permuted causal-like attention mask. The mask enforces the order "is", "This", "learning", "machine", where each token only attends those decoded before it.
A 0 i isj
(6)
i>j"
where (o()),o()) is the masking matrix for the attention maps means that the query token at index (i) is not allowed to attend to the key/value token at index oj)): ; 1 means that token (ii) can attend
to token (j). Such an attention mask can yield faithful estimates of log p(Xx(21) (x())) See
Figure Tb and Appendix Dor further discussion.
Streamlined Drafting: Ideally, do not want to train separate draft model, because it takes extra memory on the hardware and additional training cost. Furthermore, the computations from the separate draft model cannot necessarily be re-used for the oracle. If the draft model was the same as the target/oracle model, we would not incur extra memory nor training cost, and could cache computations from the draft model to accelerate the target calculations.
Two-for-One Model:
Suitable Architectures: Many popular architectures actually satisfy the aforementioned constraints, as they essentially boil down being able specify flexible attention masks according the desired factorization of the distribution. One suitable architecture family is XLNet encoders (Yang) 2019)
(discussed further in Appendix [D). GPT-style decoders (Achiam et al.) 2023 Touvron et 2023
can also be minimally modified with specialized positional encodings to become suitable AS-ARM
architectures (Pannatier et al. 2024). We run the following experiments with XLNet.
How TO MODIFY SPECULATIVE DECODING FOR AS-ARMS?
We present Any-Subset Speculative Decoding (ASSD), and prove some of its properties. See Algorithm
1 See Appendix for proofs. In the resampling step, we use the notation (f(x)) = Σ. m
(0. f(x))
Lemma 1. The first token speculated in each loop iteration will always be accepted. That is, Line 19 s conditional always evaluates to true when i = 11.
Theorem 1. Algorithm [ requires more than N - total function evaluations of p(-+)). That is. there will never be more calls to neural network than the number of tokens returned on Line 28
Theorem 2. Algorithm produces samples from the true joint distribution p(Xa(2m) (o((m)))
Based on Lemma] I and Theorem I (see proof). we should always set k > 2 (where k is the number of speculated tokens per call the draft model). We also present variant of ASSD in Appendix E.4 with a context-derivee n- n-gram as the draft model (Stewart et al. 2024). However, this variant does not fulfill [I
Comparison to Speculative Decoding: Superficially, ASSD's instructions are similar to vanilla speculative decoding (Chen et al. 2023: Leviathan et al. 2023). However, ASSD actually has
5

---

Published as a conference paper at ICLR 2026
Algorithm 1: Any-Subset Speculative Decoding
Input: number parallel tokens; N: target sequence length; a: mapping decoding order to
O-based positional index; p(-l-): any-order autoregressive model; {x(mm)² prompt tokens
Output: (=) the predicted tokens
f- m // number of tokens we' we've already decoded
while <NNoo do
1 4 min(n k,, N)
// speculate the next tokens
(Parallelized): for E [n: :t) do
fo(() ((()))(<n)) // sample from partially conditioned distribution
Po(1) - (() Xo(<n)) // get partially conditioned density
end
if n == 7 then
Za(n) + To(n) // accept the proposal return X=(2m) end
12
13
(Parallelized): for [n: do
(a(1) c (() (x((<)) {o(ns))) // get ground truth density
15
end
16
rejection sampling for i [n :t) do r ~ [[
18
if < min(1, 9-(0) then
19
Pota)
To(() To(() // accept the proposal
21
else
To(() (p(- (o()))) &afn:))) p( (o((s))) resample
23
exit from for loop
24
end
end
26
n + it 1/ update number of decoded tokens
27
end
return (2(m)
an important theoretical guarantee that vanilla speculative decoding lacks, in that Theorem upper bounds the NFEs (draft + oracle) to the number of tokens generated. In contrast, vanilla speculative decoding has no such guarantee: Leviathan et al. (2023) note that if the draft model particularly poor and/or expensive, vanilla speculative decoding can theoretically increase the total NFEs and therefore runtime (particularly when the drafts keep getting rejected). Furthermore, our algorithm has exponentially larger capacity in infilling patterns it can handle: O(2*), while vanilla speculative decoding can only handle O patterns, due to the any-subset versus left-to-right natures of the algorithms. Finally, do not require auxiliary draft models, they come for free with AS-ARMs.
TRAINING AND IMPLEMENTATION
Architectural Instantiation: We use the 110M parameter case-sensitive version of XLNet from Huggingface (Wolf et al. 2019), which is one many architectures that satisfies the criteria in Section 4 We finetune starting from the pretrained weights Yangg 2019).
Dataset: We finetune on the WebTet dataset (Gokaslan et al. 2019). Following (Sahoo et al. 2024), pack the sequences together, and split them into chunks of 512 tokens, based on XLNet's case-sensitive tokenizer with vocabulary 32, 000 possible tokens. We have separator tokens to delineate the start of new document.
Training Hyperparameters: The original XLNet Yang, 2019) was only trained to predict 85 masked tokens in sequence length of 512, corresponding to less than 20% of the sequence, which is

---

Published as a conference paper at ICLR 2026
not ideal for generative modeling tasks. We want model that predict tokens almost from scratch, so we finetune the off-the-shelf XLNet model. More hyperparameter details are in Appendix E
Teacher-Forced Joint Loss: For our training objective, we maximize the joint conditional probability in Equation 2 with cross-entropy loss:
maxe
(-Im) [log pe (Xa((m)) Xo(<m))]
where f() is distribution over integers from [0, NN) (i.e., sampling prompt length m)) and s(-(m) is
a distribution of permutations of integers from N) conditioned on prompt length m (i.e., sampling token ordering σ), where N is the sequence length. The loss has three major components: (1) joint conditional distribution; (2) expectation over token orderings: (3) expectation over prompt lengths.
Joint Conditional Objective: To justify the joint conditional distribution log p(Xx(2m) Xx(m))), assume m and 0 are fixed. We define a discrete time (absorbing state) Markov chain N
x, (x(<N-1)> Xa(<NN2) ............................. Xx(<m++))= X)(mm)) with time index t {0, 1,, ........ N m) as in Figure 2 That is. X₁ Xo(<N- To generate data, follow the time reversal of this Markov chain. To obtain the time reversal, first consider reversing a singular time step, from to
t 1. This corresponds to learning
log po (X,-1 x,)
log po (-
log po
(8)
That is, predict the next token's density. To reverse the whole process, for each timestep:
N m
log pa (o = log po (X)((x()))XXXxxm
(9)
log po (xx X (2m)
(e(m)) (2(2m)
log po (xx0((2)))
which gives us Equation [ joint conditional probability. This loss is different than the conditionally independent losses used in (Shih et al. 2022) and discrete diffusion models (Sahoo al. 2024 Lou ct al. 2023). Notably, their architectures, due to the lack of causal-like attention masking, could not support joint losses.
Expectations over Token Orderings and Prompt Lengths: the objective (Equation 7 we do not make assumptions about the distribution of the prompt length m f()) nor the distribution of the prompt ordering a ~ s(-/m) (so long as a follows the decomposition protocol laid out in Equation 4 task-dependent. For instance, in regular autoregressive tasks,
In general, these distributions would be deterministic, i.e., the identity function.
For this paper, m 21(0.01 * NN 0.10 * N]. To get a, we first sample ope ~ USS)), where Sy represents all the possible permutations of the integers from [0, N). But, remembering Equation 4 efficient masking protocol, we sort each of Opre (< m) and pre (2 m) in ascending order to get This eliminates the ambiguity in paths the model has to learn to calculate given joint, while maintaining which positions are in the prompt and which need to be predicted. Appendices E. and Contain more details.
interested in generating text from near-scratch, we train model where
EXPERIMENTS
Our first experiment (Section 7.J empirically verifies that ASSD with AS-ARMs indeed preserves the output distribution while being faster, as predicted by our theoretical analysis. Our other experiments (Section 7.2 show that on both natural language and coding benchmarks, AS-ARMs, even with fraction of the training resources, beat other model classes of comparable size, and are competitive with models orders of magnitude larger.
7.1 CORRECTNESS AND SPEED OF ANY-SUBSET SPECULATIVE DECODING
We prompt the model with 640 masked sequences from the WikiText test dataset (Merity et al. 2016). As in training, sequences packed together into chunks of 512 tokens. We randomly mask out 95% of each sequence, leaving 5% of tokens (uniformly scattered throughout the sequence) as the prompt. We evaluate our finetuned model. See Table for results. Appendix shows sample outputs.
7

---

Published as a conference paper at ICLR 2026
Sampler
Gen PPL
Entropy
Model NFE
Aux NFE
Time (s)
Sequential
107.9 + 1.6
7.65 = 0.01
486.0 + 0.0
0.0
18.21 0.00
ASSD (N-Gram)
111.7 2.0
7.64 0.01
422.0 = 0.7
422.0
16.80 = 0.03
ASSD (Self)
107.6 1.6
7.64 + 0.01
434.1 0.4
0.0 +.0
16.50 + 0.02
Table I: Comparison of Speculative and Sequential Decoding: Left-to-right, entries show mean and standard error generative perplexity (judge: GPT-2 Large), Shannon entropy, number of AS-ARM function evaluations, number of auxiliary draft model calls, and wall clock time. ASSD (Self) is from Algorithm [ We also modify Algorithm with context-derived N-Grams (Stewart et al.
2024
as a draft model (see Appendix E.4
We set k = 5 for ASSD.
Model
Size
Code Tokens
Lang. Tokens
Pass @ 1
XLNet-Code
110M
15B
OB
38.59
DiffuLLaMA
6738M
19B
46B
40.68
Table 2: Performance HumanEval Infilling: Comparison of code infilling abilities of XLNet finetuned on code versus DiffuLLaMA (Gong al. 2024). We use HumanEval's single-line infilling task (Bavarian et al. 2022), evaluated by pass @ (each attempt counts, instead of only the best attempt) on 5165 trials.
Empirical Correctness of Distribution: As expected from Theorem? 2 the output distribution from speculative decoding is statistically the same as the output distribution from sequential decoding, measured by generative perplexity and entropy.
Speed-Up: Finally, both variants of ASSD (parallel sampling from self as draft, context n-gram (Stewart et al., (2024) as draft) provide statistically significant speedup over regular sequential decoding, as predicted by Lemma (When decoding sequentially, the number of calls the neural network is the same as the number of masked tokens.) ASSD with paralllel sampling is the fastest method, and requires the least total NFEs. ASSD with context n- n-gram is not that far behind: the intuition is that although it gives lower-qualite drafts, it is very cheap. This was also observed in similar studies for AR models (Stewart al., (0224). However, only 1.15 tokens get generated per iteration when using context n-gram, as opposed to 2.24 tokens with parallel sampling as draft.
7.2 INFILLING BENCHMARK TASKS
Code Generation: In coding, bidirectional context is crucial, making appealing case for AS-ARMs. Table 2 shows that AS-ARMs finetuned on code are competitive with models that are orders of magnitude larger. Furthermore, AS-ARMs use fewer finetuning tokens. We evaluate against DiffuLLaMA as a representative baseline, as other discrete diffusion and autoregressive models of comparable size were shown be non-competitive (Gong al. 2024). See Appendix E.6
Model
Size
Tokens
Infill 1/5
Infill 3/5
ROUGE 1/2/L
NFE
ROUGE 1/2/L
NFE
GPT2-S
127M
n/
9.5/0.4/8.7
10.7 2.8
13.5/0.6/10.2
31.8 - 6.0
SEDD-S
170M
210B
11.6/0.8/10.7
3.0
16.2/1.3/12.2
64.0=0.0
MDLM
130M
262B
11.6/1.1/10.7
3.0
13.3/1.0/10.4
64.0 0.0
DiffuGPT-S
127M
+130B
14.0/1.5/13.0
3.0
16.4/2.0/14.
6.00=000
AS-ARM-PT
110M
27
14.4/1.7/13.1
8.7
7.7/0.6/6.4
18.5460
AS-ARM-FT
110M
+12B
13.1/1.1/12.0
10.4 2.8
18.0/1.4/13.2
30.4 = 6.0
Table 3: Performance on ROCStories Infilling:
We test on 1871 short stories of five sentences each,
and get completions (trials) per story. We compare ROUGE (t). "Tokens" is the number of training
tokens (excluding those for the pretrained initialization). "Infill 1/5" inputs sentences {1, 2, 4, 5) and infills (3). "Infill 3/5" imputs sentences (1, 5 and infills (2, 3, 4). We report μ + a NFEs. "+" indicates additional finetuning tokens, after the pretrained initialization. See footnote
8

---

Published as a conference paper at ICLR 2026
Natural Language: We also investigate whether AS-ARMs can outperform discrete diffusion models and regular autoregressive models on natural language infilling tasks, in which bidirectional context is key. We follow Gong et al. (2024) 's setup ROCStories (Mostatazadeh et al. 2016). Table shows that AS-ARMs are generally better than the baselines, while using the fewest parameters and finetuning resources |]
AS-ARM-PT (pretrained XLeNet weights (Yang 2019) from Huggingface, without finetuning) is the best at infilling a single missing sentence. This corresponds to a ~ 20% masking ratio, which is roughly what was pretrained on (see Appendix (.2). Finetuning on a wider distribution of masking ratios as in Equation 7 splits finite model capacity among multiple tasks. Unsurprisingly. when infilling three out of five sentences, our AS-ARM-FT (finetuned AS-ARM) surpasses all the other models on ROUGE-I, and is behind only DiffuGPT-S on ROUGE-2 and ROUGE-L. Overall, AS-ARMs are best on four oul of six metrics in Table
7.3 OTHER EXPERIMENTS
See Appendices G and F for more experiments, including: ablation on number of speculated tokens, extension to sequential benchmark tasks, ablation on sequence length, ablation on acceptance threshold, KV caching, increased model size, training setup ablation.
RELATED WORKS
The defining characteristic of any-order autoregressive models is their ability to estimate the probability density of arbitrary marginal and/or conditional queries. This generalizes vanilla autoregressive models. We primarily base our work off of XINNet (Yang) 2019) and MAC (Shih et al. 2022).. From XLNet, we adopt the idea of causal attention: crucially. this allows us to estimate probability density in parallelized single step, i.e., O(1) time (Yang 2019). From MAC, we adopt recursive decomposition of queries on binary lattice, which reduces the permutations needed to be learned from N! to 2 any-subset autoregressive model.
Any-Order Autoregressive Models:
(Shih et al. 2022): this made MAC the first
More recent AO-ARMs (Hoogeboom et al. 2021 Strauss & Oliva 2021 Shih et al. 2022) abandoned
causal attention, in favor of full attention with absorbing state tokens to represent "aakked" queries. This increases runtime of joint density estimation from (1) to O(N): to satisfy the factorization rule, each token's 's density is estimated one-by-one so it conditions only on the preceding tokens.
Discrete Diffusion: Discrete diffusion models data generation as the reversal of an inhomogeneous continuous-time Markov chain (CTMC). Empirically, the best performing CTMC an absorbing state process. Discrete diffusion can generate tokens in parallel, with arbitary prompt locations. The seminal work is D3PM (Austin et al., 2021). Later, LDR (Campbell et al., 2022] formalized
discrete diffusion under the lens of CTMC theory. More recently, SEDD (Lou al. 2023). MDLM
(Sahoo et al.) 2024). DiffuLLaMA (Gong et al. (002), and L.LaDA (Nie et al. 2025 created discrete diffusion models that rivaled surpassed autoregressive models.
One limitation of discrete diffusion models is their inability to do joint density estimation. Also, although tokens can be generated in parallel with large discretized timesteps, they are generated under the conditional independence assumption, which may not adhere to the limiting joint distribution. Furthermore, the architectures perform full attention across all tokens, even the masks (Lou et al.. 2023; Sahoo et al. 2024); due to this lack of causal attention masking, it is not straightforward to apply KV-caching.
Speculative decoding is an algorithm developed for autoregressive models that enables quick token generation from an inexpensive draft model. Via some techniques related to rejection sampling, the outputs provably adhere to the joint distribution of the target language model (Leviathan et al. 2023; Chen et al. 2022). As compared to other speculative decoding works, our method has the advantages of: (1) theoretical guarantee to never increase NFEs; (2) provides the speculator "for free" without training any auxiliary models; (3) addresses the any any-subset infilling
Speculative Decoding:
'To our knowledge, the pretraining details for the public weights of XLNet- XLNet-Base (which we call AS-ARM-PT: pretrained) were never released. If we assume that XLNet-Bas uses the same pretraining hyperparameters as those reported for XLNNet- Large, it would have heen trained on around two trillion tokens (Yang) 2019. GPT-2 also did not release their full training recipe (Radford et al. 2019).

---

Published as a conference paper at ICLR 2026
problem, which has O(2²) tasks, exponentially more than the O(N) left-to-right tasks addressed in other works. Detailed comparison to other related works are in Appendix
9 CONCLUSION
Our results suggest that the long-forgotten AS-ARMs are promising language modeling method- ology. We theoretically and empirically show that the longstanding problem of principled parallel token generation from the joint distribution is easily solved once we combine AS-ARMs parallelized density estimation and generation with speculative decoding. We can essentially get the speedup as a "free lunch", theoretically guaranteed to without degradation in output quality (as opposed to parallel decoding methods in discrete diffusion models). Furthermore, benchmark tasks indicate that AS-ARMs can match or surpass the currently dominant language modeling paradigms. Future work includes scaling to billion-parameter models.
REPRODUCIBILITY STATEMENT
experimental setup. Mod- https//hhhhoooooborooroyyyyeeeee
Beyond the main text, Appendix contains details els available at
Code
available at
ETHICS STATEMENT
Our work does not have any ethical concerns beyond the usual concerns associated with generative language modeling research.
ACKNOWLEDGEMENTS
This material is based upon work supported by the U.S. Department of Energy, Office Science, Office of Advanced Scientific Computing Research, Department of Energy Computational Science Graduate Fellowship under Award Number DE-SC0025528 to Gabe Guo. We thank Amil Merchant, Tristan Saidi, Luke Bailey, and Ajay Sridhar for helpful discussion.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint 2rXiv:2303.08774 2023.
Zachary Ankner, Rishab Parthasarathy, Aniruddha Nrusimha, Christopher Rinard, Jonathan Ragan- Kelley, and William Brandon. Hydra: Sequentially-dependen draft heads for medusa decoding. arXiv preprini ax2.v2200..05109, 2024.
Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne Van Den Berg. Structured denoising diffusion models in discrete state-spaces. Advances in Neural Information Processing Systems, 34:17988 14:17981-17993, 2021.
Mohammad Bavarian, Heewoo Jun, Nikolas Tezak, John Schulman, Christine McLeavey, Jerry Tworek, and Mark Chen. Efficient training of language models to fill in the middle. arXiv preprint arXiv:2207. 14255, 2022.
Luca Beurer-Kellner, Marc Fischer, and Martin Vechev. Guiding llms the right way: Fast, non-invasive constrained generation. arXiv preprint arXiv:2403.0668 06988, 2024.
Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng. Jason D Lee, Deming Chen, and Tri Dao. Medusa: Simple Ilm inference acceleration framework with multiple decoding heads. arXiv preprint 4rXiv:2401.1077 2024.
10

---

Published as a conference paper at ICLR 2026
Andrew Campbell, Joe Benton, Valentin De Bortoli, Thomas Rainforth, George Deligiannidis, and Arnaud Doucet. A continuous time frumework for discrete denoising models. Advances in Neural Information Processing Systems, 35:28266-2822 2022.
Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William Freeman. Maskgit: Masked generative image transformer. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 1131--1132,, 2022.
Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean- -apt-sst Lespiau, Laurent Sifre, and John Jumper. Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.011/8, 2023.
Jacob K Christopher, Brian R Bartoldson, Tal Ben-Nun, Michael Cardei, Bhavya Kailkhura, and Ferdinando Fioretto. Speculative diffusion decoding: Accelerating language generation through diffusion. arXiv preprini arXiv:2408.0555 035666666,,, 2024.
Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory- efficient exact attention with io-awarensss. Advances in neural information processing systems, 35: 16344-16359, 2022.
Nicola De Cao, Gautier Izacard, Sebastian Riedel, and Fabio Petroni. Autoregressive entity retrieval. arXiv preprint 1:Xiv20000009000 2020.
Justin Deschenaux and Caglar Gulcehre. Beyond autoregression: Fast Ilms via self-distillation through time, arXiv preprint arXiv:2410221035, 2024.
Tianqi Du, Yifei Wang, and Yisen Wang. Bridging autoregressive and masked modeling for en- http://wwwwww....foreoeeeeee
hanced visual representation learning, 2024. URL
id=KUz8gXAqFV
Mostafa Elhoushi, Akshat Shrivastava, Diana Liskovich, Basil Hosmer, Bram Wasti, Liangzhen Lai, Anas Mahmoud, Bilge Acun, Saurabh Agarwal, Ahmed Roman, et al. Layerskip: Enabling early exit inference and self-speculative decoding. arXiv preprint arXiv:2404.16710, 2024.
Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Wen-tau Yih, Luke Zettlemoyer, and Mike Lewis. Incoder: A generative model for code infilling and synthesis. arXiv preprint urXiv:2204.005 059999, 2022.
Yichao Fu, Peter Bailis, Ion Stoica, and Hao Zhang. Break the sequential dependency of Ilm inference using lookahead decoding. arXiv preprint arXiv:2402. 02057 2024.
Saibo Geng, Martin Josifoski, Maxime Peyrard, and Robert West. Grammar--onstrained decoding for structured nlp tasks without finetuning. arXiv preprint arXiv:2305.13971, 2023.
Mathieu Germain, Karol Gregor, Iain Murray, and Hugo Larochelle. Made: Masked autoencoder for distribution estimation. In International conference on machine learning, pp. 881-889. PMLR, 2015.
Aaron Gokaslan, Vanya Cohen, Ellie Pavlick, and Stefanie Tellex. Openwebtext corpus. //kyllion077.ithhul htt.://w.......eeeeeeeeeeeeooooe
http
2019.
Shansan Gong, Shivam Agarwal, Yizhe Zhang, Jiacheng Ye, Lin Zheng, Mukai Li, Chenxin An, Peilin Zhao, Wei Bi, Jiawei Han, al. Scaling diffusion language models via adaptation from autoregressive models. arXiv preprint arXiv:2410.1 17891, 2024.
Emiel Hoogehoom, Alexey A Gritsenko, Jasmijn Bastings, Ben Poole, Rianne van den Berg, and Tim Salimans. Autoregressive diffusion models. arXiv preprint arX0v:2110.02037, 2021.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 6980, 2014.
Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. In International Conference Machine Learning, pp. 19274-19286. PMLR, 2023.
11

---

Published as a conference paper at ICLR 2026
Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Mare Marone, Christopher Akiki, Jia Li, Jenny Chim, et al. Starcoder: may the source be with you! arXiv preprint arXivv2305.06166, 2023.
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng. Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:24/2.19437 2024a.
Sulin Liu, Juno Nam, Andrew Campbell, Hannes Stärk, Yilun Xu, Tommi Jaakkola, and Rafael Gómez-Bombarelli. Think while you generate: Discrete diffusion with planned denoising. arXiv preprint arXiv:2410.0622 1.0.66644, 2024b.
Ilya Loshchilov, Frank Hutter, et al. Fixing weight decay regularization in adam. arXiv preprint 1711.51711.0000/1 5:5, 2017.
Aaron Lou, Chenlin Meng, and Stefano Ermon. Discrete diffusion language modeling by estimating the ratios of the data distribution. 2023.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models, 2016.
Ning Miao, Hao Zhou, Lili Mou, Rui Yan, and Lei Li. Cgmh: Constrained sentence generation by metropolis-hastinge sampling. In Proceedings of the AAAI Conference on Artificial Intelligence. volume 33, pp. 6834-6842, 2019.
Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vander- wende, Pushmeet Kohli, and James Allen. A corpus and cloze evaluation for deeper understanding of commonsense stories. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pp. 839-849, 2016.
Shen Nie, Fengqi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, Jun Zhou, Yankai Lin, Ji- Rong Wen, and Chongxuan Li. Large language diffusion models. arXiv preprint Xiv:25522.0992. 2025.
Arnaud Pannatier, Evann Courdier, and François Fleuret. o-gpts: A new approach to autoregressive models. In Joint European Conference Machine Learning and Knowledge Discovery in Databuses, pp. 143-159. Springer, 2024.
Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The lambada dataset: Word prediction requiring broad discourse context. arXiv preprint 1rXiv:1606.06031, 2016.
YH Park, Chieh-Hsin Lai, Hayakawa, Y Takida, and Y Mitsufujii. Jump your steps: Optimizing sampling schedule of discrete diffusion models. arXiv preprint arXiv:2410.07761, 2024.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, et al. Code llama: Open foundation models for code. arXiv preprint PrXIv:2308.12955 2023.
Subham Sekhar Sahoo, Marianne Arriola, Yair Schiff, Aaron Gokaslan, Edgar Marroquin, Justin T Chiu, Alexander Rush, and Volodymyr Kuleshov. Simple and effective masked diffusion language models. arXiv preprint 2rx2i:2406.07524 2024.
Claude E Shannon. A mathematical theory communication. The Bell system technical journal, 27 (3):379-423, 1948.
Andy Shih, Dorsa Sadigh, and Stefano Ermon. Training and inference on any-order autoregressive models the right way. Advances in Neural Information Processing Systems, 2022.
12

---

Published as a conference paper at ICLR 2026
Lawrence Stewart, Matthew Trager. Sujan Kumar Gonugondla, and Stefano Soatto. The n-grammys. Accelerating autoregressive inference with learning-frr free batched speculation. arXiv preprint arXiv:2411.03776, 2024.
Ryan Strauss and Junier Oliva. Arbitrary conditional distributions with energy, Advances in Neural Information Processing Systems, 34:752-763, 2021.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew Dai, Anja Hauth, Katie Millican, al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312. 18805 2023.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint (IXIv:2302.1377,, 20233
Benigno Uria, lain Murray, and Hugo Larochelle. deep and tractable density estimator. In International Conference on Machine Learning, pp. 467-475, PMLR, 2014.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowic7, et al. Huggingface's transformers: Sfae--of-theaare natural language processing. arXiv preprint 190.v:1910.03771, 2019.
Minkai Xu, Tomas Geffner, Karsten Kreis, Weili Nie, Yilun Xu, Jure Leskovee, Stefano Ermon, and Arash Vahdat. Energy-based diffusion language models for text generation. arXiv preprint arXiv:2410.2 2024.
Zhilin Yang. Xlnet: Generalized autoregressive pretraining for language understanding. arXiv preprint prXiv:1906.022.0 2019.
Haotian Ye, Himanshu Jain, Chong You, Ananda Theertha Suresh, Haowei Lin, James Zou, and Felix Yu. Efficient and asymptotically unbiased constrained decoding for large language models. arXiv preprint arXiv:2504. 25040009550 2025.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv: 1905.07300 2019.
Jun Zhang, Jue Wang. Huan Li, Lidan Shou, Ke Chen, Gang Chen, and Sharad Mchrotra. Draft & verify: Lossless large language model acceleration via self-speculative decoding. arXiv preprint arXiv:2300.081 2023.
Yixiu Zhao, Jiaxin Shi, Lester Mackey, and Scott Linderman. Informed correctors for discrete diffusion models. arXiv preprint arXiv:2407.21243, 2024.
Kaiwen Zheng, Yongxin Chen, Hanzi Mao, Ming- Yu Liu, Jun Zhu, and Qinsheng Zhang. Masked diffusion models are secretly time-agnostic masked models and exploit inaccurate categorical sampling. arXiv preprint arXiv:2409.02900 2024.
13

---

Published as a conference paper at ICLR 2026
A PROOFS
Line numbers refer to Algorithm
Lemma The first token speculated in each loop iteration will always accepted.
Proof. When i = on Line 13
(x(l)
p(Za(i) [x(())) [ Ko(ns))
(10)
p(ão(i) (el()))))ofn
(11)
p(Fo(() Xo((n))
(12)
Pa(i))
(13)
So, r < (e(t)
on Line 19 and thus, when = To(n) will always be accepted on Line 20
n
Pa(s)
Theorem D Algorithm I requires more than N m total function evaluations of p(-I-). That is, there will never be more calls to neural network than the number of tokens decoded.
Proof. The idea underlying this proof is that each iteration of the while loop Line 2 has two function evaluations. The first function evaluation on Lines too to speculate tokens. The second function evaluation is Lines to 15 evaluate the oracle density.
We then just need to show that at least two tokens will be decoded, making for minimum one token generated per function evaluation.
By Lemma T the first token (at i = n)) is always accepted. So, we can move on to i = n 1.
When + regardless of whether accept reject (n++))) will still obtain value for
Fo(n)) whether it is (n)()) Line 20 resampling from the adjusted ()+ distribution Line 22 (which does not require additional calls to p(-1-), it makes use of already calculated distributions from Ginesoandd
So, on each loop iteration, giving us the requisite two tokens.
guaranteed to get token values for at least To(n)) Zo(n+1)) thereby
On the last loop iteration, if N - 1, there is edge case where only one token can be decoded (Line D. But, as previously shown in Lemma this speculated token is mathematically guaranteed to be accepted, since it was the first and only one to be speculated. So, step for this final loop iteration. Thus, to generate this final token, we still only need one function evaluation, maintaining the lower bound of one token generated per function evaluation.
forgo the verification
Theorem
Algorithm produces samples from the true joint distribution ((x)(2)) Xo(<m))
Proof. The only differences between our algorithm and speculative decoding are:
1. The use of p as its own speculator. This is valid, because speculative decoding is agnostic to the speculator function, long gives probability density estimates.
2. The omission of the verification step when decoding the last token n = T on Line 9 This is justified by Lemmal] [ which shows that when i = T 1, Z.(an) would always meet the acceptance criteria were to go through the verification on Line 19
3. We do not sample extra token if all of the speculations are accepted at the end of each loop iteration. However, this does not affect the distribution of the other tokens outputted. We skip this extra token sampling because, shown in Lemma [] the first token speculated in the next iteration will always be accepted, which "makes up" for the omission the extra sample.
14

---

Published as a conference paper at ICLR 2026
Other than these differences, this algorithm is simply speculative decoding, but with different We can permute/sort x in increasing order of oi). and define algorithm is mathematically equivalent to speculative decoding with the
N ordering than o = [0, ......... 1]. the result y. Then, inputs y as the sequence, and the draft model the same as the target model. The rest of the proof of correctness of our algorithm is therefore the same in (Leviathan al. 2023 CCen al. (202)). For completeness, we reproduce their proof in Appendix
B with some additional commentary.
B MODIFIED REJECTION SAMPLING CORRECTNESS
We provide an extended proof of correctness for Theorem 2 as it relates the modified rejection sampling step in Algorithm IT] Note that the main ideas and layout of this proof are copied from (Chen et al. 2023) and (Leviathan et al. 2023). We only include here for the reader's convenience and consistency with our notation.
Proof. WLOG, consider the random variable (token) Ta((). returned by Algorithm the iteration where i < n + k S N, i.e., the iteration that generates Zz(()-
D (ASSD). Consider
For the algorithm to be correct, we desire that (() x)) = p(zo(() 2 2|xx ) where (( a)) is the probability that ASSD generates token value x at position (i). and p(F = x x is the probability that regular sequential decoding would generate x at position (i). In other words, the distribution of ASSD is the same in sequential decoding. Note that {)(<)) (x(x)) O (mns) in both sequential decoding and our algorithm, because they both use generations from previous iterations (in addition to the prompt) conditioning for future iterations.
Let v (shorthand for (o) be the token value generated from the conditionally independent draft in Line 6 When Zz(() = x is true, there are two mutually exclusive and collectively exhaustive possibilities (from the if-else statement): (1) I was accepted, and F = I (2) ã was rejected, and was the result of the resampling.
P(Za(1) = x² = P accepted, ã = x) + (( = x, x rejected) IP accepted [ = x P) = x) + (i rejected))P(x.()) = z| rejected)
(14)
Analyzing the first term, look to the "if" (accept) clause on Line 19
(P accepted [ P( x²- min 1, [o( (x)
(z)
Por(s)
(15)
min (Po(o (4), (z) (2))
Here, (o(1) (x) = p(za(() (<x)) & ). i.e. the oracle density from Line 14 and Pa(i) (z)
p(Za(() = x/Xo(<n))) i.e., the speculator density from Line
Regarding the second term, we look at the "else" (reject) clause on Line 21 We analyze each item in the product separately, the expressions are long:
PP( rejected) = - P(ï accepted)
Σ
P(ro(1) = x', accepted)
Σ min (Poci) (I'), 9o()(()))
Σ (o)))(a - Σ min (pa(o) (x'), (x))))(()()))))
(16)
Σ (ooc)aa)) - min (po (2)()(())) (())
Σ max (x)()((')) Po(i) (x (x0) - 2o(0) (z'))
Σ max (go(3) (x') - (o)))
15

---

Published as a conference paper at ICLR 2026
From Line 22
(( 2/6 rejected) = (p(To(()) = (<n)) &o(na)) P(Fo(() = =)(x(<<)))
(1o() - (z)
(17)
max (9)(() Pa((i) (2), 0)
Σn' max (1n()(2²)) Pr((i) (z'), 0.
Following Equations 16 and 17
we have
P rejected) (Fo()) x| rejected) max (fo())()) (p)()))0),
(18)
Putting all together back into Equation [4]
P(Ta()) r) = min (p)()) (x() + max (Io(1) (z)))
= (x(()((z)
(19)
(((F(()) x x((onen) 8o(m:s)
=p(No(() x(()(()))
The last line Equation 19 is true, because have gotten index in the accept-reject loop (Line 17), we must have accepted Xa(n:]] i.e., Xafnss) Xojna) Therefore, we have shown that ASSD gives the same per-token distribution as in sequential decoding. Induction over E [m N) will easily show that ASSD gives the correct joint distribution.
C PROBABILISTIC GRAPHICAL MODEL.
See Figure 2 for probabilistic graphical model of the data generation process.
Forward: Absorbing State DTMC
== 0: really love machine learning
== 1: really love machine
1 3: machine
machine
Mecensy m1
X xc000)
X Maccal
Pg Pocool Foren 1))
Pg Coremmty Xoyemy
Backward: Data Generation
Figure 2: Probabilistic Graphical Model:
Shows the discrete-time Markov chain for the forward noising process, and its time reversal (i.e., data generation). This justifies Equation
D CAUSAL-LIKE ATTENTION MASKING
We provide further discussion the masking introduced in Section 4
In particular, there is a subtlety in attention mechanisms: even if a token does not attend to itself, still "sees" its content because it constructs query representation compare against the keys. At first glance, this seemingly breaks the ban (Equation 6) on a token attending to itself. The solution separate the positional and content information into streams, as done in XLNeet (Yang) 2019).
16

---

Published as a conference paper at ICLR 2026
The positional stream (which only has positional representations input) is used to calculate the queries, i.e., row index the attention map. The content stream (which has token value information) used to calculate the key/value information, L.., column index of the attention map. Thus, the evaluation of the positional queries does not "cheat" by looking at the ground truth content at its own position. Conceptually, this can be said be more like cross-attention than the self-attention common in discrete diffusion models (Austin et al. 2021).
In contrast, architectures commonly used for discrete diffusion models (Lou al. 2023 Deschenaux 2024; Sahoo et al. 2024] Austin et al. 202T) and other any-order autoregressive models
& Gulcehre (Shih et al. 2022
Hoogeboom et al. 2021) do not support this kind of masking. In these architectures,
Vi,j Aa()).)()) = 1,
(20)
which means that every token is allowed to attend to every token including itself when calculating its probability. From probabilistic graphical modeling perspective, the outputted probabilities very roughly correspond to log p(Za(() x), which would not give us principled estimates on the rrreadyvisiible tokens.
E
ADDITIONAL EXPERIMENTAL DETAILS
E.1
MASK DISTRIBUTION
We also use a similar low-discrepancy sampler as (Sahoo al. 2024) reduce variance among prompt lengths within batch.
E.2 TRAINING HYPERPARA AMPETRSRAMETERS
In finetuning XLNet, we use maximum learning rate 10-4 and batch size 320 (16 per device, accumulations, devices). We have linear learning rate warmup for 5000 steps, and linear learning rate decay for 70, 000 additional steps, making for total of 2.4 10" samples seen. (Notably, this is far fewer than the 2.56 10 samples that DiffuGPT-S saw (Gong et al. 2024).) We start at 15% masking rate, then linearly increase the minimum masking rate to 90% and the maximum masking rate to 99% over 5000 steps.
We train on NVIDIA RTX A6000 Ada devices, which have 48 GB of RAM each. Training lasts around four days. We calculate validation loop metrics every 500 steps on 64 samples. We use the A (Loshchilov et al. 2017 Kingma & Ba 2014] optimizer.
We also tried training for more epochs, but we found that performance saturated after 10B training tokens.
E.3 METRICS
We are focused generation quality. To that end, we have two metrics.
To quantify how likely the generated outputs of our model are, we calculate generative perplexity, where lower values are better. It is calculated
(eI, log ((x. 11/N
PPLgen
(21)
where is an oracle language model (we use GPT-2 Large (Radford et al. 2019) and is generated sequence.
To quantify the diversity in tokens, we calculate Shannon entropy (Shannon, 1948 where higher values are better. The formula is
H(x) - Σ (pg,
(22)
DxEx
where p(x:) Hilx,=x.]], i.e. the frequency of a token in the sequence. [x]
Typically, there is a trade-off between generative perplexity and Shannon entropy - highly repetitive sequences of common words like "a" have low generative perplexity, but also low entropy. Random sequences of gibberish have high entropy and high generative perplexity. We want to generate sequences with high entropy and low generative perplexity.
17

---

Published as a conference paper at ICLR 2026
E.4 SPECULATIVE DECODING VARIANTS
We compare sequential decoding to two variants of ASSD: the first is described in Algorithm | The second uses context-based n-grams, which decoding (Stewart et al. 20224), although they easily fit into our framework. To adopt it to arbitrary order. replace the speculator p(-1-) in Lines 5 Algorithm[] a context-based n-gram model e(-l-), as in Algorithm 2 There, c(a is the probability over the partially decoded sequence that bigram starting in b ends in a, as in Equation 23
initially proposed for left-to-right speculative
c(a|b) 40. [[x (a, [).
(23)
H. [2, = bl
We initialize a(-l-) by sweeping over the prompt, then update it iteratively, as the sequence is decoded.
Algorithm Speculation with Context-Band N-Grams
Input: same as Algorithm
Output: see Algorithm
for i in :) do
xa())- * MASK then
Feond (1)
else
Zcond f -11--1
end
(t) c(-(s) // sample from partially conditioned distribution
Po(i) + c( (eoed) // get partially conditioned density
end
Theorem 3. When 2 1, Algorithm Sasthihmmaalweee Jalways sets Foond to valid non-MASK value.
Proof. We conduct strong induction on 2 m. where m is the given prompt length.
For given value of i, we have two cases:
1. To(0)- + MASK. This case is trivial.
2. -o(1)-1 = MASK. This reduces to showing that (1) * MASK. Firstly, there exists some
j such that o = (i) - 1, which trivially means (j) < (i).. From Equation 4 this is
equivalent saying j < i. By the inductive hypothesis, To(s) A MASK should have been
speculated for all j < i.
We note that context- based n-grams lack the guarantees of Lemma I they could increase the total number function evaluations (AS-ARMs n-gram draft) beyond the sequence length. However, n-grams typically cheap to evaluate, which could make up for the increased function evaluations.
E.5 INFILLING BENCHMARK
We run 5 trials over the dataset of 1871 stories, making for 9355 samples per masking level (short or long) per model. The models compared are: GPT-2 (Radford et al., 2019), MDLM (Sahoo et al.) 2024), SEDD (Lou et al., 2023), DiffuGPT (Gong et al.) 2024), AS-ARM-PT (pretrained XLNet weights) YYngg 2019), and AR--FF (finetuned by us. starting from Huggingface initialization). We do not compare against models like LLaDA (Nie al.) 2025) because they have access to significantly more training budget than us, not to mention their orders-of-maan -faagntu--m larger model
size.
For the AS-ARM models, we use ASSD. We set = 15 for our speculative decoding. Without speculative decoding on AS-ARMs, "Infill 1/5" requires 10.9+2.99 and "Infill 3/5" requires 32.4 6.2
18

---

Published as a conference paper at ICLR 2026
Sampler
Gen PPL
Entropy
NFEs
Time (S)
Sequential
59.08 + 5.61
3.119 + 0.065
486.0 0.0
18.04 = 0.00
Speculative
59.12 5.26
3.206 + 0.064
247.3 1.8
9.36 = 0.07
Difference
+0.08%
+2.78%
- -49.12%
-48.09%
Table Comparison of ASSD (Algorithm) 1) and Sequential Decoding in Off-the-Shelf Model: The entries show mean and standard error of generative perplexity (judge: GPT-2 Large), Shannon entropy, number of network function evaluations, and wall clock time. Metries are calculated over 640 decoded WikiText sequences of length 512, where 95% is randomly masked out. We set & 5 in speculative decoding.
evaluations. For GPT, we use sequential decoding, since it cannot be its own speculator. Following (Gong et al. 2024), we only give GPT the left conditioning, as is not straightforward to give rightward conditioning without instruction-tuning. This experiment can be run on a 16GB GPU,
E.6 CODE GENERATION
We finetune our AS-ARM on 14.7 billion tokens from Starcoder's Python data (Li al. 2023). This corresponds to 75, 000 training steps with a batch size of 384 (16 per device, accumulation steps. GPUs) of 512 tokens each. We started from learning rate 0, warmed up for 20, 000 steps to learning rate 1.2 10-1 - then scheduled to decay linearly for 63,333 333 steps (which corresponds to 32 10 total samples seen), although the run crashed slightly earlier than this. We start from 5% masking rate, then warm up over 20, 000 steps to a 15% minimum masking rate, and a 99% maximum masking rate, sampled uniformly.
Since the XLNet tokenizer does not support whitespaces, we replace whitespace tokens with special characters during training (In <els>, \t <sep>, space. When generating code, we reverse the special character mapping and truncate the leading space from the generated lines, since the tokenizer by default inserts a leading space to each word.
<unk>,
<pad>), where is the
We evalute on the HumanEval single-line infilling task on 1033 Python test cases, following (Gong et al, 2024). We generated 5 completions for each test for sample size of 5165. We evaluate with the pass@ metric, i.e., we count the failure or success of each of the five completions for test case (rather than taking the best out of the five completions).
For the baseline, we compare the publicly available Difful.LaM LAMA (Gong et al. 2024) model. We find that the results from (Gong et al. 2024) actually under-reported. When manually inspecting the outputs, we found that there were often generations that were correct, except that the number of leading spaces was off by one. Our investigation suggests that the LLaMA 7B tokenizer also seems to have an issue with counting prefix spaces, etc. Since this is not a problem with model capacity, we relaxed the evaluation, and manually inserted the ground truth indentation (i.e., correct number of leading spaces) to DiffuLLaMA's outputs. After doing so, the pass@) I rate increased from about 16% to 40%.
This experiment can be run on a 16GB GPU.
F ADDITIONAL RESULTS
F.1 ANY-SUBSET SPECULATIVE DECODING: COMPARISON OF FINETUNED AND OFF-THE-SHEIL
See Table This shows extended results from Table I with the off-the-shelf model from Hugging- face.
While the generative perplexity of the off-the-sheel model is much lower than the generative perplexity of our finetuned model, this comes the cost of very low entropy. Indeed, when manually inspected the outputs, we found that the off-the-shelf model generated highly repetitive, nonsensical sequences of few common words (see Appendix K. On the other hand, the finetuned model
19

---

Published as a conference paper at ICLR 2026
generated sentences that were, for the most part, coherent both semantically and syntatically (see Appendix L
Additionally, the aff-the-shehe model gains larger boost in runtime from speculative decoding. Ex- amining the outputted sequences, it again appears be due to its highly repetitive output distributions, these would be easier speculate with mean field model.
F.2
How MANY TOKENS TO SPECULATE?
k
2
3
5
6
8
10
15
20
Length: 128
Speedup (%)
-4.0
6.7
8.8
8.9
9.1
8.7
8.4
9.0
9.0
PPL p- paVuu
0.66
0.37
0.22
0.49
0.12
0.41
0.48
0.22
0.83
Entropy p-Value
0.14
0.20
0.73
0.48
1.00
0.28
0.76
0.52
0.17
Length: 256
Speedup (%)
-2.7
6.6
8.4
8.8
9.3
8.9
8.4
8.5
8.2
PPL p- Value
0.31
0.03
0.32
0.13
0.47
0.74
0.77
0.39
0.78
Entropy p- p-Value
0.43
0.60
0.49
0.15
0.17
0.48
0.56
0.95
0.22
Table Tuning Generally, ASSD not sensitive to the value of & (number of tokens speculated per call to draft model), so long it is at least 4.
We have ablations on the effect of in Table 5 Generally, our selected value of A was close-to-optimal choice. The difference in perplexity and entropy, according to t-tests, is also not statistically significant (see p-values), as predicted by Theorem 2 (which does not depend on k). Results in the below tables are calculated from 750 sequences, where we infill 95% of the tokens per sequence length 128 and 256.
As had predicted in Section 5 choice of k = 2 slowed down the algorithm, since the algorithm, no matter what, is guaranteed to generate at least two tokens (with two function evaluations: speculate and verify) per loop iteration. So, if only two tokens speculated, there could never be speedup over the sequential capabilities, and fact, the overhead time from running speculative decoding would slow it down.
F.3 EFFECT OF SEQUENCE LENGTH
Length
Speedup (%)
PPL p-value
Entropy p-value
128
8.94
0.49
0.48
256
8.84
0.13
0.15
512
9.44
0.71
0.96
640
10.45
0,23
0.24
768
10.59
0.80
0.35
896
10.90
0.94
0).88
1024
11.00
0.78
0.50
Table 6: Effect of Sequence Length on Speedup: Set k = 5, infill 95% of the sequence.
As we increase the sequence length, the speedup becomes more pronounced. The differences in output distribution from sequential decoding are not statistically different, as predicted by Theorem See T
F.4 KV-CACHE
We implement KV caching. The prompt (which has full self-attention) is fixed. As we generate new tokens, they do not become part of the prompt: that way, can keep their causal attention and not have to recompute the prompt hidden states, while still caching the new token representations. We do recompute positional encodings, since XLN uses relative positional encodings (Yang) 2019)
20

---

Published as a conference paper at ICLR 2026
We generally found that KV caching not helpful for smaller model sizes and sequence lengths. But, does help as we scale up the problem size. When using a 340M parameter XLNet (off- sh,ll,, we did not have the resources retrain) to infill 128 tokens out of sequence length 1024 (as opposed to 10M parameter on sequence length 512), we observe that KV caching yields a 50% speedup in model inference time. See Table
Caching
Time (s)
KV Cache
0.0706 + 0.0006 0.1407 +: 0.0004
No Cache
Table KV Cache Benefits: Average inference time of the model (per parallel speculation NFE), averaged over 100 trials (KV cache, versus not using KV cache), Results on 340M-parameter XLNet infill 128 out of 1024 tokens.
F.5 DASILON-TOLERANE
On line 19 (Algorithm we tried adding small absolute tolerance to the gett) acceptance threshold, Po(s)
such that the decision is min(1, P.(t)
Bo(s) + <). Intuitively, having higher tolerance will lead to more
speculations being accepted, at the cost of quality.
See Table 8 We do 500 trials filling in 95% of a 256 token sequence, with k 5. (They were run with different model weights than in the other experiments, but given that the results align with our expectations, we don't 't think they would change much on different model weights.) Basically, while we could get a little more speed boost by increasing the error threshold, the generative perplexity gets lot worse.
Tolerance
Speedup (%6
Spec PPL
Spec Entropy
Sey PPL
Seg Entrupy
PPL
Entropy p-Val
2.85
127.31 3.13
0.2
1 10
8.46
120.72
2.61
6.97 + 0.01
121.76
2.76
6.98 + 0.61
0.77
0.39
123.28
3.40
6.97 + 0.01
120.84 : 2.97
10-2
9.28
128.89
2.65
6.99 0.01
124.17 2 3.12
6.98 1 0.01
0.18
0.20
5
11.05
152.95
3.92
6.99 4: 0.01
118.96
2.69
6.96 4: 0.01
0.00
0.00
4.63
7.02
6.99
Table 8: Effeet of tolerance in Algorithm 1 Line 19 "Spec" is short for speculative, "Seq" is short for sequential, "PPL" is short for generative perplexity, "p-Val" is short for p-value from the t-test.
F.6 LEFI-TO-RIGITT RESULTS
Decoding Speed: Our results extend to left-to-right prompting tasks. Firstly, the speculative decoding scheme still speeds up inference time without losing quality when evaluated with prefix prompt and being asked to fill the right. See Table 9
Method
Time (s)
NFE
Gen PPL
Entropy
Sequential
14.3 + 0.0
383.0 + 0.0
61.2
1.2
7.35 + 0.02
Speculative
13.3 13.3 0.1
349.5 + 0.7
60.0 + 1.1
7.36 + 0.02
Table 9: Sequential versus Speculative Decoding Left-to-Right Prompting: Over 700 trials on WikiText (Merity et al. 2016), we prompt with 128 token prefix, and have complete the 384 token suffix.
Here is an example prompt for the left-to-right experiment (128 tokens):
<sep><cls>

Robert Boulter =<spxxxLaxxxxxxxxxxXXXXXx/
Robert Boulter is English film television and theatre actor
He had a guest @-0 starring role on the television series The Bill in 2000 This was followed by a starring role in the play
21

---

Published as a conference paper at ICLR 2026
Herons written by Simon Stephens , which was performed in 2001 at the Royal Court Theatre He had a guest role in the television series Judge John Deed in 2002 In 2004 Boulter landed role as \" Craig \" in the episode \" Teddy 's Story \ of the television series The Long Firm
Here the continuation from Any-Subset Speculative Decoding (360 NFE to get 384 tokens):
In 2005's s drug \"The Filipino Giant \" he played Jake, a drug addict who he meets at Hospital Montgomery America's crooked drug dealer (played by William Costolo) rented out by a local dentist during Big Monkey Seasons. In 2006 Ted was a twin brother as \" Bear\" and a full-blown hectic guy conducting scenic trips with New Kingdom Railroad, In 2009 he played middle-aged murder victim in Common Core 2017 Cast edit <<sep><cl>><//>>> Lawmakers from both parties are leveraging the Lincoln- House dispute
parrow votes below certain major milestones: Maryland Governor cripples Democrat Ross RUMPRILLE, Md. \u2014 Maryland Gov. Larry Hogan used a month and a half in the Republican-cootrolleed
Senate to cut off a deadline, for Senate voters to top out the king's s yacht. If the votes blow up at midnight on Friday, Hogan told reporters, \"there's s no deadline at all.\" On Thursday, prosecutors in Montgomery said Hogan delayed issuing state pardon to Theo Roe, one of the nation's oldest pot thieves, as he and Ashley Sandrow got significantly older. A state attorney general's office in Baltimore clarified Rednesday that Hogan's assurance that replicators will act quickly as they demand pardon said he looked forward to using increasing leverage against his Republican challengez, including a deadline to avoid a lu20icClosing Chance \u201d in the veto-proof cutoff. The attorney general's office said Hogan anticipated the refusal to give a pardon that would likely set up trades in the Deloitte Group and other European and Asian investors who are sold by Washington and other payments to lethal injections. \u201cIn order to circumvent the blackout chance that moves the act out of the way if done allowing them to protect Goodwill and funds from being put
Left-to-Right Benchmarks: Results left-to-right benchmark tasks are in Table 10 On Lambada (Paperno et al. 2016), AS-ARM-PT performs the best out all the models, supporting our claim that AS-ARMs are powerful model class. SEDD-S (Gong al. 2024) performs the worst. Something interesting we see is that for both DiffuGPT (which was finetuned from GPT) and AS-ARM-FT (which was finetuned from AS-ARM-PT's XLNet weights), performance on Lambada drops after finetuning. We hypothesize that the drop-off comes from the fact that the original models were already well-suited for the Lambada task. Lambada is task predict the last word (one or a few tokens) in a passage. GPT, with the left-to-right next token prediction objective, is well-suited for that. Similarly, AS-ARM-PT, being trained infill 20% of sequence, is well-suited to tasks with dense context. On the other hand, when one fine-tunes DiffuGPT and AS-ARM-FT, the desired output distribution shifts away from that. DiffuGPT now trained do any-order prediction (via CTMC diffusion process), and AS-ARM-FT shifts the focus to language generation from near-scratch (i.e., sparse context). Since models have finite capacity, it makes sense that the performance on the specialized lost-word prediction task would drop. On HellaSwag (Zellers et al. 2019). AS-ARMs a bit behind the baselines, but not extremely so. Overall, the results on sequential tasks suggest that method is valid, even in settings different then it was intended for.
22

---

Published as a conference paper at ICLR 2026
Model
Lambada
HellaSwag
GPT-S (Gong et al. 2024
25,9
29.9
DiffuGPI-S (Gong et al. 2024)
21.6
33.4
SEDD-S (Gong al. 2024)
12.4
30.2
AS-ARM-PT
26.6
28.0
AS-ARM-FT
14.1
28.3
Table 10: Left-to-Right Right Benchmark Tasks: Comparing AS-ARMs to baselines (both autoregressive and discrete diffusion) left-to-right benchmark tasks.
(a) Entropy (t)
(b) Generative Perplexity (1)
(C) NFES (4)
Figure 3: Fixed (Recursive Binary Lattice) Versus Any Permutation Mask Decomposition:
Validation loop metrics on generated sequences from each training strategy. The curves shown are for models trained with an effective batch size of 96 across four NVIDIA RTX A4000 devices. Each validation iteration has 36 sequences of length 512 tokens.
G ABLATION
G.1 MASK DECOMPOSITION ABLATION
Figure shows the ablation of mask decomposition protocol described in Equation 4 (generation diversity) consistently better when training with the recursive binary mask decomposi- tion protocol (2 subsets, AS-ARM style) than with any of the N! permutations (AO-ARM style). As training progresses, the generative perplexity is also better with the AS-ARM style. That being said, training any of the N! permutations (as opposed to restricting to the 2 subsets) does result in fewer NFEs for ASSD generation, but this comes at the cost of lower quality outputs overall (as measured by the aforementioned entropy and generative perplexity).
The entropy
G.2 IMPACT OF MASKING DISTRIBUTION
Figure shows the effect of the realization of the distribution of prompt (i.e., masking) length from Section E. Since the validation task is to infill 95% a masked sequence, it expected that training the model exclusively
shorter prompt lengths would be better than training the model on mixture of long and short prompt lengths (which would dilute its capacity). Indeed, we see that, with respect generative perplexity, the model trained with m 24[0.01, 0.10) outperforms the model trained with ~ [00001 0.85]. where m is the percentage of the source text that is given as prompt (i.e., unmasked). On entropy, the short prompt training strategy performs marginally better at low training steps, but the gap closes training continues.
G.3 IMPACT OF MASKING WARMUP
We also conduct small-scale exploratory training run to test the sensitivity of our training scheme to hyperparameter choices, particularly the masking warmup in code generation (Section [.6} from StarCoder (Li et al.] 2023). Due to computational constraints, 4 accumulation steps, 2 A4000 GPUs). We have a final minimum masking rate of 85% and a final maximum rate of 99%. One of the started from 15% masking rate, and warmed up the final
batch size of 32 (4 per device,
23

---

Published as a conference paper at ICLR 2026
vol_gen_entropp
vALggenperppeeot
200
7.
160
120
6.5
80
Step
Step
10k 30k
70k
20k
50k
60k
(a) Entropy (1)
(b) Generative Perplexity (1)
Figure 4: Narrow (1% + 10% Versus Wide (1% + 85%) Prompting Rates: Validation loop
metrics generated sequences from each training strategy, it relates to the distribution of prompt lengths in the train set. The curves shown are for models trained with an effective batch size 320 five NVIDIA RTX 6000 Ada devices. Each validation iteration has 64 sequences of length 512 tokens from Open Web'ext, where the task to infill 95% of the masked sequence given a 5% prompt.
(a)
(c)
Figure 5: Impact of Masking Warmup: We compare code generation training loss with masking rate warmup and without masking rate warmup.
rates over 5000 steps, while the other run already started from 85% masking rate and took only a single warmup step (meaning that it effectively trained at the final range since the beginning).
Looking at Figure we see that at the beginning, the masking warmup makes a difference in the loss (since the task distribution different); but after the warmup is finished, there is negligible difference. The conclusion is that the ultimate performance is fairly insensitive to the masking rate warmup.
G.4 IMPACT OF PRETRAINED WEIGHTS
We also assess the impact of finetuning from the original XLNet pretrained weights (Yang) 2019) or training the architecture from scratch. Again, due to computational constraints, we set batch size 32 (4 per device, accumulation steps, A4000 GPUs). We have one run initialized from the pretrained weights, and the other initialized with random weights; the runs are otherwise identical. This experiment is done WebnWeTText (Gokaslan al. 2019).
See Figure 6 We see that it is very important to initialize from the pretrained weights. The training loss and generative perplexity said, we see the loss on the randomly initialized model improving, so we believe that given enough compute, it could eventually reach the desired performance.
very poor when initializing from random weights. That being
24

---

Published as a conference paper at ICLR 2026
1900
(a)
(b)
(c)
Figure 6: Impact of Initialization: We compare the performance of a model initialized with the
XLNet pretrained weights (Yang. 2019) and model initialized from scratch, both training
OpenWebText (Gokaslan et al. 2019).
H ADDITIONAL RELATED WORKS
H.1 ANY-ORDER AUTOREGRESSIVE MODELS
There some works from the pre-transformer era Vaswani et al. (017)) that deal with the problem of arbitrary density estimation, namely NADE (Uria et al. 2014) and MADE (Germain et al., 20155). AO-ARMs also bear some resemblance to generative, transformer-base masked image models
CCaag et al. 2022 Du al. 2024).
H.2 DISCRETE DIFFUSION MODELS
There is also work on improving sampling. EDLM (Xu et al. 2024) learns partition function to allow parallel sampling, while ensuring that the generated tokens adhere desired joint distribution. DDPD (Liu et al. 2024b) learns a planner network to optimize the step size taken with uniform diffusion models. JYS (Park et al. 2024) also works on optimizing the step schedule, although their method involves an optimization problem over (part of) the training dataset. (Zhao et al. (2024) work on predictor-corret cooreetor methods and introduce the -Gillesppe sampling algorithm. SDTT (Deschenaux & Gulcehre, (024) distilled discrete diffusion models to achieve comparable generative performance with fewer sampling steps. However, (Zheng et al., 2024) work revealed a numerical precision error with discrete diffusion sampling that leads to unintentional low-temperature sampling.
H.3 SPECULATIVE DECODING
Simple draft models include context-based and model-based n-grams (Stewart al. 2024). Looka- head Decoding also relics upon parallel generation of n- n-grams, and does not need to train an additional draft model (Fu al. 2024). Hydra (Ankner al. 2024) and Medusa (Cai et al.) 2024) augment the target language model with lightweight heads (which require additional training) to quickly predict additional tokens. The advantage of this approach, like ours, that at least tokens are guaranteed to be accepted on each loop iteration: the first token comes from the base model; if the next token rejected, is resampled from combination of the proposal and oracle distributions. There are also works (Zhang et al. (003) that, similar to ours, use the same model for drafting and verification; one such work is LayerSkip (Elhoushi al. 2024). This is known as speculaeee decoding. difference is that these methods skip model layers, while we skip inputs. There is also speculative diffusion decoding (Christopher al. 2024), which uses a discrete diffusion model as the drafter for larger autoregressive model. However, all these works limited to the left-to-right regime, since the autoregressive oracle can only process (NN) (all prefixes) prompting patterns this is exponentially less than the ( (all subsets) prompting patterns our method can handle.
The only work that we know that uses a speculative decoding- algorithm for non-left-to-rigeee models is o-GPT (Pannatier al. 2024). However, their algorithm actually violates Theorem [ and Theorem 2 meaning that is not theoretically guaranteed to produce tokens from the correct target distribution, and slow down sampling. See Appendix ffo details.
25

---

Published as a conference paper at ICLR 2026
H.4 CONSTRAINED DECODING
In principle, constrained decoding techniques can be combined with vanilla AR models to accomplish infilling. Some recently popular approaches to constrained decoding model the constraint formal grammar (Geng et al. 2023; Beuuer--eellll Kellner et al., 2024) and only allow the left-to-right LLM to decode sequences that are valid under this grammar parser. Other approaches use constrained beam search with trie data structures (De Cao et al. 2020) and Metropolis-Hasting Sampling (Miao et al. 2019). Recently, Ye al. (025) created an asymptotically unbiased constrained decoding algorithm based importance sampling and GPU parallelism. However, while technically compatible with infilling, these works generally focus on other tasks, like entity retrieval or information extraction. Furthermore, constrained decoding incurs computational time overhead top of the AR model, and biases the output distribution. contrast, AS-ARMs natively handle arbitrary infilling tasks without overhead and directly estimate unbiased conditional probabilities.
I COMPARISON TO SIGMA GPT
Algorithm 3: Token- based rejection sampling, reprinted from o-GPT (Pannatier et al. 2024)
Input: T: minimum target length, y: any-order autoregressive model, N,; number of orderings to sample, X: prompt of length to
t to while / 7'd do
In parallel, compute distribution conditioned on prompt p(x))t Vi T.....T In parallel, sample at every position ā; p(z,|X), Vi E L....,,?
Draw No random order a and in parallel, compute all logits q [x,[X, t Vi E ,....TT In parallel sample T t variables u, ~ 2/ 1]. vi E t...., T from uniform distribution.
In parallel, compute the acceptance decision u, min (1, (( for every (xa
order.
Select the order that accepts the most tokens before seeing a first rejection.
Keep that order and add the accepted tokens before the first rejection to the prompt.
10
Set t + a
end
GGTT a work that superficially seems to have a similar sampling algorithm as ours for any-order autoregressive models (Pannatier et al. 2224).. However, will see, their algorithm (Algorithm 3) actually has subtle yet critical mistakes that lead to violations of Theorem and Theorem 2 removing the theoretical guarantees.
1.1 AMBIGUOUS FACTORIZATION ORDER AND JOINT DISTRIBUTION
Firstly, their algorithm does not provably give the correct joint distribution, because they randomly sample multiple factorization orders (Lines 5 ( at each draft-accept cycle, given a prompt. This would re-introduce the consistency problem in Equation 3 i.e., different orderings give different joint distributions. Thus, it is unclear what the true target distribution they are trying to match actually is, violating Theorem 2
Another consequence of the multiple factorization orders sampled is that the number of NFEs could exceed the number of tokens eventually accepted, if the multi-order oracle evaluations all reject the speculated tokens. So, Theorem is violated, which means that this scheme could potentially slow down sampling.
I.2 LACK OF RESAMPLING STEP
Furthermore, their algorithm also does not have a resampling step in case of rejection. This decreases the number of tokens each iteration is guaranteed to accept from two to one (see Theorem [s proof). This means that their algorithm is not mathematically guaranteed to reduce the number of function
26

---

Published as a conference paper at ICLR 2026
evaluations, and can in theory increase it: in the case that only the first conditionally independent token from the draft is accepted, the function evaluation of the oracle did not lead to an extra token being accepted. This violates Theorem I
Furthermore, the lack of resampling potentially once again violates Theorem 2 s guarantee that the returned distribution will match the joint. Using the notation and ideas from Appendix [B] they have the "accept" (first) term in Equation [4, but they do not have the "reject" (second) term to balance the probability out. In fact, their Line cannot even be considered be proper rejection sampling, because it lacks the proper normalization for the decision threshold (Leviathan et al., 2023).
1.3 ANY-ORDER SPECULATIVE DECODING ADDRESSES PROBLEMS
In contrast, by enforcing (Shih al. 2022 's recursive binary lattice mask decomposition, ensure that given prompt, there is only one correct path to calculating the joint conditional probability of the missing tokens. This makes clear our Algorithm [ what the target distribution actually and our output provably matches that distribution (Theorem 2 Furthermore, we have resampling in Line 22 Combined with the single-pate evaluation, this mathematically guarantees that we never increase the number of function evaluations above the number of masked tokens (Theorem
J ABSOLUTE SPEED COMPARISON
The primary goals of this paper are (1) to demonstrate that we can develop a lossless parallel decoding scheme for AO-ARMs/AS-ARM, (2) show that this long-forgotter model class can be competitive with the currently dominant model classes. Indeed, we achieve these goals.
That being said, it is also useful understand how compares to acceleration methods that
designed for other model classes. We compare against GPT-2-S (autoregressive) (Radford et al. 2019) and MDLM (discrete diffusion) (Sahoo al.] (024). For GPT-2, we do regular decoding and KV caching. For MDLM, we vary the number of timesteps (e.g.. parallel inferences). For AS-ARMs, we do sequential decoding, ASSD with self-speculation, and ASSI) with n-grams. We have each model
generate 95% of 512-token sequence, prompted with the leftmost 5% (taken from Open WeenWeeText (Gokaslan et al. 2019)). We conduct 200 back-to-back trials per decoding method per model. Results in Table
Steps
Gen PPL
Entropy
Time (s)
512
96.5 3.5
7.18 0.03
5,88 1 0.01
448
7.21 0.03
5.44 + 0.01
384
101.8 3.9
7.18 0.03
4.94 - 0.01
256
117.8
7.32 0.02
3.74 - 0.01
128
137.9 + 4.1
7.37 + 0.02
2.08 + 0.00
(a) MDLM (discrete diffusion) (Sahoo et al.
2024)
KV Cache
Gen PPL
Entropy
Time (s)
No
18.4 = 0.3
7.04
6.29 = 0.00 2.92 + 0.00
Yes
18.2 + 0.3
7.01 + 0.02
(b) GPT-2 (autorcgressive) (Radford et al. 2019
Decoding Strategy
Gen PPL
Entropy
Model NFE
Aux NFE
Time (s)
Sequential
65.5 + 3.1
6.67 = 0.11
487.0 = 0.0
0.0
19.24 = 0.11
ASSD (Self)
65.1 + 3.9
6.76 + 0.10
435.8 + 3.2
0.0 0.0
17.13 + 0.13
ASSD (n-gram)
63.2 + 3.1
6.68 +: 0.12
359.6 + 6.4
359.6 + 6.4
15.09 + 0.28
(c) AS-ARM
Table 11:
Absolute Speed Comparisons: Speed and generative perplexity of decoding methods for various discrete generative model classes.
Before we get into the analysis, we want give some caveats the interpretation:
27

---

Published as a conference paper at ICLR 2026
1. When comparing speeds, should look at how one decoding scheme performs compared to another decoding scheme for the same model - this provides an aoples-to-appll compar- ison, the only thing that changes is the decoding strategy. Otherwise, the architectural engineering, training strategy, etc., tint this comparison.
2. In that vein, is somewhat flawed to compare generative perplexities for AS-ARMs with ASSD the generative perplexities for GPT-2 and MDLM, because the tokenizer used for XLNet is different than that used by GPT-2 and MDLM. (XLNet has a 32,000 word vocabulary (Yang) 2019), while GPT-2 and MDLM use the same 50,257 word vocabulary (Radford et al. 2019).) Since length affects perplexity (and tokenization scheme affects length), generating 512 tokens from XLNet inherently not comparable to generating 512 tokens from MDLM. Furthermore, because the generative perplexity judge is GPT-2 Large, could imagine that it is biased towards outputs that GPT tokenization scheme that it uses. However, we could still discern rough trends from generative PPL comparisons. Inference time per token is somewhat fairer comparison, we wish to compare across models.
generated using the same
From this evaluation, conclusions are:
1. Parallel decoding (up to a certain point), KV caching, and ASSD can all provide lossless acceleration for their respective model classes. KV caching on GPT-2 seems to be the fastest lossless accelerator the moment. That being said, KV caching is orthogonal to ASSD, so future lossless accelerators could use some combination of both.
2. N-gram speculator can sometimes be empirically faster than self-speculative ASSD, due the basically zero cost to go an n-gram lookup table. That being said, n-gram does not have the theoretical guarantees of never decreasing function evaluations that the self- speculative variant does, so should use it with caution as the problem difficulty scales.
3. MDLM and GPT-2 are currently faster than AS-ARM. This is likely related to the engineer- ing optimizations that went into their architectural implementations, namely the of Flash Attention (Dao et al.) 2022). That being said, our approach, in principle, is flexible with respect to architecture.
4. GPT-2 has the best generative perplexity, presumably because it was specially designed for left-to-right tasks. That being said, it is not necessarily better on the other benchmark tasks in this paper, and generative perplexity is known to be flawed metric that can be "hacked" (Zhnng et al. 2024).
For future studies, we would have architectures that support training as an AS-ARM, diffusion language model, or regular autoregressive model, so that we can do a headd comparison (with the same tokenizer, etc.). For this study, we did not emphasize that, the point to show that (1) lossless parallel generation is possible with AS-ARMs; (2) AS-ARMs are competitive with other model classes, and should be given more future research attention.
K OFF-THE-SHEL MODEL OUTPUTS
Text comes from the WikiText dataset (Merity et al. 2016). This is from the default XLNet model provided on Huggingface (not our finetuned), which was only trained to predict around 20% of tokens.
28

---

Published as a conference paper at ICLR 2026
Original Text
<sep><<l<><<<>> Robert Boulter =<sp>>cll><<s<<spppccssss
Robert Boulter is an English film, television and theatre actor He had a guest @-@ starring role on the television series The Bill in 2000. This was followed by a starring role in the play Herons written by Simon Stephens which performed in 2001 at the Royal Court Theatre He had a guest role the television series Judge John Deed in 2002
2004 Boulter landed role Craig in the episode Teddy 's Story of the television series The Long Firm he starred alongside actors Mark Strong and Derek Jacobi. He was cast in the 2005 theatre productions of the Philip Ridley play Mereury Fur which performed at the Drum Theatre Plymouth and the Menier Chocolate Factory in London He was directed by John Tiffany and starred alongside Ben Whishaw Shane Zaza Harry Kent, Fraser Ayres Sophie Stanton and Dominic Hall <sep><<ls><<>> In 2006 Boulter starred alongside Whishaw in the play Citizenship written by Mark Ravenhill He appeared on 2006 episode of the television series, Doctors followed by role in the 2007 theatre production of How to Curse directed by Josie Rourke How to Curse was performed at Bush Theatre in the London Borough of Hammersmith and Fulham. Boulter starred two films in 2008 Daylight Robbery by filmmaker Paris Leonti and Donkey Punch directed by Olly Blackburn In May 2008 Boulter made guest appearance @-@ part episode are of the television series Waking the Dead, followed by an appearance the television series Survivors in November 2008 He had recurring role in ten episodes of the television series Casualty in 2010, as Kieron Fletcher Boulter starred in the 2011 film Mercenaries directed by Paris Leonti <sep=<lls><<>>>>spp><ccss
two
= Career = /ttp:://w........pooo/////////// ==spp==cls=<>>>>spp>>>llsss In 2000 Boulter had a guest @-@ starring role the television series The Bill: he portrayed Scott Parry in the episode. In Safe Hands" " Boulter starred as Scott
= 2000 - 2005 =
Prompt
the television
Court
and
He
2005
Plymouth
Stanton and
starred
play Citizens
Josie
key
Merc
the
portrayed
starred
29

---

Published as a conference paper at ICLR 2026
Off the-Shelf, Sequential Decoding
300 300 300 300<eop> Ethnic or Ethnic Ethnically Ly Americans or Others or Others Beyond Ly Ly Ly Ly Ly Ly Ly Ly Ly Ly Ly Ly Ly Ly Ly Ly Ly Ly Ly Ly Ly Ly Ly the television Ly Ly Ly Ly Ly Ly Ly Ly Ly Ly Ly Ly Ly Ly in Ly Ly Ly Ly Ly Ly Ly North ) Ly Ly Ly Ly Ly Ly Ly Ly Ly Ly Ly Ly Ly Ly
Ly Ly Ly Ly Ly Ly Ly Ly Ly Ly Ly Ly Ly Ly Ly Ly Ly Ly Ly Ly Ly Ly Ly Ly" Ly Ly Ly Ly Ly Ly Ly Ly Ly Ly Ly Ly Ly Ly Ly Ly Ly Ly Ly Ly Ly Ly Ly Ly Ly Ly Plymouth by Ly Ly Ly Ly by
Ly Ly Court Ly
Ly
Ly Ly Ly He Ly
Ly Ly Ly Ly Ly Ly Ly Ly Ly Ly Ly Ly Ly Ly Stanton Ly Sy Ly play Citizens Ly Ly Ly Ly Ly Ly Ly Ly Ly Ly Ly LY Ly Ly Ly Ly Ly Ly Ly Lykey Ly
Ly Ly Ly Ly Ly Ly Ly a Ly Ly Ly Ly by
Ly Ly Ly Ly
Ly Ly Ly Ly Ly J Ly Ly
Ly Ly Ly. Ly Ly
Ly Ly Ly Ly
Ly Ly Ly Ly Ly Ly Ly Ly Ly Ly Ly Ly Ly Ly Ly
Ly Ly Ly Ly
Ly Ly Ly
Ly Ly Ly
Ly Ly Ly Ly Ly Ly = Ly
Ly Ly
Ly Ly Ly Ly Ly Ly Ly portrayed Ly Ly Ly Ly Ly Ly Ly
Ly Ly Ly Ly Ly Ly Ly Ly
Ly Ly
Ly Ly Ly
Ly; Ly Ly starred Ly Ly Ly Ly Ly
Off-the-Sheli Speculative Decoding (Algorithm l)
series and led starred starred starred starred starred starred starred starred starred starred starred starred starred starred starred starred starred starred starred starred starred starred starred starred starred starred starred starred starred starred starred starred starred starred starred and in the television major starred starred starred starred starred starred starred starred starred starred when the on in became heing the title characters into scene as filming the, and too happen which came. The Screen Court Movic"""" 1996)( were cast they are booked cast to be cast moondicly on set as the movie upon movies too news faint as of taking pres "" They cast to appear on set as of other celebrities the her shee sheeee heee and and casting as the Hee Hee she 2005 """"Uh Uh Uh Uh Uh Uh Uh, Plymouth listed on set the first here Hee hee she is casting as they are casting to appear as the scheduled here Hee hee he cast as previously its scheduled her to appear as Stanton and into the Las Vegas during draft her seen her can be cast or starred as the backup for the play Citizens Las Vegas are casting ankle to appear assed or a the next scheduled Las Vegas Las ("""=" shortly before after the next scheduled Las Vegas Las Vegas Las "" Josie Las Vegas Las ""jinohlah,x Vegas Las Vegas Las Vegas Las Vegas Las Vegas Las Vegas ". Las Vegas Las Vegas Las Vegas Las Vegas Las Vegas Las Vegas Las Vegas Las Vegas Las Vegas Las Vegas Las Vegaskey Las Vegas Las Vegas Las Vegas Las Vegas Las Vegas Las Vegas Las Vegas Las Vegas Las Vegas Las Vegas Las Vegas Las Vegas Las Vegas Las Vegas Las Vegas Las Vegas Las Vegas Las Vegas Las Vegas Las Vegas Las Vegas Las Vegas Las Vegas Las Vegas Las Vegas Las Vegas Las Vegas Las Vegas Las Vegas Las Vegas Las Vegas Las Vegas Las Vegas Las Vegas Las Vegas Las Vegas Las Vegas Las Vegas Las Vegas Las Vegas Las Vegas Las Vegas Las Vegas Las Las Vegas Las Vegas Las Merc Las Vegas Las Vegas Las Vegas Las Vegas Las Vegas Las Vegas Las Vegas Las Vegas Las Vegas = Las Vegas Las Vegas Las Vegas Las Vegas Las Vegas Las Las Vegas Las Vegas Las Vegas Las Vegas Las Vegas Las Vegas Las Vegas Las Vegas Las Vegas Las Vegas Las Vegas Las Vegas Las Vegas Las Vegas the Las Vegas Las Vegas Las Vegas Las portrayed Las Vegas Las Vegas Las Vegas Las Vegas Las Vegas Las Vegas Las Vegas Las Vegas Las Vegas Las Vegas Las Vegas Las Vegas starred Las Vegas Las Vegas Las
L FINETUNED MODEL OUTPUTS
Text comes from the WikiText dataset Meerty et al. 2016). Samples correspond to Table with our model trained from 90% 99% masking rate, described in Appendix E.2
30

---

Published as a conference paper at ICLR 2026
Original Text
<els></>> AFI is Top 10 Sports Filnessp-clss<ss<epppccsss
Legacy
In the decades since its release, The Hustler has cemented its reputation as classic. Roger Ebert, echoing earlier praise for the performances. direction and cinematography and adding laurels for editor Dede Allen cites the film one of those films where scenes have such psychic weight that they grow in our memories.. He further cites Fast Eddie Felson one of only handful of movie characters real that the audience refers to them as touchstones.. " TV Guide calls the film a dark stunner offering grim world whose only bright spot is the top of the pool table, yet with characters who maintain shabby nobility and grace. "The four leads are again lavishly praised for their performances and the film is summed up as not be missed. "<sep><<ls><<>> Paul Newman reprised his role as Fast Eddie Felson in the 1986 film The Color of Money for which he won the Academy Award for Best Actor and critics have suggested that this Oscar was in belated recognition for his performance in The Hustler. In 1997. the Library of Congress selected The Hustler for preservation in the United States National Film Registry
Leading Role number of observers
culturally historically Carroll and Rossen screenplay was selected by the Writers Guild of America 2006 as the 96th best motion picture screenplay of all time. In June 2008 AFI released its 'Ten top Ten the best ten films in ten classic American film genres - after polling over @ @ 500 people from the creative community The Hustler was acknowledged as the sixth best film the sports genre <<ep><<el>>
>>>>>>>>>>>>>>>>>> The Hustler is credited with sparking a resurgence in the popularity of pool in the United States which had been the decline for decades
aesthetically significant
The film also brought recognition to Willie Mosconi who despite having won multiple world championships, was virtually unknown the general public. Perhaps the greatest beneficiary of the film 's popularity was
Prompt
<sep> editor
direction
our
the
and lavish and film not.
role
for Library
Ten
ten also Willie
Hus
31

---

Published as a conference paper at ICLR 2026
Finetuned, Sequential Decoding
available all user accounts. Feel free to throw us a few questions and development suggestions.<sepee wetsssseppcssss/ssss. GENEVA, Switzerland - The paper will not draft further stories, and we will, after all, ignore the party of journalistic freedom, which includes the right not to evaluate claims against anyone, except dissenting direction from the scientific community. I assure you that we will continue the editor-in-chief role as quarterly magazine where there will be exemptions for writers of particular viewpoint. This will lead our competition to host an active website, WMSU, which will host an automated competition about WMSU as well as develop accredited methods of talking, researching and writing editorial articles in Swedish ("HF") and English (Roban Xismen in German). History edit Flawed motives for the art of the book have been debated much.[2] Until 2001, in ten of our 10 articles on 2001 and '02, we have used scientific forms, lavish rigor and style, and fake film. Since then, has not happened to us. ARandom House, in cooperation with Wikipedia, consists in role-plan pleepplaingg games and fantasy series literary novels with unique approach to artistic media, including "Jeressande" by Audiom les Hamerswein. See also edit Table of contents of articles for 2001-11 1 The intensive research effort of the Fabian Society and the Library and Museum of Great Britain on the RMA 1000 series includes more than nine dazzling articles, 50 "classics" and other pieces that garnered a mention from the Swedish press, astute academic, sergeant d' Hoskin's Service, an item of art on the rise for WMSU in the 2002 World FI World Games.[3] Ten more previous articles in February contributed to a ten-years archive by the L.B. Law Foundation of knowledge-urgent WB02 publication data. In 2005, Wercd et al. took the first step toward finishing "Moswart Huslas", which was published in the 1976 edition. In 2007, Munsry et al. repeated "White People's Rescue" also published events by Willie Miller and other Al Jazeera fresh and original voices. In 2003 paper in the sociology of journalism, David Blabas reporteds, "The so called 'white men's
Finetuned, Speculative Decoding (Algorithm
it learned to leave the visit." Read the folksy post-in full <sprec.<sep>>ls>></>>>>>> CSI: Drayton, Part I Sir Ken & Honi Pratt Expedition Expeditions, Ltd Research & Development Canada U.S. History and Culture 1,200 Games History Remfacile Rerail will provide nine series of digital direction, redesign and multi-distributtin services, leading original biographer and editor David Tatnow appears in the final stages of return service. "It's almost fast as walking, getting the basics off our shelves," Chief Creative Officer Tony Paschus said. "We envy Historic Stadium. Ottawa and the story we can tell with access to our digital resource. The behind-the-seeme year and were the costliest part of CSI's entire qualification process isn't the only aspect of our financial problem [yet]. Players will have access to the most advanced titles, and collectible characters of any franchise. But they will be lavished with additional digital content and new film releases, in things that will not cut through Rui's new, toned down, strong-are arm role-for example, when, in those spacesuits and used as rival generals, she have dramatic conversations with everyone on her team. (Each character continues to be introduced during missions.) UK flags are symbolic of support for this transition from digital play entertainment. Credit: Beaujorie Library Release date: 2015-01-25 Format: Remastered digital Download framerate: 57 kbps (a range 1.2 to 54 kbps) Thumbnail up to 1920x 1080 (16 in., notches at 40.1 Planting Function: BFI 16 00 Twelve Tens of catchests Producers: tenthreekorca-reporrs, gl.ca SUBSCRIBE TO Northeastern University's CSI: Drayton News Games Tour: 51 Place (including Canada), Alberta (including Canada) Huston University University College of Art and Design MacAskill University Waterloo (Note: Gamers 3 and under are welcome to sign up) See also: All About Willie Lewiscsep><<ls Posing for AC/DC's album's 's tour in October, D' Angelo Brakes of British Columbia proceeded with a classic 'DC" mixta
humblings cost the taxpayers more than $14 million last
32


Reviews:
[{"review_id": "qWH0DhpuOP", "reviewer": "Reviewer_yrNB", "summary": "This paper proposes to combine any-order models with speculative decoding. The method approaches any-order modeling by sampling a mask then using an encoder-decoder architecture to encode all previous tokens and produces remaining tokens autoregressively from left-to-right. The generation process is sped-up via self-spec-decoding, which uses parallel sampling (as in masked diffusion models) to draft and autoregressive mode to verify.\n\nEmpirical results on infilling in WikiText, coding, and ROCstories demonstrate a reasonable accuracy-speed tradeoff.", "questions": "The token comparison in Table 3 does not seem to be fair, as I believe those are MDLM pretraining tokens.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The method is reasonable, and exploring speculative decoding for infilling is novel. The writing was also easy to understand.", "weaknesses": "The main weakness is that the speedups are relatively small, and the acceptance length of 2.24 tokens per draft (Section 7.1) seems quite small. It's possible this is an artifact of model size. The evaluations are constrained by computational resources, so I will not let this affect my review too strongly. Additionally, the evaluations are a bit outdated as well. Again, this seems to be a limitation of computational resources.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sFqWctnKSn", "reviewer": "Reviewer_BarN", "summary": "The paper introduces a new algorithm called Any-Subset Speculative Decoding (ASSD) to significantly speed up text generation for a special class of models called Any-Subset Autoregressive Models (AS-ARMs). This method allows for parallel generation of multiple tokens at once without any loss in output quality, effectively solving a major bottleneck in generating text for tasks like infilling (filling in missing parts of a text).", "questions": "1. How do AO-ARMs compare with regular ARMs and do you think modeling the joint probability distribution in AO-ARMs facilitate the speculation intrinsically while compromising on model quality as compared to regular ARMs?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Lossless Generation: It is mathematically proven to produce text from the exact same distribution as slow, one-by-one sequential decoding. There is no quality degradation.\nNo Slowdown: The paper proves that the number of model calls will never exceed the number of tokens being generated. This is a critical guarantee that standard speculative decoding lacks, as a poor draft model can actually slow things down.\nFree Draft Model: Since the AS-ARM acts as its own drafter and verifier, there is no need to train or maintain a separate, smaller draft model, saving on resources.\nEnhanced Versatility: The method is designed for \"any-subset\" tasks, meaning it can handle an exponential number of infilling patterns (O(2^N)) compared to the linear number of prefix-based patterns (O(N)) handled by standard autoregressive models.", "weaknesses": "1. Scope beyond infilling use cases is not defined or compared.\n2. Detailed analysis of accuracy of these models is not mentioned.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ubDP1XVde8", "reviewer": "Reviewer_ummp", "summary": "This paper proposes ASSD, a speculative decoding method for any-subset autoregressive models (AS-ARMs). ASSD yields about a 10% speedup over standard AS-ARM sampling and, thanks to its sound joint probabilistic modeling, outperforms much larger discrete diffusion models on code generation and infilling benchmarks.", "questions": "1. On KV caching and causal masking: Lines 117-118 state that all prompt tokens attend to each other. If this is the case, KV caching should not be possible, as each new token would require recomputing previous activations due to bidirectional attention. Could the authors clarify how KV caching is handled in their experiments (appendix)? Is the prompt kept fixed during sampling, with new tokens using causal attention? Additionally, from line 213 (\"density estimation via causal-like masking\"), are the authors actually applying causal masking in the prompt?\n\n2. The results in Table 3 show that performance on infilling multiple sequences in ROCStories is better than infilling a single sequence, which is counter-intuitive. I would expect the single-sequence case to be easier. Do the authors have an explanation for this? Additionally, after fine-tuning, the XLNet model performs worse at single-sentence infilling. Could this be related to the proportion of prompt tokens during the fine-tuning or something similar? This seems important to clarify.\n\n3. Still, the speedup is much smaller than regular speculative decoding, do the authors have a sense of how to maybe push this further than 10% ?\n\n4. I understand that no loss is applied to the prompt tokens during fine-tuning. Is that correct?\n\n5. Have the authors considered training from scratch, compared to fine-tuning from an XLNet, even on some small synthetic datasets (if costs are prohibitive)? In any case, do you see some advantages or disadvantages of pre-training from scratch in terms of capabilities / inductive-biases of the final model?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The paper is very well written and easy to follow.\n\n2. Unlike speculative decoding with drafting for ARMs, ASSD is guaranteed to *not* increase the number of calls, and does not need a separate drafting model.\n\n3. Generation statistics for sequential decoding and ASSD are similar (Table 1), consistent with the theoretical guarantees (Lemma 1, Theorems 1 and 2).\n\n4. I appreciate that the authors found and applied a fix to the DiffuLlama evaluation code and chose to compare against DiffuLlama with this correction, making the baseline stronger, instead of ignoring it.", "weaknesses": "1. A direct comparison of the speed and generation perplexity of ASSD against standard AR generation (e.g. GPT-2 with and without KV caching) and masked diffusion models (e.g. MDLM) is missing. I expect ASSD to be slower, but reporting these metrics is important for a complete evaluation, and does *not* diminish the value of the contributions, even as the observed ~10% speedup is notably lower than the 2-3x gains commonly reported for speculative decoding with a drafting model (e.g. in [1]).\n\n\n2. Important details about hyperparameter choices are too often deferred to the appendix (e.g., lines 308-309 and 350-351). Since there is space available, a brief summary in the main text would be justified.\n\n3. The motivation for the chosen training hyperparameters in code generation (appendix E7) is unclear. In particular, why was masking rate warmup used? Were there experiments comparing with and without it? Including results or rationale for this decision would strengthen the work and guide future research. Currenly, the choice appears somewhat ad-hoc.\n\n4. Line 422: The \"k\" hyperparameter could be more clearly denoted as the number of speculated tokens.\n\n\n[1] Fast Inference from Transformers via Speculative Decoding, Leviathan et al.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)