Parallel Token Prediction for  Language Models
Authors: Felix Draxler
Justus Will
Farrin Marouf Sofian
Theofanis Karaletsos
Sameer Singh
Stephan Mandt
Venue: ICLR 2026 Poster (2026)
Keywords: transformer
autoregressive model
multi-token prediction
generative model
large language models
Primary area: foundation or frontier models, including LLMs
TL;DR: An LLM framework to predict multiple tokens with arbitrary dependencies in a single model call.

Abstract: Autoregressive decoding in language models is inherently slow, generating only one token per forward pass. We propose Parallel Token Prediction (PTP), a general-purpose framework for predicting multiple tokens in a single model call. PTP moves the source of randomness from post-hoc sampling to random input variables, making future tokens deterministic functions of those inputs and thus jointly predictable in a single forward pass. We prove that a single PTP call can represent arbitrary dependencies between tokens. PTP is trained by distilling an existing model or through inverse autoregressive training without a teacher. Experimentally, PTP achieves a 2.4$\times$ speedup on a diverse-task speculative decoding benchmark. We provide code and checkpoints at https://github.com/mandt-lab/ptp.

Published as a conference paper at ICLR 2026
PARALLEL TOKEN PREDICTION FOR LANGUAGE MODELS
Felix Draxler* Justus Will"
Farrin Marouf Sofian
Theofanis Karaletsos
Department of Computer Science, University of California, Irvine
Chan Chan-Zuckerberg Initiative & Pyramidal Al
Sameer Singh
Stephan Mandt
Department of Computer Science, University of California, Irvine
Computer Science & Al in Science Institute, University of California, Irvine
ABSTRACT
Autoregressive decoding in language models is inherently slow, generating only one token per forward pass. We propose Parallel Token Prediction (PTP), a general-purpost framework for predicting multiple tokens in a single model call. PTP moves the source of randomness from post-hoc sampling to ran- dom input variables, making future tokens deterministic functions of those in- puts and thus jointly predictable in a single forward pass. We prove that a single PTP call can represent arbitrary dependencies between tokens. PTP is trained by distilling an existing model or through inverse autoregressive training without teacher. Experimentally, PTP achieves 2.4x speedup diverse- task speculative decoding benchmark. We provide code and checkpoints at https: http:/ggtttuuttt thu..thu.....m com/mandt lab/ptp
INTRODUCTION
Autoregressive transformers Vaswani et al., 2017) are the foundation of today's large language models (LLMs) (Brown et al. 2020). Their sequential generation process, however, remains major forward pass of the model. This increases inference latency significantly compared to what a single transformer call would achieve.
bottleneck: predicting token requires
Many recent efforts aim bypass this bottleneck by predicting multiple tokens once. Broadly, they can be categorized into two lines of work: the first, speculative decoding, takes a systems approach, making predictions
lightweight model that is verified by large model (Leviathan et al. 2023 Chen et al.) 2023 Sun et al., 2023) Zhong et al.] (025).. The second line of work makes use of predicting several tokens independently of each other. This significantly reduces the search space for sequences and improves overall model quality (Qi et al. 2020; Gloeckle et al. 2024
ial
factorial[nm):: nue 1:
nur
return
35 5 steps
e1s6
else:
return factor nue 8
return nun
factorial
Autoregressive Model
Parallel Token Prediction (ours)
Our parallel model generates the same text as its teacher in a fraction of the steps. By the time our model (right) has generated entire function, its autoregressive teacher (left) is still busy with function header. Green tokens are generated during the time spent on PTP: gray tokens show the remaining steps autoregressive model would still need. Prompt: Write Python function that computes the factorial of a number.
Figure 1:
"Equal contribution. Corresponding authors: { (drraxler, jewi 11, ) edu

---

Published as a conference paper at ICLR 2026
DeepSeek-AL et al. 2025). Similarly, discrete diffusion iteratively refine generated sequences across several steps (Hoogeboom et al., 2021 Austin et al.) 2021). However, all of these methods still contain an irreducible sequential component to generate sequences.
Our work takes step towards filling this gap. We propose a framework that, in theory, can generate arbitrary length sequences in parallel. This is enabled by small but fundamental architectural change: instead of sampling from the distributions predicted by an autoregressive model a post- processing step, we feed the involved random variables an input the model: the model learns to sample. This enables it to anticipate which tokens will be sampled and to predict them jointly. Figure [ shows how this results significant reduction model calls to produce identical text. Similar frameworks have been formulated in the normalizing flow literature; Inverse Autoregressive Flows (Kingma al. (0166) generate samples of many continuous dimensions parallel and Free- form Flows (Draxler et al. 2024) distill a fast generator network. We transfer these concepts to sampling discrete sequences from continuous latent space.
Our contributions are as follows:
We propose Parallel Token Prediction (PTP), modeling approach for discrete data that generates multiple interdependent tokens in one model call (Section 2.1).
We prove that PTP is as expressive as autoregressive models (Theorems and 2
We propose Partial Quadratic Decoding, efficient error correction scheme that allows parallel verification of long generated sequences during sampling (Section 3)
Experimentally, we distill real-world natural language PTP models, achieving a speedup of 2.4x and 4.2 accepted tokens per speculative decoding step (Section 4
Together, our framework opens a design space to accurately predict several tokens in parallel, re- ducing latency in language model output without limiting representational power.
PARALLEL TOKEN PREDICTION
2.1 PARALLEL SAMPLING
To construct our Parallel Token Prediction framework, let us recap how a classical transformer decoder generates text, see Fig. 2 (left). This decoder iteratively predicts the categorical distribution of the next token 4 E (1,...,, based on all previous tokens Lęi = (4,,...,4i-),
P = P(I)(ee).
One then samples & P., before moving on to predicting the next token. The requirement know token t; to predict the next token f,+1 is the reason for the latency of autoregressive inference. For simplicity, we assume this distribution the final distribution that is used generate tokens, that it already reflects temperature scaling (Guo et al.) (017), top-k and top-p sampling (Holizman et al., 2020), or other approaches trading sample diversity with quality.
Neural Network
Network
Neural Network
fu
Autoregressive Model
One-Hot Parallel Token Prediction
Figure 2: Parallel Token Prediction predicts several tokens one model call. (Left) An autore- gressive model predicts a single i; per step. It predicts the distribution of each single next token, then the token is chosen with the help of an auxiliary random variable u, ~ U[00, 1] (see Fig. 3. (Right) One-Hot Parallel Token Prediction learns the sampling process by feeding the auxiliaries into the model. This makes predictions deterministic, allowing joint prediction of several tokens.
2

---

Published as a conference paper at ICLR 2026
To break the autoregressive nature, we need to understand how exactly tokens sampled from P.: One draws an auxiliary random variable u, ~ 24 1] and looks up the corresponding token from the inverse cumulative distribution function follows:
Pick(us, u4 P:)
min (j Fry uz), where Fij
Pa-
je(1.... v)
Figure 3 illlsstatees how this function jumps from to- ken to token function of u, Here, j iterates possible token choices, Pa is the probability to sam- ple 4 = 1, and Pej is the cumulative distribution sample a token 4 E (1....j).
Histogram
0 Picktu,
fros
froe
Import
leport
Note that while Eq. TT) defines distribution over possible next tokens, Eq, (2) is a deterministic rule once the auxiliary variable u, is drawn. Thus, write this rule as an explicit deterministic function:
def
dcf
Auxilaay
fp((csi u.) = Pick(u,, P(- t<))). (3)
Figure 3:
Sampling from a discrete distri- bution. Given histogram P. (left), compute the inverse cumulative distribution function (right) and look up the token at random lo- cation u, E 24(0, 1]. PTP learns the function the right instead of the histogram.
These auxiliary variables all we need to perform parallel generation of text: all information about which token t4 we are going to select is available to the model if it has access to U; as one of its in- puts. Since u; uniquely determines i, given leci, carries the same information as t; itself. The model can therefore use u, as a stand-in for & when predicting 1,+1, then u as a stand-in for fi+11 and so Feeding all auxiliary variables U....., Uy into the model once thus allows every future token to be predicted without any sequential dependency (formal proof in Appendix I)
Theorem 1. Let P denote probability distribution for next token prediction. Then, the future token tu can be selected as a deterministic function fr of previous tokens tes and auxiliary variables uk ~ 40, 1):
Ug. ,
tu fp(tesi u.....ug). Ug),
for all k 2 i.
(4)
Theorem T shows a clear path build model that can sample many tokens in parallel: instead of learning the distribution P(tx tzx), we propose to directly fit the function fP u....., uy)). which jointly predicts future tokens tg.
Figure 2 (right) visualizes how this path can be implemented with a standard transformer back- bone Vassaani al. 2017): alongside the previous tokens, simply feed random auxiliary variables ug..... uN ~ 27 into the model. Then predict P(tx | u....., ug)) for each future token in one model call. Per Theorem [ since ug. uniquely determines l, this prediction narrows down to a single token- a one-hot distribution: P(4x lex u.,...., uk.) = 1(;2 fr(ten "; ...., u.).). In practice, we take the argmax to extract that predicted token:
t = fp. (ter) U.....,, u²) argmax(P(( P(tx ttz Pei:u...... ........ ux))
(5)
We refer to this model as a One-Hot Parallel Token Prediction Model (O-PTP), O-PTPs can be trained to replicate an existing autoregressive model P, as we discuss in Section 2.2.1
O-PTPs converge to one-hot distributions, and therefore do not expose the underlying sampling dis- tributions of the model. While we gain access to the model's confidence, this prevents access to the original conditional probabilities P(, (<c)), which are required for training without a teacher, adjusting temperature (Holizman et al. 2020), and uncertainty quantification. To address this, we introduce Categorical Parallel Token Prediction (C-PTP), which recovers the full conditional dis- tribution of each token. The key idea is to predict each token ig while conditioning on all past auxiliary variables u,,..., 4 but explicitly excluding its own auxiliary variable ug. By The- orem these past auxiliaries deterministically encode the sampled history t<k- Withholding ug preserves the uncertainty over t2 rather than collapsing it to a point mass. As a result, conditioning 4 (lec Ui...., Uk-1) exactly recovers the original autoregressive conditional P(lx beck) (proof in Appendix(A.2
Let P denote probability distribution for next token prediction. Then, the distribution of a token by is fully determined by context tokens tzg and the past auxiliary variables ......; ug-17
Theorem 2.
P(1k Leis Uj.
1k-1) = P(tk t(ck), for all l 2 i.
(6)
3

---

Published as a conference paper at ICLR 2026
Note how Eq. (6) does not depend ug. Figure 4 shows how Theoremp can be used to predict the distribution of the tokens f,, tN in parallel. Just like for O-PTP, first sample all required auxiliary variables Uj. uN ~ ((0 1). and then predict all PR- P(tx l<ci, ui,... ... ug (g-1) in parallel. Sampling from these distributions is done just as in Eq. (3)
Neural Network
4
l = 15 Mp ( U,....,, us)
(7) By using causal decoder architecture, we can properly mask which token has access to which auxiliaries.
Pick(ux; P(tx tei U...... 2k
Categorical Parallel Token Prediction
Figure 4: Categorical Parallel To- ken Prediction. By withholding for the prediction of t.. C-PTP learns the same distribution as an autoregres- sive model (Theorem 2 The auxil- iary uy is used to sample token Ég.
C-PTP can be trained without teacher by iteratively solv- ing Eq. (6) for the auxiliary corresponding to given token tR. see Section 2.2.2 Like O-PTP, can also be dis- tilled from teacher.
Both One-Hot and Categorical Parallel Token Prediction al- low coordinated token prediction in single model call. By Theorems and 2 there are no fundamental restrictions as to how many tokens be jointly modeled apart from model capacity. In the next section, propose approaches for training from scratch (only C-PTP) and distilling an existing model (both O-PTP and C-PTP).
2.2 TRAINING
Before deriving the training paradigms for Parallel Token Prediction Models, let us quickly recall that autoregressive models are trained by minimizing the cross-entropy between samples from the training data t ~ P(t) and the model Po:
(()) = EE-P())
Σ log Polts tei)
(8)
Using a causal model such transformer Vaswani al.) 2017). this loss be evaluated on an entire sequence of tokens in single model call (Radford et al. 2018). We now present how to distill both One-Hot and Categorical Parallel Token Prediction Models from a trained autoregressive model. We then show how the categorical variant can be self-distileed from data alone via Eq. (8).
2.2.1 DISTILLATION
To build a PTP model Pa that speeds up inference of pretrained autoregressive model Qo. we propose to train it using distillation. Given enough model capacity, this results in fast model that generates identical text as the base model (Theorems) and 2). We defer efficiently correcting errors
arising from finite resources the subsequent Section 3
To train the student for a given training sequence = l,,..,,7: Lr, we reverse engineer the auxiliary variables u1...., ut under which the teacher would have generated the sequence. We then randomly split the sequence into context and prediction sequences, and evaluate a loss that leads the student towards the correct generation. This process is summarized in Algorithm in Appendix G.I
Auxiliary variables. We train the model based on auxiliary variables that the teacher model would use to generate the training sequence. To this end, we evaluate the teacher distributions of each training token to get the cumulative discrete distributions F,,..., Fr for each token. Inverting Eq. (2) for u we find for every token the sequence = 1,...., T)):
us E [Fi
Pick - ([tx),
Since uk is continuous, while t is discrete, we can randomly pick any compatible value. See Appendix E.3 for details.
Loss evaluation. Both O-PTP and C-PTP can be trained with standard losses given an input se- quence i, with uy extracted using Eq. (9) We randomly split the sequence an index i into context
4

---

Published as a conference paper at ICLR 2026
and prediction, and evaluate the following cross-entropy losses, choosing N the index of the last token in the completion part:
N
O-PTP: 2(0;1,0) =
Σ
log Po(4n bei Way..., 1k-1, Wk);
(10)
Aem
C-PTP: L(0;t,
Σ
log (t Wei........ uk-1).
(11)
C-PTP also be trained similar knowledge distillation (Hinton et al., 2015): to this end, explicitly match the student's Po. to the teacher distribution Qp.kk. This works for any loss d(Q, P) for the difference between categorical distributions, such as the Kullback-Leibler divergence d'
KL(Q || P) or its reverse variant d = KL(PIIQ):
C(O;L,))
Σ d(Qu(tu Po(t, bees (i....-))..
(12)
Ami
At their corresponding minima, the losses in Eqs. (10 to 12) are minimized by the optimal solutions
given Theorems M and 2
a one-hot distribution for O-PTP and the teacher distribution for C-PTP.
Training data. We can train the student using sequences from any data source, even if the teacher assigns different probabilities to them. As long as the teacher assigns non-zero probability and the dataset is sufficiently large, querying the teacher for the corresponding auxiliary variables provides full supervision and allows the student to match the teacher everywhere. We conceptually and empirically compare several training data sources (teacher samples, dataset sequences, student self- samples) in Appendix E.2 the teacher.
We find the lowest-variance option is to sample training sequences from
We now show how C-PTP can be trained on training data alone, without a teacher model available.
2.2.2 TRAINING FROM SCRATCH
Categorical Parallel Token Prediction Models can also trained directly via Eq. (8) avoiding the need to have teacher model as target. For a given training sequence 41,-..,44 br, we again split it into the context tæ and the following prediction 2z4-
Exactly for distillation, we have to find auxiliary variables that are compatible with every tg, k > We can do this by selecting, randomly, any us E [fe..te11 F. equivalently to Eq. (9). where FR. ly now is the cumulative probability under P& (instead of the teacher model) to choose l when predicting that token. As this probability depends on the previous auxiliary variables u, Ug-11 select them iteratively. Specifically, we can alternate between computing the logits of Po(ta bes; Ui,..., Uk 1), and drawing us using equation 9
Finally, train model using the cross-entropy loss
(()) E.-
Σ log (tt teis Ue...,, u.....,......))
(13)
Algorithm in Appendix G.1 summarizes the procedure. similar approach of iteratively deter- mining latent variables (our auxiliaries) was proposed by Inverse Autoregressive Flows (Kingmal al. 2016), although they considered continuous variables in an invertible neural network.
ERROR CORRECTION
Theorems I and 2 say that O-PTP and C-PTP can in principle predict coherent sequences of arbi- trary length in one model call. Practically, finite model capacity limits the length which a single transformer pass
produce coordinated text. In this section, we propose how to generate identical text to a base model (of arbitrary architecture and size) at minimum latency.
One successful strategy comes in the form of Speculative Decoding (Leviathan et al. 2023). A PTP model proposes several tokens in one model call. A subsequent call to the base model then veri- fies them. A proposed token lg is considered correct if it matches L Pick(ue, Q (4x (ck)),
5

---

Published as a conference paper at ICLR 2026
t Proposal:
IN
factorial
Verification
Proposal:
def
factorial
ignore
def
factorial
Parallel
def
factorial
In
ignore
Token
def factorial
Prediction
def factorial
num
3:
Yn
OUT
factorial
def
factorial
reject
Figure 5: Partial Quadratic Decoding performs parallel verification and proposing. Given a proposal of Npop tokens, the model evaluates multiple branches in parallel, each assuming a different number of proposed tokens correct. Green tokens are verified, red tokens mark the first mismatch, and blue tokens are new PTP proposals. Compute is allocated based on proposal confidence, and the branch with the longest verified prefix is selected.
the token the base model selects when the same auxiliary uk is applied to its own distribution. Additionally, since the base model already evaluates its distribution Qp(- tek) at every position during verification, one token can be sampled from for free at the first rejection, resulting in #accepted #correct + tokens (see Appendix C for formal definitions). We will find in Sec- tion 4.2. that PTP models outperform autoregressive models of the same size in terms of wall-clock speedup in vanilla speculative decoding since they only call the student model once.
Speculative decoding executes proposal and base models in sequence. To reduce this latency. we execute verification and drafting in parallel using Quadratic Decoding (Samragh al. 2025). Since verification has not yet completed when we begin the next drafting step, we do not know how many of the N Prop previously proposed tokens are correct. Rather than waiting, we prepare for all possible outcomes by constructing a branch for each m {O..... N Nprop J. each assuming exactly proposed tokens are correct and predicting continuation tokens conditioned on that assumption. All branches are computed concurrentyy with verification. Once the base model determines the true number m" of correct tokens, all other branches are discarded and generation continues from the precomputed continuation of branch m.. Parallelization can be achieved with separate hardware or in one merged call by leveraging Gated Low-Rank Adaption (Samragh et al. 2025) to let different model weights attend to different positions of the input.
While quadratic decoding works well for short proposal lengths, its quadratic cost in the number of predicted tokens becomes prohibitively expensive in case, where can predict larger num- ber of coherent tokens. We therefore propose Partial Quadratic Decoding, leveraging confidence estimates from the proposal model to significantly reduce the number of proposed tokens while retaining a high number of accepted tokens per step. Figure illustrates our proposed decoding scheme, details can be found Algorithm and Appendix F
The confidence estimates for Partial Quadratic Decoding come directly from the O-PTP model: for each proposed token tx. the model's output probability serves as confidence score ck. which closely tracks the base model's acceptance rate (see Fig. D in Appendix Fb.
Given context l<s and proposed tokens 44 for As 2 with confidences Gj for i, treat errors as independent and estimate branch probabilities as
{+m-1
P = m ~ (1 Ci+m)
II k=
Ck
for m < N Nprop-
(14)
Given a budget of C proposed tokens across all branches, we allocate B,, continuation tokens to branch by maximizing the expected utility
max Npopt1
Σ P(#correct = m H(B))
(15)
BE
E.,, =m=C
where (n) is monotonically increasing reward for predicting n tokens. We solve Eq. greedily, repeatedly assigning the next token to the branch with the highest marginal gain A,
15

---

Published as a conference paper at ICLR 2026
P(#correct = (H +1)-H(Bm)) We find (n)) = E P(#correct = j) works well empirically, with P(#correct estimated over the training dataset. When additional hardware allows sampling multiple independent proposal sequences in parallel, the acceptance rate can be improved further; see Appendix E.4
EXPERIMENTS
We empirically evaluate Parallel Token Prediction (PTP) by its wall-clock speedup compared to an autoregressive model. Since this is hardware-specifie, we also report the number of accepted tokens per speculative decoding step (#accepted in Section 3 as this is a hardware-inddpeendnt and method-agnostic metric.
We first show that C-PTP can be trained from data alone without access to a teacher model (Sec- tion 4.1). We then distill a 1. B.--paraeeeer model code generation and demonstrate that O-PTP achieves larger speedups than autoregressive speculative decoding by enabling the draft model to predict multiple tokens per call (Section 4.2.Ip. On the same task, we show that conditioning on auxiliary variables yields substantially more correct tokens than independently predicting multiple tokens (Section 4.2.2). Finally, on a speculative decoding benchmark spanning diverse language tasks, we finetune 7B 7B-parameter model and show that O-PTP outperforms competitive baselines, achieving wall-clock 2.4x speedup (Section 4.3)
Implementation We implement PTP by adding an embedding scheme for the continuous auxiliary tokens u to standard transformer:
embed(u) W binary(u) + b, where binary(u) E {0, 1)
(16)
Here, binary(u) maps the float 32 number into the 32 digits of binary float (Witten et al, 1987 The linear layer (W,b) maps this binary vector to the embedding space of the autoregressive transformer. For distilling a fast student model from teacher, we train O-PTP with cross-entropy (Eq. (IO)) sample training data from the teacher once and train multiple epochs on We identify these choices through an ablation and practical considerations in Appendix E
4.1 INVERSE AUTOREGRESSIVE TRAINING
We first confirm that C-PTP can be trained as a standalone generative model directly from data. We follow the procedure in Section 2.2.2 and train a model on a dataset that predicts se- quences of pick-up locations for taxis in New York City (NYC TLC, 2017).
Model
Perplexity (4)
C-PTP (Ours)
19.88
Autoregressive
19.81
Table 1: Categorical Parallel Token Prediction (C-PTP) can be successfully trained from data alone. The perplexity of C-PTP and an autore- gressive model on taxi pickup location sequences NYCTIC, 2017 are almost identical.
Table I shows that the resulting model matches the performance of an autoregressive baseline. Based on the evidence for multi-token predic- tion (Gloeckle et al., 2024; DeepSeek-AT et al.) 2025). we expect the quality of PTP models to outperform autoregressive baselines at larger scales. We leave training larger models from scratch to future work.
4.2 LIMITATIONS OF COMPETING FRAMEWORKS
Parallel Token Prediction overcomes important limitations in other frameworks to decrease latency: first, small surrogate models for speculative decoding do not leverage their parallel multi-token potential, which we unlock with our framework in Section[4.2.1 Second, modeling tokens indepen- dently produces incoherent sequences (Section 4.2.2). This also limits the representational power of discrete diffusion models (Hoogeboom et al. 202 A et al. 2021). We compare to speculative decoding and independent token prediction on CodeContests (Li et al., 2023), a dataset of coding ama challenges, by distilling Tinyll ama- 1t1.-11.1111111111111 1B-Chat-vl. (Zhang al. 2024
7

---

Published as a conference paper at ICLR 2026
4.0x
PTP draft (ours)
PTP draft PTP (finetuned)
3.5x
PTP (finetuned)
Accepted tokens
Speedup
3.0x
AR draft
2.5x
2.0x
1.5x
1.0x
2-
0.5x
4.2M
268M
1.18
66k
4.2M
268M 1.18
Number parameters
Number parameters
Figure 6: PTP allows parallelism draft models for speculative decoding, yielding larger speedups at equal model size. (Left) Wall-clock speedup relative to standard autoregressive decod- ing (Right) Average number of tokens accepted per step under teacher verification. Curves compare autoregressive draft models Parallel Token Prediction (PTP) drafts; (+) are models trained from scratch for a fixed number of epochs, and (*) is finetuned from the teacher. Across model sizes, PTP drafts achieve higher speedups by generating more than one correct token per step, whereas autoregressive drafts remain sequential.
4.2.1 SMALLER AUTOREGRESSIVE DECODERS
Vanilla speculative decoding distills small autoregressive draft models to imitate the teacher with fewer parameters (Leviathan al., 2023). Through their reduced size, these models require less compute to predict each token, and the teacher is used to verify them in parallel as in Section
Figure shows that PTP draft models outperform AR draft models because they need fewer student calls for the same number of tokens. This shifts the optimal model size towards larger models. In fact, find that the best performance is achieved by finetuning the teacher model directly. The latter may be an artifact of the smaller models being trained from scratch instead of being finetuned.
4.2.2 INDEPENDENT PREDICTION
Many approaches to predicting multiple tokens in parallel, including multi-token prediction (Qi et al.) Gloeckle et al., (024) and discrete diffusion models (Hoogeboom et al. 202T, Austin et al. 2021. assume that future tokens conditionally independent. As a result, later tokens are sampled from marginal distributions that average over incom- patible earlier choices, making semantic and syn- tactic inconsistencies unavoidable even with infinite model capacity. In code generation, this manifests as spurious token combinations such as def numpy and import find. See Appendix B for formal derivation.
Parallelization technique
#accepted (t)
2020
O-PTP (ours)
7.0 - 0.1
Independent prediction
6.2 = 0.1
Table 2: Speculative decoding with O-PTP accepts more tokens per step than inde- pendently predicting tokens. We distill draft model once with meaningful auxiliaries that determine token dependence and once with a generic [MASK] token. Error is dif- ference between two runs.
To isolate this effect, we compare models trained with informative auxiliary variables u, (our O-PTP) to models that have multiple independent heads for predicting multiple tokens (Qi et al. 2020
Gloeckle et al. 2024). As shown in Figure 7 O-PTP consistently produces meaningful token com- binations by coordinating predictions through the auxiliary variables, while independent prediction frequently produces incompatible pairs. Quantitatively, Table shows that auxiliary variables sub- stantially increase the number of tokens when used in sequential speculative decoding. Figure shows a qualitative sample of our model's predictions.
4.3 GENERAL-PURPOST TEXT GENERATION
In the following experiment, we confirm that we can finetune a large pretrained model using PTP, resulting in model that predicts several tokens per forward pass.
8

---

Published as a conference paper at ICLR 2026
Autoregressive Teacher
0-PTP (ours)
Optimal Independent
The Sestlrrr
0.5-
0.0
0.6
24
0.4 3.6
0.6
6.
0.
0.6 3.8
Figure 7: Parallel Token Prediction generates meaningful pairs of tokens. Each panel plots the first and second auxiliary variables (#1, W2) [0, 1J² on the axes; regions in this unit square correspond to the resulting two-token outputs (green: compatible, red: incompatible). (Left) In coding problem, autoregressive sampling generates meaningful token pairs, such as importing package or defining a method. (Center) Our O-PTP coding model produces sensible token pairs, but in single model call; only rarely (< 1%) does it yield spurious combinations like def sys. (Right) model that independently predicts future tokens is bound to fail: in about 60% of cases, combines incompatible tokens because the second token is not informed about the first.
We distill a One-Hot Parallel Token Prediction model (O-PTP) student from an autoregressive teacher, following the procedure in Section 2.2.1 We use Vicuna-7B (Chiang et al., 2023) as the teacher and finetune the student on ShareGPT conversational data (Chen et al. (024). Instead of finetuning the full model, we train a gated LoRA (Samragh et al., 2025) adapter, allowing us to make only the minimal changes that are necessary to correctly parse the auxiliary variables.
To reflect a practical deployment scenario for chat-oriented large language model, we evaluate the performance on SpecBench (Xia et al. 2024), containing a diverse set of tasks: multi-turn con- versation (MTC), translation (TL.,, summarization (SUM), question answering (QA), mathematical reasoning (Math), and retrieval-augmentte wall-clock speedup on a single NVIDIA RTX A6000. We also report the hardware-agee -agnosste average number of accepted tokens, this is highly correlated to the maximum speedup that can be obtained with efficient decoding schemes.
generation (RAG). Performance is measured using the
As shown in Table 3 O-PTP consistently predicts more tokens identical to its teacher than existing parallelization baselines across most tasks, resulting in a larger speedup. These results demonstrate that PTP is effective when scaled to large models and heterogeneous real- worldworld datasets. See Ap- pendix D for detailed results, and Appendix H for inference without error correction.
5 RELATED WORK
Speeding up the generation of autoregressive models and discrete sequence models in particular has been the focus of a broad body of work, see (Khoshnoodi et al. 2024) for an overview.
Our framework combines two ideas from the Normalizing Flow literature and imports them to mod- eling discrete data: Inverse Autoregressive Flows (IAF) are trained with fast prediction in mind (Kingma et al., 2016) by iteratively identifying latent variables (our auxiliary variables) that gener- ate a particular continuous one-dimensional value, and Free-Form Flows (FFF) train a single-step generating function as the inverse of encoder (Draxler et al. 2024).
In the LLM literature, speeding up generation has been approached from various angles. Specu- lative decoding takes system perspective, using small draft model to propose multiple tokens and a large target model to verify them (Leviathan et al., 2023 Chen et al. (023). Variants verify entire sequences (Sun et al. (023) or smaller verifier network Zhong et al. (2025) improve quality and speed. Latent variable methods first sample latent codes from the prompt so that the
distribution of subsequent tokens factorizes given latent codes (Gu al. 2018 Ma et al. 2019

---

Published as a conference paper at ICLR 2026
Parallelization
MTC
TL
SUM
QA
Math
RAG
Task Average
#accepted (1)
O-PTP (ours)
2.77
2.12
2.47
1.91
2.79
2.32
2.40 = 0.007
4.21 = 0.011
SAMD Hu et al. 2025
2.79
1.87
2.47
2.05
2.60
2.22
2.34 = 0.012
3,95 - 0.011
Eagle-2
Li et al. 2024a
2.76
1.88
2.19
2.08
2.75
2.06
2.29 = 0.010
3.93 - 0.008
Hydra Ankner al. 2124
2.59
1.98
1.91
2,16
2.56
1.86
2.18 1 0.007
3.40 + 0.005
Eagle ef.
20244
2.50
1.80
2.10
1.93
2.49
1.94
0.11
3.24 0.006
Recyclig (Luo cl 2025
2.08
1.77
1.93
1.83
2.21
1.73
1.93 0.007
2.38 = 0.003
Medusa (Cat
2024
2.08
1.69
1.57
1.76
2.07
1.55
1.79 0.004
1.93 0.003
PLD Saxena (2025
1.65
1.08
1.62
1.63
0.004
2.70 + 0.006
SpS CCen 2023
1.57
1.12
1.39
1.40
1.55
1.43
0.006
= 0.005
Lookahead (Fu 20124
1.46
1.13
1.26
1.24
1.55
1.17
1.30 0.005
1.57 + 0.003
None
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
Table 3: Wall-clock speedups under teacher verification (1). We compare One-Hot Parallel
Token Prediction (O-PTP) distilled from Vicuna-7B (Chiang ct al. 2023) to concurrent paralleliza- tion techniques on the diverse tasks of SpecBench (Xia al. 2024 see main text for abbreviations. Errors indicate the standard error over three runs. Temperature is set to 0.7. Across most tasks, O- PTP achieves the largest speed-up. We also report #accepted (1), the number of accepted tokens per base model call. Here, the errors are standard deviations over all steps across three runs. 'aarallll verification trades number of accepted tokens for total speedup so we report sequential verification here. For more detailed results see Appendix D
Discrete Diffusion Models leave autoregressive sampling behind by iteratively refining the text
starting from noisy masked variant (Hoogeboom et al. 2021 Austin et al. 202I). Multi-head output models predict several next tokens independent of each other (Qi et al. 2020 Gloeckle et al. 2024 DeepSeek-AL et al., 2025), narrowing down the possible set of next tokens. Both discrete diffusion and multi-head models assume independence of tokens, which is fundamentally limited in modeling capacity, compare Section and Feng et al. (2025). Recent work discrete diffusion models leverages copula models to capture dependencies in an additional model (Liu et al. 2025). We model dependencies between several tokens without an additional model at inference time.
In contrast to the above, our framework is universal in the sense that it can approximate arbitrary dependence between several tokens in a single model call. Our new method is complementary to existing approaches, and we leave exploring these combinations open for future research.
CONCLUSION
In this paper, introduce Parallel Token Prediction, framework that permits consistent generation of several tokens in a single autoregressive model call. It eliminates the independence assumptions that limited prior approaches, allowing one model multiple tokens with arbitrary dependency between them. Empirically, we show that existing models can be distilled into efficient parallel samplers. With error correction, these models produce identical output as a base model while sig- nificantly increasing how many tokens are obtained model call.
This speedup makes language models more practical for real-time applications. Future work in- cludes extending our framework to large scale models, multimodal generation, combining it with complementary accelcration strategies, and exploring thcorctical limits on parallelization.
Overall, our results suggest that the sequential bottleneck in autoregressive transformers is not inher- ent, and that universal, efficient parallel generation within reach. We think that our experiments in Section Jonly scratch the surface of the possibilities enabled by our Theorems [and We envi- sion future work to train large models from scratch that think in long sequences, and speculate that planning over longer spans may improve downstream performance.
Limitations
The experiments in the present paper focus on distilling existing models, leaving open the question whether model quality will improve when training from scratch (as is observed for multi-token prediction (Gloeckle et al. ))244). The inference speedup only manifests given heavily parallelized hardware, where increasing the number predicted tokens only marginally increases computational cost.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
Justus Will acknowledges support from the HPI Research Center in Machine Learning and Data Science at UC Irvine. Sameer Singh acknowledges funding from the National Science Founda- tion (NSF) through an NSF CAREER award IIS 20460444 Stephan Mandt acknowledges funding from the National Science Foundation (NSF) through an NSF CAREER Award IIS-2047418, IIS- 2007719, the NSF LEAP Center, and the Hasso Plattner Research Center at UCI, and the Chan Zuckerberg Initiative.
ETHICS STATEMENT
Our work focuses on reducing the inference time of Large Language Models, enabling more com- putations per unit time and supporting large-scale real-time applications. While this can improve responsiveness and resource efficiency, it may also increase the potential for misuse, such as gen- erating misinformation or automated spam higher volumes. Faster inference does not mitigate underlying model biases, so responsible deployment, monitoring, and safeguards are critical to bal- ance performance gains with societal risks.
REPRODUCIBILITY STATEMENT
We include proofs for all theoretical results introduced the main text Appendix A We include further experimental and implementation details (including model architectures and other hyperparameter choices) in Appendix and Appendix G Our code available at https://github.
http://tthuu..cccmcccoeeeeee com/mandt-lab//e
USAGE OF LARGE LANGUAGE MODELS (LLMS)
We used Large Language Models improve the writing of this paper, by pasting text and asking for feedback and spell checking; we also used them for code editing, particular bug resolving. We also used LLMs for discovering related work. All ideas presented in this paper are our own.
REFERENCES
Zachary Ankner, Rishab Parthasarathy, Aniruddha Nrusimha, Christopher Rinard, Jonathan Ragan- Kelley, and William Brandon. Hydra: eequentially-dependee draft heads for medusa decoding. arXiv preprint arXhv:202.00510, 2024.
Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne Van Den Berg. Structured denoising diffusion models in discrete state-spaces. Advances in neural information processing systems, 34:17981- 11777981-777 11993, 2021.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert-Voss Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCan- dlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learn- In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in neural information processing systems, volume 33, pp. 1877-1901. Curran Associates, Inc., 2020. URL bttps://prooeeddnsss.eeeee/ File htlp://8fccee0000077010066666660
http:/ppppraaaaaapapapappeeea files/paper/2020/
Tianle Cai, Yuhong Li. Zhengyang Geng, Hongwu Peng, Jason D Lee, Deming Chen, and Tri Dao. Medusa: Simple Ilm inference acceleration framework with multiple decoding heads. arXiv preprint arXiv:2401. 10774, 2024.

---

Published as a conference paper at ICLR 2026
Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean- -aptiise Lespiau, Laurent Sifre, and John Jumper. Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.01318 2023.
Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. In European Confer- ence on Computer Vision, pp. 370-387. Springer, 2024.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph Gonzalez, lon Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL https:
03 30 vicuna/
DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Cheng- gang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongje Ji, Erhang Li, Fangyun Lin, Fucong Dai. Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang. Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J. I.. Cai, Jian Liang, Jianzhong Guo, Jiaqi Ni, Jiashi Li. Jiawei Wang, Jin Chen, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, Junxiao Song, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Lei Xu, Leyi Xia, Liang Zhao, Litong Wang. Liyue Zhang, Meng Li, Miaojun Wang, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Mingming Li, Ning Tian, Panpan Huang. Peiyi Wang. Peng Zhang, Qiancheng Wang. Qihao Zhu, Qinyu Chen, Qiushi Du, R. J. Chen, R. L. Jin, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, Runxin Xu, Ruoyu Zhang, Ruyi Chen, S. S. Li, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shaoqing Wu, Shengfeng Ye, Shengfeng Ye, Shirong Ma, Shiyu Wang, Shuang Zhou, Shuiping Yu, Shunfeng Zhou, Shut- ing Pan, T. Wang, Tao Yun, Tian Pei. Tianyu Sun, W. L. Xiao, Wangding Zeng, Wanjia Zhao, Wei An, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, X. Q. Li, Xiangyue Jin, Xianzu Wang, Xiao Bi, Xiaodong Liu, Xiaohan Wang, Xiaojin Shen, Xiaokang Chen, Xi- aokang Zhang, Xiaosha Chen, Xiaotao Nie, Xiaowen Sun, Xiaoxiang Wang. Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xingkai Yu, Xinnan Song, Xinxia Shan, Xinyi Zhou, Xinyu Yang. Xinyuan Li, Xuecheng Su, Xuheng Lin, Y. K. Li, Y. Q. Wang. Y. X. Wei, Y. X. Zhu, Yang Zhang, Yanhong Xu, Yanhong Xu, Yanping Huang, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Li, Yaohui Wang. Yi Yu, Yi Zheng, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Ying Tang, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma Yiyuan Liu, Yongqiang Guo, Yu Wu, Yuan Ou, Yuchen Zhu, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yukun Zha Yunfan Xiong, Yunxian Ma. Yuting Yan, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Z. F. Wu, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhen Huang, Zhen Zhang. Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhibin Gou, Zhicheng Ma, Zhigang Yan, Zhihong Shao, Zhipeng Xu, Zhiyu Wu, Zhongyu Zhang, Zhuoshu Li, Zihui Gu, Zijia Zhu, Zijum Liu, Zilin Li, Ziwei Xie, Ziyang Song. Ziyi Gao, and Zizheng Pan. DeepSeek- V3 Technical Report, February 2025. URL http://arx: or. abs 2412 19437
Felix Draxler, Peter Sorrenson, Lea Zimmermann, Armand Rousselot, and Ullrich Köthe. Free- form Flows: Make Any Architecture a Normalizing Flow. In Artificial Intelligence and Statistics, 2024.
Felix Draxler, Yang Meng, Kai Nelson, Lukas Laskowski, Yibo Yang, Theofanis Karaletsos, and Stephan Mandt. Transformers for mixed-type event sequences. In The Thirty-ninth Annual Conference on Newral Information Processing Systems, 2025. URL.tttsss /tttps:///penreviee tttt/forrumidddMtto
2hf
William Falcon and The PyTorch Lightning team. PyTorch lightning, March 2019.
Guhao Feng, Yihan Geng, Jian Guan, Wei Wu. Liwei Wang, and Di He. Theoretical benefit and limi- tation of diffusion language model. The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. URL https: httpps://operree http////eeeevvveeeeee/ ew net t
Yichao Fu, Peter Bailis, Ion Stoica, and Hao Zhang. Break the sequential dependency of Ilm infer- ence using lookahead decoding. arXiv preprint arXiv:2402.02005 2024.
12

---

Published as a conference paper at ICLR 2026
Fabian Gloeckle, Badr Youbi Idrissi, Baptiste Rozière, David Lopez-Paz, and Gabriel Synnaeve. Better & faster large language models via multi-token prediction. Proceedings of the s1st international conference machine learning, pp. 15706-15734 2024.
Jiatao Gu, James Bradbury, Caiming Xiong, Victor O. Li, and Richard Socher. Non-autoregressive
neural machine translation. In International conference on learning representations, 2018. URL
net/ worumoideeoooo............. www.mmmmd-BB1Btt......
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International conference on machine learning, pp. 1321-1330. PMLR, 2017.
Charles R. Harris, K. Jarrod Millman, Stéfan J. van der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel J. Smith, Robert Kern, Matti Picus, Stephan Hoyer, Marten H. van Kerkwijk, Matthew Brett, Allan Haldane, Jaime Fernández del Río, Mark Wiebe, Pearu Peterson, Pierre Marchan,,,, Kevin Sheppard, Tyler Reddy, Warren Weckesser, Hameer Abbasi, Christoph Gohlke, and Travis E. Oliphant. Array program- ming with NumPy. Nature, 585(7825):357-366 2020.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint arXiv:1503.0253: 2015.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. International conference on learning representations, 2020.
Emiel Hoogeboom, Didrik Nielsen, Priyank Jaini, Patrick Forré, and Max Welling, Argmax flows and multinomial diffusion: Learning categorical distributions. In A. Beygelzimer, Y. Dauphin, P. Liang. and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL httptttsts///eeee.eeeeeee
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022.
Yuxuan Hu, Ke Wang, Xiaokang Zhang, Fanjin Zhang, Cuiping Li, Hong Chen, and Jing Zhang. Sam decoding: Speculative decoding via suffix automaton. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 12187- 12204, 2025.
J. D. Hunter. Matplotlib: A 2D graphics environment. Computing in Science & Engineering, 9(3): 90-95, 2007.
Mahsa Khoshnoodi, Vinija Jain, Mingye Gao, Malavika Srikanth, and Aman Chadha. A Compre- hensive Survey of Accelerated Generation Techniques in Large Language Models, May 2024. URL
22055/2255... 13019 arXiv:2405.13015 [cs].
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. International Conference on Learning Representations (ICLR), 2015.
Durk P Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Ilya Sutskever, and Max Welling. Improved variational inference with inverse autoregressive flow. D. Lee, M. Sugiyama, U. Luxburg. I. Guyon, and R. Garnett (eds.). Advances in neural in- formation processing systems, volume 29. Curran Associates, Inc., 2016. URL httppppprr.lleepppppr//////////1
https: ptttps:/prroceedings.o neurips ddeebdeefabbreeeeaaalleee -appedeeee----appp.... -apeeaapppppp
Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative
decoding. In Proceedings of the 40th international conference on machine learning. volume 202 of Proceedings of machine learning research, pp. 19274-19286. PMLR, July 2023. URL htttps:/pprocest wtw..ggg......ss......... wtt..ppeess/2..... www.eeeeaaaaaaaaaaaaaaa htmll
R Li, LB Allal, Y Zi. N Muennighoff, D Kocetkov, C Mou, M Marone, C Akiki, J Li, J Chim, and others. StarCoder: May the source be with you! Transactions on machine learning research, 2023. Publisher: OpenReview.
13

---

Published as a conference paper at ICLR 2026
Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. Eagle-2: Faster inference of language models with dynamic draft trees. arXiv preprint arXiv:2406. 2024a.
Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. Eagle: Speculative sampling requires rethinking feature uncertainty. arXiv preprint urXiv:2401. 2024b.
Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cy- prien de Masson d' Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James Molloy, Daniel J. Mankowitz, Esme Sutherland Rob- son, Pushmeet Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals. Competition- level code generation with AlphaCode. Science, 37866624) 2022. doi: 10.1126/ 1100..1126///0 1ttp://o1.111////ceeeeeees
science.abql URL https:: wwwwwwwww.. www.wscieaee www.nceieencccc.ccc..oo wwt.:o//g/oooooooo abql158
Anji Liu, Oliver Broadrick, Mathias Niepert, and Guy Van den Broeck. Discrete copula diffusion.
In The Thirteenth International Conference on Learning Representations, 2025. URL https:
//openzeview.co net. http://ormmmmdd==/pooo
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations (ICLR), 2019.
Xianzhen Luo, Yixuan Wang. Qingfu Zhu, Zhiming Zhang, Xuanyu Zhang, Qing Yang, and Dongliang Xu. Turning trash into treasure: Accelcrating inference of large language models with token recycling. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 6816-6831, 2025.
Xuezhe Ma, Chunting Zhou, Xian Li, Graham Neubig, and Eduard Hovy. FlowSeq: Non- autoregressive conditional sequence generation with generative flow. In Proceedings of the 2019 conference on empirical methods in natural language processing, Hong Kong, November 2019.
Wes McKinney. Data Structures for Statistical Computing in Python. In Stéfan van der Walt and Jarrod Millman (eds.), 9th Python in Science Conference, 2010.
New York City Taxi and Limousine Commission. 2016 yellow taxi trip data, 2017. City of New York, OpenData portal.
Aaron Oord, Yazhe Li, Igor Babuschkin, Karen Simonyan, Oriol Vinyals, Koray Kavukcuoglu, George Driessche, Edward Lockhart, Luis Cobo, Florian Stimberg, et al. Parallel wavenet: Fast high-fidelity speech synthesis. In International conference on machine learning, pp. 3918-3926. PMLR, 2018
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. In Advances in Neural Information Processing Systems, 2019.
Weizhen Qi, Yu Yan, Yeyun Gong, Dayiheng Liu, Nan Duan, Jiusheng Chen, Ruofei Zhang, and Ming Zhou. ProphetNet: Predicting future n-gram for eqqueeneeo-Ssqqueenetttrrtrnnnn Trevor Cohn, Yulan He, and Yang Liu (eds.), Findings of the association for computational lin- guistics: EMNLP 2020, pp. 2401-24100 Online, November 2020. Association for Computational Linguistics. doi: 18658v/3/v 1/2020. 1/2020.findings.. findinng-eenll.2111. URL
In
prg/2020. inddiii ngs emn p 2177]
Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, al. Improving language under- standing by generative pre-training, OpenAl, 2018.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAl Technical Report, 2019.
Mohammad Samragh, Arnav Kundu, David Harrison, Kumari Nishu, Devang Naik. Minsik Cho, and Mehrdad Farajtabar. Your Ilm knows the future: Uncovering its multi-toke token prediction potential. arXiv preprint 110. 2025.
14

---

Published as a conference paper at ICLR 2026
Apoorv Saxena. Prompt lookup decoding, November 2023. URL https://github. com/ antoo///aann/rrrrnt--ooo----eeee
Ziteng Sun, Ananda Theertha Suresh, Jac Hun Ro, Ahmad Beirami, Himanshu Jain, and Fe- lix Yu. SpecTr: Fast speculative decoding via optimal transport. In A. Oh, T. Nau- mann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in neu- ral information processing systems, volume 36, pp. 30222-30242. Curran Associates, Inc., newripss............p.......... cc cc/paper flles/paper/2023/
2023. URL
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Daniel Bikel, Lukas Blecher, Nikolay Bogoychev, William Brannon, Anthony Brohan, Humberto Caballero, Andy Chadwick, Jenny Lee, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprini arXiv:2307.09288, 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural informa- fion processing systems, 30, 2017.
Ian H. Witten, Radford M. Neal, and John G. Cleary. Arithmetic Coding for Data Compression, volume 30. Communications of the ACM, 1987.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, al. Huggingface" transformers: State-of-the-ar natural language processing. arXiv preprint arXiv:1910.0377 2019.
Heming Xia, Zhe Yang, Qingxiu Dong, Peiyi Wang, Yongqi Li, Tao Ge, Tianyu Liu, Wenjie Li, and Zhifang Sui. Unlocking efficiency in large language model inference: comprehensive survey of speculative decoding. arXiv preprint arXiv:2401. 20.0011,11,,, 2024.
Siqiao Xue, Xiaoming Shi, Zhixuan Chu, Yan Wang, Hongyan Hao, Fan Zhou, Caigao Jiang, Chen Pan, James Y. Zhang, Qingsong Wen, Jun Zhou, and Hongyuan Mei. EasyTPP: Towards open benchmarking temporal point processes. In International conference learning representations
(ICLR), 2024.
absp//bb//233/ 08097
Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. TinyLlama: An open-source small language model, 2024, arXiv: 2401.02385 [cs.CL].
Meiyu Zhong, Noel Teku, and Ravi Tandon. Speeding up speculative decoding via sequential ap- proximate verification. In ES- Mo III: 3rd workshop on efficient systems for foundation models, 2025. URL htttps://oeenrviieee
net wwtp:/eeeeee//aaannnneeeeaoaaoee
15

---

Published as a conference paper at ICLR 2026
A PROOFS
A.1 PROOF OF THEOREM 1
Proof. We prove by induction over k.
For = i, the function is given by Eq. 43)
assume the statement holds for i,..., This gives us access all f1...., 4-11 Ue (by induction assumption). (from the context), and all ...... le using , ,
For &
Then, we can compute Pk+1 = P( beis li .,,,.,,,)).........).. and pick the next token deterministically via Eq. 2
fx+1 Pick(ux+1i Pk+1) = fp 4-11 Ua
ugy Uk uk+1).
(17)
A.2 PROOF OF THEOREM 2
Proof. We prove by induction over k, i.
For = i, there is nothing to show, since there are no auxiliaries involved in the statement.
the distribution Pj: P(tx t<x)== = P(t t<c ug.... -x-1) of the token fy. Since token t is uniquely determined from P& and uy via Eq. (3). any distribution conditioning on P&. ty can instead condition Pk. uy via the law of total probability, so that P(t Aei, us..... ux) = P(t++1 &ck, fx).
For l:
1, assume the statement holds for k. This gives
B FORMAL LIMITATIONS OF INDEPENDENT PREDICTION
Without the auxiliary variables, the distribution of all tokens k > i+ is not informed about the choice we make for tokens 4441,... 1g-1- Instead, they are marginalized out, which explains the spurious code snippets:
Independent: P(ts tes)
Σ
P(tx P(t.. tk tee) * P(tx ((z).
(18)
Dependent (ours): P(ts
tess Uiy
1k-1)
thm. P(t
tek))
(19)
This limits how many tokens can be sampled in parallel without auxiliary variables, even for infinite model capacity. Our Theorems only restriction.
[ and allow for coordinating tokens, making model capacity the
C AVERAGE NUMBER OF CORRECT TOKENS
To quantify the effectiveness of parallel token generation, we measure the number of correctly pre- dicted tokens. Given a student output sequence (() (8) (S) and a teacher output sequence
.t2 ,...,
(TT) of the same length, this metric is defined as (T) (T) ,...,
#correct = max {
= (T) visk].
(20)
That is, #correct equals the length of the longest prefix of the student's output that exactly matches the teacher's output. We further define #accepted the number of tokens accepted per base model call. Often, for example with parallel verification, we have #correct = #accepted. Note, however, that with sequential error correction we obtain
#accepted
#corrett + 1,7'.
(21)
Intuitively, this measures how many tokens the student generate in parallel before the first dis- agreement, and directly corresponds to the number of tokens that can be safely accepted without enror correction.
16

---

Published as a conference paper at ICLR 2026
D DETAILED COMPARISON ON SPECBENCH
We report more extensive results from our experiment in Section 4 See Table for average number
of accepted tokens and Table 4 for wall-clock speed-ups.
Parallelization
MTC
TL
SUM
QA
Math
RAG
Task-Average
0.PTP (ours)
2.77
2.12 0.027
2.47 0.019
1.91
0.019
2.79 0.011
0.021
SAMD
[Tu al
2025
2.79 1: 0.026
1.87 1: 0.016
2.47 1 0,043
2.05
:: 0.007
2.60 1: 0.018
2.22
0.056
2.34
0.012
Eagle-2
202241
2.76 0.049
1.88 + 0.008
2.19 0.014
2.08
00015
0.036
2.06
0.024
2.29
0.010
Hydra
Ankner al 2024
2.59 1: 0.022
1.98 1: 0.006
1.91 +: 0.010
2.16 0.011
2.56 0.033
1,86 0.030
2.18
0.007
Eagle
[L
20243
2.50 1: 0.057
1,80 1: 0.009
2.10 1: 0.005
1,93
0.028
2.49 1.011
1.44 0.017
2.13
0.010
Recycliny
M225
0.045
1.93 + 0.004
1.83
0.007
2.21 0.010
0,013
Medusa
Cii ar 2023
2.08 1: 0.011
1.89 1. 0.008
1.57 1 0,002
1,76 4 0.019
2.07 1. 0.008
1.55 1 0.015
1.79 D.004
(Sakers
0.006
2.50 1: 0.004
1,20 0.011
0.005
SpS
ç PU28
1.57 =: 0.018
1,12 + 0.007
1.54 + 0.006
1.39
= 0.008
1.40 + 0.021
1.55 = 0.023
1.43 0.006
Lookateau
2024
1.46 0,021
1.13 1 0.024
1.26 1 0.005
1.24 0.010
1.55 0.003
1.17 0.007
1.30 0.005
Nonc
1.00
1,00
1.00
1,00
1,00
1.00
1.00
Table 4: Wall-clock speedups under teacher verification (t). Errors indicate the standard error over three runs. Temperature set to 0.7. Across most tasks, O-PTP achieves the largest speed-up.
Parillelizatinn
MTC
T1.
SUM
QA
Math
RAG
Task-Averige
O.PTP, sequentaa (ouns)
0.019
0.041
4.53 0.027
3.42
4.62
O-PTP, parallel (ours)
3.00 1: 0.015
2.59 1: 0.037
3.12 1 0,024
2.23
0.017
3.44 1: 0.027
2.93 1 0.032
2,89
0.009
SAMD Hu
2025
0.032
0.036
3,44
0.028
4.37 0.024
3,86
Eagle-2
Te al 20244
4,44 +- 0.011
3,20 + 0.031
3.79 + 0.017
3.40
0.019
4.62
0.021
3.89
0.024
3,93
0.008
1ydra Ankner
2024
3.90 0.006
2.88 1: 0.023
2.95 1 0.010
3,21
0.013
3.90
0.009
3.37 0,014
3.40
0.005
Eagle
Li 20210
3.63 0.008
2.72 + 0.024
3.17 1: 0,013
288
= 0.15
3.76 = 0.014
3.14:00011
3,24
0.006
Recychng [100 2t 2025
2.74 0.007
2.46 1. 0.029
2.68 1. 0.013
2.60 1 0.014
3.06 1 0.014
2.60 0.017
2.70
0.006
Medusa Cär
0.005
2.12
2.24
2.67
2.26
SpS (Cenn oF 20:5
2.10 = 0.006
1,40 + 0.009
2.15 + 0.011
1.79 = 0.008
1.89 + 0.008
2.17 0.014
1.93 0.003
PLD (Susens
1.67 1: 0 0.007
1,10 1: 0.007
2.73 1: 0.024
1.37 0.010
1,80 0.012
1.71 0,016
1.73
U. 0.005
Lookaixau
Fu al. 2024
1.69 0.004
1.24 0.010
1.54 + 0.006
1.56 = 0.007
1,92 0.008
1.42 0.008
1.57 0.003
None
1,00
1.00
1.00
1.00
1.00
1.00
1.00
Table 5: Average number of accepted tokens under teacher verification (1). For each method and task, we list the average length of the longest prefix matching the teacher's ' output, which directly determines decoding speedup. Errors indicate the standard etror over total number of model calls. Temperature is set to 0.7. Across nearly all tasks, O-PTP achieves the highest number of accepted tokens. Parallel verification trades number of accepted tokens for total speedup, so report both for our model.
E ADDITIONAL ABLATION RESULTS
Based on a dataset (NYC TLC. 2017) that contains latitudes and longitudes for pick-up locations for all taxi rides in 2016, we divide the city into 25 neighborhoods via k-Means clustering to obtain a discrete-valued time-series that we can split into overlapping chunks of length N. This is a com- mon benchmark dataset in the literature of marked temporal point processes (Xue et al. 2024). and autoregressive transformers are a common architecture (Draxler et al. 2025)
We now provide some of the specific choices we made when implementing the general framework of Parallel Token Prediction. Specifically, we discuss the empirical difference between O-PTP and C-PTP and which specific loss to choose. We will specify our model architecture and how to embed both tokens and auxiliary variables in the same embedding space, and lastly compare the proposal distributions our training sequences can be sampled from.
We test our framework by training model that predicts pick-up locations for taxis in New York City. Based on a dataset (NYC NLC, TLC, that contains latitudes and longitudes for pick-up locations for all taxi rides in 2016, we divide the city into 25 neighborhoods via k-Means clustering to obtain discrete-value time-series that we can split into overlapping chunks of length N. This is a common benchmark dataset in the literature of marked temporal point processes (Xue et al. 2024).
As a teacher model, we pretrain a 99M-paraeeter autoregressive causal transformer based on the architecture of GPT-2 (Radford et al. 2019), using the cross-entropy loss in Eq. (8) For our PLM
17

---

Published as a conference paper at ICLR 2026
choose the same GPT-style transformer architecture as the teacher. This allows us to the teacher's s parameters as a warm-start. We evaluate all our parallel models in terms of the average number of leading tokens predicted by student model that are identical to the teacher. In the end, this is the quantity that limits the maximum latency reduction that can be achieved, see Section 3
E AUXILIARY VARIABLE EMBEDDINGS
In our experiments we use transformers that embed tokens into a higher-dimensional embedding space via a learned embedding before adding a positional embedding. This doesn't work out-of-the- box for our auxiliary variables since they are one-dimensional continuous variables. Thus we learn separate embedding. We combine two components, for each of which we test several variants: (I) A learned affine linear transform [lin] fully connected neural network [NN]. (2) Feed either the ej = 1fu u I j/n] [th], or an n-dimensional
scalar u [fl], an n-dimensional threshold-embeddeee
embedding e, 1(u2) mod 1 < 3.5} [ar] inspired by arithmetic coding (Witten et al. 1987).
Empirically, all methods work reasonably well, but structured embedding leads to faster and more stable training convergence. This is similar to the transformer's positional embedding where both learned and fixed embeddings work well but the latter is preferred in practice (Vaswani al., (017). For further experiments we use the [ar + lin] embedding. Table shows the detailed effect of different embedding strategies.
E.2 DISTILLATION LOSSES AND PROPOSAL. DISTRIBUTIONS
Equations (T0p to [2) leave freedom as to how to choose the source of training sequences. this section, we discuss and empirically compare several options.
our goal is deploy our parallelized student as a drop-in replacement of our teacher model, the lowest-variance option is to sample training sequences from the teacher. Another possibility is to directly sample training sequences from dataset, such as the one that was used to train the teacher model in the first place. This has the advantage that can compute the teacher predictions (tx t<*) in parallel over full sequence instead of iteratively having to generate it. Finally, we can sample sequences directly from the student model by first sampling auxiliary variables ui..-.. uT [0, 1] and then using our student model in its current state to sample training sequences in parallel. As the student's prediction gets closer to that of the teacher during training, this approaches the same training sequence distribution as if we had sampled the teacher directly.
Conceptually, the latter is similar the techniques used in distilling the autoregressive WaveNet into a parallel model (Oord et al. 2018). In practice, following their approach, which equates to using a reverse KL loss as noted above, is efficient but proved be too unstable. Parallel WaveNet stabilizes training with several auxiliary losses, like a perceptual loss, which we do not have in our setting, in general.
We now empirically compare the distillation losses (Section 2.2.1 focusing on KL. and cross- entropy losses in Eqs. (TO) and (12). Specifically the KL loss (k)), reverse KL loss (kl-rev), binary cross-entropy loss (bee), and categorical cross-entropy loss (ce). During training we sample training sequences from a dataset and continuations 12; either from the teacher model Q.px the student model Pe. or directly from a dataset. Table shows the results for different losses. Empirically we note, that O-PTPs are casier to train than C-PTPs and achieve a higher number of average correct tokens. This is most likely due the fact that O-PTPs do not have predict the full token distribution accurately, which includes tail behavior, as long as they learn which token is the most likely given the auxiliary variable. In the following, choose sample training sequences from the teacher model for best results.
E.3 SAMPLING OF AUXILIARY VARIABLES
Our framework conditions, for a prompt Leg, not token t directly but on the auxiliary variable uk E [Fk.4.-1, Fk...) that contains the same information. During inference we sample uy ~ 24(0, 1] as not bias our predictions. During training on the other hand, we have more flexibility and can sample the permissible interval using 44 FR -1 + ūg [F Fr.,. 1]. where ūt ~ Beta(b, b). For this simplifies uniform distribution while b o puts more less weight on predic-
18

---

Published as a conference paper at ICLR 2026
Proposal Distribution P(()
kl
kl-rev
bee
ce
MTP
Teacher
40
45
44
10.1
Student
44
39
45
45
10.1
Dataset
29
36
44
43
10.1
Table Our framework compatible with several losses. Average number of correct tokens (1) on the taxi dataset, evaluated on 16000 samples. O-PTP are distilled with KI. reverse KI. loss (kl, kl-rev), C-PTP with binary or categorical cross entropy loss (bee, ce). Independent prediction (MTP) (Gloeckle et al. 2024) achieves 10.1. Numbers rounded to reflect level of statistical certainty.
Model
fl + NN
th lin
th NN
lin
NN
MТP
O-PTP
35.9
40.9
39.1
45.4
46.1
10.1
C-PTP
28.8
36.8
36.6
40.4
35.7
10.1
Table 7: Structured embeddings auxiliary variables uy are more stable than fully-learned embeddings. Average number of correct tokens (f) on the taxi dataset, evaluated on 16000 samples. Trained using the KL loss (C-PTP) and binary cross-entropy loss (O-PTP), respectively. Indepen- dent prediction (MTP) (Gloeckle et al. 2024) achieves 10.1. Numbers rounded to reflect level of statistical certainty.
tions that land closer to the border of the permissible interval and thus are more difficult to predict. Training results for different values of can be found in Table [ Empirically, we find that while the choice of b does not seem affect the final average number of correct samples, a larger might speed up the earlier stages training while a smaller might yield slightly better sample quality during inference, measured by model perplexity.
E.4 ABUNDANT COMPUTATIONAL RESOURCES
Another way to leverage additional compute (via additional hardware or longer predictions) pure inference setting is improve the expected number of correct tokens directly. Specifically, for a fixed context we can compute M independent predictions using independently drawn auxiliary variables m....... UNN for ...... M. By choosing the best prediction, i.e. the one that gives us the best chance of higher number of correct tokens, we can improve latency further.
Crucially, we have choose the best prediction in a way that doesn't 't bias the marginal distribution over future tokens. If we, for example, naively choose the sequence that is correct for the most amount of tokens, we will bias our prediction towards sequences that are easier to predict. One way to achieve bias-free improvements is to pick the set of auxiliary variables that lands, aver- age, closest to the center of a token's s valid interval 1y(tz) = [Fx. F Fk.4y-1) where Fx,te is the cumulative probability under Q₂ to choose ts when predicting that token. Specifically, choose
Uk.m FR.
argmax,
(22)
Fx.te,m-1 Fk.ee.
This does not bias the marginal distribution but does bias the distribution of the selected we to be closer to the center of its interval (1x),, making the prediction less prone to small differences in
b
0.5
MTP
О-PTР
12.9
13.9
13.8
8.4
C-PTP
13.6
13.8
13.5
8.4
Table 8: Different sampling strategies for are available. Average number of correct tokens (f) for üg Bota(b, b) on the taxi dataset, evaluated on 16000 samples, with N 16. Trained using the KL loss (C-PTP) and binary cross-entropy loss (O-PTP), respectively. Independent prediction (MTP) Glooeckl al. 2024) achieves 8.4. Numbers rounded to reflect level of statistical certainty.
19

---

Published as a conference paper at ICLR 2026
the teacher's s and student's logits. In the limit M - yielding an upper bound to the possible improvement. Table shows the performance gains on the taxi dataset.
always select the middle point of I& (tx),
M
16
64
256
1024
10
00
Avg. correct tokens
45.36
49.67
54.76
55.74
56.91
59.79
65.42
90.17
Table Additional compute increases correctness. Average number of correct tokens (1) for M predicted sequences, computed in parallel on the taxi dataset, with sequence length N 100.
E.5 RESTRICTED COMPUTATIONAL RESOURCES
Limiting the number N of tokens PTP predicts number of floating point operations, increasing energy efficiency. This, of course, negatively affects the possible latency gains, especially since N is an upper bound on the average number of correct tokens. Table 10 shows the result for different values of N on the taxi dataset.
smaller number will reduce the total
N
1
2
4
16
64
100
Avg. correct tokens
1.00
1.99
3.93
7.59
13.90
36.60
45.36
48.76 78.76
MTP
1.00
1.91
3.53
5.86
8.40
10.08
10.07
10.20
Table 10:
Less compute decreases correctness. Average number of correct tokens (1) for limited number of predicted tokens N per O-PTP call on the taxi dataset, M = 1.
F EFFICIENT ERROR CORRECTION
Algorithm shows how to correct errors with sequential calls to proposal model and a base model.
Algorithm shows how correct errors by running proposal model and base model in parallel. This can also be achieved in a single model call when leveraging Gated LoRA (Samragh et al. 2025): Algorithm can be executed in single forward pass by structuring the attention mask as shown in Fig. T0 the verification rows attend only to the prefix (base model weights active), while each proposal row attends to hypothetical prefix plus all preceding proposal tokens (student weights active), enabling the base and student computations to run in parallel within one model call.
Algorithm Sampling with sequential error correction
Require: Prompt Lzz. one-hot or categorical PTP Pg, verification model Q target length T' Sample ug 24[0, 1] for all & 2 i.
while < '''d do
if Po one-hot PTP then
Pk + Po tzi, ui..... ux), jointly for all & 2
Student one-hot distributions
ta argmax, Pht for all k 2i
Eq. (5)
else
Pk - Po(la less ui (a-1)) jointly for all & 2 i Student categorical distributions Pick(uk, Pk) for all k >
Eq.
end
Q (te [tek), jointly for all l: 2
Teacher categorical distributions
& Pick(ux, Qk) for all k 2
Teacher tokens
i + minazi (k: la # 1x)
D cf. Eq. (0
--i
i i+i+1
> #accepted = #correct +
end while
20

---

Published as a conference paper at ICLR 2026
Algorithm 2 Sampling with Partial Quadratic Decoding
Require: Prompt E<c, one-hot PTP Pe. verification model 22 Beken-budget C, utility function H,
target length T"
Sample ug 24 1] for all k 2 i.
Es f- Eaa for all & 2 i Ch +- for all
P Only allocate to m = in first iteration
while < 7 do
Compute optimal allocation of tokens from c under Pmn - Po(tx+m &<t+m: U4+ms.... Uk+m), jointly for all i Bm
Eq. 15
mnk - argmax,PPil
Eq.
Cmk max Pmiki
Proposal Confidence Qu and Pg can be run in parallel
QN + Qu(ta (tee): jointly for all k 2
L t Pick(ux, Qa) for all k 2 i
n f minw2. {k: ty # {x] -1
D cf. Eq. 20
txtm t tmm for + Bm k 2 i
Ok+m < Cmkk for i + Bm > 2
i +- ittm
4 + l
i i+1
#accepted correct
end while
AR Stop
AR Step 2
AR Step
AR Stop 4
AR Step
001
def Tactal
def facterial
def factoriall
def facteristrius
PTP Step
PTP Stop
PTP Step
PTP Step
PTP Step
der factorsald niln
def factoriat ounnts
def facteriatl num)) 4f man we 12
fnctorialiuunt man 1: return:25
def facteriatteumii con 1f nua 11in
wee 9530
LB 80
recuressse
returnitiin
retum 1nn
Figure 8: Detailed generation trajectory for Fig. Green is accepted, red is rejected by the base model call in the next step.
1.0
Teacher verification
0.8
0.6
0.0
0.0
0.2
0.4
0.6
0.8
1.0
Student confidence Polte tei Un Ug)
Figure 9: The learned probabilities Po(ts Ezz u...., Ug) of One-Hot Parallel Token Prediction (O-PTP) are reliable estimators as to whether a token will be accepted upon teacher verification.
21

---

Published as a conference paper at ICLR 2026
Attention mask Partial Quadratic Decoding (N=4)
Context (base model, cache)
Verification (base model]
Proposals (student model)
Masked
Context (IV cacha)
Query (altending)
Verification
Branch (B=2)
Branch (B-4)
Branch
Branch ()=3)
Context (KV cache)
Verification (N-4)
Branch
Branch 2 (8-2) (B-4)
Branch (B-4)
Branch (8-3)
Key (attended to)
Figure 10: Attention mask used for efficient parallel decoding, when the PTP model adapted from
its base model via gated LoRA (Samragh et al., 2025). The attention mask corresponds to Fig. [
The color in each row indicates what model weights are active: not active because contained KV
cache, base model for verification, and student model for proposals.
22

---

Published as a conference paper at ICLR 2026
G EXPERIMENTAL DETAILS
G.1 TRAINING ALGORITHMS
Algorithm shows how distill PTP from teacher, Algorithm shows how train directly from data.
Algorithm Training PTP (distillation)
Require:
Sequence proposal distribution P()) (teacher, student, dataset, or combination), cutoff
distribution i( (). teacher model (4), one-hot categorical PTP Pp.
while not converged do
Sample P(t)
Sample P(i t).
Choose such that < N ST.
Ps = 2 ttcc) in single model call for all k = i,....N.
Sample E [FR.4a-1, FR.A.) for all k i,.... N.
Compute (oL(0, using one of Eqs. 110
12
Gradient step.
end while
Algorithm 4 Training PTP (inverse autoregressive)
Require:
Dataset P(t), cutoff distribution P(i f), categorical PTP Pg.
while not converged do
Sample training sequence f1....,tr tr.
Sample split position i {1, (1,..-,77 T). Choose such that < N T.
for k i....,N do
Po(t tzi, us..... Uk- 1), with auxiliary available from previous iterations. Sample 4y E 11 Fk ).
end for
Compute VgL(e, using Eq. Gradient step.
end while
G.2 TRAINING DETAILS
The model used in Section 4.1 and Appendix Elis GPT-2-style transformer language model with transformer layers, a hidden size of 1536, and approximately 29 million trainable parameters. Each layer follows the standard GPT-2 architecture, consisting of multi-head self-attention and position- wise feedforward sublayers, combined with residual connections and layer normalization. The vocabulary size is set to 25. Unless otherwise noted, all other hyperparameters and initialization schemes follow the original GPT-2 specification (Radford et al. 2019). During training and infer- ence of our student model we don't provide any context and evaluate the correctness of the next N = 100 tokens, by comparing (t+ f(c) and Po(t)). ). For results on a smaller N = 16, see Appendix H.5 We train every model for 150k steps with a batch size of 32 with the Adam optimizer Kingma& & Ba 2015) and learning rate 0.0001.
The teacher model used in Section 4.2 dialogue-tune variant of the TinyLlama Zhang al. 2024) 1.1 billion parameter model, adopting the same architecture and tokenizer as ILaMM
The model uses a transformer archi- tecture comprising 22 transformer layers, each with standard multi-head self-attention, SwiGLU feedforward blocks, residual connections, and layer normalization. The embedding and hid- den dimension is 2048, and the intermediate (feedforward) dimension is 5632, consistent with LLaMA-style scaling. The vocabulary size is 32, 000. The parameters are available via https: //buggingfacce wwccee..cc.... www.c///nniiie Tt//TinnyLamm aal.lmma/aaa hat.maaaaiinaaaaaaaaaaaaaa 1at.-1111-hhht11.....
Touvron al., 2023): TinyLl ama-1 1B-Chat-v1.
We train an O-PTP on predicting 16 additional tokens. We adopt the strategy of finetuning us- ing gated low-rank adaptation, where changed parameters are only applied for completion
23

---

Published as a conference paper at ICLR 2026
kens (Samragh et al. 2025) We choose LoRA (Hu al. 2022) rank of 256, although com- parable speedups can be achieved already with r = 64. We train every model for 1.3M steps with a batch size of 56 with the AdamW optimizer (Loshchilov & Hutter 2019) on Eq. (IO) and learning rate 0.0001. We generate training and validation data by generating Python code completions on CodeContests (Li al., (002)) from the teacher, splitting generated sequences randomly into input and completion. We use teacher sampling temperature of 0.7, top-k = 50 and top-p 0.9, as is recommended for this model.
For the MTP baseline, we use Eq. with uninformative us in otherwise identical code for a fair comparison.
For the scaling experiments in Section 4.2.1 we parameterize models of different scales as in Ta- We adopt the same architecture choices as TinyLlama.
ble 11
For the natural language tasks on SpecBench (Xia et al. 2024), we finetune Vicuna-7B-vl. (Chiang
al. ) on ShareGPT (Chen al. 2024). We train it with the same gated r 128 LoRA setup as for TinyLlame but with batch size 64 and for 1M steps.
Model Size
Hidden Size
Hidden Layers
Attention Heads
4k
16
66k
64
525k
128
4.2M
256
4
34M
512
268M
1024
16
16
1.1B (teacher)
2048
22
32
Table 11: Model configuration per parameter count for scaling experiment
We base code trans formers (Wolf et al.
2019
PyTorch (Paszke et al. 2019), PyTorch
Lightning (Falcon & The Py PyTochh Lightning team
2019),
Numpy (Harris et al. (020),, Matplotlib
(Hunter 2007) for plotting and Pandas (McKinney 2010 for data evaluation.
H SELF- VERIFICATION
The experiments in Section 4.3
separate teacher model for verification, guaranteeing output identical in distribution to the teacher. Here we explore a faster alternative that requires no teacher all: the PTP student model verifies its own proposals.
After drafting tokens with auxiliary variables the student evaluates its marginal distribution Po teg) with the gated adapter weights active but without any auxiliary variable input. Per Theorem 2 this recovers the standard autoregressive distribution of the student. A proposed token t accepted if matches Pick(ux, (ta tcxx). Since both proposal and verification use the same model in single forward pass, separate teacher call is required.
Under self- verifiitiii on SpecBench, the average number accepted tokens is #accepted 6.1, compared to #accepted = 4.2 under teacher verification (Table 3. The higher acceptance rate arises because the student's drafts are already well-aligned with its own marginal distribution. This translates to significantly higher potential speedup than what is reported in the main text, at the cost of no longer guaranteeing identity with the teacher's 's output distribution.
Below, show uncurated outputs from SpecBench prompts (Xia et al. 2024). Despite the absence of teacher verification, the student produces coherent, instruction followwin responses.
H.1 CODING
Prompt: Write C++ program to find the nth Fibonacci number using recursion.
Response:
24

---

Published as a conference paper at ICLR 2026
Here is simple program that finds the nth Fibonacci number using recursion in C++:
#include <icstream> using namespace std;
int fibonacci (int n) {
it (n <= 0) I
return 0;
10
} else
11
return 1;
12
} else if (n = 2)
return 2;
else
15
return fibonacci (n-1) fibenacci (n-2)
16
17
int main() () 19
20
int n;
21
cout << "Enter the number of Fibonacci numbers to find: ";
22
ein >> ni cout << "The first <<
Fibonacei numbers are:\nt
24
for (int cout << "Fibonacci (* i << ") fibonacci (i) endl;
0; i n; i++) [
25
26
return 0;
29 ...
30
This program asks the user for the number of Fibonacci numbers they want to find, and then prints out the first Fibenacci numbers. The function "fibonacei" is called recursively Fibonacei number, with the base cases being 0, 1, and 2. The 'main' function is used to handle the user input and print out the results.
compute the nth
31
Note:
This is very simple implementation of the ribonacci sequence, and may not suitable for larger values of For larger values n, you may need to use a different algorithm, such as the Golden Ratio algorithm or the Matrix Exponential algorithm. Additionally, this implementation assunes that the input value is positive, may need to handle negative input values appropriately.
H.2 EXTRACTION
Evaluate the following movie reviews on scale of to 5, with being very negative, being neutral, and 5 being very positive: 1. This movie released on Nov. 18, 2019, was phenomenal. The cinematography, the acting, the plot everything was top-notch. so disappointed with movie. The plot In my opinion, this movie is the worst one to have been released 2022. 3. The movie was okay. There were parts enjoyed, but there were also parts that felt lackluster. This is movie that was released in Feb 2018 and seems to be quite ordinary. Return the answer as JSON array of integers.
Prompt:
Never before have been predictable and the characters were one-dimensional.
Response:
[1, 4, 3]
H.3 MATH
Prompt: When rolling two dice, what is the probability that you roll total number that is least
3?
25

---

Published as a conference paper at ICLR 2026
Response:
There are two ways to roll total number that is at least 3, 3 and 3, and 2. So the probability is 1/36.
To calculate this, you can use the formula for combinations, which is:
(2+1)/2 = P (A)
where n is the number choices, and ((A) is the probability of occurring. In this case, n 2, so the probability of rolling total namber that is at least 3 is 1/2\+1/2 1/36.
Alternatively, you can use the fermula for the binomial distribution, which is:
10
11
(()) = (n! (n A (1-p) kk \ p'n
13
where is the number trials, is the number successes, and p is the probability of success. In this case, n 2, k 1, and p 1/36. Plugging these values into the formula, we get (A) - (2) (2-1)!)
\ (1-1/36)* 1 \ 36/36 = 1/36.
15
hope this helps! Let me know if you have any more questions.
H.4 MATH REASONING
Prompt: A company wanted buy 500 computers and had $700 for each computer. However, when they went to order, they realized the price of each computer was 10% higher than they had planned for. Calculate the total amount of money they paid buy the computers at the new prices.
Response:
The price of each computer was 108 higher than planned, so the new price of each computer was 700 * 1.10 $770.
The company wanted to buy 500 computers, $38500 for the computers.
they paid $770 \ 500
Therefore, the company paid $38, 500 to buy the computers at the new prices.
It's important to note that this is just example, and the actual amount of money paid by the company would depend the specifics of purchase, such any discounts or taxes that may have been applied. The calculation provided here is just general example of how to calculate the total amount of money paid for the computers at the new prices.
H.5 QA
Prompt:
Name of the last part of harry potter?
Response:
The last part of Harry Potter is "Harry Potter and the Deathly Hallows, Part 2. " It was released in 2007 and is the final installment the Harry Potter series by J.K. Rowling, The film adaptation of the book was directed by David Yates and starred Daniel Radcliffe, Emma Watson and Rupert Grint. The Deathly Hallows Part 2 follows the story of Harry Potter and his friends as they confront the dark lord Voldemort and the Deathly Hallows, a powerful and dangerous object that grants its possessor immense power. The film was critical and commercial success, and it brought an end to the Earry Potter series, which had captivated vated audiences vorldwide for over decade. The series whole is considered one of the most successful and influential book
26

---

Published as a conference paper at ICLR 2026
and film series in recent history, and its legacy continues to be felt popular culture.
H.6 REASONING
Prompt: One morning after sunrise, Suresh standing facing pole. The shadow of the pole fell exactly to his right. Can you tell me the direction towards which the shadow was pointing cast, south, west, north? Explain your reasoning steps.
Response:
To determin ne the di direction towards which the shadow was pointing, we need to use the relationship between the angle of the shadow and the angle of the sunrise.
Firstly, we know that the sun rises in the east and sets in the west. So, if the shadow is pointing towards the east, then the pole is pointing towards the west.
Now, we need to determine the angle between the shadow and the pole. Since the shadow is pointing towards the east, we can draw an arrow from the shadow to the east and another arrow from the pole to the west.
The angle between these two arrows will give us the angle of the shadow. Since the shadow is pointing towards the east, the angle between the arrow and the pole will be right angle, or 90 degrees.
Therefore, the direction towards which the shadow was pointing was east.
10
II In summary, the pole was pointing tewards the east, and the shadow was pointing towards the cast well. The angle of the shadow was 90 degrees, which moans the pole and shadow were 1ine with each other. So, the direction towards which the shadow was pointing was east.
H.7 ROLEPLAY
Embrace the role of Sheldon from "The Big Bang Theory" as we delve into our conversa- tion. Don't start with phrases like "As Sheldon". Let's kick things off with the following question: "What is your opinion on hand dryers?"
Prompt:
Response:
As Sheldon, have bit of love-hate relationship with hand dryers. one hand, find then especially when compared to paper towels or cloth towels. Howover, on the other hand, I feel that they can be quite loud and disruptive, especially in public restrooms. Overall, I think they are decent choice for drying your hands, but it's important be mindful of others around you and try not to use then situations where they may be a nuisance.
quite efficient and convenient,
H.8 STEM
How have the Alps and Rhine River influenced settlement and agriculture Western Europe? List three impacts.
Prompt:
Response:
27

---

Published as a conference paper at ICLR 2026
The Alps and Rhine River have had significant impact settlement and agriculture Western Burope, Here are three ways which this occurred:
1. Climate: The Alps act natural barrier, blocking the flow of warm air and creating cooler climate in the region. This has made it difficult to grow certain crops, such as wheat, which requires a warmer climate. Howevez, the cooler climate has also made it easier to grow other crops, such as fruit and vegetables, which thrive in cooler temperatures.
2. Soil: The Rhine River has created fertile soil deposit along its banks, making it an ideal location for agriculture. The river also provides source of water for irrigation, which helps to support the growth of crops.
3. Trace:
The Rhine River has been an important trade route connecting Western Europe with other parts of the continent. This has made it casier for farmers in the region to sell their crops and has helped to support the growth of settlements along the river. The Alps, en the other hand, have created natural barriers that have made more difficult travel and trade goods. However, the presence the mountains has also made it easier for farmers to protect their crops from invasions and other threats.
centuries
Overall, the Alps and Rhine River have had significant impact settlement and agriculture in Western Europe, shaping the region's climate, soil, and trade routes, and influencing the types of crops that can be grown and the settlements that can be built. These factors have contributed to the development of distinctive agricultural and settlement patterns in the region.
H.9 TRANSLATION
Prompt: Translate German to English: Nicht zu vergessen die richtige Kosmetik und Nagelpflege.
Response:
Don't forget about the right cosmetics and nail
H.10 WRITING
Prompt:
Compose an engaging travel blog post about a recent trip to Hawaii, highlighting cultural experiences and must-see attractions.
Response:
Title:
Hawaií Adventure: Exploring the Islands' Rich Culture and Natural Beauty
Introauction:
Bawaii is tropical paradise that has captivated the hearts of many travelers with its stunning natural wonders, crystal-clear waters, and lush landscapes. But what sets Hawaii apart from other vacation spots is its rich cuiture and history. In this blog post, we'll take journey through the islands' cultural experiences and must-see attractions, leaving you with a deeper appreciation for the land of the setting sun.
Day 1: Oahu
Visited the USS Arizona Memorial at Pearl Harbor, a solemn reminder of the tragic event that Led to World War II. Explored the Nuuanu Pali Trail, offering breathtaking views of the island and the surrounding ocean.
28

---

Published as a conference paper at ICLR 2026
Enjoyed traditional Hawaiian luau dinner, where learned about Hawalian culture, history, traditions.
II
Day 2: Maui
12
Visited the Hanauma Bay Nature Preserve, a stunning marine sanctuary where could swim and snorkel among colorful fish and vibrant coral
reefs.
14
Explored the iconic Road to Hana, stopping at picturesque viewpoints and waterfalls along the way. Went whale watching and witnessed the majestic creatures in their natural habitat.
16
17
Day 3: Kauai
18
Explored the Napali Coastline, stunning series of cliffs, waterfalls,
and beaches that offer breathtaking views of the ocean. Visited the Hanalei Bay, where we could swim, sunbathe on the beach, and watch the sunset.
20
Enjoyed sunset sail, where cculd watch the sunset while enjoying the cool ocean breeze.
22
Day 4: Big Island
23
24
Visitea the Hawaii Volcances National Park, where could see the Kilauea Volcano and explore the park's s lush rainforests and waterfalls.
25
Visited the Akaka Falls State Park, where we could see the stunning waterfall and its surrounding natural beauty.
Enjoyed farm tour and tasting at local organic farm, where could learn about sustainable farming practices and enjoy delicious fresh produce.
27
28 Conclusion:
Hawaii is more than just vacation spot, it's cultural and historical adventure. From the USS Arizona Memorial to the Akaka Falls state Park, there something ng for everyone Hawaii. whether you' a nature lover or cul curtural enthusiast, Hawai has something special to offer. So pack your baga, grab your camera, and get ready for adventure that you'l1 never forget.
29


Reviews:
[{"review_id": "D1Jwce86m8", "reviewer": "Reviewer_125r", "summary": "Modern autoregressive models estimate the conditional probability of the next token given all the past tokens available in the context. One limitation of this class of models is that generating a sequence of $N$ tokens requires $N$ forward passes, hence resulting in possibly high latency. The paper proposes a solution to this problem by introducing a new approach, Parallel Token Prediction (PTP), enabling parallel generation of multiple tokens in parallel in a single forward pass. By incorporating auxiliary random variables into the model -- borrowing ideas from inverse autoregressive flows --  in PTP the generated tokens are not generated independently, in contrast to existing multi-token-prediction models. PTP models can be either trained from scratch or via distillation from standard autoregressive models. The experiments show some evidence that PTP models can effectively match the performance of their autoregressive counterparts yet significantly reducing latency.", "questions": "- In Eq. (6), shouldn't it be $P_{i+1} = P(t_{i+1}|t_{<i}, u_i)$?\n- The l.h.s. of Eq. (7) does not seem correct. Rather, I think it should be written as $P(t_{k}|t_{<i}, u_i, ..., u_{k-1})$. Could the authors clarify?\n- Could the authors provide intuition or evidence for potential performance gains relative to [1] (see weaknesses)?\n- Have you explored how the performance of PTP evolves with scale, and how that compares to AR baselines?\n- Can a PTP model trained to predict $N$ tokens be adapted at inference time to predict $M$ (with $M \\neq N$) tokens?\n\n[1] Liu, Aixin, et al. \"Deepseek-v3 technical report.\" arXiv preprint arXiv:2412.19437 (2024).", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The paper presents a simple yet, to the best of the reviewer’s knowledge, novel approach in which the model learns to sample directly by conditioning on random variables, enabling it to anticipate and jointly predict future tokens.", "weaknesses": "- Paper clarity: the presentation of the paper can be improved in multiple points. See questions.\n- The experimental evaluation is quite limited, focusing on a toy dataset and a small-scale distillation task. Given the potential of the idea, stronger empirical evidence would help validate its impact.\n- While the provided arguments about the theoretical speedups guaranteed by PTP seem convincing, I believe the paper would greatly benefit from an empirical comparison between the latency of standard autoregressive models compared to the proposed frameworks. \n- At multiple points in the paper (lines: 054-056, 441-442, 465-466) it is mentioned that the model proposed in [1] generates multiple independent tokens. I am not sure this is the case as in section 2.2 of [1] the authors state that \"Different from Gloeckle et al. (2024), which parallelly predicts $D$ additional tokens using independent output heads, we sequentially predict additional tokens and keep the complete causal chain at each prediction depth\". I would appreciate it if the authors can clarify this point.\n\n\n[1] Liu, Aixin, et al. \"Deepseek-v3 technical report.\" arXiv preprint arXiv:2412.19437 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "G9EUaUZCK5", "reviewer": "Reviewer_kW1t", "summary": "The paper proposes a method to improve **multi-token prediction (MTP)** by using $k$ auxiliary variables in the context which “represent” the next $k$ tokens. These auxiliary variables (sampled uniformly in $[0,1]$) contain information about the next generated tokens and are used to sample the $k$-th token ahead from the corresponding cumulative distribution function.\n\nTo train the model, the authors propose methods to retrieve these auxiliary random variables (e.g., from a teacher autoregressive model). The student model (the multi-token predictor) learns to reverse-engineer the process by predicting the same tokens as the teacher with the additional auxiliary variables. This way, training and inference match.\n\nExperiments show improvements over standard MTP baselines.", "questions": "How does performance change as the number of auxiliary variables (i.e., prediction horizon) increases?\nIs there a point at which adding more auxiliary variables yields diminishing returns?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The approach is conceptually sound and addresses a relevant challenge: reducing autoregressive inference latency by enabling multiple token predictions in one forward pass.\n\n- The method provides the model with additional structured information about future sampling decisions, potentially allowing more informed multi-token predictions than standard MTP approaches.\n\n- Several design dimensions are explored, including loss functions and auxiliary-variable extraction strategies.", "weaknesses": "- **(Major) Role of auxiliary variables.** A stronger baseline would include the same number of additional “special” tokens as placeholders for future tokens (similar in spirit to the Pause-token approach [https://arxiv.org/abs/2310.02226]), trained to predict future tokens directly. Such a baseline would help clarify whether the auxiliary variables provide unique modeling benefits or primarily serve as additional learned positions.\n\n- **(Minor) Training efficiency.** The highest-performing configuration relies on generating full sequences from a teacher model in order to extract auxiliary variables, which is computationally expensive.\n\n- While the evaluation demonstrates gains on the selected setups, the method appears broadly applicable; including results on standard reasoning and coding benchmarks (e.g., TruthfulQA, HumanEval, GSM8K) would strengthen the empirical validation and illustrate generality.\n\nSmall typo: “which jointly predicts future tokens $t_k$” → “which jointly predicts future tokens $t_i, \\ldots, t_k$”.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dRdzUGKuCQ", "reviewer": "Reviewer_Efeg", "summary": "The paper introduces a framework called Parallel Token Prediction (PTP) to accelerate the inference speed of autoregressive language models like transformers. The authors' key insight is to reframe the generation process: instead of the model predicting a probability distribution from which a token is then sampled, they make the sampling process itself part of the model.\n\nBy feeding a sequence of random numbers (called auxiliary variables) as input along with the text prompt, the model learns to deterministically convert these random numbers into a sequence of multiple output tokens in a single forward pass. This breaks the strict one-by-one generation bottleneck and allows for significant parallelization without making the limiting independence assumptions that plagued previous parallel decoding methods.", "questions": "1. How do you guarantee accuracy of the teacher model during PTP?", "limitations": "", "rating": 2, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "On a coding dataset, they distill a TinyLlama-1.1B model. Their resulting PTP model can generate an average of 5.1 correct tokens that are identical to the teacher's output, all within a single forward pass. This represents a 5x speedup over the standard autoregressive baseline.\nThey show that their method significantly outperforms naive parallel methods that assume token independence, which produce invalid token pairs over 60% of the time in their example.", "weaknesses": "Novelty: This approach has been adopted by a few other works like: \n1. BiTA: Bi-Directional Tuning for Lossless Acceleration in Large Language Models\n2. Your LLM Knows the Future: Uncovering Its Multi-Token Prediction Potential \nI believe the authors are doing the same thing here.\n\nTypically such multi-token prediction tasks are evaluated on Spec-Bench. These results are necessary to justify real gains at inference time.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MjMfDYm8Xp", "reviewer": "Reviewer_5jCL", "summary": "This paper introduces a novel architecture for parallel token generation, says $K$ next token simultaneously. Conventional large language models (LLMs) generate text in an autoregressive manner—given $T$ input tokens, the model predicts the probability distribution of the next token, then repeats this process sequentially. In contrast, the authors observe that the sampling step can be reformulated: rather than drawing directly from a categorical distribution, one can apply the inverse cumulative distribution function (CDF) to a uniform random variable, enabling a more parallelizable generation procedure. So, given a $K$ uniform random variable as input to the model (hypothetically), we can fully determine the next $K$ tokens without the need to perform autoregressive sampling. This paper then presents a training paradigm to train such architecture: input $T$ tokens and $K$ uniform random variables, outputs the next $K$ tokens.", "questions": "In section 3.1, why are we using the taxi dataset instead of other standard dataset for finetuning LLM such as GSM8K for ablation?\n\nIs there any tradeoff between the number parallel token to be generated and the downstream performance?\n\nSuppose I would like to do greedy sampling or changing the sampling temperature, is there any way to do this with your framework? Is there anyway to compute the uncertainty in this framework? \n\nIs it correct to say that the C-PTP framework can let us recover this information? \n\n\nSection 2.3 seems to suggest that the model can produces mismatch between the generated tokens and the actual probability. The error correction part looks very similar to the speculative decoding process, is there anyway to get rid of this?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "Up to my knowledge, the idea presented in this work is original and interesting, drawing inspiration from the probability/sampling theory. \n\nThe paper distinguished itself clearly from other multi-token generation frameworks, including speculative decoding and discrete diffusion model. Though I'm not an expert in discrete diffusion, I think the independent generation issues of prior approach detailed in section 2.4 is an important contribution.", "weaknesses": "No wall-clock time experiment/ speed comparison reported.\n\nThe discussion on model architecture should be emphasized more.\n\nIt is a bit unclear what is the tradeoff between the number parallel token to be generated affects the performance. A small graph showing this would strengthen the paper.\n\nThe error correction part (section 2.3) seems to say to the use of this framework is to serve as a draft model in speculative decoding (correct me if I'm wrong). If this is the case, there should be an ablation on this. \n\nOverall, I believe this is a neat idea. There might be some challenges training such model so I suggest the experiments to be more focused instead of spreading over multiple separate issues.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)