LANTERN: Accelerating Visual Autoregressive Models with Relaxed Speculative Decoding
Authors: Doohyuk Jang
Sihwan Park
June Yong Yang
Yeonsung Jung
Jihun Yun
Souvik Kundu
Sung-Yub Kim
Eunho Yang
Venue: ICLR 2025 Poster (2025)
Keywords: Speculative decoding
Visual Autoregressive Models
Primary area: generative models

Abstract: Auto-Regressive (AR) models have recently gained prominence in image generation, often matching or even surpassing the performance of diffusion models. However, one major limitation of AR models is their sequential nature, which processes tokens one at a time, slowing down generation compared to models like GANs or diffusion-based methods that operate more efficiently. While speculative decoding has proven effective for accelerating LLMs by generating multiple tokens in a single forward, its application in visual AR models remains largely unexplored. In this work, we identify a challenge in this setting, which we term \textit{token selection ambiguity}, wherein visual AR models frequently assign uniformly low probabilities to tokens, hampering the performance of speculative decoding. To overcome this challenge, we propose a relaxed acceptance condition referred to as LANTERN that leverages the interchangeability of tokens in latent space. This relaxation restores the effectiveness of speculative decoding in visual AR models by enabling more flexible use of candidate tokens that would otherwise be prematurely rejected. Furthermore, by incorporating a total variation distance bound, we ensure that these speed gains are achieved without significantly compromising image quality or semantic coherence. Experimental results demonstrate the efficacy of our method in providing a substantial speed-up over speculative decoding. In specific, compared to a na\"ive application of the state-of-the-art speculative decoding, LANTERN increases speed-ups by $\mathbf{1.75}\times$ and $\mathbf{1.82}\times$, as compared to greedy decoding and random sampling, respectively, when applied to LlamaGen, a contemporary visual AR model. The code is publicly available at \url{https://github.com/jadohu/LANTERN}.

Published as a conference paper at ICLR 2025
LANTERN: ACCELERATING VISUAL AUTOREGRES- SIVE MODELS WITH RELAXED SPECULATIVE DECOD-
ING
Doohyuk Jang" Sihwan Park June Yong Yang'
Yeonsung Jung!
Jihun Yun' Souvik Kundu² Sungyub Kim Eunho Yang!
'KAIST Pnteel Labs PAITRICS (jadohu, sihwan. parwan.park, sungyub. kim, junhoy)@kaist.comm
ABSTRACT
Auto-Regressive (AR) models have recently gained prominence in image gener- ation, often matching or even surpassing the performance of diffusion models. However, one major limitation of AR models is their sequential nature, which processes tokens one at time, slowing down generation compared to models like GANs or diffusion based methods that operate more efficiently. While specula- tive decoding has proven effective for accelerating LLMs by generating multiple tokens in a single forward, its application in visual AR models remains largely unexplored. In this work, we identify a challenge in this setting, which we term token selection ambiguity, wherein visual AR models frequently assign uniformly low probabilities to tokens, hampering the performance of speculative decoding. To overcome this challenge, propose relaxed acceptance condition referred to as LANTERN that leverages the interchangeability of tokens in latent space. This relaxation restores the effectiveness of speculative decoding in visual AR models by enabling more flexible use of candidate tokens that would otherwise be prematurely rejected. Furthermore, by incorporating total variation distance bound, we ensure that these speed gains are achieved without significantly com- promising image quality or semantic coherence. Experimental results demonstrate the efficacy of our method in providing a substantial speed-up over speculative decoding. specific, compared naïve application of the sfate-of-the-rr specu- lative decoding, LANTERN increases speed-ups by 1.75x and 1.82x, com- pared to greedy decoding and random sampling, respectively, when applied to LlamaGen, contemporary visual AR model. The code is publicly available https::/ g thub. com/ Sadohu/LANTERS LANTERN.
INTRODUCTION
unto-Regressive (AR) models have recently gained significant traction in image generation (Ramesh et al., 2021; Chen et al., 2020; Tian et al., 2024; Sun et al., 2024) due to their competitive performance, often matching or even surpassing diffusion models (Ho et al., 2020; Rombach et al., 2022). Notable examples include iGPT (Chen et al., 2020), DALL-E (Ramesh et al., 2021), VAR (Tian et al., 2024), and LlamaGen (Sun et al., 2024), which showcase the potential of AR models in image generation. Moreover, recent studies like Lu et al. (2023); Team (2024); Chern et al. (2024) have demonstrated that AR modeling can handle multi-modal data, including language and images, within a single unified framework. Given the remarkable success of AR models in language modeling, leading the era of large language models (LLMs) (Brown et al., 2020; Touvron et al., 2023; Jiang et al., 2023), anticipated that AR modeling will emerge dominant paradigm for unifying multiple modalities into a single model in the near future.
Despite the promising potential of AR models, their sequential nature poses significant bottleneck for both efficiency and scalability since they generate a single token per forward pass. In contrast,
"quual Contribution
Corresponding Authors

---

Published as a conference paper at ICLR 2025
rreentie sice the har, offlcttegt the snee- capped puaks shone, with irall sooedin
Adous-p of
darilled and parnting wi ettawan with hrar ondersand differme sectune such flower art badigroond.
10 medendd
dreadiecion creddare acpmedion anda partoreed handara
focteed and thick ctring dinectty at camerrs apaindt backgrredd
uith dower ciows, ser sopersdiie cairs with rutliatic pain sestine lightngg
Figure 1: Images generated by vanilla decoding (top) and lossy speculative decoding with relaxed acceptance condition (boltom) on the text-conditioned LlamaGen-XI. Stage II (Sun et al., 2024). The mean accepted length for each image is displayed in white at the bottom right corner of each image.
GANs (Goodfellow et al., 2014; Karras et al., 2019), which generate images in single forward pass, naturally avoid this issue, and diffusion models have benefited from extensive research aimed at improving their speed (Song et al., 2022; Sauer al., 2023; Heek al., 2024). However, transferring these acceleration techniques to visual AR models is far from straightforward due to fundamental differences in the underlying mechanisms of these models.
One notable acceleration technique for AR models is speculative decoding (Leviathan et al., 2023; Chen et al., 2023; Cai et al., 2024; Li al., 2024b), which has demonstrated its effectiveness in LLMs. Initially introduced by Leviathan et al. (2023), speculative decoding addresses the sequential bottleneck of AR models by introducing a draft and verify mechanism. this framework, smaller model (the drafter) predicts the next few tokens, which are then verified by the larger target model. If the drafter's predictions are accurate, multiple tokens can be generated from a single forward pass, resulting in substantial inference speed-up. This method has proven highly effective in accelerating LLM inference, making it leading option for reducing the latency associated with AR models.
While speculative decoding shows great promise for accelerating AR models, its application to visual AR models remains largely unexplored. Therefore, in this paper, we take the first step toward addressing this gap by migrating speculative decoding to visual AR models. Interestingly, our findings reveal that the naïve application of existing speculative decoding methods falls short in visual AR models. Specifically, we identify key problem, namely the token selection ambiguity. that hampers the effective migration of speculative decoding to visual AR models.
To mitigate such an obstacle in speculative decoding, we propose solution dubbed as LANTERN (Latent Neighbor Token Acceptance Relaxation) that leverages the interchangeabillty of image tokens in latent space for the relaxation of acceptance condition. By relaxing the acceptance in speculative decoding, we allow for more effective utilization of draft (candidate) tokens that would otherwise be frequently rejected despite their potential usefulness. However, our relaxation introduces some distortion the target model's 's distribution, which may cause the generated images to deviate from the original target output. To mitigate this, we further incorporate total variation distance bound which ensures that the deviation remains controlled.
Our main contributions are summarized as below:
To the best of our knowledge, we are the first to thoroughly investigate speculative decoding in visual AR models, identifying the token selection ambiguity problem, where near-uniform token probability distributions hinder token prioritization, causing existing methods to fail in improving speed. Based on our insights, then propose LANTERN, novel relaxation of acceptance condition for the speculative decoding that addresses the token selection ambiguity problem, successfully enabling the effective application of speculative decoding visual AR models.
2

---

Published as a conference paper at ICLR 2025
Our experiments using LlamaGen (Sun et al., 2024) as the target and EAGLE-2 (Li al., 2024a) as the base speculative decoding method demonstrate significant speed-ups, im- proving from 1.29x to 2.26x in greedy decoding and from 0.93 to 1 x.69x in random sampling. compared to the naive application of EAGLEE22, without substantial performance drop in terms of image quality.
PRELIMINARIES
Notations In this paper. the target model refers to the visual AR model we aim to accelerate. In contrast, the drafter model is a supplementary model used to generate draft tokens. The probability distributions modeled by the drafter and target models Individual tokens are denoted lowercase 2, and sequences are represented by uppercase X; for instance, Xuy represents the sequence (Ii, (Zi,...,, 2.)). The concatenation of sequences XI:N and YIMw denoted by (XI:N, YI:M) ) = (II,.... ,...; NN M1....... yM)) For simplicity, we allow certain notational liberties; for instance, expressions like X1.0 are treated empty sequences to avoid unnecessary complexity in notation.
represented by p(-l-) 1.) and q(-l-). respectively.
Visual Autoregressive Modeling In visual AR models, image generation involves two main stages: generating image tokens through auto-regression and decoding the image tokens into actual image patches. text-to-image generation setting, given a tokenized text prompt XI.N. the model generates a sequence of image tokens X N based on the following probability modeling:
K
P(XN+EN+K XI:N
II P(EN+el X1:N+6-1), t=1
where K represents the total number of image tokens corresponding the height and width of the image feature map. Since each token is predicted based solely on its preceding tokens, visual AR models require K sequential steps generate all K image tokens.
Once the K image tokens are generated, they are mapped to visual representation by referring to the codebook C. Specifically, the codebook C fci,...., Cr.} consists of codes c; Rd, where each code is a latent feature used by the image encoder-decoder pair (such as VQVAE (Van Den Oord al., 2017) VQGAN (Esser al., 2021)). and d is the dimensionality of these features. As the text tokenization, since the image tokens represent indices in the codebook C, each image token IN+ maps to Cax+.. which then rearranged into h x w X d shaped tensor in raster-scan order (from top-left to bottom-right), for h = H/J f and w W/f when f is down-sampling factor. After that, rearranged latent is fed into the image decoder gxxwxd + HHxWxC to construct an actual RGB image.
Speculative Decoding We briefly introduce the basics of speculative decoding, which were pro- posed by Leviathan et al. (2023). Given sequence of tokens X1-N (#1,... ......... . model generates y draft tokens X1:7 as speculations for the next gy tokens following X1-N. Each draft token ã; is sampled from the drafter distribution (Xx](XX,.) 1:N, X1=)-)) for each = 1,..., thus requiring y forward steps to generate 7 draft tokens.
IN), ), the drafter
After obtaining the draft tokens, the concatenated sequence (XINN, X1:7) is fed into the target model, which calculates the likelihood of each draft token (Xiz)((i.,, X in parallel within a single forward pass. Each draft token 24 is accepted with probability:
min (1. ( ( X
('p(Fs (XIN, X1+-1))
If 3; is accepted, it is immediately set as EN+: 2;. Otherwise, all subsequent tokens X are discarded, and IN+4 resampled from distribution defined by XX.. p(x1(X1..v X1:-1))) where [+] denotes normalization over positive values only. As proven by Leviathan et al. (2023), this approach ensures that the distribution of the generated token sequence matches the distribution produced by the target model. Related works on the visual AR models and speculative decoding are presented in Appendix A.
3

---

Published as a conference paper at ICLR 2025
4.58
14.45
Moun Agcospted keugh
Vicuna78 LlamaGen-38
Top-1 Accurasy
Top-3 Accuracy
1-c
Tap-l Probubillyy
Top Prosabiltty
2.36
2.52
2.13
Tex: Accuraky
60-
Probab lity
0.500
0.4
02
0.004
Decoding
E
0-
Viana-138
LianaCer-Xe XL
Anole
06
Vionaa78 78
Ulmaaen-XL
Arol:
(a) Naive Application Performance
(b) Drafter Test Accuracy
(c) Average Top-1.10 probabilities
Figure 2: (a) Mean accepted length of naive application of existing speculative decoding methods on visual AR model and LLM counterpart. (b) Top-1 and top-3 accuracy of learned drafter model for predicting the target model's outputs. (c) An average top-1 and top- 10 probabilities the next token prediction.
TOKEN SELECTION AMBIGUITY LIMITS SPECULATIVE DECODING IN VISUAL AR MODELS
Although speculative decoding has been highly successful in LLMs, it fails to deliver comparable speed improvements when naively applied to visual AR models. As illustrated in Figure 2(a), the visual AR model (LlamaGen-3B (Sun et al., 2024)) exhibits 41% 58% reduction in mean accepted length compared to their performance LLM (Vicuna-7B (Zheng et al., 2023)), a consistent decline observed across different speculative decoding methods. Toward an in-depth exploration of this performance degradation, we train drafters for various visual AR models and conduct additional analyses on their predictions.
Our analysis reveals that the drafter in visual AR models (LlamaGGen-X and Anole (Chern et al., 2024)) struggles to predict the target model's outputs accurately. Specifically, as demonstrated in Figure 2(b), the trained drafters for visual AR models fail to capture the target model's prediction precisely. whereas the drafter LLMs exhibits considerably higher accuracy. Such a low accuracy of the drafters for visual AR models leads to a significant reduction in the speed-up uup achievable via speculative decoding.
To delve into the root cause of the drafter's performance degradation, we analyze the next token probabilities of visual AR models and identify unique problem we term token selection ambiguity. As shown in Figure 2(c), visual AR models present substantially lower average top-1 and top-10 probabilities in next token predictive distributions compared to language models, indicating that visual AR models are more ambiguous when selecting the next token. This lack of prioritization among tokens reflects the model's limited confidence in any single option.
We hypothesize that this issue arises from fundamental differences between image and language data. In language models, tokens represent discrete units, such as words or subwords, that form structured and predictable sequences governed by grammar and syntax (Zipf, 1935). Consequently, the next-token probabilities are more concentrated, and the model has high confidence in the most likely token. contrast, visual AR models treat pixels or patches as tokens, forming continuous and highly complex sequences. As a result, these models exhibit more dispersed next token probabilities and face higher uncertainty and ambiguity in predicting subsequent tokens. This token selection ambiguity problem in visual AR models hinders the drafter's predictive accuracy, thereby limiting the effectiveness of speculative decoding. Further details on the setup of experiments can be found in Appendix B.
4 DETOURING TOKEN SELECTION AMBIGUITY THROUGH LATENT SPACE
In this section, we propose LANTERN, a simple yet effective method that permits detouring the failure of speculative decoding caused by the token selection ambiguity problem by relaxing acceptance condition in Speculative Decoding (Leviathan et al., 2023). In Section 4.1, we introduce concept of latent proximity which asserts close image tokens in the latent space are interchangeable and examines its validity on the generated images. Section 4.2 describes how relax the acceptance
4

---

Published as a conference paper at ICLR 2025
haseridd wish
Achoss of host mesh
tndkeape with
civee- urta carty and barc stonde filling tra fact, genta enile.
A choeeep out with parringg gettin syes. Ultusinttd uit agane dort.
wilthowa, a snall staan sixting
inclading applis, ocunger and popee. wist tight satty highlighing soctrn
coull anitiig corn gloe from stidirss, crabe pertly ning foon
trough, homertha flttrinng and dirnes polke
Figure 3: Image generated by text-conditioned LlamaGen-XX Stage II model (Sun et al., 2024). The images are generated by either standard sampling method (top) or sampling with random replacement within 100-closest tokens in the latent space (bottom).
condition based the interchangeability. In Section 4.3, present another component to ensure that the distribution of generated images does not catastrophically deviate from the original distribution.
4.1 LATENT PROXIMITY PERMITS TOKEN INTERCHANGEABILITY
We introduce latent proximity, a property in visual AR models that asserts tokens close to one another in latent space interchangeable without significantly affecting the visual semanties overall image quality. This means that replacing a token with another nearby token in latent space results in minimal changes the generated image.
This property arises from the tokenization process unique to visual AR models. Unlike text tokeniza- tion, which is straightforward due its discrete nature, images spatially continuous, making tokenization more complex Esser et al. (2021). To handle this, models like VQVAES (Van Den Oord al., 2017) and VQGANs (Esser et al., 2021) are used discretize the latent embeddings of images, as introduced in Section 2. These embeddings maintain a continuous mapping between changes in latent space and the visual semanties of the generated images (Kingma & Welling, 2022; Goodfellow et al., 2014; Karras et al., 2019). As result, small shifts in latent space lead to minor shifts in the image, supporting the idea that tokens close in the latent space are effectively interchangeable.
To demonstrate this concept empirically, we perform experiment in which, after each token is sampled, re-sampled uniformly from the 100 closest tokens latent space. Figure reveals that the images generated by this procedure closely resemble those produced using the original sampling method. This confirms that tokens close in latent space can be treated as interchangeable, allowing for flexible token replacement without significantly compromising the visual semantics overall image quality. more detailed analysis latent proximity can be found in Appendix C.
4.2 LANTERN: RELAXATION OF ACCEPTANCE CONDITION
Building on our findings about latent proximity, we introduce LANTERN, a simple yet effective solution that leverages the interchangeability of proximate tokens in latent space. By treating neigh- boring tokens as commutable, LANTERN effectively resolves the token selection ambiguity problem, significantly boosting the acceptance probability of candidate tokens and enabling the successful application of speculative decoding.
We start with revisiting the original acceptance condition from Leviathan et al. (2023), introduced in Section 2. The drafter model samples draft token ă ~ p(z|s) given preceding sequence
g(ā(s) and if rejected, the next XLAN The draft token is accepted with probability min (1, p(z
5

---

Published as a conference paper at ICLR 2025
Table Average accept probabilities of LANTERN, We only use accept probability of the first draft
token. An average accept probability of EAGLE-2 (Li et al., 2024a) is 0.0402.
Average Accept Probability
k
8 0.05
8 = 0.1
8 0.2
5 = 0.4
100
0,0725
0.1096
0.1703
0.2595
300
0.0759
0.1166
0.1892
0.3267
1000
0,0786
0.1186
0.2000
0.3657
token is re-sampled from () p(=[s)]++ Note that the acceptance depends on the alignment of probabilities between the drafter and target models.
However, this acceptance condition results in sharp decline in accept probability when encountering the token selection ambiguity problem. As mentioned Table 1, the EAGLE-2 drafter exhibits an average accept probability of 0.0402, meaning only 4% of drafts are accepted when applied visual AR models. This issue arises because the target model assigns low probabilities to individual tokens and frequently misaligns with the drafter's s distribution, leading to frequent rejections of candidate tokens and reducing the overall effectiveness of speculative decoding.
To alleviate this problem, we exploit the latent proximity by aggregating the probabilities of candidate token's 's nearest neighbors, treating them as proxies. This approach effectively increases the acceptance probability by utilizing the combined likelihood of similar tokens, mitigating the impact of the token selection ambiguity problem and reducing unnecessary rejections.
Specifically, we define the neighborhood B (z) as the of k-nearest tokens in latent space, including मे itself. The accept probability is then adjusted to
min (1, Σ xc (B(w)) B. (([ss)
p(7(s)
Because T is always included in B& (7), this new accept probability is guaranteed be equal or higher than the original acceptance condition. As demonstrated in Table 1, applying LANTERN significantly increases the average acceptance probability, reaching values as high as 0.37. This improvement allows us to recover candidate tokens that would have otherwise been unjustifiably rejected.
4.3 WITH LIMITED DISTRIBUTIONAL DIVERGENCE
Although the relaxed acceptance condition (1) effectively permits speculative decoding in visual AR models by significantly raising the accept probability, it inevitably distorts the target distribution. In particular, when we condition the target distribution on the candidate token T, it becomes:
ExeB.(D) (<x(s)
i =
2h (x|s D - -
0
if E (x otherwise
g<]|s)
where is random variable representing the candidate token and qu denotes the distorted target distribution. In contrast, under the original acceptance condition, the target distribution remains unchanged regardless of the candidate token. For this reason, (1) may excessively distort the target distribution, leading to generating images that diverge significantly from those generated by the target model.
To mitigate this distortion, we impose upper bound on the distributional divergence using total variation distance (TVD). Since the distortion results from redistributing probability mass, TVD effectively measures the extent of this shift, allowing control the magnitude of the divergence. This can achieved by adjusting the neighborhood B (ī)) used in the relaxation as follows.
Since the relaxation be analogously derived using any neighborhood of ē. borhood that ensures the TVD between the target distribution and the distorted target distribution induced by the neighborhood is below specific threshold. To formulate this approach, define
find neigh-

---

Published as a conference paper at ICLR 2025
the neighborhood (z) of ø for given TVD bound ò > 0 and Z Ag,o(z) (z) is the largest subset of (x) such that for the total variation distance Drv,
Drv (x), = 2) (((2( s, D = =)) = Drv ( D D=)), g()ls)) <
where 2k,6 denotes the distorted target distribution induced by (k.
We construct AR.o (z) by incrementally adding tokens from (x) to ( starting with the closest ones to I. and stopping when adding another token would exceed the TVD threshold ô. This procedure allows us relax the acceptance condition by incorporating probabilities of similar tokens while keeping the divergence within predefined boundary.
By integrating the TVD constraint into the acceptance condition (1), we arrive the final relaxed acceptance condition of LANTERN:
Accept I with probability min (1, S (z)())
p())(8)
Else re-sample I ~ [q,, = p(x|s)]]
For greedy decoding. LANTERN can be simply reduced to accept if ā = arg max, (2]s, I). The algorithms for LANTERN and construction of Ak,5 can be found in Appendix D.
5 EXPERIMENTS
5.1 EXPERIMENTAL SETUP
To validate our method LANTERN, we conduct experiments on text-conditioned LlamaGen-XL Stage I model (Sun et al., 2024) as target model that establishes the best performance among visual AR models without iision-specife modifications. We utilize the MS-COCO validation captions (Lin al., 2014) to generate images and evaluate the image quality with the ground-true truth images. For the base speculative decoding method, we employ EAGLE-2 (Li et al., 2024a), which has demonstrated state-of-the-art performance in speculative decoding in the language domain. We employ the &2 distance to quantify latent proximity and utilize the TVD as a metric for divergence bound.
For the assessment of speed-ups, we use 1000 MS-COCO validation captions because this sample size provides results that are nearly identical to those obtained when evaluating the entire dataset. Actual speed-up is measured by the inference time ratio between each method and standard regreseeeee decoding. The mean accepted length is determined by the average number of tokens accepted in each forward step the target model. We evaluate each method both the greedy decoding setting with = 0 and the sampling with = 1. The statistical analysis on actual speed-up and number of captions can be found in Appendix F.1.
Since LANTERN can impact the quality of generated images, we evaluate image quality using FID (Heusel et al., 2017), CLIP score (Hessel et al., 2021), Precision and Recall (Kynkäänniemí et al., 2019), and HPS v2 (Wu et al., 2023) with 30K samples. For each evaluation, we do not evaluate image quality on EAGLE-2 since theoretically guarantees exact distribution matching with the target model. Further details can be found in Appendix B.2.
5.2
MAIN RESULTS
In this section, we demonstrate qualitative and quantitative results of speculative decoding with our relaxed acceptance condition. First of all, Section 5.2.1 demonstrates how much speed-up can be achieved through our method. Next, Section 5.2.2 showcases that our method retains image quality within a similar level by showing qualitative samples. Afterward, Section 5.2.3 presents the trade-off between performance and efficiency in our method.
5.2.1 SPEED UP COMPARISON
To confirm that LANTERN provides notable speed improvements over the baseline while maintaining image quality, compare method with baselines under both greedy decoding and random
7

---

Published as a conference paper at ICLR 2025
Table Actual speed-up. MAL (Mean Accepted Length), FID, CLIP score, Precision Recall, and HPS v2 for each method. 0 refers to the greedy decoding and refers to the sampling with temperature for generation. The actual speed-up measured on single RTX 3090.
0
Method
Acceleration (t)
Image Quality Metrics
Speed-up
MAL
FID ())
CLIP score (1)
Precision (1)
Recall (?)
HPSv2 (1)
Vanilla AR (Sun et al., 2024)
1.00x
1.00
28.63
0.3169
0.4232
0,3517
23,18
EAGLE-2 (Li ct al. 2024a)
1.29x
1.60
LANTERN 0.05, k 1000)
1.56x
2.02
29.77
0.3164
0.3158 0.2773
22.62
LANTERN (6 0.2, k 1000)
2.26x
2.89
30,78
0.3154
0.4771
21.69
Method
Acceleration (1)
Image Quality Metries
Speed-up
MAL
FID ())
CLIP score (f)
Precision (?)
Recall (T)
HPSv2 (1)
Vanilla AR (Sun et al., 2024)
1.00x
1.00
15.22
0.3203
0.4781
0.5633
24.11
EAGLEE- (1. et al.,, 2024a)
0.93x
1.20
LANTERN (6 0.1, 1000)
1.13x
1.75
16.17
0.3208
0.4869
0,5172
23.75
LANTERN 0.4, 1000)
0.3206
0.4909
0.4497
23.22
Figure 4: Qualitative samples generated by LlamaGen- XL Stage model for LANTERN and standard autoregressive decoding. From top bottom, the images generated by standard autoregressive decoding, LANTERN (8 = 0.2, 0.4) where k is fixed at 1000, and images in the same column are generated using the same text prompt. Text prompts for the images are provided in Appendix
sampling situations. Table 2 demonstrates the speed-up up and image quality across different meth- ods. LANTERN shows sigmificant acceleration, even with slight degradation in image quality, when compared to standard autoregressive decoding and EAGLE-2 (Li et al., 2024a). Our method, LANTERN demonstrates significant improvements in both mean accepted length and actual speed-up up. In terms of mean accepted length, LANTERN outperforms both baselines by reaching 2.40x and it translate to substantial actual speed-up in practice, with LANTERN achieving 1.69 x actual speed-up.
While LANTERN's s acceleration comes with trade-off in image quality, the degradation remains minimal and well within acceptable boundaries. For instance, the HPS v2 score decreases by less than 1.5 for both greedy decoding and sampling when compared to standard autoregressive decoding. Precision remains stable or slightly improved, whereas recall shows a modest decline, indicating a slight reduction in image diversity but with individual image quality largely preserved. Other metrics, such as FID and CLIP score, further confirm that LANTERN maintains competitive image quality despite the significant acceleration.
As a result, by allowing a degree of flexibility in token selection, our method strikes a favorable balance between speed and quality, outperforming both standard autoregressive decoding and EAGLE-2 in terms of practical efficiency. Experimental results on other visual AR models including LlamaGen-XL Stage model and Anole (Chern al., 2024) are presented Appendix Also, detailed analyses of latency, including latency on Intel Gaudi 2, are provided in Appendix F.
5.2.2 QUALITATIVE RESULTS
To confirm that image quality is preserved with LANTERN, as indicated by the various image quality metrics, conduct qualitative analysis. Figure 4 demonstrates that, despite the modification of
8

---

Published as a conference paper at ICLR 2025
1000
300
100
LO0A 0.
500 02
190.
000) 0.1
500.01
100
1000
01
300
0.
100.011
(90)
100) (00)
1000 045
390
0.05
100, 006
18
30.5
17-
30.0-
29.5-
16.
Vanilla
290
Vanilla
15
28.5
1.0
1.5
2.0
2.5
20
2.5
3.0
Accepted Length
Meen Accepted Langth
(a) Tradc-off curve with r (sampling)
(b) Trade-off curve with r 0 (greedy)
Figure 5: Trade-off curves show the relationship between performance (FID) and acceleration (mean accepted length). The results with the same k are annotated with the same color, while the same 5 values are marked with identical symbols. In the legend, the values are separated by commas, indicating i: and 6, respectively.
the target model's probability distribution achieve acceleration, our method effectively preserves image quality. Notably, even under the setting of 5 = 0.4 and k = 1000, which achieve about 1 1.64x speed-up compared to the standard autoregressive decoding, generated images retain both content and style at level comparable to standard autoregressive decoding. These qualitative results, along with the fact that LANTERN avoids significant degradation in image quality metries as shown previous section, demonstrate that it effectively preserves image quality while increasing efficiency. More qualitative examples can be found in Appendix
5.2.3 TRADE-OFF BETWEEN PERFORMANCE EFFICIENCY
LANTERN provides various options between quality and efficiency to the end users. Therefore, we explore this trade-off across different hyperparameters by adjusting k and 5. Figure illustrates the relationship between image quality, which measured by FID, and speed-up, which is assessed with mean accepted length for various settings of 8 and k, under both sampling (r = 1) and greedy decoding (r = 0). The trade-off curves highlight that increasing and & generally improves speed-up. but the expense of image quality.
The results underline that LANTERN allows for flexible tuning to balance acceleration and image quality depending specific practical requirements. For instance, in the sampling (7 = 1), con- figurations with smaller 8 or & prioritize image quality, maintaining lower FID scores, while larger 5 i achieve greater acceleration with controlled loss in quality. A similar trend is observed for greedy decoding (r = 0), where the trade-off is more pronounced at higher mean accepted lengths. Importantly, across all hyperparameter configurations, LANTERN demonstrates consistent and predictable trade-offs, enabling users to fine-tune the method based on their preferred balance between speed and quality. Trade-offs with other image quality metrics can be found Appendix H.
5.3 ABLATION STUDY
In this section, we conduct ablation studies for LANTERN. In Section 5.3. assess the impact of the metric used to measure latent proximity on performance. Then, in Section 5.3.2, we provide an ablation study the effect of the metric for measuring the distance from the modified probability distribution.
5.3.1 NEAREST LATENT SELECTION
To explore the impact of various metrics for measuring latent proximity, we conduct an ablation study using representative distance metrics commonly used to measure latent proximity. Table 3 summarizes the comparisons among different strategies for selecting the nearest latent tokens, including lz distance, cosine similarity, and random selection under sampling (r 1). This experiment aims to assess the role of proximity-based selection token aggregation, with & = 1000 used across all methods.

---

Published as a conference paper at ICLR 2025
Table Ablation study for latent proximity measure and probability distribution distance metries on LlamaGen-XLL Stage model under sampling (r = 1). For latent proximity measures, & distance, cosine similarity, and random selection are used, and TVD and JSD are used as probability distribution distance metrics.
Latent Proximity Measure
Distance Metric
Mean Accepted Length
FID
CLIP score
Cosine similarity (8 0.2)
2.09
17.46
0.3206
&g distance (8 = 0.2)
2.09
17.43
0.3208
& distance (6 = 0.05)
1,50
15.71
0,3203
Random (6 0.2)
1.26
15.62
0.3203
Probability Distribution Distance
Distance Metric
Mcan Accepted Length
FID
CLIP score
TVD (Ō 0.3)
2.29
18.27
0.3206
ISD (8 0.2)
2.29
18.21
0.3206
TVD (Ō = 0.2)
2.09
17.43
0.3208
JSD (6 = 0.13)
2.09
17.48
0.3206
As shown in Table 3, the random selection significantly underperforms in terms of acceleration, achieving only a 1. 1.26 mean accepted length, which is notably lower than both &g distance and cosine similarity. Additionally, the random selection shows inferior acceleration compared to the f2 distance with 8 = 0.05, while maintaining a similar FID, which highlights the importance of token selection based on latent proximity. Comparing l distance and cosine similarity, both methods demonstrate nearly identical performance in terms of mean accepted length, FID, and CLIP score, suggesting robustness and flexibility in our approach to proximity measurement. These results emphasize that selecting tokens based on latent proximity is critical for achieving balance between acceleration and image quality, offering a clear advantage over random selection. An analysis on the size of latent proximity set is presented in Appendix G.
5.3.2 DISTANCE BETWEEN PROBABILITY DISTRIBUTION
To ensure that the modified target distribution remains within an acceptable range of divergence from the original, we introduce as upper bound for distributional divergence. To validate the impact of the divergence metric, we evaluate two different metrics to measure the divergence: Total Variation Distance (TVD) and used as it is asymmetric and not a valid metric for distance in mathematical sense.
Divergence (ISD). Kullback-Leibler Divergence (KLD) is not
To compare the effectiveness of these distance metries, fix k 1000 and adjust å achieve similar mean accepted lengths across the different divergence metrics. The results shown in Table demonstrate that, although the two metrics require different for the same level acceleration (mea- sured by mean accepted length), they produce nearly identical image quality metrics at comparable acceleration levels.
These results confirm that our method consistently functions as a robust trade-off controller regardless of the chosen distance metric. Since the difference performance between two metries marginal, we opt use TVD, as it is computationally lighter and thus more efficient for large-scale implementations. The detailed analysis on the latency differences between TVD and JSD provided in Appendix F.3.
CONCLUSIONS
In this paper, we explored the application of speculative decoding to visual AR models for the first time. We revealed that the naïve application of existing methods fails due to the token selection ambiguity problem. To address this, we proposed LANTERN, a novel relaxed acceptance condition that effectively resolves this problem. Our experiments using the state-of-the-art visual AR model and speculative decoding method demonstrated that LANTERN successfully enables speculative decoding in visual AR models, achieving substantial speed-ups with minimal compromise in image generation performance. For future work, we plan to design a drafter specifically tailored to visual AR models, aiming achieve acceleration without sacrificing the generation performance.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was partly supported by Institute for Information & communications Technology Pro- (ITP)
Artificial Intelligence Graduate School Program(KAISTT, No.2022-0-0071 Mett-learning applicable to real-world problems. search Hub Project) and National Research Foundation of Korea (NRF) (No.RS-2023. 00209060, A Study on Optimization and Network Interpretation Method for Large-Scale Machine Learning) grant funded by the Korea government (MSIT). This research was supported in part by the NAVER-Intel Co-Lab. The work was conducted by KAIST and reviewed by both NAVER and Intel.
24-2024-00457882, Al Re-
REFERENCES
Zachary Ankner, Rishab Parthasarathy, Aniruddha Nrusimha, Christopher Rinard, Jonathan Ragan- Kelley, and William Brandon. Hydra: Sequentially-dependeee draft heads for medusa decoding. 2024. URL attps://arxixs org/ abs /tabbs/4400 22022440222.00 05109.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert- Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCan- dlish, Alec Radford, Iya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Ad- vances in Neural Information Processing Systems, volume 33, pp. 1877-1901. Curran Asso- ciates, Inc., 2020. URL httpctt::/pprrocceeeeeee ings neurips cc cc/paper___ppe fllfffils/pppee files/paper/ pdf.
Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D. Lee, Deming Chen, and Tri Dao. Medusa: Simple LLM inference acceleration framework with multiple decoding heads. In Forty-first Internationall Conference on Machine Learning, 2024. URL https: /openeview.. net / /t///oormmmm/////// leeerummid-PPpppeee Popood
Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean- Baptiste Lespiau, Laurent Sifre, and John Jumper. Accelerating large language model decoding with speculative sampling. arXiv preprint 1xXiv:2302.01318 2023.
Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learming Research, pp. 1691-1703. PMLR, 13-18 Jul 2020. URL https: htttts::/prrceeddngg mlr. www.ppresssvv... /ttsss/vvl///// chen20s html.
Ethan Chern, Jiadi Su, Yan Ma, and Pengfei Liu. Anole: An open, autoregressive, native large multimodal models for interleaved image-text generation, 2024. URL 06135.
Christoph Chuhmann, Andreas Köpf, Richard Vencu, Theo Coombes, and Romain Beaumont. Laion coco: 600m synthetic captions from laion2b-en, 2022. URL aattps://laiiee ai/blog/ malioncccco/ September 27th, 2024.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang. Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ros, Marie Pellat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. Scaling instruction finettnndd language models, 2022. URL https: / arxiv...ie org/ aws//aa///aa 2210 11416.
Jia Deng, Wei Dong, Richard Socher. Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hier- archical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248-255, 2009. doi: 10. W00////P.20099020000

---

Published as a conference paper at ICLR 2025
Ming Ding, Zhuoyi Yang, Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang, and Jie Tang. Cogview: Mastering text-to-image generation via trans- formers. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 19822-19835. Curran Asso- ciates, Inc., 2021. URL https: / pto.eedoooeeeeoeeeoooooooccoeeee ngs neuri ps cc cc/paper___p.ee.eeeeee wilesspaape htt.:/iieeeaaaa22eaeaeae/ T
Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 12873-12883, 2021.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley. Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural informa- tion processing systems, pp. 2672 2680, 2014.
Jonathan Heek, Emiel Hoogeboom, and Tim Salimans. Multistep consistency models, 2024. URL https: / a waa/axiiv. www.w.oogg.baa/////ooo/o/// awtpg////////// /203 06007.
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. CLIPScore:
A reference- free evaluation metrie for image captioning. In Maric-Frandn Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (cds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 7514-7528, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10. 18633v 1 1/2021 http:///22221/pp.. .emnlp-main. 595. URL https: /aaaaaaaaaanooooooaaaaaaoaooolaa wwg/oog///// emn ma main...... 595.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochre- iter. Gans trained by two time-scale update rule converge to local nash equilibrium. In Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Gar- nett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Asso- cttpp//eeeppepeee/e//eeeeeeeeeee www.ff/pepppaeeeee
ciates, Inc., 2017. URL https: httt::::///c/ccccoooooocoo www.einiiiiiii http://www...ur.......o.ooe http.//Rttttsss////rreaaeaa
8 11111169470000 htt.:f/lleeeaaaeeeeeeeeeeeeeeee2
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NeurlPS 202/ Workshop on Deep Generative Models and Downstream Applications, 2021. URL htttps://operrevieen
net/ forum? d==gwAKKxXYbX
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neu- ral Information Processing Systems, volume 33, pp. 6840-6851. Curran Associates. Inc., 2020. URL https: http:s::/p/ccceeddeeoooo wwwurip..ccppaaee htte:/ccccccccc555eeo006f1111111
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023. URL https: //axxiv. org/abs / 22100311022 06825.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes, 2022. URL https: wttarii...rg/bbs////a/aao /13////.b//311..11.... 6114.
Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models, 2019. URL htttpss//arxiv.ort Abss/904, 06991.
Doyup Lee, Chiheon Kim, Sachoon Kim, Minsu Cho, and Wook-Shin Han. Autoregressive image generation using residual quantization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11523-11532. 2022.
12

---

Published as a conference paper at ICLR 2025
Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning, Pp. 19274-19286. PMLR, 2023.
Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. Eagle-2: 2: Faster inference of language models with dynamic draft trees. arXiv preprint arXiv:2406.16858 2024a.
Yuhui Li, Fangyun Wei, Chao Zhang. and Hongyang Zhang. Eagle: Speculative sampling requires rethinking feature uncertainty, 2024b.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitmick. Microsoft coco: Common objects in context. In David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars (eds.), Computer Vision ECCV 2014, pp. 740-755, Cham, 2014. Springer International Publishing. ISBN 978-3-319-10602-1..
Hya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations, 2019. URL https: http://ppeeeeeeeeeee ew. net/ forum? didd Bkg6RiCag Cqy7.
Jiasen Lu, Christopher Clark, Sangho Lee, Zichen Zhang, Savya Khosla, Ryan Marten, Derek Hoiem, and Aniruddha Kembhavi. Unified-io 2: Scaling autoregressive multimodal models with vision, language, audio, and action. arXiv preprint arXiv:2312. 171722 2023.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1-67, 2020.
2020.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation, 2021. URL https:://ttpi......///// bbss/1000 12092.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pp. 10684-10633 10695, 2022.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural information processing systems, 35:36479-36494, 2022.
Axel Sauer, Dominik Lorenz, Andreas Blattmann, and Robin Rombach. Adversarial diffusion distillation, 2023. URL httpt: /ttpppp:://axxv..///e/ org/ abs /bss/311.........b..bbbbbbbbb.tb 2211..11122. 17042.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models, 2022, URL https: www.wiiworgg....e...........ooot wttgg//b///////// 001.2/211..0222222.
Peize Sun, Yi Jiang, Shoufa Chen, Shilong Zhang, Bingyue Peng, Ping Luo, and Zehuan Yuan. Autoregressive model beats diffusion: Llama for scalable image generation. arXiv preprint arXiv:2406..05555 2024.
Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models. arXiv preprint arXiv:2405.09818. 2024.
Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng, and Liwei Wang. Visual autoregressive modeling: Scalable image generation via next-scale prediction. arXiv preprint a7X4v:240.002905 2024.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie- Anne Lachaux, Timothée Lacroix. Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023. URL httpt:ttppeecm //t/axxivvv org/ abs 22002 13971.
Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
13

---

Published as a conference paper at ICLR 2025
Xiaoshi Wu, Yiming Hao, Keqiang Sun, Yixiong Chen, Feng Zhu, Rui Zhao, and Hongsheng Li. Human preference score v2: A solid benchmark for evaluating human preferences of text-to-image synthesis. 2023. URL https: / arxiv. org/ abs/ ////2335 22066 09341.
Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang, Ruoming Pang, James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge, and Yonghui Wu. Vector-quantized image modeling with improved vegan. arXiv preprint arXiv:2110.04627 2021.
Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, et al. Scaling autoregressive models for content- rich text-to-image generation. arXiv preprin! arXiv:2206. 10789, 2(3):5, 2022.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595-46623, 2023.
Yongchao Zhou, Kaifeng Lyu, Ankit Singh Rawat, Aditya Krishna Menon, Afshin Rostamizadch, San- jiv Kumar, Jean--aanaaaaaaeeee FranFFrnçoos Kagy, and Rishabh Agarwal. Distillspec: Improving speculative decoding via knowledge distillation. In The Twelfth International Conference on Learning Representations.
GK Zipf. The psycho-biology of language: introduction dynamic philology. 1935.
14

---

Published as a conference paper at ICLR 2025
APPENDIX
A RELATED WORKS
Visual auto-regressive models The development of auto-regressive (AR) models in language modeling (Brown et al., 2020; Touvron et al., 2023; Jiang et al., 2023) has paved the way for AR- based approaches in generative vision tasks, where images are tokenized into discrete tokens on a 2D grid and processed as a unidirectional sequence. Early studies (Esser et al., 2021; Yu et al., 2021; Lee et al., 2022) explored different sequence orders such as row-major raster seans, spirals, and z-curves for image generation. However, these approaches often suffer from computational inefficiencies and underperform compared to diffusion models (Ho et al., 2020; Song et al., 2022; Rombach et al., 2022).
Building upon fundamental mechanism of visual AR models from DALL-E (Ramesh et al., 2021) and CogView (Ding al., 2021), where images discretized into tokens using vector-quantized VAEs (VQVAE) (Van Den Oord et al., 2017) or VQGANs (Esser et al., 2021) for next-token prediction, recently, visual AR models have emerged strong competitors to diffusion models. For instance, Parti (Yu et al., 2022) and LlamaGen (Sun et al., 2024) utilize a verbose encoder-decoder architecture that incorporates frozen T5 (Raffel al., 2020) text features via cross-attention or prefix- filling methods, drawing on insights from Imagen (Saharia et al., 2022). More recently, Chameleon (Team, 2024) has proposed a unified AR framework that enables fully token- representations for both image and text modalities. Additionally, model like Anole (Chern et al., 2024), which build on Chameleon, further enhance image generation capabilities, demonstrating versatility across diverse tasks. In this work, we conduct experiments on state-of-the-ar visual AR Laamaa- and Anole-for for text-to-image generation to validate the effectiveness of our acceleration scheme.
Speculative decoding for AR models The foundational concept of speculative decoding first introduced by Leviathan et al. (2023) to address the sequential constraints of the AR framework in language modeling. Unlike the standard AR modeling, which generates one token per forward pass, speculative decoding aims to generate multiple tokens in a single forward pass by speculating a series of tokens, referred to as draft tokens. To generate draft tokens efficiently, most speculative decoding methods rely on separate, typically smaller, model called the drafter model. Since generating draft tokens with the drafter model is much faster than the target model, this approach can reduce the overall token generation time.
In Leviathan al. (2023), the drafter model a smaller version within the same architecture family, trained on similar data and objectives as the target model. An alternative approach, DistillSpec (Zhou et al.), uses knowledge distillation to develop efficient drafter models. Recognizing that the latency of generating draft tokens is crucial for accelerating, recent works have explored lightweight designs for drafter models. Medusa (Cai et al., 2024), for instance, employs multiple separate language model heads as drafter models instead of fully independent models and leverages tree drafting and decoding to enhance the chance of accepting draft tokens. Hydra (Ankner et al., 2024) builds on Medusa by incorporating sequential dependency, yielding additional acceleration benefits. EAGLE (Li et al., 2024b;a) extends these approaches by introducing feature uncertainty in drafting and utilizing single decoder layer as drafter models incorporate better sequential dependency. Furthermore, they have achieved state-of-the art performance in speculative decoding for language modeling by dynamically refining the tree structure for tree drafting.
15

---

Published as a conference paper at ICLR 2025
B EXPERIMENTAL DETAILS
B.1 EXPERIMENTS FOR MOTIVATING EXAMPLES
For the experiment on naive application shown in Figure 2(a), we utilize LlamaGen-L (Sun al., 2024), an MLP with two linear layers, and a single decoder layer as the drafters for Speculative Decoding (Leviathan et al., 2023), Medusa (Cai al., 2024), and EAGLE-2 (Li et al., 2024a), respectively. To facilitate Speculative Decoding, which requires a smaller-sized model with the same architecture as the main model and trained on the same dataset, we employ class-conditioned LlamaGen-33 model instead of text-conditioned models. This is because text-conditioned models do not provide multiple model sizes. For each ImageNet class (Deng et al., 2009), we generate 100 images to measure the mean accepted length. Results Vicuna-7B (Zheng et al., 2023) taken directly from EAGLE-2.
The drafter test accuracies presented in Figure 2(b) correspond to the test accuracies of the drafters used in our main experiments (Table 2). Details on the training process for these drafters are provided in the following section. Additionally, the test accuracies of the learned drafter for Vicuna- sourced from EAGLE-2.
For the experiments average top-1 and top-10 probabilities shown in Figure 2(c), generate 80 responses from Vicuna-7B on the MT-Bench dataset (Zheng et al., 2023) and 1000 images from LlamaGen-Xe and Anole based on MS-COCO validation captions (Lin al., 2014).
B.2 EXPERIMENTS FOR MAIN RESULTS
To train the text-conditional model's drafter, sampled 100k images in LAION-COCO dataset (Chuhmann et al., 2022), which is used to train Stage I target model. We used the same amount of image sampled in ImageNet (Deng et al., 2009) dataset to train the class-conditional model's drafter. For Anole (Chern et al., 2024), we use 118K images sampled with target model by MS-COCC (Lin et al., 2014) train caption train drafter. We used a single-layer decoder with the same structure as the target model in the same manner as EAGLE. During training, 5% of data is set to be held out validation dataset.
Since LlamaGen (Sun et al., 2024) and Anole (Chern et al., 2024) use classifier-free guidance (Ho & Salimans, 2021) to generate images, we trained our drafter both learn conditioned input and null-conditioned input. To do so, we dropped 10% of conditional embedding during training, as same as target model training. The batch size 16, and the base learning rate is 10-4, Adam (Loshchilov & Hutter, 2019) optimizer with 31 = 0.9 and B2 0.95 is used, and Linear learning rate scheduling with warm-up is used with 2000 warm-up steps. We select the best-performing model terms of top-3 accuracy in the hold-out validation set for 20 epochs. In addition, Flan-T5 XL (Chung et al., 2022) is used to encode input text for text-conditional generation.
For LlamaGin (Sun et al., 2024) stage I and stage II, images are generated using a classifier-free guidance scale of 7.5 with top-p set to 1.0 and top-k set to 1000, which the default generation configuration of LlamaGen official implementation for text-conditional image generation. For class-condes coss-conditional generation, the classifier free guidance scale is set to 4.0, with the top-k sampling covering the entire vocabulary and the top-p sampling use a classifier-free free guidance scale of 3.0 with with top-k as 2000. For EAGLE-2 and our method, 60 candidate tokens are passed into the target model for each verification process.
1.0. For Anole (Chern et al., 2024),
16

---

Published as a conference paper at ICLR 2025
C DETAILED ANALYSIS ON LATENT PROXIMITY
C.I STATISTICAL ANALYSIS LATENT PROXIMITY
Table 4: Impact of replacing tokens with one of the k-th nearest tokens image quality. FID and CLIP Score indicate degradation in image quality as increases.
Randomly Replaced by one k-th nearest token
FID
CLIP Score
Vanilla AR
25.06
0.3214
k 50
26.88
0.3120
k 100
30.76
0.3091
k = 1000
88.03
0.2715
Table 4 provides statistical evidence supporting our earlier qualitative observations (Figure 3) that token replacement does not lead to significant degradation in image quality, particularly for smaller values of k. This experiment was conducted using the LlamaGen-XL Stage model (Sun et al., 2024) and MS-COCO 2017 validation captions (Lin al., 20,4), with FID and CLIP Score used as evaluation metrics.
As increases, the replaced token is chosen from a larger set of latent space neighbors, and its impact on image quality becomes more evident. For k 50, the FID increases slightly from 25.06 (Vanilla AR) to 26.88, and the CLIP Score decreases marginally from 0.3214 to 0.3120, indicating minimal degradation. Similarly, for k 100, the FID increases to 30.76, and the CLIP Score drops to 0.3091, showing that even with & = 100, the image quality remains relatively stable and acceptable. However, for A 1000, substantial decline is observed, with the FID increasing sharply to 88.03 and the CLIP Score dropping to 0.2715, underscoring the negative impact of replacing tokens with more distant neighbors.
These results corroborate our earlier qualitative findings, showing that token replacement up to k 100 maintains reasonable image quality, making it viable approach in generative tasks. This demonstrates the robustness of the model under controlled token replacement scenarios.
C.2 COMPARISON OF LATENT PROXIMITY IN VISUAL AND LANGUAGE AR MODELS
Trp'ss Gotled SiAN
Top'ss
(a) A big teddy bear on black motorcycle.
(b) A red and white train traveling along a mountain road.
Figure 6: Visualization of top-5 nearest token and top-5 farthest token from original token in LlamaGen-Xe Stage II model (Sun et al., 2024).
To achieve deeper understanding of latent proximity, conducted additional analyses to examine the tendencies nearest tokens in visual AR models and language models using LlamaGen-XL. Stage II (Sun et al., 2024) and Vicuna-7B (Zheng et al., 2023). For the visual AR model, we visualized the five nearest tokens and the five farthest tokens based on their L22 distances in the latent space, as shown in Figure 6. However, for the language model, since each token corresponds to subwords, it is challenging to measure proximity directly using the token itself. Therefore, we conducted the same analysis using the & distance of the input embeddings, with the results presented in Table 5.
17

---

Published as a conference paper at ICLR 2025
Table 5: Top-5 nearest and farthest tokens from the original tokens based on &S distance of input embeddings in Vicuna-7B (Zheng et al., 2023). [OBJ] indicates the Unicode object replacement character.
Token: hi
Token: act
Top-k
Nearest
Farthest An
Nearest
Farthest
Poortáy
Mediabestanden oreferrer
[OBJ]
Mediabestandeen
[OBJ]
Bruno
3
<0x4C>
lnffnnttll
<0x24>
Ernst
<0x6B>
firewall
<0x71>
Santos
<0x49>
_sooner
<0x54>
firewall
This result indicates that in the visual AR model, tokens with high latent proximity are decoded into image patches with similar visual appearances, whereas tokens with low latent proximity correspond to image patches with distinctly different appearances. On the other hand, in the language model, the similarity based on the input embeddings did not reveal clear relationship between the nearest and farthest tokens. These results support the conclusion that latent proximity serves as an effective metric for identifying visually similar tokens in visual AR models.
D ALGORITHMS
D.1 SPECULATIVE DECODING WITH LANTERN
Algorithm 1 LANTERN
1: Input: Target model q(-1.), draft model p(-l-), initial sequence 2p: 2t, drafted sequence length I., minimum target sequence length 7, Drv tolerance 5 >> 0, and maximum cardinality of latent neighborhood k.
2: Initialize: n1 + t.
3: while n T' do
4: for - 1.... L do
5: Sample draft autoregressively ūe p(x|xo..... ................................ Ini T1....,, .(-1)
6: end for
7: In parallel, compute + sets of logits from drafts TI,...,
g0 ... nn g(0/200 ... Fn. Fi), g(0/200 ... Fny Fly...,
8:
for = 1.... ...
L do Find the neighborhood (T+)
9:
10:
Sample UO. from uniform distribution.
Σ=EAL g( ((el=o....o+t11)) then
11:
ifr
min
plua 20t1:))
12:
Set Unit + Tz and n + n n
13:
else
14:
Sample _n-t
Inte-1. = Zx) - p(zl/o...... -. xn+(-1)) and
exit the loop
15:
end if end for
17:
If all drafts are accepted, sample an extra token E.41+1 ~ q(x/oo Fn+1).
18:
end while
19: Output: Entlirer Intr. Intly In4L+1
18

---

Published as a conference paper at ICLR 2025
D.2 PROXIMITY SET CONSTRUCTION
Algorithm 2 Proximity Set Construction for LANTERN
Require: Latent space representation of tokens, number of neighbors DrV tolerance 6
1: Precompute Bg (?) for all tokens:
2: for each token ž in the quantized latent space do
3: Compute distances between T and all other tokens.
4: Identify nearest tokens (including itself) based on & distance.
5: Store these tokens as the set (2).
6: end for
7: Dynamically calculate (z) during inference:
8: for each token sampled during decoding do
9: Initialize 5 + 0.
10: Initialize cumulative TVD, DrV
11:
for each token x E B3: (in increasing distance order) do
12:
Compute potential TVD increment ADrv for adding x to A (T).
if DTv + ADTV < then
14:
Add x to (T,s(T).
Update Drv Drv + ADrv.
16:
else Break.
18:
end if
end for
20:
end for
21:
Return proximity sets By (precomputed) and Ax,s (dynamically calculated).
E
EXPERIMENTAL RESULTS ON OTHER VISUAL AR MODELS
Table Actual speed-up, MAL (Mean Accepted Length), FID, CLIP score, Precision Recall, and HPS v2 for each method on LlamaGen-XL Stage II and Anole.
Acceleration (1)
Image Quality Metrics
Method
Speed-up
MAL
FID (4)
CLIP score (1)
Prec (1)
Rec (1)
HPS v2
TlamaGen-XI XT. Stage II
1.00x
1.00
47.60
0.2939
0.4138
0.5648
23.84
EAGLE-2 (Li al., 2024a)
0.96x
1.22
LANTERN 0.4, 1000)
1.64x
2,24
46.10
0.2925
0.4704
0.5222
23.06
Anole
1.00x
1.00
20.27
0.3215
0.6552
0.6398
23.52
EAGLE-2
0.73x
1.10
LANTERN 0.5, k 100)
1.17x
1.83
23.40
0.3186
0.6026
0.6178
22.92
Table 6 presents additional results comparing LANTERN EAGLE-2 and standard decoding LlamaGen-XI Stage II (Sun et al., 2024) and Anole (Chern et al., 2024). The table reports both actual speed-up (measured on RTX 3090 for LlamaGen and A100 80GB SXM for Anole) and image quality metrics, including FID, CLIP score, Precision/Recall, and HPS v2.
As shown in the Table 6, LANTERN consistently achieves superior acceleration compared to EAGLE-2 across multiple visual AR models, including LlamaGen- XL Stage II and Anole. While the performance of LANTERN varies slightly depending on the model. consistently outperforms EAGLEE22 in terms of both speed-up and mean accepted length. For instance, on the Lla Stage II model, LANTERN achieves a speed-up of 1.64 and mean accepted length of 2.24, significantly higher than EAGLE-2' EE22s 0.96 x speed-up and 1.22 mean accepted length. Similarly, for Anole, LANTERN achieves a speed-up of 1.17x with a mean accepted length of 1.83, compared to EAGLE-2's 0. x.73x speed-up up and 1.10 mean accepted length.
In terms of image quality, LANTERN demonstrates minimal degradation despite the improved acceleration. For the LlamaGen-XX GLL Stage II model, the FID decreases slightly 46.10 compared to the baseline of .6, while the CLIP score remains stable. Precision and recall metrics indicate that LANTERN maintains balance between individual image quality (precision) and diversity (recall).
19

---

Published as a conference paper at ICLR 2025
On Anole, LANTERN similarly exhibits competitive image quality metrics, with CLIP score of 0.3186 and a moderate decrease in HPS v2.
These results highlight the versatility of LANTERN, showcasing its ability to provide significant ac- celeration benefits while maintaining acceptable image quality across diverse models. This reinforces LANTERN's effectiveness practical and efficient approach for visual AR models.
F DETAILED LATENCY ANALYSIS FOR LANTERN
F.1 ANALYSIS ON THE NUMBER OF CAPTIONS
Table Actual speedup results for LANTERN across different settings of r, k, and 8, with varying numbers of captions.
Num Captions
Actual Speed-up
Actual Speed-up
Actual Speed-up
Actual Speed-up
k 1000, 0.05)
[T [r=00,k
1030, 6 0.2)
(r 1, 1000,
0.1)
1,k 1000, 6 0.4)
100
1.56x
2.33x
1.13x
1.73x
1000
1.56x
2.26x
1.13x
1.69x
2000
1,57x
2,27
1.13x
1.69x
5000
1,56x
2.26x
1.13x
1.69x
Table provides a statistical analysis of actual speed-up measurements conducted with different numbers of captions for MS-COCO validation captions (Lin al., 2014) LlamaGen-XL Stage I model (Sun et al., 2024). Across all configurations of т, k and 6, the actual speed-ups remain consistent beyond 1000 captions, with negligible differences observed. For example, for = 0, k 1000, = 0.05, the speed-up values remain between 1.56x and 1.57 across 1000, 2000, and 5000 captions. Similarly, other configurations also exhibit minimal variation, confirming the reliability of using 1000 captions as a sample size.
This statistical stability justifies our choice use 1000 captions in the main experiments, as provides an accurate and computationally efficient estimate of actual speed-up without sacrificing reliability.
F.2 COMPONENT-LEVEL LATENCY ANALYSIS FOR LANTERN
Table 8: Component-level latency analysis for LANTERN. Each component's 's latency is averaged over single draft-and-verijy process, with measurements conducted on a single RTX 3090 GPU.
Method
Target Forward
Drafter Forward
Proximity Set A Calculation
LANTERN(Ó = 0.1, k = 1000)
1.57 x 10-3s 1.19 10
3.80 10-2s 2s
1.08 10-#s
LANTERN(Ó 0.4, k 1000)
In this section, conducted experiments evaluate the contribution of each component of LANTERN to overall latency. Specifically, we identified three primary components that signifi- cantly impact latency: (I) the target model forward pass, (2) the drafter model forward pass, and (3) the computation of the proximity set A. Using the LlamaGen-XI Stage I model, we measured the average latency of each component during single draft-and draft-and-verile and-verify process across 1,000 prompts from MS-COCO validation captions (Lin et al., 2014). Experiments is conducted under two settings, = 1000, 8 0.1 and 1000, 0.4, with the results summarized in Table
Among the three key contributors to latency. the proximity set computation, which is introduced by LANTERN, required more time lower values. This can be attributed the increased number of rejections lower ö, necessitating multiple proximity set computations within single step. However, even the & = 1000, 8 = 0.4 setting, the latency of proximity set computation is 24x smaller than that of the target model forward passes and about 7x smaller than the drafter model forward pass.
Overall, while LANTERN introduces additional computational overhead compared to traditional speculative decoding methods, the trade-off results in significantly greater speed-up. Thus, approach demonstrates superior actual speed-up, establishing its advantage over existing speculative decoding methods.
20

---

Published as a conference paper at ICLR 2025
F.3 LATENCY COMPARISON ON PROBABILITY DISTANCE METRICS
Table 9: Computation time comparison between TVD and JSD in LANTERN. Each component's latency is averaged over single
verify process, with measurements conducted on a single RTX 3090 GPU. The hyperparameters for LANTERN are fixed to i 1000, and 8 0.4 for TVD and 8 = 0.2 for JSD.
Distance Metric
Computation Time for Distance
Total Computation Time
TVD JSD
1.19 x 10 2s
4.89 10 10-2s 1.92 x 10 2s
1.03 x 10 8s
Table 9 provides comparison of computation times between TVD) and JSD, supporting our decision to use TVD as the preferred distance metric. We use LlamaGen-XX Stage with randomly sampled 1000 captions in MS-COCO validation captions (Lin et al., 2014) for this experiment. While both metrics produce nearly identical results in terms of mean accepted length and image quality, TVD is significantly more computationally efficient. Specifically, the computation time for TVD 1.19 10-3 seconds, which is approximately three times faster than JSD's 's computation time of 4.03 x 10-3 seconds.
Although the total computation time for each decoding step (including other processes) shows smaller difference, 4.89 x 10-2 seconds for TVD and 4.92 10-2 seconds for ISD, this difference accumulates over multiple decoding steps. For large-scale applications or models generating long sequences, this efficiency advantage becomes increasingly significant. Therefore, given the negligible impact quality metrics and the consistent computational advantage, we choose TVD over JSD as the more practical and efficient metric for our method.
Note that the hyperparameters do not affect the computation time for the distance metric itself but do impact the total computation time. Therefore, we set the values of 8 for TVD and JSD to achieve the same level of mean accepted length.
F.4 LATENCY RESULTS ON INTEL GAUDI ACCELERATOR
Table 10: Actual speed-up. MAL (Mean Accepted Length) for each method. r 0 refers to the greedy decoding and t refers to the sampling with temperature for generation. The actual speed-up are measured single Intel Gaudi 2 (96GB) accelerator and NVIDIA RTX 3090.
7=0
Method
Acceleration (*)
Speed-up up (Intel Gaudi 2)
Speed-up (NVIDIA RTX 3090)
MAL
LlamaGen-Xe -L Stage
1.00x
1.00x
1.00
EAGLE-2
0.80x
1.29
1.60
LANTERN 0.05, Io 1000)
1.01
1,56x
2.02
LANTERN 0.2, ks 1000)
1.32x
2.26
2.89
Method
Acceleration (*)
Speed-up (Intel Gaudi 2)
Speed-up (NVIDIA RTX 3090)
MAL.
LlamaGen-Xe Stage
1.00
1.00x
1.00
EAGLE-2
0.55x
0.93
1.20
LANTERN (8 0.05. A: 1000)
0.74x
1.13x
1.75
LANTERN (5 = 0.2, & = 1000)
1.02x
1.69
2.40
To demonstrate the adaptability of LANTERN across different hardware platforms, we extend our evaluation beyond NVIDIA GPUs to include the Intel Gaudi 2 accelerator (96GB). Table 10 presents the actual speed-up and Mean Accepted Length (MAL) for different methods, comparing performance both Intel Gaudi 2 and NVIDIA RTX 3090. While the primary results in the main paper focus on LUDA-based hardware, this analysis highlights that LANTERN can effectively accelerate auto-regressive generation Gaudi 2 as well.
21

---

Published as a conference paper at ICLR 2025
The observed difference in speed-up between Gaudi 2 and NVIDIA GPUs is not indicative of general performance gap but rather reflects the unique computational characteristics of speculative decoding. Unlike standard inference and training workloads, where Gaudi 2 has demonstrated competitive performance, speculative decoding introduces irregular execution patterns, including interleaving model forwards and non-trivial memory access patterns. These factors make speculative decoding particularly sensitive to low-level implementation details, which can vary across different accelerator architectures.
Despite the absence of Gaudi-specific optimizations in our implementation, LANTERN still achieves meaningful acceleration on this hardware. While EAGLE-2 achieves only 0.80x speed-up at and 0.55x x at 1, LANTERN improves these figures to x.32x and 1.02x, respectively. This demonstrates that speculative decoding can be effectively deployed on Gaudi and that the core principles of LANTERN remain beneficial across diverse hardware platforms.
These results suggest that Gaudi 2 is well-suited for auto-regressive generation, including techniques such speculative decoding. While hardware-speciffc optimizations could further enhance perfor- mance, the existing results already indicate that speculative decoding can be efficiently executed on Gaudi 2. As speculative decoding continues to gain traction in large-scale model deployment, further refinement of its implementation across different accelerator architectures will help maximize its benefits in diverse hardware environments.
G SIZE OF LATENT PROXIMITY SETS ACROSS POSITIONS
Anarge Sum Prolrsty Salik..
Anarge Prainnsy SetAA.
(a) 8 = 0.1, A: 1000
(b) 8 0.4,
1000
Figure 7: Average size of A&,5 on different position. Note that first token is generated right after pre-fill stage, we do not conduct speculation for first token.
We conduct experiments to better understand the behavior of LANTERN by examining how the size of the latent proximity set A varies depending on the position in images. For 1000 and 5 = 0.1, 0.4, we generate 100 images based on MS-COCO validation captions (Lin et al., 2014) with the LlamaGen-XI.. Stage model for each setting. The calculated average size of on the different positions can be found in Figure 7.
As expected, larger values generally led to a larger size of regardless position. Across various 5 values, the size of the proximity set A is consistently larger at the left edge of the image. We hypothesize that this phenomenon occurs because the left edge of the image corresponds to positions immediately following line break, where uncertainty is higher compared other positions. This increased uncertainty results in generally lower probabilities assigned to individual tokens, allowing larger number of tokens meet the threshold and be included in the set A.
22

---

Published as a conference paper at ICLR 2025
Table 11: Precision and recall (Kynkäänniemi et al., 2019) values for varying ò and l settings. The vanilla AR decoding score is also provided for reference.
Precision / Recall
8 0.05
0 0.1
8 0.2
8 0.4
Vanilla AR
0.4232 / 0.3517
k 100
0,4424/0.32/
0,4510 0.2998
0,4661 / 0.2754
0,4689 / 0.2881
k 300
0,4488 / 0.3195
0,4619 0.2960
0,4696 / 0.2802
0,4750 / 0.2753
k 1000
0.4484 0.3158
0.4659 / 0.2939
0.4771 / 0.2773
0.4854 / 0.2682
T 1
Precision Recall
8 0.05
8 0.1
0.2
0.4
Vanilla AR
0.4781 / 0.5633
k: 100
0.4867 / 0.5389
0.4796
0.5303
0.4789 / 0.5140
0.4825
0.4946
A 300
0.4856 / 0.5367
0.4834 0.5231
0.4894 / 0.4901
0.4895 / 0.4719
k: 1000
0.4865 0.5334
0.4869 0.5172
0.4880 / 0.4888
0.4909 / 0.4497
Table 12: HPS v2 (Wu et al., 2023) score for different 8 and & settings. The baseline score for Vanilla AR decoding is provided for reference.
HPS v2
==
0.05
ö == 0.1
0.2
0.4
Vanilla AR
23.18
A: 100
22.73
22.41
22.13
21.96
À 300
22.66
22.25
21.92
21.69
& 1000
22.62
22.14
21.69
21.39
HPS v2
= 0.05
0.1
= 0.2
= 0.4
Vanilla AR
24.11
k 100
24.01
23.94
23.86
23.75
300
23.97
23.85
23.70
23.55
& = 1000
23.91
23.75
23.47
23.22
H TRADE-OFFS WITH ADDITIONAL METRICS
To gain a deeper understanding of the trade-offs in LANTERN, we examined precision & re- call (Kynkäänniemi al., 2019), and HPS v2 (Wu al., 2023) in addition to FID (Heusel al., 2017), and the results are summarized in Table 11 and 12. The experimental settings are identical to the main experiments in Table 2 and Figure 5.
The result in Table 11 shows that LANTERN achieves comparable or slightly improved precision relative the baseline across various settings, reflecting its ability maintain high-quality token generation. While recall decreases marginally with increasing 6, this is an expected trade-off due the relaxed acceptance condition, which prioritizes sampling speed. The precision/recall results demonstrate that our method strikes a reasonable balance between quality and diversity across different hyperparameter configurations.
To further quantify the aesthetic quality of generated images, we evaluate our approach using HPS v2. Table 12 shows that while there is a slight reduction in aesthetic quality compared to the baseline, the trade-off is well-justified by the significant improvements in generation speed. This aligns with the intended design of LANTERN, which emphasizes efficiency while preserving acceptable quality.
23

---

Published as a conference paper at ICLR 2025
I TEXT PROMPTS FOR QUALITATIVE SAMPLES
The prompts listed below were used to generate Figure 4, with the images in the figure generated sequentially from left right using Prompt I through Prompt 9.
Prompt
serene lake reflecting the colors of a sunset sky with distant mountains and few birds flying across the sky.
Prompt
cozy bedroom with soft, warm lighting, a bed with fluffy pillows, a small bookshelf filled with books, and potted plant the windowsill.
Prompt
beautiful stained glass window with sunlight streaming through a church, casting colorful patterns on the stone floor and pews.
Prompt
close-up of single pink rose in full bloom, with soft, layered petals and gentle sunlight illuminating its delicate curves.
Prompt
classic still life a bowl fresh fruit, including apples, oranges, and grapes, with light softly highlighting the textures of each fruit.
Prompt
high-fashion portrait of woman with vibrant, artistic makeup and bold accessories, wearing modern, avant-garde outfit against plain studio background.
Prompt
close-up of man gazing thoughtfully of rain-covered window, with soft reflections and water droplets creating a melancholic, introspective atmosphere.
Prompt
close-up of small bird perched snow-covered branch, with soft, fluffy feathers and delicate beak, illuminated by gentle winter sunlight.
Prompt
A close-up of wolf with intense, focused eyes and thick gray fur, staring directly at the camera, set against blurred forest background.
24

---

Published as a conference paper at ICLR 2025
J QUALITATIVE RESULTS
Vanilla
300
1000
8 0.05
8 0.2
0.4
Figure 8: Qualitative sample for the changes in the generated images according to various 5 (0.05, 0.1, 0.2, 0.4) and k (100, 300, 1000) at 1. Input prompt kitchen with refrigerator, stove and oven with cabinets'. Target model is LlamaGen-XI. Stage I.
25

---

Published as a conference paper at ICLR 2025
Vanilla
100
= 300
k 1000
8 = 0.05
8 = 0.2
0.4
Figure 9: Qualitative sample for the changes in the generated images according to various 8 (0.05, 0.2, 0.4) and & E {100, 300, 1000) at r 0. The input prompt is 'A motorcycle parked in parking space next to another motorcycle.', The target model is LlamaGen-XL Stage I.
26

---

Published as a conference paper at ICLR 2025
Vordo
Figure 10: Additional qualitative samples with various and 8 at = The target model is LlamaGen-XL Stage and MS-COCO validation captions are used. Images within the same column are generated using the same text prompt.
27

---

Published as a conference paper at ICLR 2025
Wordo
2006
DIBERTOn!
Figure 11: Additional qualitative samples with various à: and 5 at r = 0. The target model is LlamaGen-XL Stage and MS-COCO validation captions are used. Images within the same column are generated using the same text prompt.
28

---

Published as a conference paper at ICLR 2025
Vanilla
k 100
k 300
= 1000
8 0.05
8 == 0.1
8 = 0.2
0.4
Figure 12: Qualitative sample for the changes in the generated images according to various (0.05, 0.1, 0.2, 0.4} and & {100, 300, 1000} crates topped with yellow bananas." The target model is LlamaGen- -L Stage II.
1. The input prompt is 'A pile of oranges in
29

---

Published as a conference paper at ICLR 2025
Vanda
Scm 2000
2-24
Figure 13: Additional qualitative samples with various and 8 at r = The target model is LlamaGen-XL Stage II, and MS-COCO validation captions are used, Images within the same column are generated using the same text prompt.
30


Reviews:
[{"review_id": "zecsjaMhVL", "reviewer": "Reviewer_v6x3", "summary": "In this paper, the authors propose LANTERN, a sampling strategy to speed up the image generation without losing too much quality. The method is implemented by accumulating probabilities from nearby tokens of the current sampling token. The authors further propose a thresholding technique to prevent the accumulated distribution deviating too much from the original one. Experiment results seem to be effective, however, the analysis is intuitive and empirical without deep insights.", "questions": "- See the weakness.\n- The claim of interchangeability in Section 3.1 is only empirically explored via a few examples. Are there some theoretical insights or statistics supporting this claim?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The proposed method is intuitive and easy to follow.\n\n- Experimental results seem to verify the proposed method in terms of speeding up image generation without significant quality loss.", "weaknesses": "- There is a gap between the introduction and methodologies in writing. The related works are deferred to the appendix and problem definition and preliminaries such as Speculative Decoding are missing, resulting in difficulties in understanding the problem and challenge for readers who are not exactly working on this domain. On the other hand, Section 2.1 and 2.2 have some overlaps about the experiments and observation, which can be more concise.\n\n- The evaluation seems to be insufficient. For the testing data, it is ideal to use the same setting as the vanilla baseline (i.e., LlamaGen) rather than just 100 captions from MSCOCO. The statement “Since measuring speedup with more than 100 samples shows no significant difference, we use 100 captions for efficiency” is not very convincing to me. In addition, other models are suggested for evaluation as well including other variations of LlamaGen.\n\n- Some analyses in experiments are not sufficient. In sampling, the authors only mentioned that when $\\delta$ is small, using larger k results in speeding up without significant degradation in performance, yet the reason for it is less explored. Section 4.3.2 also lacks further analysis why using TVD is better than JSD except some empirical results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "y38AWFxofP", "reviewer": "Reviewer_56i9", "summary": "The work presents a novel approach to enhance the efficiency of image generation using Auto-Regressive (AR) models, which traditionally suffer from slow sequential processing. The authors introduce LANTERN, a method that leverages speculative decoding with a relaxed acceptance condition to significantly speed up inference while maintaining image quality. By utilizing a smaller drafter model to predict multiple tokens simultaneously, LANTERN addresses the challenges of token selection ambiguity inherent in visual AR models. The results demonstrate notable improvements in speed and efficiency compared to baseline methods, highlighting the potential of LANTERN to advance the capabilities of AR models in generating high-quality images.", "questions": "Please see the weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The methodology of this paper is clear and comprehensible, and the research question it addresses is interesting. Although the overall approach is relatively straightforward, I believe the findings are still valuable.", "weaknesses": "1. My primary concern is that the paper lacks an evaluation of the image generation quality. The authors present several generated images, but these images are stylistically very similar and exhibit poor consistency with the accompanying generated text. Some images even contain clear visual errors. I recommend that the authors conduct a more comprehensive assessment of image generation quality to better demonstrate the effectiveness of the proposed method.\n\n2. The problem the paper aims to address is the low Mean Accepted Length that occurs when applying speculative decoding to AR image generation models. Specifically, although speculative decoding is used, a large number of predictions generated by the lightweight model are rejected, forcing the larger model to regenerate many tokens. As a result, the expected efficiency gains from speculative decoding are not realized. While the authors present a complete exploration from two perspectives, I still feel that, given the inherent ambiguity in token selection, improving the acceptance rate alone might suffice. A large set of tokens could be considered acceptable. This raises the question: Is speculative decoding still necessary in such a case? What would the quality and efficiency be if tokens were instead randomly generated within certain constraints?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rokszdMqQj", "reviewer": "Reviewer_Axtn", "summary": "This paper proposed speculative decoding for AR Image generation, AR models tend to assign uniformly low probabilities across a wide range of tokens, making it difficult to select the most appropriate token during decoding.\n\nTo solve this problem, the author introduce LANETERN (Latent Neighbor Token Acceptance Relaxation), leveraging the interchangeability of the tokens and relaxing the acceptance for decoding. \n\nThe benefits of LANETREN can be found at generating speed with comparable quality.", "questions": "see weakness. \n\nIn summary, the author presents a promising sampling scheme for AR models, which improves sampling speed at the cost of some image quality.\n\nThe primary concern is that the author’s experiments lack sufficient evidence to demonstrate that the loss in image quality is within an acceptable range. Given this limitation in quality, I believe the impact of the work may be constrained. Therefore, I am inclined to recommend rejection.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The motivation behind the author’s proposed approach is straightforward: in standard sampling processes, an excessively high token rejection rate can significantly slow down generation. The author suggests a more relaxed policy to mitigate this issue, thereby improving generation speed.\n\n2. To mitigate the distributional shift introduced by the lenient sampling approach, the author proposes a constraint rule for sampling based on total variation.\n\n3. Through extensive ablation experiments, the author demonstrates how the lenient sampling scheme influences sampling speed and elucidates the specific roles of various hyperparameters within this scheme.", "weaknesses": "My main concern lies in whether the metrics employed in the experimental section sufficiently capture the effectiveness of the proposed approach, particularly in Table 3 and Section 4.2.3. For evaluating basic generation quality, would it be possible to provide precision and recall (P/R) instead of FID? This change could allow for a clearer view of how LANTERN and limited distribution divergence affect and restore distribution. Additionally, since LlamaGen is used as the base model, could the authors include some modern image quality scores, such as PickScore or HPS, to better quantify aesthetic quality loss?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2qL4bZyZA8", "reviewer": "Reviewer_NFR2", "summary": "This paper tackles the challenge of transferring speculative decoding from LLMs to autoregressive image generation models. The authors observe that the nature of image data causes image tokens to exhibit selection ambiguity, resulting in high rejection rates and poor acceleration effects in speculative decoding. Hence  they propose using the proximity set of image tokens as proxies to speculate on accepting a predicted image token, and introduce a combinatorial optimization strategy for selecting the set to ensure image quality remains intact.  Extensive experiments confirm significant acceleration with minimal quality loss.", "questions": "1.There's a suggestion that the authors could provide the details of \"Find the neighborhood\" in Appendix A.\n\n2.A suggestion is (since the experimental section of the current paper is already sufficient, this is just a gentle suggestion) that the authors could discuss more about the differences in token ambiguity between text generation and image generation. For example, in LLaMAGen, what common properties do tokens within the same set have? What are the patterns in the sizes of sets A and B corresponding to different image tokens? These discussions could help in designing further algorithms.\n\n3.Could validation experiments be conducted on other AR models? This is also a optional suggestion aimed at enhancing the generalization and robustness of the method.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper's structure and presentation are well-crafted. It identifies the phenomenon of \"token ambiguity\" and provides a viable solution. Extensive experiments, particularly the ablation studies, demonstrate the effectiveness and scalability of the method. It offers valuable insights for research on inference acceleration in AR text-to-image generation. Although the idea of using a set to proxy a single point has been extensively researched, the authors have elegantly applied it to the sampling decision process and achieved excellent results. \n Simplicity yet effectiveness is crucial.", "weaknesses": "1.For a specific image token, the algorithm for finding its proximity set , specifically \"Find the neighborhood (i.e., Appendix B, Line 9),\" lacks detailed discussion on efficiency. For example, for each quantized image token, are the corresponding sets A and B precomputed or dynamically calculated during inference? How much time does this part take? I recommend the authors supplement this section with analysis and discussion.\n\n2.In Line 105, there might be a misstatement. EAGLE-2 is an acceleration decoding method, not a drafter.\n\n3.To demonstrate the generalizability of the observation of \"token ambiguity,\" i.e., that this phenomenon is present in most AR image generation models, the paper should include experiments on a broader range of AR image generation models beyond just LLaMAGen. This would enhance the experimental completeness and provide stronger evidence for the generality of the observed issue.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)