LANTERN: Accelerating Visual Autoregressive Models with Relaxed Speculative Decoding

Authors: Doohyuk Jang, Sihwan Park, June Yong Yang, Yeonsung Jung, Jihun Yun, Souvik Kundu, Sung-Yub Kim, Eunho Yang
Primary area: generative models
Keywords: Speculative decoding, Visual Autoregressive Models

Abstract:
Auto-Regressive (AR) models have recently gained prominence in image generation, often matching or even surpassing the performance of diffusion models. However, one major limitation of AR models is their sequential nature, which processes tokens one at a time, slowing down generation compared to models like GANs or diffusion-based methods that operate more efficiently. While speculative decoding has proven effective for accelerating LLMs by generating multiple tokens in a single forward, its application in visual AR models remains largely unexplored. In this work, we identify a challenge in this setting, which we term \textit{token selection ambiguity}, wherein visual AR models frequently assign uniformly low probabilities to tokens, hampering the performance of speculative decoding. To overcome this challenge, we propose a relaxed acceptance condition referred to as LANTERN that leverages the interchangeability of tokens in latent space. This relaxation restores the effectiveness of speculative decoding in visual AR models by enabling more flexible use of candidate tokens that would otherwise be prematurely rejected. Furthermore, by incorporating a total variation distance bound, we ensure that these speed gains are achieved without significantly compromising image quality or semantic coherence. Experimental results demonstrate the efficacy of our method in providing a substantial speed-up over speculative decoding. In specific, compared to a na\"ive application of the state-of-the-art speculative decoding, LANTERN increases speed-ups by $\mathbf{1.75}\times$ and $\mathbf{1.82}\times$, as compared to greedy decoding and random sampling, respectively, when applied to LlamaGen, a contemporary visual AR model. The code is publicly available at \url{https://github.com/jadohu/LANTERN}.

Decision: accept (poster)
Reviewer scores: 6, 6, 6, 8

Reviews and discussion:
<thread>
creation date:
1730364183709

thread_type:
official_review

<thread_object>
creator: reviewer_nfr2

<content>
summary:
This paper tackles the challenge of transferring speculative decoding from LLMs to autoregressive image generation models. The authors observe that the nature of image data causes image tokens to exhibit selection ambiguity, resulting in high rejection rates and poor acceleration effects in speculative decoding. Hence  they propose using the proximity set of image tokens as proxies to speculate on accepting a predicted image token, and introduce a combinatorial optimization strategy for selecting the set to ensure image quality remains intact.  Extensive experiments confirm significant acceleration with minimal quality loss.

strengths:
The paper's structure and presentation are well-crafted. It identifies the phenomenon of "token ambiguity" and provides a viable solution. Extensive experiments, particularly the ablation studies, demonstrate the effectiveness and scalability of the method. It offers valuable insights for research on inference acceleration in AR text-to-image generation. Although the idea of using a set to proxy a single point has been extensively researched, the authors have elegantly applied it to the sampling decision process and achieved excellent results. 
 Simplicity yet effectiveness is crucial.

weaknesses:
1.For a specific image token, the algorithm for finding its proximity set , specifically "Find the neighborhood (i.e., Appendix B, Line 9)," lacks detailed discussion on efficiency. For example, for each quantized image token, are the corresponding sets A and B precomputed or dynamically calculated during inference? How much time does this part take? I recommend the authors supplement this section with analysis and discussion.

2.In Line 105, there might be a misstatement. EAGLE-2 is an acceleration decoding method, not a drafter.

3.To demonstrate the generalizability of the observation of "token ambiguity," i.e., that this phenomenon is present in most AR image generation models, the paper should include experiments on a broader range of AR image generation models beyond just LLaMAGen. This would enhance the experimental completeness and provide stronger evidence for the generality of the observed issue.

questions:
1.There's a suggestion that the authors could provide the details of "Find the neighborhood" in Appendix A.

2.A suggestion is (since the experimental section of the current paper is already sufficient, this is just a gentle suggestion) that the authors could discuss more about the differences in token ambiguity between text generation and image generation. For example, in LLaMAGen, what common properties do tokens within the same set have? What are the patterns in the sizes of sets A and B corresponding to different image tokens? These discussions could help in designing further algorithms.

3.Could validation experiments be conducted on other AR models? This is also a optional suggestion aimed at enhancing the generalization and robustness of the method.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730440252476

thread_type:
official_review

<thread_object>
creator: reviewer_axtn

<content>
summary:
This paper proposed speculative decoding for AR Image generation, AR models tend to assign uniformly low probabilities across a wide range of tokens, making it difficult to select the most appropriate token during decoding.

To solve this problem, the author introduce LANETERN (Latent Neighbor Token Acceptance Relaxation), leveraging the interchangeability of the tokens and relaxing the acceptance for decoding. 

The benefits of LANETREN can be found at generating speed with comparable quality.

strengths:
1. The motivation behind the author’s proposed approach is straightforward: in standard sampling processes, an excessively high token rejection rate can significantly slow down generation. The author suggests a more relaxed policy to mitigate this issue, thereby improving generation speed.

2. To mitigate the distributional shift introduced by the lenient sampling approach, the author proposes a constraint rule for sampling based on total variation.

3. Through extensive ablation experiments, the author demonstrates how the lenient sampling scheme influences sampling speed and elucidates the specific roles of various hyperparameters within this scheme.

weaknesses:
My main concern lies in whether the metrics employed in the experimental section sufficiently capture the effectiveness of the proposed approach, particularly in Table 3 and Section 4.2.3. For evaluating basic generation quality, would it be possible to provide precision and recall (P/R) instead of FID? This change could allow for a clearer view of how LANTERN and limited distribution divergence affect and restore distribution. Additionally, since LlamaGen is used as the base model, could the authors include some modern image quality scores, such as PickScore or HPS, to better quantify aesthetic quality loss?

questions:
see weakness. 

In summary, the author presents a promising sampling scheme for AR models, which improves sampling speed at the cost of some image quality.

The primary concern is that the author’s experiments lack sufficient evidence to demonstrate that the loss in image quality is within an acceptable range. Given this limitation in quality, I believe the impact of the work may be constrained. Therefore, I am inclined to recommend rejection.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730473789077

thread_type:
official_review

<thread_object>
creator: reviewer_56i9

<content>
summary:
The work presents a novel approach to enhance the efficiency of image generation using Auto-Regressive (AR) models, which traditionally suffer from slow sequential processing. The authors introduce LANTERN, a method that leverages speculative decoding with a relaxed acceptance condition to significantly speed up inference while maintaining image quality. By utilizing a smaller drafter model to predict multiple tokens simultaneously, LANTERN addresses the challenges of token selection ambiguity inherent in visual AR models. The results demonstrate notable improvements in speed and efficiency compared to baseline methods, highlighting the potential of LANTERN to advance the capabilities of AR models in generating high-quality images.

strengths:
The methodology of this paper is clear and comprehensible, and the research question it addresses is interesting. Although the overall approach is relatively straightforward, I believe the findings are still valuable.

weaknesses:
1. My primary concern is that the paper lacks an evaluation of the image generation quality. The authors present several generated images, but these images are stylistically very similar and exhibit poor consistency with the accompanying generated text. Some images even contain clear visual errors. I recommend that the authors conduct a more comprehensive assessment of image generation quality to better demonstrate the effectiveness of the proposed method.

2. The problem the paper aims to address is the low Mean Accepted Length that occurs when applying speculative decoding to AR image generation models. Specifically, although speculative decoding is used, a large number of predictions generated by the lightweight model are rejected, forcing the larger model to regenerate many tokens. As a result, the expected efficiency gains from speculative decoding are not realized. While the authors present a complete exploration from two perspectives, I still feel that, given the inherent ambiguity in token selection, improving the acceptance rate alone might suffice. A large set of tokens could be considered acceptable. This raises the question: Is speculative decoding still necessary in such a case? What would the quality and efficiency be if tokens were instead randomly generated within certain constraints?

questions:
Please see the weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730509345985

thread_type:
official_review

<thread_object>
creator: reviewer_v6x3

<content>
summary:
In this paper, the authors propose LANTERN, a sampling strategy to speed up the image generation without losing too much quality. The method is implemented by accumulating probabilities from nearby tokens of the current sampling token. The authors further propose a thresholding technique to prevent the accumulated distribution deviating too much from the original one. Experiment results seem to be effective, however, the analysis is intuitive and empirical without deep insights.

strengths:
- The proposed method is intuitive and easy to follow.

- Experimental results seem to verify the proposed method in terms of speeding up image generation without significant quality loss.

weaknesses:
- There is a gap between the introduction and methodologies in writing. The related works are deferred to the appendix and problem definition and preliminaries such as Speculative Decoding are missing, resulting in difficulties in understanding the problem and challenge for readers who are not exactly working on this domain. On the other hand, Section 2.1 and 2.2 have some overlaps about the experiments and observation, which can be more concise.

- The evaluation seems to be insufficient. For the testing data, it is ideal to use the same setting as the vanilla baseline (i.e., LlamaGen) rather than just 100 captions from MSCOCO. The statement “Since measuring speedup with more than 100 samples shows no significant difference, we use 100 captions for efficiency” is not very convincing to me. In addition, other models are suggested for evaluation as well including other variations of LlamaGen.

- Some analyses in experiments are not sufficient. In sampling, the authors only mentioned that when $\delta$ is small, using larger k results in speeding up without significant degradation in performance, yet the reason for it is less explored. Section 4.3.2 also lacks further analysis why using TVD is better than JSD except some empirical results.

questions:
- See the weakness.
- The claim of interchangeability in Section 3.1 is only empirically explored via a few examples. Are there some theoretical insights or statistics supporting this claim?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732199287434

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
We sincerely thank the reviewers for their thoughtful feedback and valuable suggestions. Your insights have been instrumental in improving the clarity and quality of our work, and we greatly appreciate the opportunity to address your concerns. In this response, we have included a **General Response** for questions raised by two or more reviewers, ensuring consistency and transparency. For individual responses, we have provided tailored responses, with relevant excerpts from the general response included where appropriate.

---
### **Broader range of image quality evaluation**

To further validate LANTERN's ability to maintain image quality, we conducted additional evaluations using two text-to-image quality metrics: (1) Precision and Recall [1], and (2) Human Preference Score (HPS) v2 [2], with same setting as our main experiment. The results below extend our original main table (Table 3 in original paper, Table 2 in the revised version) to include these metrics. We are currently working on the evaluation for greedy decoding ($\tau=0$), and the results will be updated as soon as available.

| $\tau=1$ | Speedup | Mean Accepted Length | FID | CLIPScore | Precision / Recall | HPS v2 |
| --- | --- | --- | --- | --- | --- | --- |
| Vanilla AR | $\times1.00$ | $\times1.00$ | $15.22$ | $0.3203$ | $0.4781$ / $0.5633$ | $24.11$ |
| EAGLE-2 | $\times0.93$ | $\times1.20$ | - | - | - | - |
| LANTERN ($\delta=0.10, k=1000)$ | $\times1.13$ | $\times1.75$ | $16.17$ | $0.3208$ | $0.4869$  / $0.5172$ | $23.75$ |
| LANTERN ($\delta=0.40, k=1000)$ | $\times1.69$ | $\times2.40$ | $18.76$ | $0.3206$ | $0.4909$  / $0.4497$ | $23.22$ |

As shown in the table above, LANTERN demonstrates comparable or slightly improved precision while showing a slight decrease in recall. This can be interpreted as maintaining the quality of individual images while slightly reducing diversity. This slight reduction in recall can be attributed to the challenges associated with token selection ambiguity in the drafter. When the drafter is suboptimally trained, it may struggle to provide sufficiently diverse predictions or precise alternatives. Consequently, the increased acceptance probability, while improving acceleration, can lead to a modest decline in the diversity of generated images. However, it is important to note that this trade-off between acceleration and diversity does not significantly impact the overall image quality, as evidenced by the consistent FID, Precision and HPS v2 scores. Future work could address this by improving the drafter’s training process to better handle token selection ambiguity, thereby further enhancing recall without compromising LANTERN’s efficiency. Additionally, the HPS v2 score, which is derived from a preference model trained on a human preference dataset, does not exhibit any significant degradation, further supporting the robustness of LANTERN's performance. As a result, LANTERN prove its effectiveness over various metric, by maintaining its score in acceptable range. The results have been updated to the Table 2 in the revised version.

---
### **Evaluation on other visual AR models**

To evaluate LANTERN on additional visual AR models, we extended our experiments to include the LlamaGen Stage II model and Anole [3]. These experiments focused on random sampling ($\tau=1$) for the following reasons: (1) Random sampling generally produces higher-quality images than greedy decoding ($\tau=0$), (2) Our prior results (Table 3 in the paper) indicate random sampling is a more challenging case for acceleration, and (3) Resource constraints prevented comprehensive experiments for both settings. We use MS-COCO 2017 Validation set to evaluate LlamaGen stage II and Anole’s image generation performance. In addition, we measure actual speedup of LlamaGen model with RTX3090 and A100 for Anole. The results are summarized below.

| LlamaGen Stage II, $\tau=1$ | Speedup | Mean Accepted Length | FID | CLIPScore | Precision / Recall | HPS v2 |
| --- | --- | --- | --- | --- | --- | --- |
| Vanilla AR | $1.00\times$ | $1.00\times$ | $47.60$ | $0.2939$ | $0.4138$  / $0.5648$ | $23.84$ |
| EAGLE-2 | $0.96\times$ | $1.22\times$ | - | - | - | - |
| LANTERN ($\delta=0.40, k=1000)$ | $1.64\times$ | $2.24\times$ | $46.10$ | $0.2925$ | $0.4704$  / $0.5222$ | $23.06$ |

| Anole, $\tau=1$ | Speedup | Mean Accepted Length | FID | CLIPScore | Precision / Recall | HPS v2 |
| --- | --- | --- | --- | --- | --- | --- |
| Vanilla AR | $1.00\times$ | $1.00\times$ | $20.27$ | $0.3215$ | $0.6552$  / $0.6398$ | $23.52$ |
| EAGLE-2 | $0.73\times$ | $1.10\times$ | - | - | - | - |
| LANTERN ($\delta=0.50, k=100)$ | $1.17\times$ | $1.83\times$ | $23.40$ | $0.3186$ | $0.6026$  / $0.6178$ | $22.92$ |
</content>
</thread_object>

</thread>
<thread>
creation date:
1733188761408

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Gratitude and Final Remarks on Paper #12935

comment:
Dear Reviewers, AC, and SAC of paper #12935,

As we approach the conclusion of the author-reviewer discussion phase, we would like to highlight few points:

1. We are sincerely grateful to the reviewers for their constructive feedback, which has significantly contributed to improving the clarity and depth of the manuscript. The thoughtful comments and suggestions have been invaluable in refining our work.

2. During the discussion period, we made every effort to address all concerns raised by the reviewers, including conducting additional experiments, collecting data, and providing thorough responses. While we are glad that this process helped garner more favorable assessments from some reviewers, we regret that two reviewers were unable to engage further despite our polite reminders. Out of respect for their time, we chose not to send repeated reminders, trusting that our detailed rebuttal and the positive feedback from other reviewers sufficiently highlight the merits of our work. Nevertheless, if circumstances allow, we would still greatly appreciate their feedback, which could further enrich the evaluation process.

3. Understanding the limitations and potential for speculative decoding (SD) for vision AR models has been an open problem. Our paper "LANTERN" not only takes the first attempt to understand the limitations of SD for vision AR but also identifies the associated potential causes and presents a novel solution to mitigate that. We demonstrate LANTERN can yield inference speed up by up to $1.82\times$ compared to any existing alternative. We believe this work would inspire the community to adopt SD for the emerging vision AR use cases for latency-critical applications. 

We remain hopeful and trustful in the reviewing system and grateful for the reviews and discussions (though only two out of four reviewers provided them). We hope our efforts are acknowledged and taken into account during the decision.

Best regards,

The Authors

Paper ID #12935
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper introduces LANTERN, a method that accelerates visual autoregressive models by adapting speculative decoding -- a mechanism originally proposed for large language models -- to the domain of visual autoregressive generation. The proposed approach demonstrates the ability to achieve speed gains while maintaining image quality. This work opens up new possibilities in enhancing the efficiency of autoregressive visual generative models and highlights how mechanisms from large language models can be effectively tailored for visual-specific applications. Considering the positive feedback from all reviewers, I recommend the acceptance of this paper.