Amphista: Accelerate LLM Inference with Bi-directional Multiple Drafting Heads in a Non-autoregressive Style
Authors: Zeping Li, Xinlong Yang, Ziheng Gao, Ji Liu, Zhuang Liu, Dong Li, Jinzhang Peng, Lu Tian, Emad Barsoum


Reviewer paper summaries: The paper introduces Amphista, an innovative speculative decoding algorithm designed to accelerate inference in large language models (LLMs) through a non-autoregressive paradigm. This approach enhances parallelism and efficiency compared to traditional autoregressive methods. The key components of Amphista include the Auto-embedding Block, which uses bi-directional self-attention to facilitate interaction among multiple drafting heads, and Staged Adaptation Layers that transform semantic information from the base model to the drafting heads. Extensive experiments on the Vicuna 33B model using MT-Bench and Spec-Bench benchmarks demonstrate that Amphista outperforms existing methods like Medusa and Hydra in speed improvements.
Summary of strengths: - The introduction of non-autoregressive decoding significantly enhances the overall parallelism, better leveraging modern GPU accelerators more effectively, leading to notable improvements in inference speed.
- Extensive experiments on various benchmarks, including MT-Bench and Spec-Bench, validate the method's effectiveness.
Summary of weaknesses: 1. Limited Batch Size Exploration: The paper focuses primarily on scenarios with a batch size of one, leaving the performance on larger batch sizes underexplored.
2. The added complexity of the Auto-embedding Block and Staged Adaptation Layers might pose challenges for practical implementation and deployment.
3. The evaluation of generation quality needs further efforts.
3. The writing of the method part can be further improved.
Comments, suggestions and typos: **Questions**
1. Table 5. From [1], we know that the Average Accepted Length depends on not only the proposal of the draft model but also the posterior threshold. So, do the models listed in Table 5 use the same posterior threshold?
2. About the generation quality. I am really worried about the quality of generated text from a non-autoregressive model. However, the paper only includes the evaluation of summarization (Table 3).  Why not include an evaluation of more scenarios you have experimented with, such as MT-bench, and MATH? 

[1] [Hydra: Sequentially-Dependent Draft Heads for Medusa Decoding](https://arxiv.org/abs/2402.05109)

**Typos**
1. L153. "lm_head" can be replaced with a better form.
2. L222. What is "FAL"?
3. L244. "Where" should be "where" when it is an incomplete sentence and as an explanation of Eq. (2).
4. L398. "2.10$\times$-2.31$\times$" should be "2.10$\times - 2.31\times$"
Overall assessment scores: 3