Speculative Streaming: Fast LLM Inference without Auxiliary Models
Authors: Nikhil Bhendawade, Irina Belousova, Qichen Fu, Henry Mason, Mohammad Rastegari, Mahyar Najibi


Reviewer paper summaries: The paper discusses large language model generation acceleration and introduces a speculative decoding method that incorporates future token planning into the fine-tuning objective. It leverages speculating of multiple candidate future tokens, followed by verification of all in parallel. The work claims to be the first parameter-efficient approach that also achieves improved downstream metrics. Empirical results show up to 3X decoding speedup with $10^4$X fewer extra parameters. The reduction in parameter size gives an added benefit in deployment on edge devices.

# Comments after Author Discussion
Please refer to my official comment on this thread. I retain my scores.
Summary of strengths: - The work is well-motivated and leverages the speculation of multiple candidate future tokens and verification of them in parallel.
- Empirical evidence show consistent speedup achieved in decoding with multiple LLMs of different model size variants.
- The paper discusses design decisions in a depth that can be useful for future research, such as in the insertion of pruning layer in the network (reduced latency with early insertion and better token quality with late insertion).
Summary of weaknesses: - While the paper claims an edge in deployment on resource-constrained devices, the experiments conducted primarily deals with 7B+ parameter models (with the exception of OPT-1.3B) that are usually considered unsuitable for such use cases. Perhaps, the work can benefit from further exploration with quantized LLMs to observe if the findings are consistent with those.
Comments, suggestions and typos: - Line 453: Erroneous capitalization of "It's" after "Finally,".
Overall assessment scores: 3.5