Cerberus: Efficient Inference with Adaptive Parallel Decoding and Sequential Knowledge Enhancement
Authors: Yuxuan Liu, Zhongxing Ming, Wenyuan Li, Laizhong Cui


Reviewer paper summaries: This paper addresses the issues of low accuracy and overhead in parallel decoding by proposing a learned gated network to determine whether to use parallel inference. Additionally, it enhances accuracy and reduces latency through residuals. The proposed method was tested on MT-Bench using Vicuna-7B. Experimental results demonstrate that the method achieves performance close to greedy decoding while offering faster latency.
Summary of strengths: - The research problem focused on this paper has practical value.
- The motivation behind the paper is straightforward, and the proposed method is reasonable.
Summary of weaknesses: - The baseline Medusa compared in the paper is not a common method for parallel decoding; it is only used for speculative decoding to generate draft input, not directly for decoding.
- The paper lacks comparisons and discussions with some essential baselines, such as the one described in "Better & Faster Large Language Models via Multi-token Prediction, ICML 2024."
- The experimental section is limited, with tests conducted only on MT-Bench using Vicuna-7B, making it difficult to demonstrate the generalizability of the proposed method.
Comments, suggestions and typos: In Figure 6, the font size of the legend can be adjusted to be slightly smaller.
Overall assessment scores: 2.5