Towards Fast Multilingual LLM Inference: Speculative Decoding and
  Specialized Drafters
Authors: name: Euiin Yi; name: Taehyeon Kim; name: Hongseok Jeung; name: Du-Seong Chang; name: Se-Young Yun
Published: 2024-06-24 16:06:50

Abstract: Large language models (LLMs) have revolutionized natural language processing
and broadened their applicability across diverse commercial applications.
However, the deployment of these models is constrained by high inference time
in multilingual settings. To mitigate this challenge, this paper explores a
training recipe of an assistant model in speculative decoding, which are
leveraged to draft and-then its future tokens are verified by the target LLM.
We show that language-specific draft models, optimized through a targeted
pretrain-and-finetune strategy, substantially brings a speedup of inference
time compared to the previous methods. We validate these models across various
languages in inference time, out-of-domain speedup, and GPT-4o evaluation.