Llasa: Scaling Train-Time and Inference-Time Compute for Llama-based
  Speech Synthesis
Authors: name: Zhen Ye, name: Xinfa Zhu, name: Chi-Min Chan, name: Xinsheng Wang, name: Xu Tan, name: Jiahe Lei, name: Yi Peng, name: Haohe Liu, name: Yizhu Jin, name: Zheqi DAI, name: Hongzhan Lin, name: Jianyi Chen, name: Xingjian Du, name: Liumeng Xue, name: Yunlin Chen, name: Zhifei Li, name: Lei Xie, name: Qiuqiang Kong, name: Yike Guo, name: Wei Xue

Recent advances in text-based large language models (LLMs), particularly in
the GPT series and the o1 model, have demonstrated the effectiveness of scaling
both training-time and inference-time compute. However, current
state-of-the-art TTS systems leveraging LLMs are often multi-stage, requiring
separate models (e.g., diffusion models after LLM), complicating the decision
of whether to scale a particular model during training or testing. This work
makes the following contributions: First, we explore the scaling of train-time
and inference-time compute for speech synthesis. Second, we propose a simple
framework Llasa for speech synthesis that employs a single-layer vector
quantizer (VQ) codec and a single Transformer architecture to fully align with
standard LLMs such as Llama. Our experiments reveal that scaling train-time
compute for Llasa consistently improves the naturalness of synthesized speech
and enables the generation of more complex and accurate prosody patterns.
Furthermore, from the perspective of scaling inference-time compute, we employ
speech understanding models as verifiers during the search, finding that
scaling inference-time compute shifts the sampling modes toward the preferences
of specific verifiers, thereby improving emotional expressiveness, timbre
consistency, and content accuracy. In addition, we released the checkpoint and
training code for our TTS model (1B, 3B, 8B) and codec model publicly
available.