Takin: A Cohort of Superior Quality Zero-shot Speech Generation Models
Authors: name: EverestAI; name: Sijin Chen; name: Yuan Feng; name: Laipeng He; name: Tianwei He; name: Wendi He; name: Yanni Hu; name: Bin Lin; name: Yiting Lin; name: Pengfei Tan; name: Chengwei Tian; name: Chen Wang; name: Zhicheng Wang; name: Ruoye Xie; name: Jingjing Yin; name: Jianhao Ye; name: Jixun Yao; name: Quanlei Yan; name: Yuguang Yang
Published: 2024-09-18 17:03:12

Abstract: With the advent of the big data and large language model era, zero-shot
personalized rapid customization has emerged as a significant trend. In this
report, we introduce Takin AudioLLM, a series of techniques and models, mainly
including Takin TTS, Takin VC, and Takin Morphing, specifically designed for
audiobook production. These models are capable of zero-shot speech production,
generating high-quality speech that is nearly indistinguishable from real human
speech and facilitating individuals to customize the speech content according
to their own needs. Specifically, we first introduce Takin TTS, a neural codec
language model that builds upon an enhanced neural speech codec and a
multi-task training framework, capable of generating high-fidelity natural
speech in a zero-shot way. For Takin VC, we advocate an effective content and
timbre joint modeling approach to improve the speaker similarity, while
advocating for a conditional flow matching based decoder to further enhance its
naturalness and expressiveness. Last, we propose the Takin Morphing system with
highly decoupled and advanced timbre and prosody modeling approaches, which
enables individuals to customize speech production with their preferred timbre
and prosody in a precise and controllable manner. Extensive experiments
validate the effectiveness and robustness of our Takin AudioLLM series models.
For detailed demos, please refer to https://takinaudiollm.github.io.