Seed-TTS: A Family of High-Quality Versatile Speech Generation Models
Authors: name: Philip Anastassiou; name: Jiawei Chen; name: Jitong Chen; name: Yuanzhe Chen; name: Zhuo Chen; name: Ziyi Chen; name: Jian Cong; name: Lelai Deng; name: Chuang Ding; name: Lu Gao; name: Mingqing Gong; name: Peisong Huang; name: Qingqing Huang; name: Zhiying Huang; name: Yuanyuan Huo; name: Dongya Jia; name: Chumin Li; name: Feiya Li; name: Hui Li; name: Jiaxin Li; name: Xiaoyang Li; name: Xingxing Li; name: Lin Liu; name: Shouda Liu; name: Sichao Liu; name: Xudong Liu; name: Yuchen Liu; name: Zhengxi Liu; name: Lu Lu; name: Junjie Pan; name: Xin Wang; name: Yuping Wang; name: Yuxuan Wang; name: Zhen Wei; name: Jian Wu; name: Chao Yao; name: Yifeng Yang; name: Yuanhao Yi; name: Junteng Zhang; name: Qidi Zhang; name: Shuo Zhang; name: Wenjie Zhang; name: Yang Zhang; name: Zilin Zhao; name: Dejian Zhong; name: Xiaobin Zhuang
Published: 2024-06-04 15:48:29

Abstract: We introduce Seed-TTS, a family of large-scale autoregressive text-to-speech
(TTS) models capable of generating speech that is virtually indistinguishable
from human speech. Seed-TTS serves as a foundation model for speech generation
and excels in speech in-context learning, achieving performance in speaker
similarity and naturalness that matches ground truth human speech in both
objective and subjective evaluations. With fine-tuning, we achieve even higher
subjective scores across these metrics. Seed-TTS offers superior
controllability over various speech attributes such as emotion and is capable
of generating highly expressive and diverse speech for speakers in the wild.
Furthermore, we propose a self-distillation method for speech factorization, as
well as a reinforcement learning approach to enhance model robustness, speaker
similarity, and controllability. We additionally present a non-autoregressive
(NAR) variant of the Seed-TTS model, named Seed-TTS_DiT, which
utilizes a fully diffusion-based architecture. Unlike previous NAR-based TTS
systems, Seed-TTS_DiT does not depend on pre-estimated phoneme
durations and performs speech generation through end-to-end processing. We
demonstrate that this variant achieves comparable performance to the language
model-based variant and showcase its effectiveness in speech editing. We
encourage readers to listen to demos at
https://bytedancespeech.github.io/seedtts_tech_report.