NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and
  Diffusion Models
Authors: name: Zeqian Ju; name: Yuancheng Wang; name: Kai Shen; name: Xu Tan; name: Detai Xin; name: Dongchao Yang; name: Yanqing Liu; name: Yichong Leng; name: Kaitao Song; name: Siliang Tang; name: Zhizheng Wu; name: Tao Qin; name: Xiang-Yang Li; name: Wei Ye; name: Shikun Zhang; name: Jiang Bian; name: Lei He; name: Jinyu Li; name: Sheng Zhao
Published: 2024-03-05 16:35:25

Abstract: While recent large-scale text-to-speech (TTS) models have achieved
significant progress, they still fall short in speech quality, similarity, and
prosody. Considering speech intricately encompasses various attributes (e.g.,
content, prosody, timbre, and acoustic details) that pose significant
challenges for generation, a natural idea is to factorize speech into
individual subspaces representing different attributes and generate them
individually. Motivated by it, we propose NaturalSpeech 3, a TTS system with
novel factorized diffusion models to generate natural speech in a zero-shot
way. Specifically, 1) we design a neural codec with factorized vector
quantization (FVQ) to disentangle speech waveform into subspaces of content,
prosody, timbre, and acoustic details; 2) we propose a factorized diffusion
model to generate attributes in each subspace following its corresponding
prompt. With this factorization design, NaturalSpeech 3 can effectively and
efficiently model the intricate speech with disentangled subspaces in a
divide-and-conquer way. Experiments show that NaturalSpeech 3 outperforms the
state-of-the-art TTS systems on quality, similarity, prosody, and
intelligibility. Furthermore, we achieve better performance by scaling to 1B
parameters and 200K hours of training data.