Autoregressive Speech Synthesis without Vector Quantization
Authors: name: Lingwei Meng; name: Long Zhou; name: Shujie Liu; name: Sanyuan Chen; name: Bing Han; name: Shujie Hu; name: Yanqing Liu; name: Jinyu Li; name: Sheng Zhao; name: Xixin Wu; name: Helen Meng; name: Furu Wei
Published: 2024-07-11 14:36:53

Abstract: We present MELLE, a novel continuous-valued tokens based language modeling
approach for text to speech synthesis (TTS). MELLE autoregressively generates
continuous mel-spectrogram frames directly from text condition, bypassing the
need for vector quantization, which are originally designed for audio
compression and sacrifice fidelity compared to mel-spectrograms. Specifically,
(i) instead of cross-entropy loss, we apply regression loss with a proposed
spectrogram flux loss function to model the probability distribution of the
continuous-valued tokens. (ii) we have incorporated variational inference into
MELLE to facilitate sampling mechanisms, thereby enhancing the output diversity
and model robustness. Experiments demonstrate that, compared to the two-stage
codec language models VALL-E and its variants, the single-stage MELLE mitigates
robustness issues by avoiding the inherent flaws of sampling discrete codes,
achieves superior performance across multiple metrics, and, most importantly,
offers a more streamlined paradigm. See https://aka.ms/melle for demos of our
work.