F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow
  Matching
Authors: name: Yushen Chen; name: Zhikang Niu; name: Ziyang Ma; name: Keqi Deng; name: Chunhui Wang; name: Jian Zhao; name: Kai Yu; name: Xie Chen
Published: 2024-10-09 13:46:34

Abstract: This paper introduces F5-TTS, a fully non-autoregressive text-to-speech
system based on flow matching with Diffusion Transformer (DiT). Without
requiring complex designs such as duration model, text encoder, and phoneme
alignment, the text input is simply padded with filler tokens to the same
length as input speech, and then the denoising is performed for speech
generation, which was originally proved feasible by E2 TTS. However, the
original design of E2 TTS makes it hard to follow due to its slow convergence
and low robustness. To address these issues, we first model the input with
ConvNeXt to refine the text representation, making it easy to align with the
speech. We further propose an inference-time Sway Sampling strategy, which
significantly improves our model's performance and efficiency. This sampling
strategy for flow step can be easily applied to existing flow matching based
models without retraining. Our design allows faster training and achieves an
inference RTF of 0.15, which is greatly improved compared to state-of-the-art
diffusion-based TTS models. Trained on a public 100K hours multilingual
dataset, our Fairytaler Fakes Fluent and Faithful speech with Flow matching
(F5-TTS) exhibits highly natural and expressive zero-shot ability, seamless
code-switching capability, and speed control efficiency. Demo samples can be
found at https://SWivid.github.io/F5-TTS. We release all code and checkpoints
to promote community development.