F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching

## Abstract

Abstract This paper introduces F5-TTS, a fully non-autoregressive text-to-speech system based on flow matching with Diffusion Transformer (DiT).
Without requiring complex designs such as duration model, text encoder, and phoneme alignment, the text input is simply padded with filler tokens to the same length as input speech, and then the denoising is performed for speech generation, which was originally proved feasible by E2 TTS.
However, the original design of E2 TTS makes it hard to follow due to its slow convergence and low robustness.
To address these issues, we first model the input with ConvNeXt to refine the text representation, making it easy to align with the speech.
We further propose an inference-time Sway Sampling strategy, which significantly improves our model’s performance and efficiency. This sampling strategy for flow step can be easily applied to existing flow matching based models without retraining.
Our design allows faster training and achieves an inference RTF of 0.15, which is greatly improved compared to state-of-the-art diffusion-based TTS models.
Trained on a public 100K hours multilingual dataset, our Fairytaler Fakes Fluent and Faithful speech with Flow matching (F5-TTS) exhibits highly natural and expressive zero-shot ability, seamless code-switching capability, and speed control efficiency. Demo samples can be found at https://SWivid.github.io/F5-TTS .
We release all code and checkpoints to promote community development 1 1 1 https://github.com/SWivid/F5-TTS .

## 1 Introduction

Recent research in Text-to-Speech (TTS) has experienced great advancement [, , , , , , , ]. With a few seconds of audio prompt, current TTS models are able to synthesize speech for any given text and mimic the speaker of audio prompt [, ]. The synthesized speech can achieve high fidelity and naturalness that they are almost indistinguishable from human speech [, , , ].

While autoregressive (AR) based TTS models exhibit an intuitive way of consecutively predicting the next token(s) and have achieved promising zero-shot TTS capability, the inherent limitations of AR modeling require extra efforts addressing issues such as inference latency and exposure bias [, , , , ]. Moreover, the quality of speech tokenizer is essential for AR models to achieve high-fidelity synthesis [, , , , , , ]. Thus, there have been studies exploring direct modeling in continuous space [, , ] to enhance synthesized speech quality recently.

Although AR models demonstrate impressive zero-shot performance as they perform implicit duration modeling and can leverage diverse sampling strategies, non-autoregressive (NAR) models benefit from fast inference through parallel processing, and effectively balance synthesis quality and latency. Notably, diffusion models [, ] contribute most to the success of current NAR speech models [, ]. In particular, Flow Matching with Optimal Transport path (FM-OT) [] is widely used in recent research fields not only text-to-speech [, , , , ] but also image generation [] and music generation [].

Figure: Figure 1: An overview of F5-TTS training (left) and inference (right). The model is trained on the text-guided speech-infilling task and condition flow matching loss. The input text is converted to a character sequence, padded with filler tokens to the same length as input speech, and refined by ConvNeXt blocks before concatenation with speech input. The inference leverages Sway Sampling for flow steps, with the model and an ODE solver to generate speech from sampled noise.
Refer to caption: x1.png

Unlike AR-based models, the alignment modeling between input text and synthesized speech is crucial and challenging for NAR-based models. While NaturalSpeech 3 [] and Voicebox [] use frame-wise phoneme alignment; Matcha-TTS [] adopts monotonic alignment search and relies on phoneme-level duration model; recent works find that introducing such rigid and inflexible alignment between text and speech hinders the model from generating results with higher naturalness [, ].

E3 TTS [] abandons phoneme-level duration and applies cross-attention on the input sequence but yields limited audio quality. DiTTo-TTS [] uses Diffusion Transformer (DiT) [] with cross-attention conditioned on encoded text from a pretrained language model. To further enhance alignment, it uses the pretrained language model to finetune the neural audio codec, infusing semantic information into the generated representations. In contrast, E2 TTS [], based on Voicebox [], adopts a simpler way, which removes the phoneme and duration predictor and directly uses characters padded with filler tokens to the length of mel spectrograms as input. This simple scheme also achieves very natural and realistic synthesized results. However, we found that robustness issues exist in E2 TTS for the text and speech alignment. Seed-TTS [] employs a similar strategy and achieves excellent results, though not elaborated in model details. In these ways of not explicitly modeling phoneme-level duration, models learn to assign the length of each word or phoneme according to the given total sequence length, resulting in improved prosody and rhythm.

In this paper, we propose F5-TTS, a Fairytaler that Fakes Fluent and Faithful speech with Flow matching. Maintaining the simplicity of pipeline without phoneme alignment, duration predictor, text encoder, and semantically infused codec model, F5-TTS leverages the Diffusion Transformer with ConvNeXt V2 [] to better tackle text-speech alignment during in-context learning. We stress the deep entanglement of semantic and acoustic features in the E2 TTS model design, which has inherent problems and will pose alignment failure issues that could not simply be solved with re-ranking. With in-depth ablation studies, our proposed F5-TTS demonstrates stronger robustness, in generating more faithful speech to the text prompt, while maintaining comparable speaker similarity. Additionally, we introduce an inference-time sampling strategy for flow steps substantially improving naturalness, intelligibility, and speaker similarity of generation. This approach can be seamlessly integrated into existing flow matching based models without retraining.

## 2 Preliminaries

### 2.1 Flow Matching

The Flow Matching (FM) objective is to match a probability path $p_{t}$ from a simple distribution $\displaystyle p_{0}$, e.g., the standard normal distribution $p(x)=\mathcal{N}(x|0,I)$, to $\displaystyle p_{1}$ approximating the data distribution $q$. In short, the FM loss regresses the vector field $u_{t}$ with a neural network $v_{t}$ as

$$ $\mathcal{L}_{FM}(\theta)=E_{t,p_{t}(x)}\left\|v_{t}(x)-u_{t}(x)\right\|^{2},$ (1) $$

where $\theta$ parameterizes the neural network, $t\sim\mathcal{U}[0,1]$ and $x\sim p_{t}(x)$. The model $v_{t}$ is trained over the entire flow step and data range, ensuring it learns to handle the entire transformation process from the initial distribution to the target distribution.

As we have no prior knowledge of how to approximate $p_{t}$ and $u_{t}$, a conditional probability path $p_{t}(x|x_{1})=\mathcal{N}(x\ |\ \mu_{t}(x_{1}),\sigma_{t}(x_{1})^{2}I)$ is considered in actual training, and the Conditional Flow Matching (CFM) loss is proved to have identical gradients w.r.t. $\theta$ []. $x_{1}$ is the random variable corresponding to training data. $\mu$ and $\sigma$ is the time-dependent mean and scalar standard deviation of Gaussian distribution.

Remember that the goal is to construct target distribution (data samples) from initial simple distribution, e.g., Gaussian noise. With the conditional form, the flow map $\psi_{t}(x)=\sigma_{t}(x_{1})x+\mu_{t}(x_{1})$ with $\mu_{0}(x_{1})=0$ and $\sigma_{0}(x_{1})=1$, $\mu_{1}(x_{1})=x_{1}$ and $\sigma_{1}(x_{1})=0$ is made to have all conditional probability paths converging to $p_{0}$ and $p_{1}$ at the start and end. The flow thus provides a vector field $d\psi_{t}(x_{0})/dt=u_{t}(\psi_{t}(x_{0})|x_{1})$. Reparameterize $p_{t}(x|x_{1})$ with $x_{0}$, we have

$$ $\mathcal{L}_{\text{CFM}}(\theta)=E_{t,q(x_{1}),p(x_{0})}\|v_{t}(\psi_{t}(x_{0} ))-\frac{d}{dt}\psi_{t}(x_{0})\|^{2}.$ (2) $$

Further leveraging Optimal Transport form $\psi_{t}(x)=(1-t)x+tx_{1}$, we have the OT-CFM loss,

$$ $\mathcal{L}_{\text{CFM}}(\theta)=E_{t,q(x_{1}),p(x_{0})}\|v_{t}((1-t)x_{0}+tx_ {1})-(x_{1}-x_{0})\|^{2}.$ (3) $$

To view in a more general way [], if formulating the loss in terms of log signal-to-noise ratio (log-SNR) $\lambda$ instead of flow step $t$, and parameterizing to predict $x_{0}$ ($\epsilon$, commonly stated in diffusion model) instead of predict $x_{1}-x_{0}$, the CFM loss is equivalent to the $v$-prediction [] loss with cosine schedule.

For inference, given sampled noise $x_{0}$ from initial distribution $p_{0}$, flow step $t\in[0,1]$ and condition with respect to generation task, the ordinary differential equation (ODE) solver [] is used to evaluate $\psi_{1}(x_{0})$ the integration of $d\psi_{t}(x_{0})/dt$ with $\psi_{0}(x_{0})=x_{0}$. The number of function evaluations (NFE) is the times going through the neural network as we may provide multiple flow step values from 0 to 1 as input to approximate the integration. Higher NFE will produce more accurate results and certainly take more calculation time.

### 2.2 Classifier-Free Guidance

Classifier Guidance (CG) is proposed by [], functions by adding the gradient of an additional classifier, while such an explicit way to condition the generation process may have several problems. Extra training of the classifier is required and the generation result is directly affected by the quality of the classifier. Adversarial attacks might also occur as the guidance is introduced through the way of updating the gradient. Thus deceptive images with imperceptible details to human eyes may be generated, which are not conditional.

Classifier-Free Guidance (CFG) [] proposes to replace the explicit classifier with an implicit classifier without directly computing the explicit classifier and its gradient. The gradient of a classifier can be expressed as a combination of conditional generation probability and unconditional generation probability. By dropping the condition with a certain rate during training, and linear extrapolating the inference outputs with and without condition $c$, the final guided result is obtained. We could balance between fidelity and diversity of the generated samples with

$$ $v_{t,CFG}=v_{t}(\psi_{t}(x_{0}),c)+\alpha(v_{t}(\psi_{t}(x_{0}),c)-v_{t}(\psi_ {t}(x_{0})))$ (4) $$

in CFM case, where $\alpha$ is the CFG strength.(^2^22Note that the inference time will be doubled if CFG. Model $v_{t}$ will execute the forward process twice, once with condition, and once without.)

## 3 Method

This work aims to build a high-level text-to-speech synthesis system. We trained our model on the text-guided speech-infilling task [, ]. Based on recent research [, , ], it is promising to train without phoneme-level duration predictor and can achieve higher naturalness in zero-shot generation deprecating explicit phoneme-level alignment. We adopt a similar pipeline as E2 TTS [] and propose our advanced architecture F5-TTS, addressing the slow convergence (timbre learned well at an early stage but struggled to learn alignment) and robustness issues (failures on hard case generation) of E2 TTS. We also propose a Sway Sampling strategy for flow steps at inference, which significantly improves our model’s performance in faithfulness to reference text and speaker similarity.

### 3.1 Pipeline

Training The infilling task is to predict a segment of speech given its surrounding audio and full text (for both surrounding transcription and the part to generate). For simplicity, we reuse the symbol $x$ to denote an audio sample and $y$ the corresponding transcript for a data pair $(x,y)$. As shown in Fig.1 (left), the acoustic input for training is an extracted mel spectrogram features $x_{1}\in\mathbb{R}^{F\times N}$ from the audio sample $x$, where $F$ is mel dimension and $N$ is the sequence length. In the scope of CFM, we pass in the model the noisy speech $(1-t)x_{0}+tx_{1}$ and the masked speech $(1-m)\odot x_{1}$, where $x_{0}$ denotes sampled Gaussian noise, $t$ is sampled flow step, and $m\in\{0,1\}^{F\times N}$ represents a binary temporal mask.

Following E2 TTS, we directly use alphabets and symbols for English. We opt for full pinyin to facilitate Chinese zero-shot generation. By breaking the raw text into such character sequence and padding it with filler tokens to the same length as mel frames, we form an extended sequence $z$ with $c_{i}$ denoting the $i$-th character:

$$ $z=(c_{1},c_{2},\ldots,c_{M},\underbrace{\langle F\rangle,\ldots,\langle F \rangle}_{(N-M)\text{ times}}).$ (5) $$

The model is trained to reconstruct $m\odot x_{1}$ with $(1-m)\odot x_{1}$ and $z$, which equals to learn the target distribution $p_{1}$ in form of $P(m\odot x_{1}|(1-m)\odot x_{1},z)$ approximating real data distribution $q$.

Inference To generate a speech with the desired content, we have the audio prompt’s mel spectrogram features $x_{ref}$, its transcription $y_{ref}$, and a text prompt $y_{gen}$. Audio prompt serves to provide speaker characteristics and text prompt is to guide the content of generated speech.

The sequence length $N$, or duration, has now become a pivotal factor that necessitates informing the model of the desired length for sample generation. One could train a separate model to predict and deliver the duration based on $x_{ref}$, $y_{ref}$ and $y_{gen}$. Here we simply estimate the duration based on the ratio of the number of characters in $y_{gen}$ and $y_{ref}$. We assume that the sum-up length of characters is no longer than mel length, thus padding with filler tokens is done as during training.

To sample from the learned distribution, the converted mel features $x_{ref}$, along with concatenated and extended character sequence $z_{ref\cdot gen}$ serve as the condition in Eq.4. We have

$$ $v_{t}(\psi_{t}(x_{0}),c)=v_{t}((1-t)x_{0}+tx_{1}|x_{ref},z_{ref\cdot gen}),$ (6) $$

See from Fig.1 (right), we start from a sampled noise $x_{0}$, and what we want is the other end of flow $x_{1}$. Thus we use the ODE solver to gradually integrate from $\psi_{0}(x_{0})=x_{0}$ to $\psi_{1}(x_{0})=x_{1}$, given $d\psi_{t}(x_{0})/dt=v_{t}(\psi_{t}(x_{0}),x_{ref},z_{ref\cdot gen})$. During inference, the flow steps are provided in an ordered way, $e.g.$, uniformly sampled a certain number from 0 to 1 according to the NFE setting.

After getting the generated mel with model $v_{t}$ and ODE solver, we discard the part of $x_{ref}$. Then we leverage a vocoder to convert the mel back to speech signal.

### 3.2 F5-TTS

E2 TTS directly concatenates the padded character sequence with input speech, thus deeply entangling semantic and acoustic features with a large length gap of effective information, which is the underlying cause of hard training and poses several problems in a zero-shot scenario (Sec.5.1). To alleviate the problem of slow convergence and low robustness, we propose F5-TTS which accelerates training and inference and shows a strong robustness in generation. Also, an inference-time Sway Sampling is introduced, which allows inference faster (using less NFE) while maintaining performance. This sampling way of flow step can be directly applied to other CFM models.

Model As shown in Fig.1, we use latent Diffusion Transformer (DiT) [] as backbone. To be specific, we use DiT blocks with zero-initialized adaptive Layer Norm (adaLN-zero). To enhance the model’s alignment ability, we also leverage ConvNeXt V2 blocks []. Its predecessor ConvNeXt V1 [] is used in many works and shows a strong temporal modeling capability in speech domain tasks [, ].

As described in Sec.3.1, the model input is character sequence, noisy speech, and masked speech. Before concatenation in the feature dimension, the character sequence first goes through ConvNeXt blocks. Experiments have shown that this way of providing individual modeling space allows text input to better prepare itself before later in-context learning. Unlike the phoneme-level force alignment done in Voicebox, a rigid boundary for text is not explicitly introduced. The semantic and acoustic features are jointly learned with the entire model. And unlike the way of feeding the model with inputs of significant length difference (length with effective information) as E2 TTS does, our design mitigates such gap.

The flow step $t$ for CFM is provided as the condition of adaLN-zero rather than appended to the concatenated input sequence in Voicebox. We found that an additional mean pooled token of text sequence for adaLN condition is not essential for the TTS task, maybe because the TTS task requires more rigorously guided results and the mean pooled text token is more coarse.

We adopt some position embedding settings in Voicebox. The flow step is embedded with a sinusoidal position. The concatenated input sequence is added with a convolutional position embedding. We apply a rotary position embedding (RoPE) [] for self-attention rather than symmetric bi-directional ALiBi bias []. And for extended character sequence $\hat{y}$, we also add it with an absolute sinusoidal position embedding before feeding it into ConvNeXt blocks.

Compared with Voicebox and E2 TTS, we abandoned the U-Net [] style skip connection structure and switched to using DiT with adaLN-zero. Without a phoneme-level duration predictor and explicit alignment process, and nor with extra text encoder and semantically infused neural codec model in DiTTo-TTS, we give the text input a little freedom (individual modeling space) to let it prepare itself before concatenation and in-context learning with speech input.

Sampling As stated in Sec.2.1, the CFM could be viewed as v-prediction with a cosine schedule. For image synthesis, [] propose to further schedule the flow step with a single-peak logit-normal [] sampling, in order to give more weight to intermediate flow steps by sampling them more frequently. We speculate that such sampling distributes the model’s learning difficulty more evenly over different flow step $t\in[0,1]$.

In contrast, we train our model with traditional uniformly sampled flow step $t\sim\mathcal{U}[0,1]$ but apply a non-uniform sampling during inference. In specific, we define a Sway Sampling function as

$$ $f_{sway}(u;s)=u+s\cdot(\cos(\frac{\pi}{2}u)-1+u),$ (7) $$

which is monotonic with coefficient $s\in[-1,\frac{2}{\pi-2}]$. We first sample $u\sim\mathcal{U}[0,1]$, then apply this function to obtain sway sampled flow step $t$. With $s<0$, the sampling is sway to left; with $s>0$, the sampling is sway to right; and $s=0$ case equals to uniform sampling. Fig.3 shows the probability density function of Sway Sampling on flow step $t$.

Conceptually, CFM models focus more on sketching the contours of speech in the early stage ($t\to 0$) from pure noise and later focus more on the embellishment of fine-grained details. Therefore, the alignment between speech and text will be determined based on the first few generated results. With a scale parameter $s<0$, we make model inference more with smaller $t$, thus providing the ODE solver with more startup information to evaluate more precisely in initial integration steps.

## 4 Experimental Setup

Datasets We utilize the in-the-wild multilingual speech dataset Emilia [] to train our base models. After simply filtering out transcription failure and misclassified language speech, we retain approximately 95K hours of English and Chinese data. We also trained small models for ablation study and architecture search on WenetSpeech4TTS [] Premium subset, consisting of a 945 hours Mandarin corpus. Base model configurations are introduced below, and small model configurations are in Appendix B.1. Three test sets are adopted for evaluation, which are LibriSpeech-PC test-clean [], Seed-TTS test-en [] with 1088 samples from Common Voice [], and Seed-TTS test-zh with 2020 samples from DiDiSpeech [](^3^33[https://github.com/BytedanceSpeech/seed-tts-eval](https://github.com/BytedanceSpeech/seed-tts-eval)). Most of the previous English-only models are evaluated on different subsets of LibriSpeech test-clean while the used prompt list is not released, which makes fair comparison difficult. Thus we build and release a 4-to-10-second LibriSpeech-PC subset with 1127 samples to facilitate community comparisons.

Training Our base models are trained to 1.2M updates with a batch size of 307,200 audio frames (0.91 hours), for over one week on 8 NVIDIA A100 80G GPUs. The AdamW optimizer [] is used with a peak learning rate of 7.5e-5, linearly warmed up for 20K updates, and linearly decays over the rest of the training. We set 1 for the max gradient norm clip.
The F5-TTS base model has 22 layers, 16 attention heads, 1024/2048 embedding/feed-forward network (FFN) dimension for DiT; and 4 layers, 512/1024 embedding/FFN dimension for ConvNeXt V2; in total 335.8M parameters. The reproduced E2 TTS, a 333.2M flat U-Net equipped Transformer, has 24 layers, 16 attention heads, and 1024/4096 embedding/FFN dimension. Both models use RoPE as mentioned in Sec.3.2, a dropout rate of 0.1 for attention and FFN, the same convolutional position embedding as in Voicebox[].

We directly use alphabets and symbols for English, use jieba(^4^44[https://github.com/fxsjy/jieba](https://github.com/fxsjy/jieba)) and pypinyin(^5^55[https://github.com/mozillazg/python-pinyin](https://github.com/mozillazg/python-pinyin)) to process raw Chinese characters to full pinyins. The character embedding vocabulary size is 2546, counting in the special filler token and all other language characters exist in the Emilia dataset as there are many code-switched sentences. For audio samples we use 100-dimensional log mel-filterbank features with 24 kHz sampling rate and hop length 256. A random 70% to 100% of mel frames is masked for infilling task training. For CFG (Sec.2.2) training, first the masked speech input is dropped with a rate of 0.3, then the masked speech again but with text input together is dropped with a rate of 0.2. We assume that the two-stage control of CFG training may have the model learn more with text alignment.

Inference The inference process is mainly elaborated in Sec.3.1. We use the Exponential Moving Averaged (EMA) [] weights for inference, and the Euler ODE solver for F5-TTS (midpoint for E2 TTS as described in []). We use the pretrained vocoder Vocos [] to convert generated log mel spectrograms to audio signals.

Baselines We compare our models with leading TTS systems including, (mainly)
autoregressive models: VALL-E 2 [], MELLE [], FireRedTTS [] and CosyVoice [];
non-autoregressive models: Voicebox [], NaturalSpeech 3 [], DiTTo-TTS [], MaskGCT [], Seed-TTSDiT [] and our reproduced E2 TTS []. Details of compared models see Appendix A.

Metrics We measure the performances under cross-sentence task. The model is given a reference text, a short speech prompt, and its transcription, and made to synthesize a speech reading the reference text mimicking the speech prompt speaker. In specific, we report Word Error Rate (WER) and speaker Similarity between generated and the original target speeches (SIM-o) for objective evaluation. For WER, we employ Whisper-large-v3 [] to transcribe English and Paraformer-zh [] for Chinese, following []. For SIM-o, we use a WavLM-large-based [] speaker verification model to extract speaker embeddings for calculating the cosine similarity of synthesized and ground truth speeches.
We use Comparative Mean Opinion Scores (CMOS) and Similarity Mean Opinion Scores (SMOS) for subjective evaluation. For CMOS, human evaluators are given randomly ordered synthesized speech and ground truth, and are to decide how higher the naturalness of the better one surpasses the counterpart, w.r.t. prompt speech. For SMOS, human evaluators are to score the similarity between the synthesized and prompt.

## 5 Experimental Results

Tab.5 and 5 show the main results of objective and subjective evaluations. We report the average score of three random seed generation results with our model and open-sourced baselines. We use by default a CFG strength of 2 and a Sway Sampling coefficient of $-1$ for our F5-TTS.

**Table 1: Results on LibriSpeech test-clean and LibriSpeech-PC test-clean. The boldface indicates the best result, and * denotes the score reported in baseline papers with different subsets for evaluation. The Real-Time Factor (RTF) is computed with the inference time of 10s speech. #Param. stands for the number of learnable parameters and #Data refers to the used training dataset in hours.**
| Model             #Param. #Data(hrs) WER(%)↓ | SIM-o ↑ | RTF ↓ |  |  |  |
| --- | --- | --- | --- | --- | --- |
| LibriSpeech test-clean |  |  |  |  |  |
| Ground Truth (2.2 hours subset) | - | - | 2.2* | 0.754* | - |
| \cdashline1-6<br>VALL-E 2 [] | - | 50K EN | 2.44* | 0.643* | 0.732* |
| MELLE [] | - | 50K EN | 2.10* | 0.625* | 0.549* |
| MELLE-R2 [] | - | 50K EN | 2.14* | 0.608* | 0.276* |
| Voicebox [] | 330M | 60K EN | 1.9* | 0.662* | 0.64* |
| DiTTo-TTS [] | 740M | 55K EN | 2.56* | 0.627* | 0.162* |
| Ground Truth (40 samples subset) | - | - | 1.94* | 0.68* | - |
| \cdashline1-6<br>Voicebox [] | 330M | 60K EN | 2.03* | 0.64* | 0.64* |
| NaturalSpeech 3 [] | 500M | 60K EN | 1.94* | 0.67* | 0.296* |
| MaskGCT [] | 1048M | 100K Multi. | 2.634* | 0.687* | - |
| LibriSpeech-PC test-clean |  |  |  |  |  |
| Ground Truth (1127 samples 2 hrs) | - | - | 2.23 | 0.69 | - |
| Vocoder Resynthesized | - | - | 2.32 | 0.66 | - |
| \cdashline1-6<br>CosyVoice [] | $\sim$300M | 170K Multi. | 3.59 | 0.66 | 0.92 |
| FireRedTTS [] | $\sim$580M | 248K Multi. | 2.69 | 0.47 | 0.84 |
| E2 TTS (32 NFE) [] | 333M | 100K Multi. | 2.95 | 0.69 | 0.68 |
| \cdashline1-6<br>F5-TTS (16 NFE) | 336M | 100K Multi. | 2.53 | 0.66 | 0.15 |
| F5-TTS (32 NFE) | 336M | 100K Multi. | 2.42 | 0.66 | 0.31 |

### 5.1 Ablation of Model Architecture

To clarify our F5-TTS’s efficiency and stress the limitation of E2 TTS. We conduct in-depth ablation studies. We trained small models to 800K updates (each on 8 NVIDIA RTX 3090 GPUs for one week), all scaled to around 155M parameters, on the WenetSpeech4TTS Premium 945 hours Mandarin dataset with half the batch size and the same optimizer and scheduler as base models. Details of small model configurations see Appendix B.1.

We first experiment with pure adaLN DiT (F5-TTS$-$Conv2Text), which fails to learn alignment given simply padded character sequences. Based on the concept of refining the input text representation to better align with speech modality, and keep the simplicity of system design, we propose to add jointly learned structure to the input context. Specifically, we leverage ConvNeXt’s capabilities of capturing local connections, multi-scale features, and spatial invariance for the input text, which is our F5-TTS. And we ablate with adding the same branch for input speech, denoted F5-TTS$+$Conv2Audio. We further conduct experiments to figure out whether the long skip connection and the pre-refinement of input text are beneficial to the counterpart backbone, i.e. F5-TTS and E2 TTS, named F5-TTS$+$LongSkip and E2 TTS$+$Conv2Text respectively. We also tried with the Multi-Modal DiT (MMDiT) [] a double-stream joint-attention structure for the TTS task which learned fast and collapsed fast, resulting in severe repeated utterance with wild timbre and prosody. We assume that the pure MMDiT structure is far too flexible for rigorous task e.g. TTS which needs more faithful generation following the prompt guidance.

Figure: Figure 2: Ablation studies on model architecture. Seed-TTS test-zh evaluation results of 155M small models trained with WenetSpeech4TTS Premium a 945 hours Mandarin Corpus.
Refer to caption: x2.png

Fig.2 shows the overall trend of small models’ WER and SIM scores evaluated on Seed-TTS test-zh. Trained with only 945 hours of data, F5-TTS (32 NFE w/o SS) achieves a WER of 4.17 and a SIM of 0.54 at 800K updates, while E2 TTS is 9.63 and 0.53. F5-TTS$+$Conv2Audio trades much alignment robustness (+1.61 WER) with a slightly higher speaker similarity (+0.01 SIM), which is not ideal for scaling up. We found that the long skip connection structure can not simply fit into DiT to improve speaker similarity, while the ConvNeXt for input text refinement can not directly apply to the flat U-Net Transformer to improve WER as well, both showing significant degradation of performance. To further analyze the unsatisfactory results with E2 TTS, we studied the consistent failure (unable to solve with re-ranking) on a 7% of the test set (WER$>$50%) all along the training process. We found that E2 TTS typically struggles with around 140 samples which we speculate to have a large distribution gap with the train set, while F5-TTS easily tackles this issue.

We investigate the models’ behaviors with different input conditions to illustrate the advantages of F5-TTS further and disclose the possible reasons for E2 TTS’s deficiency. See from Tab.4 in Appendix B.2, providing the ground truth duration allows more gains on WER for F5-TTS than E2 TTS, showing its robustness in alignment. By dropping the audio prompt, and synthesizing speech solely with the text prompt, E2 TTS is free of failures. This phenomenon implied a deep entanglement of semantic and acoustic features within E2 TTS’s model design. From Tab.3 GFLOPs statistics, F5-TTS carries out faster training and inference than E2 TTS.

The aforementioned limitations of E2 TTS greatly hinder real-world application as the failed generation cannot be solved with re-ranking. Supervised fine-tuning facing out-of-domain data or a tremendous pretraining scale is mandatory for E2 TTS, which is inconvenient for industrial deployment. On the contrary, our F5-TTS better handles zero-shot generation, showing stronger robustness.

### 5.2 Ablation of Sway Sampling

It is clear from Fig.3 that a Sway Sampling with a negative $s$ improves the generation results. Further with a more negative $s$, models achieve lower WER and higher SIM scores. We additionally include comparing results on base models with and without Sway Sampling in Appendix B.4.

As stated at the end of Sec.3.2, Sway Sampling with $s<0$ scales more flow step toward early-stage inference ($t\to 0$), thus having CFM models capture more startup information to sketch the contours of target speech better. To be more concrete, we conduct a “leak and override" experiment. We first replace the Gaussian noise input $x_{0}$ at inference time with a ground-truth-information-leaked input $(1-t^{\prime})x_{0}+t^{\prime}x_{ref}^{\prime}$, where $t^{\prime}=0.1$ and $x_{ref}^{\prime}$ is a duplicate of the audio prompt mel features. Then, we provide a text prompt different from the duplicated audio transcript and let the model continue the subsequent inference (skip the flow steps before $t^{\prime}$). The model succeeds in overriding leaked utterances and producing speech following the text prompt if Sway Sampling is used, and fails without. Uniformly sampled flow steps will have the model producing speech dominated by leaked information, speaking the duplicated audio prompt’s context. Similarly, a leaked timbre can be overridden with another speaker’s utterance as an audio prompt, leveraging Sway Sampling.

The experiment result is a shred of strong evidence proving that the early flow steps are crucial for sketching the silhouette of target speech based on given prompts faithfully, the later steps focus more on formed intermediate noisy output, where our sway-to-left sampling ($s<0$) finds the profitable niche and takes advantage of it. We emphasize that our inference-time Sway Sampling can be easily applied to existing CFM-based models without retraining. And we will work in the future to combine it with training-time noise schedulers and distillation techniques to further boost efficiency.

Figure: Figure 3: Probability density function of Sway Sampling on flow step $t$ with different coefficient $s$ (left), and small models’ performance on Seed-TTS test-zh with Sway Sampling (right).
Refer to caption: x3.png

## 6 Conclusion

This work introduces F5-TTS, a fully non-autoregressive text-to-speech system based on flow matching with diffusion transformer (DiT).
With a tidy pipeline, literally text in and speech out, F5-TTS achieves state-of-the-art zero-shot ability compared to existing works trained on industry-scale data. We adopt ConvNeXt for text modeling and propose the test-time Sway Sampling strategy to further improve the robustness of speech generation and inference efficiency.
Our design allows faster training and inference, by achieving a test-time RTF of 0.15, which is competitive with other heavily optimized TTS models of similar performance.
We open-source our code, and models, to enhance transparency and facilitate reproducible research in this area.

#### Acknowledgments

The authors would like to express our gratitude to Tianrui Wang, Xiaofei Wang, Yakun Song, Yifan Yang, Yiwei Guo, and Yunchong Xiao for the valuable discussions.

#### Ethics Statements

This work is purely a research project. F5-TTS is trained on large-scale public multilingual speech data and could synthesize speech of high naturalness and speaker similarity. Given the potential risks in the misuse of the model, such as spoofing voice identification, it should be imperative to implement watermarks and develop a detection model to identify audio outputs.

## Appendix A Baseline Details

VALL-E 2 [] A large-scale TTS model shares the same architecture as VALL-E [] but employs a repetition-aware sampling strategy that promotes more deliberate sampling choices, trained on Libriheavy [] 50K hours English dataset. We compared with results reported in [].

MELLE [] An autoregressive large-scale model leverages continuous-valued tokens with variational inference for text-to-speech synthesis. Its variants allow to prediction of multiple mel-spectrogram frames at each time step, noted by MELLE-Rx with x denotes reduction factor. The model is trained on Libriheavy [] 50K hours English dataset. We compared with results reported in [].

Voicebox [] A non-autoregressive large-scale model based on flow matching trained with infilling task. We compared with the 330M parameters trained on 60K hours dataset English-only model’s results reported in [] and  [].

NaturalSpeech 3 [] A non-autoregressive large-scale TTS system leverages a factorized neural codec to decouple speech representations and a factorized diffusion model to generate speech based on disentangled attributes. The 500M base model is trained on Librilight [] a 60K hours English dataset. We compared with scores reported in [].

DiTTo-TTS [] A large-scale non-autoregressive TTS model uses a cross-attention Diffusion Transformer and leverages a pretrained language model to enhance the alignment. We compare with DiTTo-en-XL, a 740M model trained on 55K hours English-only dataset, using scores reported in [].

FireRedTTS [] A foundation TTS framework for industry-level generative speech applications. The autoregressive text-to-semantic token model has 400M parameters and the token-to-waveform generation model has about half the parameters. The system is trained with 248K hours of labeled speech data. We use the official code and pre-trained checkpoint to evaluate(^6^66[https://github.com/FireRedTeam/FireRedTTS](https://github.com/FireRedTeam/FireRedTTS)).

MaskGCT [] A large-scale non-autoregressive TTS model without precise alignment information between text and speech following the mask-and-predict learning paradigm. The model is multi-stage, with a 695M text-to-semantic model (T2S) and then a 353M semantic-to-acoustic (S2A) model. The model is trained on Emilia [] dataset with around 100K Chinese and English in-the-wild speech data. We compare with results reported in [].

Seed-TTS [] A family of high-quality versatile speech generation models trained on unknown tremendously large data that is of orders of magnitudes larger than the previously largest TTS systems []. Seed-TTSDiT is a large-scale fully non-autoregressive model. We compare with results reported in [].

E2 TTS [] A fully non-autoregressive TTS system proposes to model without the phoneme-level alignment in Voicebox, originally trained on Libriheavy [] 50K English dataset. We compare with our reproduced 333M multilingual E2 TTS trained on Emilia [] dataset with around 100K Chinese and English in-the-wild speech data.

CosyVoice [] A two-stage large-scale TTS system, first autoregressive text-to-token, then a flow matching diffusion model. The model is of around 300M parameters, trained on 170K hours of multilingual speech data. We obtain the evaluation result with the official code and pre-trained checkpoint(^7^77[https://huggingface.co/model-scope/CosyVoice-300M](https://huggingface.co/model-scope/CosyVoice-300M)).

## Appendix B Experimental Result Supplements

### B.1 Small Model Configuration

The detailed configuration of small models is shown in Tab.3. In the Transformer column, the numbers denote the Model Dimension, the Number of Layers, the Number of Heads, and the multiples of Hidden Size. In the ConvNeXt column, the numbers denote the Model Dimension, the Number of Layers, and the multiples of Hidden Size. GFLOPs are evaluated using the thop Python package.

As mentioned in Sec.3.2, F5-TTS leverages an adaLN DiT backbone, while E2 TTS is a flat U-Net equipped Transformer. F5-TTS$+$LongSkip adds an additional long skip structure connecting the first to the last layer in the Transformer. For the Multi-Model Diffusion Transformer (MMDiT) [], a double stream transformer, the setting denotes one stream configuration.

**Table 3: Details of small model configurations.**
| Model | Transformer | ConvNeXt | #Param. | GFLOPs |
| --- | --- | --- | --- | --- |
| F5-TTS | 768,18,12,2 | 512,4,2 | 158M | 173 |
| F5-TTS$-$Conv2Text | 768,18,12,2 | - | 153M | 164 |
| F5-TTS$+$Conv2Audio | 768,16,12,2 | 512,4,2 | 163M | 181 |
| F5-TTS$+$LongSkip | 768,18,12,2 | 512,4,2 | 159M | 175 |
| E2 TTS | 768,20,12,4 | - | 157M | 293 |
| E2 TTS$+$Conv2Text | 768,20,12,4 | 512,4,2 | 161M | 301 |
| MMDiT [] | 512,16,16,2 | - | 151M | 104 |

### B.2 Ablation study on Input Condition

The ablation study on different input conditions is conducted with three settings: common input with text and audio prompts, providing ground truth duration information rather than an estimate, and retaining only text input dropping audio prompt. In Tab.4, all evaluations take the 155M small models’ checkpoints trained on WenetSpeech4TTS Premium at 800K updates.

**Table 4: Ablation study on different input conditions. The boldface indicates the best result, and the underline denotes the second best. All scores are the average of three random seed results.**
| Model | Common Input | Ground Truth Dur. | Drop Audio Prompt |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
|  | WER ↓ | SIM ↑ | WER ↓ | SIM ↑ | WER ↓ | SIM ↑ |
| F5-TTS | 4.17 | 0.54 | 3.87 | 0.54 | 3.22 | 0.21 |
| F5-TTS$+$Conv2Audio | 5.78 | 0.55 | 5.28 | 0.55 | 3.78 | 0.21 |
| F5-TTS$+$LongSkip | 5.17 | 0.53 | 5.03 | 0.53 | 3.35 | 0.21 |
| E2 TTS | 9.63 | 0.53 | 9.48 | 0.53 | 3.48 | 0.21 |
| E2 TTS$+$Conv2Text | 18.10 | 0.49 | 17.94 | 0.49 | 3.06 | 0.21 |

### B.3 Comparison of ODE Solvers

The comparison results of using the Euler or midpoint ODE solver during F5-TTS inference are shown in Tab.B.3. The Euler is inherently faster (first-order) and performs slightly better typically for larger NFE inference with Sway Sampling (otherwise the Euler solver results in degradation).

**Table 5: Evaluation results of F5-TTS (F5) on LibriSpeech-PC test-clean, Seed-TTS test-en and Seed-TTS test-zh, employing the Euler or midpoint ODE solver, and with different Sway Sampling $s$ values. The Real-Time Factor (RTF) is computed with the inference time of 10s speech.**
| Model   LibriSpeech-PC test-clean | Seed-TTS test-en | Seed-TTS test-zh |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| WER(%)↓ | SIM-o ↑ | WER↓ | SIM-o ↑ | WER↓ | SIM-o ↑ | RTF ↓ |  |
| Ground Truth | 2.23 | 0.69 | 2.06 | 0.73 | 1.26 | 0.76 | - |
| \cdashline1-8<br>$s=-1$ |  |  |  |  |  |  |  |
| F5 (16 NFE Euler) | 2.53 | 0.66 | 1.89 | 0.67 | 1.74 | 0.75 | 0.15 |
| F5 (16 NFE midpoint) | 2.43 | 0.66 | 1.88 | 0.66 | 1.61 | 0.75 | 0.26 |
| F5 (32 NFE Euler) | 2.42 | 0.66 | 1.83 | 0.67 | 1.56 | 0.76 | 0.31 |
| F5 (32 NFE midpoint) | 2.41 | 0.66 | 1.87 | 0.66 | 1.58 | 0.75 | 0.53 |
| \cdashline1-8<br>$s=-0.8$ |  |  |  |  |  |  |  |
| F5 (16 NFE Euler) | 2.82 | 0.65 | 2.14 | 0.65 | 2.28 | 0.72 | 0.15 |
| F5 (16 NFE midpoint) | 2.58 | 0.65 | 1.86 | 0.65 | 1.70 | 0.73 | 0.26 |
| F5 (32 NFE Euler) | 2.50 | 0.66 | 1.81 | 0.67 | 1.62 | 0.75 | 0.31 |
| F5 (32 NFE midpoint) | 2.42 | 0.66 | 1.84 | 0.66 | 1.62 | 0.75 | 0.53 |

### B.4 Sway Sampling Effectiveness on Base Models

From Tab.B.3, it is clear that our Sway Sampling strategy for test-time flow steps consistently improves the zero-shot generation performance in aspects of faithfulness to prompt text (WER) and speaker similarity (SIM). The gain of applying Sway Sampling to E2 TTS [] proves that our Sway Sampling strategy is universally applicable to existing flow matching based TTS models.

### B.5 ELLA-V Hard Sentences Evaluation

ELLA-V [] proposed a challenging set containing 100 difficult textual patterns evaluating the robustness of the TTS model. Following previous works [, , ], we include generated samples in our demo page(^8^88[https://SWivid.github.io/F5-TTS](https://SWivid.github.io/F5-TTS)). We additionally compare our model with the objective evaluation results reported in E1 TTS [].

StyleTTS 2 is a TTS model leveraging style diffusion and adversarial training with large speech language models. CosyVoice is a two-stage large-scale TTS system, consisting of a text-to-token AR model and a token-to-speech flow matching model. Concurrent with our work, E1 TTSDMD is a diffusion-based NAR model with a distribution matching distillation technique to achieve one-step TTS generation.
Since the prompts used by E1 TTSDMD are not released, we randomly sample 3-second-long speeches in our LibriSpeech-PC test-clean set as audio prompts.
The evaluation result is in Tab.7. We evaluate the reproduced E2 TTS and our F5-TTS with 32 NFE and Sway Sampling and report the averaged score of three random seed results.

**Table 7: Results of zero-shot TTS WER on ELLA-V hard sentences. The asterisk * denotes the score reported in E1 TTS. Sub. for substitution, Del. for Deletion, and Ins. for Insertion.**
| Model | WER(%))↓ | Sub.(%)↓ | Del.(%)↓ | Ins.(%)↓ |
| --- | --- | --- | --- | --- |
| StyleTTS 2 [] | 4.83* | 2.17* | 2.03* | 0.61* |
| CosyVoice [] | 8.30* | 3.47* | 2.74* | 1.93* |
| E1 TTSDMD [] | 4.29* | 1.89* | 1.62* | 0.74* |
| E2 TTS [] | 8.58 | 3.70 | 4.82 | 0.06 |
| F5-TTS | 4.40 | 1.81 | 2.40 | 0.18 |

We note that a higher WER compared to the results on commonly used test sets is partially due to mispronunciation (yogis to yojus, cavorts to caverts, etc.). The high Deletion rate indicates a word-skipping phenomenon when our model encounters a stack of repeating words. The low Insertion rate makes it clear that our model is free of endless repetition. We further emphasize that prompts from different speakers will spell very distinct utterances, where the ASR model transcribes correctly for one, and fails for another (e.g. quokkas to Cocos).

## Peer Reviews

confidence: 5
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. If s is getting smaller (e.g. -1, -0.9), what will be the performance of the sway sampling?

2. Although this is an empirical engineering study, I'm quite curious why ConvNeXt did not result in a performance improvement for the original E2-TTS, whereas the experiments showed that it was beneficial for DiT blocks.

3. Why Conv2audio is harmful for F5-TTS, but it is helpful for text, interms of WER?

4. Sway sampling is interesting. Have you tried to observe the mel spectrogram in different sampling stages, to verify your assumptions that "that the early flow steps are crucial for sketching the silhouette of target speech based on given prompts faithfully, the later steps focus more on formed intermediate noisy output"?
soundness: 3
strengths: The model performance is good and the audio sample is impressive. Experimental validation is sufficient and rigorous, which well demonstrates the effectiveness of each technique. Overall it's a good exploring exmrical engineering work of E2-TTS. The writing is clear and well-structured.
summary: The work is an incremental engineering effort based on E2-TTS.It replaces E2-TTS's original Transformer + U-Net structure with a Diffusion Transformer. Additionally, it employs a ConvNeXt V2 to encode text sequences before passing them to the transformer. Sway Sampling strategy is introduced during model inference, which performs non-uniform sampling during inference, assigning more weight to the early stages of th steps. It can enhance both the performance and efficiency of the model.
weaknesses: Although this work demonstrates impressive performance, it appears to be primarily an incremental engineering exploration of E2-TTS, rather than a novel contribution. 

The major contribution lies in replacing the transformer with DiT blocks and incorporating a ConvNeXt for text encoding. Most of the model structure settings are based on empirical ablation performance (see Section 5.1).
While the ablation study shows improvements brought by these techniques, the authors do not provide a theoretical analysis of their effectiveness, which overall makes the work contribution relatively weak. 

Furthermore, the discussion section does not offer insightful analysis.

---

confidence: 5
contribution: 1
initial_rating: 3
presentation: 2
questions: - The phrase "Fairytaler Fakes Fluent" in the title does not clearly convey the content or main contributions of the paper.
- It is worth examining whether entanglement is a structural limitation in both E2-TTS and F5-TTS. To clarify, the paper could include an ablation study where, under the same configuration and data setting as F5-TTS, a modified model uses cross-attention for text conditioning instead of adding filler-padded text to speech. This would help determine if the existing entanglement issues are reduced by changing the conditioning method.
- The paper claims that the two-stage control of CFG aids text alignment, yet it does not experimentally demonstrate this. It would be beneficial to show, through experiments, what specific alignment issues arise with the original CFG training method and how the two-stage control addresses these problems.
- Line 357: Information about the Mean Opinion Score (MOS) evaluation process is insufficient. Clarifying whether a platform like MTurk was used, the number and qualifications of evaluators hired, and providing examples of evaluation scripts would improve transparency. Additionally, mentioning studies that used similar MOS evaluation standards could provide a basis for comparison.
- Line 426: The paper refers to the performance of "F5-TTS−Conv2Text," but no related data appears in any tables or figures. The phrase "fails to learn" suggests that this model entirely failed at learning alignment, which is puzzling given its similarity to E2-TTS in setup. An explanation of why this model fails, in contrast to E2-TTS, is needed to understand the effectiveness and limitations of ConvNeXt within the model.

typo

- While the captions of Tables 1 and 2 mention results *w/o* SS, the tables themselves do not include these values.
- Line 192 should explicitly designate the filler token as <F>.
soundness: 2
strengths: The paper is clearly written and easy to follow, effectively outlining its research goal of addressing the slow convergence speed and low robustness of E2-TTS, a prior work in the emerging TTS paradigm. By increasing the capacity of text representation and improving inference sampling, it presents a clear approach to tackling these issues.
summary: The paper introduces F5-TTS, a non-autoregressive text-to-speech (TTS) system utilizing flow matching with Diffusion Transformer (DiT). F5-TTS uses ConvNeXt to enhance text representation and introduces a Sway Sampling strategy to boost performance and efficiency compared to E2-TTS model.
weaknesses: We expect a scientific paper rather than a technical report. Based on this expectation, the paper lacks the following content.

Originality and Significance:

- The paper's approach to addressing the issues of slow convergence and low robustness primarily involves augmenting the model with ConvNeXt for enhanced text representation. This raises concerns about whether simply adding ConvNeXt constitutes a sufficient contribution to the TTS research community. Fundamentally, it remains unclear why E2-TTS-like architecture is capable of learning some degree of alignment without cross-attention mechanisms. Is this alignment ability a consequence of scaling up the dataset?
- Beyond presenting favorable results, the paper lacks intuitive explanations for why these improvements occur. The paper does not provide performance comparisons with other architectures that naturally emerge as alternatives to ConvNeXt, such as Transformers [1], Conformers [2], or Mamba [3].
- The proposed Sway Sampling strategy appears to be a variation of the cosine schedule, which limits its novelty. The significance of inference sampling schedules in diffusion models is already well-known and has been extensively researched in prior works like DDPM [4], DDIM [5], and DPM-Solver [6].

Quality:

- There is a significant discrepancy between the scores in the original E2-TTS paper and the reproduced one. While it is understandable that reimplementation is necessary, this raises doubts about whether E2-TTS has been accurately reproduced.
- Also, only the reimplementation's metrics are compared for E2-TTS. It would be more consistent to compare either the reported metrics from all original papers or metrics from accurately reproduced implementations. Notably, when comparing with the original E2-TTS reported metrics, F5-TTS exhibits a higher WER.
- The use of Whisper-large-v3 for measuring WER differs from the evaluation models used in other papers. This raises concerns about the fairness and comparability of the results.
- Pronunciation errors still occur in demo samples—such as failing to pronounce "y'know" in the F5-TTS sample for "Disgust" in the Emotion section of the demo. Additionally, the prosody can become completely flat with abnormal input lengths, as seen in the 1.3x Speed sample for the first prompt in the "Speed Control" section of the E2-TTS demo page. It is challenging to ascertain how much these issues are mitigated compared to the original E2-TTS.
- For the above reasons, claiming state-of-the-art performance in the conclusion may be an overstatement.
- The paper mentions that it "simply estimates the duration based on the ratio," yet there appears to be a significant deviation when ground truth is provided. Given the performance gap, it might be beneficial to include additional modules to improve duration prediction, even if it compromises some efficiency. Using duration predictors from existing methods—such as summing phoneme durations to generate the total length, directly matching the total length—could potentially enhance performance.

Clarity:

- The paper lacks clear explanations for how several hyperparameters were determined. For example, why is the CFG scale set to 2? How were the ratios selected in the two-stage control of CFG? Referring to Figure 3, it seems that values of s < −0.8 might yield better metrics.
- In line 267, the paper mentions that having some gain when providing GT implies robustness. However, robustness is typically indicated when the difference in WER between using GT and not using GT is minimal. This reasoning needs further clarification.
- The term "re-ranking" used in line 461 is not adequately explained. The content described in lines 461-464 is difficult to interpret, and its significance and connection to the overall conclusions of the paper are not clear.
- In line 241, it seems that F5-TTS still exhibits significant length differences, similar to E2-TTS.
- There are several instances where proper citations are necessary to clarify references to existing work:
    - Lines 86-87: Is the paper referring to the monotonic alignment search method from Glow-TTS?
    - Line 317: Does this line relate to the cross-sentence task introduced in Voicebox?
    - Line 319: Is the paper referencing SIM-o from Voicebox?
    - Line 431: Does this pertain to the long skip connections used in DiTTo-TTS?

[1] Vaswani, A. (2017). Attention is all you need. *Advances in Neural Information Processing Systems*.

[2] Gulati, A., Qin, J., Chiu, C. C., Parmar, N., Zhang, Y., Yu, J., ... & Pang, R. (2020). Conformer: Convolution-augmented transformer for speech recognition. *arXiv preprint arXiv:2005.08100*.

[3] Gu, A., & Dao, T. (2023). Mamba: Linear-time sequence modeling with selective state spaces. *arXiv preprint arXiv:2312.00752*.

[4] Ho, J., Jain, A., & Abbeel, P. (2020). Denoising diffusion probabilistic models. *Advances in neural information processing systems*, *33*, 6840-6851.

[5] Song, J., Meng, C., & Ermon, S. (2020). Denoising diffusion implicit models. *arXiv preprint arXiv:2010.02502*.

[6] Lu, C., Zhou, Y., Bao, F., Chen, J., Li, C., & Zhu, J. (2022). Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. *Advances in Neural Information Processing Systems*, *35*, 5775-5787.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: - I would like to inquire if comparisons with other sampling strategies from the TTS field or related domains could be provided—starting from more straightforward methods like the Heun sampler and beyond. 

- Given the lightweight approach to encoding text using an additional layer (ConvNeXt) compared to a language model, my impression is that this method may require a substantial amount of data to robustly learn pronunciation and speech-text alignment. As the authors highlight the stability of alignment training as a strength of this work, I am curious to know the lower limits of the data threshold while still maintaining stable training (although I understand that the paper's primary focus may not be on stability relative to data quantity; this question is posed out of pure curiosity). For instance, would it be feasible to achieve stable character-speech alignment training with a single-speaker dataset such as LJSpeech, which contains around 20 hours of data? Additionally, how much training time is typically needed to achieve stable pronunciation accuracy? Including results that reflect the impact of varying training data amounts would add valuable insights.

If these questions are addressed, I would like to increase the rating.
soundness: 3
strengths: - The proposed elements are clearly articulated, and each is thoroughly examined through ablation studies.

- The authors have made commendable efforts to compare their model against a wide range of baselines and benchmarks. Recently, many zero-shot TTS models, particularly those trained on large-scale datasets, have refrained from open-sourcing for various reasons, making it challenging to evaluate new models. In this regard, it is a distinct strength that the authors not only disclose their evaluation data and methods, such as LibriSpeech-PC, but also conduct comprehensive comparisons with diverse models.

- Similarly, the paper states that they plan to release their code and checkpoints soon, which is another clear advantage. Given the limited availability of large-scale models, making their code and model accessible would greatly benefit other researchers in the field.
summary: The authors propose F5-TTS, a zero-shot text-to-speech model that operates without explicit alignment or a pre-trained text encoder. Key distinguishing features from previous research include: (1) the use of a DiT-based structure as an estimator, (2) an additional module (ConvNeXt) for character encoding, and (3) the introduction of the Sway Sampling strategy to improve sampling efficiency. The model demonstrates respectable speaker similarity, audio quality, and pronunciation accuracy in English and Chinese benchmarks. The necessity of the proposed components is further substantiated through ablation studies.
weaknesses: - While I do not personally doubt the effectiveness of the proposed methodology, the contributions feel somewhat limited. This is not intended to downplay the model's design and structure for stable training. However, as the authors themselves mention, adopting a DiT structure or performing encoding to compress textual information has been already proposed in previous studies.

- Regarding Sway Sampling, a richer comparison with other sampling strategies could have highlighted its effectiveness more distinctly, especially given the recent influx of sampling techniques based on flow matching and diffusion. In addition to a thorough hyperparameter search within Sway Sampling or evaluating the effects of switching it on and off, comparisons with alternative sampling strategies would have further strengthened the impact of the proposed method.

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: 1. Even though the supplemental results show WER and RTF improvements over E2-TTS, in the main text, it appears F5 performs on par with previous methods. Especially, in Table 1, the WER seems to be worse than the closely related two models. Could you please explain the comparison more clearly in the main text? 

2. Regarding the speaker similarity metrics vs. the ablation in Section 5.2: Do the authors think that Sway sampling has the advantage which is mentioned in Section 5.2 but at the same time hurting speaker similarity to some extend? 

3. Since in Table 1 the paper compares the punctuation added and capitalized version of Librispeech with the widely known original version, have the authors tried a model without including punctuation in their vocabulary? If yes, how does it compare to the presented results?
soundness: 3
strengths: Originality, 

- Sufficiently original: The paper builds on the idea of E2-TTS which pads the character sequence up to the number of speech feature frames and with different modeling choices (transformer instead of Unet, and ConveNext based text frontend), the proposed system can handle better speech and text alignment. 

- The paper also proposes Sway sampling strategy to the inference instead of uniform time step sampling. The paper suggests that one advantage of this strategy is that it can prevent speaker information leaks in the cases where a speaker or context change occurs.

- The way the paper uses time step representation in the training setup is different than prior studies.

Quality, 

- Several ablation experiments are performed. Some of those results are included in the Appendix. 


Clarity,

- Writing is clear.

- Experimental setup details are mentioned in detail.  


Significance

- Not requiring additional modules (e.g. a duration model) is an advantage of the proposed method.
summary: This study uses flow-matching based diffusion model for TTS. As opposed to the earlier studies which usually require additional modules such as an alignment model (e.g. in Voicebox), the proposed model pads the character sequence up to the length of the speech features and uses a ConvNext-like model to let the text features align with speech during the training process. The paper also proposes to use Sway sampling during model inference. This strategy samples the timestep non-uniformly. According to the experiments on Librispeech and a Chinese set, the proposed model results in comparable WER on the test data with previous models (Voicebox and E2-TTS). The RTF also improves in the proposed model.

After reading the rebuttal, I would like to maintain my score.
weaknesses: 1. One problem in experimental settings is the lack of direct comparison between Voicebox, E2-TTS and the current study. The paper already mentions that the prompting test sets are not known and hence it is hard to compare, a reproduction of the setups could have solved this problem. In the Appendix, it seems that there are some comparisons with E2. It would be better to include some of those comparisons in the main text. 

2. Table 1, the upper section related to Librispeech test-clean does not show the proposed system's results.

3. There are some minor issues in table captions: Tables 1 and 2 mention that (w/o SS) means without Sway Sampling, but this phrase is not used anywhere in these tables.

Meta-review: The paper proposes to use a different embedding model and a different sampling procedure to improve a nonautoregressive synthesis model, E2 TTS.

I recommend a rejection because, despite the improvement and impressive performance, the root cause of the improvement is unclear. Other than reviewer X8VX, much of the argument among reviewers and authors is around this issue. Without identifying the root cause, the paper reads as if the motivation is missing (reviewer abUH) and the contribution is limited (reviewer mAVA and 7LV6). Lengthy discussion among the reviewers and the authors were still not enough to convince the reviewers.

I have to emphasize this is not about ablation study, as several ablation experiments are already included in the paper. it's about providing a deeper understanding of the problem. For example, the paper confirms from experiments that proposed approach improves convergence, but we do not know **why** convergence is slow to begin with and **how** convergence becomes faster once we employ the changes.

This paper has great potential and I encourage the authors to carefully consider the feedback from reviewers.

Decision: Reject