DiTTo-TTS: Diffusion Transformers for Scalable Text-to-Speech without Domain-Specific Factors

## Abstract

Abstract Large-scale diffusion models have shown outstanding generative abilities across multiple modalities including images, videos, and audio. However, text-to-speech (TTS) systems typically involve domain-specific modeling factors (e.g., phonemes and phoneme-level durations) to ensure precise temporal alignments between text and speech, which hinders the efficiency and scalability of diffusion models for TTS.
In this work, we present an efficient and scalable Diffusion Transformer (DiT) that utilizes off-the-shelf pre-trained text and speech encoders. Our approach addresses the challenge of text-speech alignment via cross-attention mechanisms with the prediction of the total length of speech representations. To achieve this, we enhance the DiT architecture to suit TTS and improve the alignment by incorporating semantic guidance into the latent space of speech. We scale the training dataset and the model size to 82K hours and 790M parameters, respectively.
Our extensive experiments demonstrate that the large-scale diffusion model for TTS without domain-specific modeling not only simplifies the training pipeline but also yields superior or comparable zero-shot performance to state-of-the-art TTS models in terms of naturalness, intelligibility, and speaker similarity. Our speech samples are available at https://ditto-tts.github.io .

## 1 Introduction

Large-scale diffusion models have demonstrated impressive generative abilities in a wide range of fields including images [, ], videos [, ], and audio [, ]. The integration of latent diffusion models (LDMs) [] further amplifies their popularity [, , ], as these models significantly enhance computational efficiency. This efficiency is achieved through the reduction of input dimensionality using autoencoders, which also enables the diffusion models to focus on the most critical features of data []. However, applying LDMs to text-to-speech (TTS) presents unique challenges because TTS requires precise alignment between text and generated speech over time. Hence, their application to TTS often requires a complex pipeline, incorporating speech domain-specfic modeling such as phoneme and duration [, ]. Without these components, generation performance tends to be suboptimal [, ], while their inclusion hinders the model efficiency and scalability.

In this work, we present a latent diffusion model for TTS that is integrated with off-the-shelf pre-trained text and speech encoders without relying on the speech domain-specific modeling. Our method addresses the challenge of aligning text and speech solely through cross-attention mechanisms. Furthermore, we introduce a module that predicts the total duration of the generated speech from a given text and speech prompt, rather than determining the duration of each individual input token. To accomplish this, we conduct a network architecture ablation to identify a model specifically suited for TTS applications. Consequently, we adopt the Diffusion Transformer (DiT) [] to TTS, naming our method DiTTo-TTS (or simply DiTTo). We also explore the significance of leveraging aligned text and speech representations, showing that performance can be enhanced by either using a text encoder jointly trained with speech data or a speech autoencoder with an auxiliary language modeling objective alongside the reconstruction objective.

Our comprehensive experiments on English-only and multilingual evaluation demonstrate that our model not only simplifies the training process but also achieves superior or comparable zero-shot performance to state-of-the-art models in terms of naturalness, intelligibility, and speaker similarity. The base-sized DiTTo surpasses a state-of-the-art autoregressive model [], offering an inference speed 4.6 times faster and a model size 3.84 times smaller. Additionally, we demonstrate that our model scales effectively with increases in both data and model sizes.

Figure: Figure 1: An overview of DiTTo-TTS. (middle) The LDM backbone is trained to denoise a span-masked noisy segment given its contextual counterpart, without utilizing phoneme and phoneme-level duration (see Section 3.1). (left) The inner structure of the DiT block incorporates multi-head cross-attention with global AdaLN (see Section 3.3). (right) The speech length predictor is based on causal transformer blocks (see Section 3.2). Both DiT blocks and the speech length predictor employ cross-attention to condition on text representation. Additionally, DiT blocks utilize AdaLN with the mean-pooled text embedding. ‘+’ denotes addition, and ‘c’ represents concatenation. ‘text’ in sky blue oval represents the embedding from text encoder. The red line indicates a long skip connection between the states before and after the DiT blocks. The effectiveness of our design choices is demonstrated in our architecture search (see Section 6.1).
Refer to caption: x1.png

## 2 Related Work

#### Large-scale TTS

Recently, large-scale TTS research progresses actively in two main directions: LLM-based autoregressive (AR) TTS and non-autoregressive (Non-AR) TTS. A prominent feature of LLMs is the scalability [, ] and their proficiency in zero-shot learning tasks, demonstrating significant capabilities without prior specific training on those tasks [, , , ]. Efforts to replicate LLM’s capability in different modalities have shown progress, including vision [, , ] and audio [, , , ]. VALL-E [] employs EnCodec [] for speech-to-token mapping, posing TTS tasks as AR language modeling tasks, thus enabling zero-shot capabilities in the speech domain. CLaM-TTS [] introduces Mel-VAE to achieve superior token length compression, and enables a language model to predict multiple tokens simultaneously. Although this approach removes the need for cascaded modeling [] to manage the number of token streams, their resource-intensive inference processes limit their applications []. On the other hand, Non-AR generative models are employed to enhance the efficiency of TTS systems. Voicebox [] utilizes a flow matching [] to generate speech, effectively casting the TTS task into a speech infilling task. NaturalSpeech series [, ], building upon recent advances in the Latent Diffusion Model (LDM) [], incorporate auxiliary modules for controllability of various speech attribute such as content, prosody, and timbre. However, requiring supplementary data beyond speech-transcription pairs hinders scalability. Simple-TTS [] simplifies the data preparation and training process by removing auxiliary modules and the need for phoneme-level durations. Similarly, E3-TTS [] follows this approach but does not utilize a pre-trained latent autoencoder. However, both models have limitations in audio quality and impose a fixed length on the target audio, which is a significant constraint for speech generation.

#### Latent Diffusion Model (LDM)

LDM [] improves modeling efficiency of the diffusion process model [, ] by operating in a latent space, achieving remarkable performance in generating realistic samples. Initially applied in image generation, their success is attributed to the reduced dimensionality of the latent space, facilitating efficient training and sampling []. Notably, guided diffusion [, ] has been expanded to various applications of LDMs, such as image editing [] and image retrieval []. In the field of audio signals, techniques such as style transfer, inpainting, and super-resolution have been explored, along with text-guided audio and speech generation [, ]. In the context of TTS, however, applying LDMs to TTS [, ] necessitates domain-specific elements such as phonemes, phoneme-level durations, and pitch. This is primarily due to the need for precise temporal alignment between text and speech, as well as the higher fidelity requirements inherent in audio data.

#### Neural Audio Codec

Neural audio codecs, which effectively compress various types of audio using neural networks, are used as part of many TTS systems [, , ]. Recent advancements employ an encoder-decoder architecture coupled with Residual Vector Quantization (RVQ) [, , ] to transform raw audio waves into discretized tokens. For example, EnCodec [] converts 24,000 Hz mono waveforms into 75 Hz latents. With a similar architecture, by focusing the compression specifically on speech rather than general audio signals, Mel-VAE [] achieves approximately 10.76 Hz latents by compressing the mel-spectrogram. This reduction significantly lowers the computational cost of the speech generation module.
Another research direction of improving neural audio codecs for TTS systems is injecting semantic information using large language models (LLMs) [].

## 3 Method

We present a latent diffusion model (LDM) for TTS that enables the use of off-the-shelf pre-trained text and speech encoders without relying on speech domain-specific modeling, such as phoneme and duration. Toward this goal, we employ the following two approaches: 1) introducing a speech length predictor that predicts the total length of the generated speech without relying on phoneme-level durations or requiring a fixed speech length; and 2) fine-tuning the pre-trained neural audio codec using a pre-trained language model to enhance the alignment between text and speech embeddings.

### 3.1 Preliminary

Diffusion models [, ] are a class of generative models that iteratively transform a simple noise distribution into a complex data distribution through a stochastic denoising process. They define a forward process that progressively adds Gaussian noise to the input data as time step increases. The reverse generative process then estimates the added noise to reconstruct the original data. Conditional diffusion models enhance this framework by incorporating additional information, such as text descriptions in text-to-image generation [, ] or phonemes and their durations in TTS [, ]. While diffusion models can operate directly on real-world data, many of them are applied in the latent space [, , , ]. Thanks to the reduced dimensionality, this approach improves computational efficiency and output quality by allowing diffusion models to focus on the semantic information of the data while the autoencoder handles the high-frequency details that are less perceptible [].

In our setting, a conditional LDM can be formulated as follows. Given speech audio, an autoencoder produces its latent representation ${\bm{z}}_{speech}$, and the diffusion model is trained to predict ${\bm{z}}_{speech}$ at each diffusion step $t\in[1,T]$ conditioned on a text token sequence ${\bm{x}}$.
Specifically, the noised latent ${\bm{z}}^{(t)}$ is expressed as $\alpha_{t}{\bm{z}}_{speech}+\sigma_{t}\bm{\epsilon}$, where $\bm{\epsilon}$ is sampled from the standard normal distribution and $\alpha_{t}$ and $\sigma_{t}$ are defined by a noise schedule. Note that ${\bm{z}}^{(1)}$ is ${\bm{z}}_{speech}$ and ${\bm{z}}^{(T)}$ follows the standard normal distribution. We use ${{\bm{v}}}$-prediction [] as our model output ${{\bm{v}}_{\theta}}({{\bm{z}}^{(t)}},{\bm{x}},t)$, which predicts ${{\bm{v}}^{(t)}}:=\alpha_{t}\bm{\epsilon}-\sigma_{t}{{\bm{z}}^{(t)}}$. This setup provides a mean squared error objective as the training loss:

$$ $\mathcal{L}_{\text{diffusion}}=\mathbb{E}_{t\sim\mathcal{U}(1,T),\bm{\epsilon} \sim\mathcal{N}(0,I)}\left[\|{{\bm{v}}^{(t)}}-{{\bm{v}}_{\theta}}({{\bm{z}}^{( t)}},{\bm{x}},t)\|^{2}\right].$ $$

To enrich the contextual information and facilitate zero-shot audio prompting, we incorporate a random span masking into the model training following [, ].
We input ${\bm{z}}_{mask}^{(t)}:=\bm{m}\odot{\bm{z}}^{(t)}+(1-\bm{m})\odot{\bm{z}}_{speech}$ to the model,
where $\odot$ indicates element-wise multiplication and the binary masking $\bm{m}$ fully masks with the probability of 0.1 or partially masks a random contiguous segment whose size is between 70% and 100% of data. We also use the binary span masking as an additional input to the model. This allows the model to explicitly identify which part needs to be generated. The inclusion of masking modifies the training loss to:

$$ $\mathcal{L}_{\text{diffusion}}=\mathbb{E}_{t\sim\mathcal{U}(1,T),\bm{\epsilon} \sim\mathcal{N}(0,I)}\left[\|\bm{m}\odot({{\bm{v}}^{(t)}}-{{\bm{v}}_{\theta}}( {{\bm{z}}_{mask}^{(t)}},\bm{m},{\bm{x}},t))\|^{2}\right].$ (1) $$

### 3.2 Model and Training

An overview of our proposed method is presented in Figure 1.

#### Text Encoder

We employ a text encoder from a pre-trained large language model $p_{\phi}$, which is parameterized by $\phi$.
The model was pre-trained to maximize the log-likelihood of the text token sequence $\log p_{\phi}({\bm{x}})$.
The parameters of the model are kept frozen while training the diffusion model for TTS. We denote the output of the text encoder by ${\bm{z}}_{text}$.

#### Neural Audio Codec

A neural audio codec, which is parameterized by $\psi$, comprises of three components: 1) an encoder that maps a speech into a sequence of latent representations ${{\bm{z}}}_{speech}$; 2) a vector quantizer converting the latent vector into the discrete code representation; and 3) a decoder that reconstructs the speech from a sequence of the quantized latent representations ${\hat{{\bm{z}}}}_{speech}$. To enhance alignment between text and speech embeddings, we fine-tune the neural audio codec using the pre-trained language model. We introduce a learnable linear projection $f(\cdot)$ to match the dimension of the latent representation ${{\bm{z}}}_{speech}$ to the language model’s hidden space. Subsequently, we use this projected embedding in place of the embedding from the pre-trained text encoder within the cross-attention operation of the language model’s decoder. The neural audio codec is fine-tuned with auxiliary loss that infuse semantic content into the generated representations:

$$ $\displaystyle\mathcal{L}(\psi)$ $\displaystyle=\mathcal{L}_{NAC}(\psi)+\lambda\mathcal{L}_{LM}(\psi),\quad \mathcal{L}_{LM}(\psi)=-\log{p_{\phi}(\bm{x}|f({{\bm{z}}_{speech}}))},$ (2) $$

where $\mathcal{L}_{NAC}(\psi)$ indicate the loss function which is used when the neural audio codec is pre-trained []. $\lambda$ controls the contribution of $\mathcal{L}_{LM}$, with $\lambda=0$ indicating the pre-training phase. When $\lambda>0$, a pre-trained language decoder performs causal language modeling on text token sequences ${{\bm{x}}}$, based on the speech latent vector ${{\bm{z}}_{speech}}$.
While the parameters of language model decoder are fixed, gradients are backpropagated to adjust the linear mappings $f({{\bm{z}}_{speech}})$. This training strategy aligns the speech latents with the linguistic latents of the pretrained language model during autoencoding.

#### Diffusion Model

We are given text embedding ${\bm{z}}_{text}$ and speech embedding ${\bm{z}}_{speech}$. We train the diffusion model $\bf{v_{\theta}}(\cdot)$ using the objective in Eq. (1), replacing ${\bm{x}}$ with ${\bm{z}}_{text}$:

$$ $\mathcal{L}_{\text{diffusion}}=\mathbb{E}_{t\sim\mathcal{U}(1,T),\bm{\epsilon} \sim\mathcal{N}(0,I)}\left[\|\bm{m}\odot({{\bm{v}}^{(t)}}-{{\bm{v}}_{\theta}}( {{\bm{z}}_{mask}^{(t)}},\bm{m},{\bm{z}}_{text},t))\|^{2}\right],$ (3) $$

where ${\bm{z}}_{mask}^{(t)}=\bm{m}\odot{\bm{z}}^{(t)}+(1-\bm{m})\odot{\bm{z}}_{speech}$ is the masked input latent, $\bm{m}$ is the binary span masking, and $t$ is the diffusion time step. We apply classifier-free guidance (CFG) [] and adopt the diffusion noise schedule from [].

#### Speech Length Predictor

We introduce a model designed to predict *the total length* of a generated speech for a given text rather than to estimate each phoneme’s duration, and the input noise of diffusion model is set by the length from the speech length predictor at inference time.
As shown in Figure 1, we employ an encoder-decoder transformer for the speech length predictor. The encoder processes text input bidirectionally to capture comprehensive textual context, while the decoder, equipped with causal masking to prevent future lookahead, receives an audio token sequence from the encoder of the neural codec for speech prompting at inference time.
We use cross-attention mechanisms to integrate text features from the encoder.
We use the softmax activation in the final layer to predict the number of tokens to be generated within the given maximum length $N$. Specifically, the ground truth label for the remaining audio length decreases by one at each subsequent time step. The model is trained separate from the diffusion model, using the cross-entropy loss function.

### 3.3 Model Architecture

We conduct a comprehensive model architecture search to identify the most suitable diffusion-based model for TTS, resulting in the adoption of the Diffusion Transformer (DiT) [] model (see Section 6.1). We adopt the DiT model in TTS while incorporating recent architectural advancements for transformer variants, such as the gated linear unit with GELU activation [], rotary position embeddings [], and cross-attention with global adaptive layer normalization (AdaLN) []. For the latent space, we employ Mel-VAE introduced in [] which is able to compress audio sequences approximately seven times more than EnCodec [], yet maintaining superior quality. Due to space limitations, additional details regarding model configurations are provided in Appendix 8.4. We also detail down our noise scheduler and CFG in Appendix 8.6.

## 4 Experimental Setup

#### Dataset

We employ 82K hours of over 12K unique speakers’ speech-transcript datasets spanning nine languages: English, Korean, German, Dutch, French, Spanish, Italian, Portuguese, and Polish. We train two models: (1) DiTTo-en, a model trained on 55K hour English-only dataset, and (2) DiTTo-multi, a multilingual model trained on 82K hour datasets. Details of each dataset are provided in Appendix 8.1.
We follow the data preprocessing methodology described in [], except that we include all samples without any filtering and exclude speaker metadata from the text prompts. It enables the on-the-fly processing of data with different sampling rates at a uniform rate of 22,050 Hz, approximating audio resampling. We use the same datasets for the speech length predictor with additional LibriSpeech [] sets: train-clean-100, train-clean-360, and train-other-500. We find that it helps minimize reliance on text normalization (see Appendix 8.1 for discussion). For the evaluation of DiTTo-en and baseline models, we use the test-clean subset of LibriSpeech, which consists of speech clips ranging from 4 to 10 seconds with transcripts. For DiTTo-multi, we randomly select 100 examples from the test set of each language dataset, with clip durations ranging from 4 to 20 seconds.

#### Training

Following DiT [], we train four different sizes of DiTTo: Small (S), Base (B), Large (L), and XLarge (XL). All models are trained on 4 NVIDIA A100 40GB GPUs, and use $T=$ 1,000 discrete diffusion steps. The S and B models of DiTTo-en are trained with a maximum token size of 5,120 and a gradient accumulation step of 2 over 1M steps. The L and XL models are trained with a maximum token size of 1,280 and a gradient accumulation step of 4 over 1M steps. The DiTTo-multi model is trained only in the XL configuration, with a maximum token size of 320 and a gradient accumulation step of 4 over 1M steps. The trainable model parameters for DiTTo-en S, B, L, and XL are 41.89M, 151.58M, 507.99M, and 739.97M, respectively, and 790.44M for DiTTo-multi. For the text encoder, we employ SpeechT5 [] (^1^11We use TTS fine-tuned SpeechT5 model from [https://huggingface.co/microsoft/speecht5_tts](https://huggingface.co/microsoft/speecht5_tts).) (as in VoiceLDM []) and ByT5 [] in DiTTo-en and DiTTo-multi, respectively. We use the AdamW optimizer [] with the learning late of 1e-4, beta values of (0.9, 0.999), and a weight decay of 0.0. We use a cosine learning rate scheduler with a warmup of 1K steps. Further details of the speech length predictor is provided in Appendix 8.5.

#### Inference

To generate speech, we first input the text and speech prompt into the speech length predictor, which determines the total length of the speech $L$ by adding speech prompt length with the predicted length. Further details about the speech length predictor can be found in Appendix 8.5. The diffusion backbone then generates the latent speech of length $L$ using the same text and speech prompt pair. The generated latent is decoded into mel-spectrograms using the Mel-VAE decoder, and then converted to raw waveform using a pre-trained vocoder, BigVGAN [].

#### Metrics

We use following objective metrics to evaluate the models: Character Error Rate (CER), Word Error Rate (WER), and Speaker Similarity (SIM) following the procedure outlined in [] and []. CER and WER indicate how intelligible and robust the model is, while SIM represents how well the model captures the speaker’s identity. For subjective metrics, we employ Similarity MOS (SMOS) to measure the speaker similarity between the prompt and the generated speech, and Comparative MOS (CMOS) to assess relative naturalness and audio quality. Details of each metric and evaluation can be found in Appendix 8.2.

#### Baselines

We compare the proposed model with state-of-the-art TTS models including (1) *autoregressive models*: VALL-E [], SPEAR-TTS [] , CLaM-TTS []; (2) *non-autoregressive models*: YourTTS [], VoiceBox []; and (3) *simple diffusion-based models*: Simple-TTS [].
Since Voicebox [], VALL-E [], NaturalSpeech 2 [], NaturalSpeech 3 [], Mega-TTS [], and E3-TTS [] are not publicly available, we bring the scores reported in the respective paper or samples provided in demo page. Please refer to Appendix 8.3 for details of each model.

#### Tasks

We evaluate our model on two tasks: 1) continuation:
given a text and an initial 3-second segment of corresponding ground truth speech, the task is to synthesize seamless subsequent speech that reads the text in the style of the provided speech segment;
2) *cross-sentence*: given a text, a 3-second speech segment, and its corresponding transcript (which differs from the given text), the task is to synthesize speech that reads the text in the style of the provided 3-second speech.

## 5 Experimental Results

### 5.1 Comparison with Baselines

**Table 1: Performances for the English-only *continuation* task. The boldface indicates the best result, the underline denotes the second best, and the asterisk denotes the score reported in the baseline paper. The inference time indicates the generation time of 10s speech, and #Param. refers to the number of learnable parameters (model size).**
|  | Objective Metrics |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Model | WER $\downarrow$ | CER $\downarrow$ | SIM-o $\uparrow$ | SIM-r $\uparrow$ | Inference Time $\downarrow$ | #Param. |
| Ground Truth | 2.2* | 0.61* | 0.754* | 0.754* | n/a | n/a |
| YourTTS [] | 7.57 | 3.06 | 0.3928 | - | - | - |
| VALL-E [] | 3.8* | - | 0.452* | 0.508* | $\sim$6.2s* | 302M |
| Voicebox [] | 2.0* | - | 0.593* | 0.616* | $\sim$6.4s* (64 NFE) | 364M |
| CLaM-TTS [] | 2.36* | 0.79* | 0.4767* | 0.5128* | 4.15s* | 584M |
| Simple-TTS [] | 3.86 | 2.24 | 0.4413 | 0.4668 | 17.897s (250 steps) | 243M |
| DiTTo-en-S | 2.01 | 0.60 | 0.4544 | 0.4935 | 0.884s | 42M |
| DiTTo-en-B | 1.87 | 0.52 | 0.5535 | 0.5855 | 0.903s | 152M |
| DiTTo-en-L | 1.85 | 0.50 | 0.5596 | 0.5913 | 1.479s | 508M |
| DiTTo-en-XL | 1.78 | 0.48 | 0.5773 | 0.6075 | 1.616s | 740M |

**Table 2: Performances for the English-only *cross-sentence* task.**
| Model | WER $\downarrow$ | CER $\downarrow$ | SIM-o $\uparrow$ | SIM-r $\uparrow$ |
| --- | --- | --- | --- | --- |
| YourTTS [] | 7.92 (7.7*) | 3.18 | 0.3755 (0.337*) | - |
| VALL-E [] | 5.9* | - | - | 0.580* |
| SPEAR-TTS [] | - | 1.92* | - | 0.560* |
| Voicebox [] | 1.9* | - | 0.662* | 0.681* |
| CLaM-TTS [] | 5.11* | 2.87* | 0.4951* | 0.5382* |
| Simple-TTS [] | 4.09 (3.4*) | 2.11 | 0.5026 | 0.5305 (0.514*) |
| DiTTo-en-S | 3.07 | 1.08 | 0.4984 | 0.5373 |
| DiTTo-en-B | 2.74 | 0.98 | 0.5977 | 0.6281 |
| DiTTo-en-L | 2.69 | 0.91 | 0.6050 | 0.6355 |
| DiTTo-en-XL | 2.56 | 0.89 | 0.6270 | 0.6554 |

For *English-only Evaluation*, we evaluate the performances of DiTTo-en across *continuation* and *cross-sentence* tasks.
Recall that DiTTo-en is the model trained on 55K English-only dataset. Following the evaluation setting in [], we employ a subset of the LibriSpeech test-clean dataset. This subset comprises speech clips ranging from 4 to 10 seconds, each with a corresponding transcript. We use open-sourced checkpoints of YourTTS (^2^22[https://github.com/Edresson/YourTTS](https://github.com/Edresson/YourTTS)) and Simple-TTS (^3^33[https://github.com/asappresearch/simple-tts](https://github.com/asappresearch/simple-tts)) for evaluations, and the official checkpoint of CLaM-TTS. When conducting subjective evaluations, all samples are resampled to 16,000 Hz. Further details can be found in Appendix 8.3.

Table 1 and 2 present the results of the *continuation* and *cross-sentence* tasks, respectively. Our model demonstrates excellent performance across all measures, consistently ranking either first or second. Specifically, the DiTTo-en base (B) model outperforms CLaM-TTS, a state-of-the-art (SOTA) autoregressive (AR) model, in terms of naturalness, intelligibility, and speaker similarity, while achieving an inference speed that is 4.6 times faster with 3.84 times smaller model size. Despite its simplicity, our model achieves performance comparable to more complex a non-autoregressive (Non-AR) SOTA TTS model which use a phoneme-level duration modeling. Additionally, DiTTo-en surpasses a simple diffusion-based model, Simple-TTS, further demonstrating its effectiveness.

Table 3 presents the results of subjective evaluations.
DiTTo-en significantly outperforms the baseline models, Simple-TTS and CLaM-en, and achieves performance comparable to the ground truth in terms of speaker similarity (as measured by SMOS) as well as naturalness, quality, and intelligibility (as assessed by CMOS). In Appendix 8.8, Table  9 shows the results of the subjective evaluation compared to the baseline demo samples. We use the provided voice samples as is (as detailed down in Appendix 8.3), and our model operates at 22,050 Hz. Our model surpasses all models except NaturalSpeech 3 in SMOS and all models except Voicebox in CMOS.

For *Multilingual Evaluations*, we provide the result in Appendix 8.9 due to space constraints. Table 10 presents the result of the *continuation* task for DiTTo-multi.
DiTTo-multi shows better or comparable performances compared to CLaM-TTS performances, as reported in Table 11.

**Table 3: Human evaluations on *cross-sentence* task with 40 LibriSpeech test-clean speakers show DiTTo-en-XL surpasses the baseline in quality, intelligibility, similarity, and naturalness, nearing Ground Truth. SMOS scores include a 95% confidence interval. The boldface indicates the best result and recon indicates the reconstructed audio by Mel-VAE followed by vocoder.**
| Model | SMOS | CMOS |
| --- | --- | --- |
| Simple-TTS [] | 2.15$\pm$0.19 | -1.64 |
| CLaM-en [] | 3.42$\pm$0.16 | -0.52 |
| DiTTo-en-XL | 3.91$\pm$0.16 | 0.00 |
| Ground Truth (recon) | 4.07$\pm$0.14 | +0.11 |
| Ground Truth | 4.08$\pm$0.14 | +0.13 |

**Table 4: Performances of English-only *cross-sentence* task for Mel-VAE++. All models are trained on MLS English and consumed ground truth length during sampling. The boldface indicates the best result within the same text encoder setting, and ^† represents the best result across all cases.**
| Text Encoder | Neural Audio Codec | WER $\downarrow$ | CER $\downarrow$ | SIM-o $\uparrow$ | SIM-r $\uparrow$ |
| --- | --- | --- | --- | --- | --- |
| SpeechT5 | Mel-VAE | 3.07 | 1.15 | 0.5423 | 0.5858 |
| SpeechT5 | Mel-VAE++ | $\bf{2.99}^{\dagger}$ | $\bf{1.06}^{\dagger}$ | 0.5364 | $\bf{0.5982}^{\dagger}$ |
| ByT5-base | Mel-VAE | 6.22 | 3.82 | $\bf{0.5482}^{\dagger}$ | 0.5945 |
| ByT5-base | Mel-VAE++ | 3.11 | 1.17 | 0.5323 | 0.5965 |

### 5.2 Scaling Model Size

We train 4 models of different size on 55K hour English-only datasets, which are referred to as small (S), base (B), Large (L), and XLarge (XL). Detailed model configurations are provided in Appendix 8.4. As shown in Table 3, we observe the performance improvement as the model size increases.

### 5.3 Aligned Text-Speech Embeddings Improve Performances

DiTTo uses cross-attention for text conditioning, which can be influenced by the distance between text and speech representations. To validate the effect of aligned text-speech embeddings, we train DiTTo variants for 200K steps on the MLS English subset (referred to as DiTTo-mls) with two different text encoders: 1) SpeechT5 [], which is trained jointly on English-only text and corresponding speech, and 2) ByT5 [] (we use ByT5-base), which is trained solely on a large corpus of multilingual text. In Table 4, we observe that DiTTo-mls trained with SpeechT5 outperforms the model trained with ByT5-base. Given that the encoder size for SpeechT5 is 85M and for ByT5-base is 415M, and that ByT5 trains on a significantly larger dataset, this indicates that aligning text embeddings with speech embeddings effectively enhances TTS performance, independent of the model or data size.

Building on this insight, we fine-tune Mel-VAE on the same MLS dataset for 200K steps by setting $\lambda>0$ in Eq. (2). This process results in two fine-tuned models, referred to as Mel-VAE++, each aligned with SpeechT5 and ByT5, respectively. Further experiments, presented in Appendix 8.7, demonstrate that DiTTo-mls using Mel-VAE++ outperforms the version using Mel-VAE. This effect is more pronounced with the DiTTo-mls and Mel-VAE++ pair aligned with ByT5 than with SpeechT5, because SpeechT5’s text embeddings are already aligned with speech embeddings during pre-training. Aggregating all results, we can conclude that fine-tuning the neural audio codec, rather than the language model, is sufficient to enhance alignment and subsequently improve TTS performance. This approach offers a significant advantage since fine-tuning the language model is more time-consuming.

## 6 Ablation Study

We conduct an ablation study to analyze the proposed method. First, we evaluate the performance changes associated with various options for the model structure, confirming the validity of our choices. Next, we perform a search for hyperparameters, including the number of diffusion steps. All models train for 200K steps using the MLS English subset in the base (B) configuration. We adopt the same span masking strategy, utilize L1 loss, and constrain the training data to 20 seconds as described in []. Other hyperparameters remain consistent with those used in DiTTo training.

### 6.1 Model Architecture Search

**Table 5: Performances of English-only cross-sentence task for ablation study. All models use SpeechT5 as text encoder and original Mel-VAE as audio latent codec, and comsume ground truth length during sampling. We set the diffusion steps to 250 for WER and SIM-r and 25 for Inference Time, noise schedule scale shift to 0.3, and classifier-free guidance scale to 5.0. Results confirm the effectiveness of the architectural design of our model. The boldface indicates the best result.**
| Model | WER $\downarrow$ | SIM-r $\uparrow$ | Inference Time $\downarrow$ |
| --- | --- | --- | --- |
| U-Net | 3.70 | 0.3890 | 1.328s |
| Flat-U-Net | 2.97 | 0.5471 | 1.310s |
| DiTTo-local-adaln | 3.38 | 0.5673 | 0.937s |
| DiTTo-uvit-skip | 3.17 | 0.5848 | 0.940s |
| DiTTo-no-skip | 3.30 | 0.5727 | 0.905s |
| Simple-TTS (from Table 2) | 4.09 | 0.5305 | 1.982s |
| DiTTo | 2.93 | 0.5877 | 0.903s |
| DiTTo-encodec | 4.19 | 0.5460 | n/a |
| DiTTo-fixed-length (trimmed) | 3.72 | 0.5707 | 1.303s |

As listed in Table 5, we consider several model architecture options, and our final design demonstrates optimal performance in WER and SIM-r while showing the fastest inference time. We begin with the U-Net diffusion model (referred to as U-Net). We train the U-Audio Transformer (U-AT) proposed by [] on the Mel-VAE latent, where the input latent undergoes an 8-fold down/up sampling. Given that Mel-VAE already compresses about seven times more than EnCodec, this additional compression results in significant information loss and poor output quality. Next, we eliminate the down/up sampling (referred to as Flat-U-Net), which proves to be more suitable choice than U-Net but still leaves room for improvement. Subsequently, we remove the residual and convolution block, making our model resemble the DiT architecture. We then modify AdaLN to be shared (referred to as DiTTo-local-adaln), as implemented in Pixart-alpha []. This reduces model parameters by 30.5% (for XL models) and surprisingly improves performance as well.

Additionally, we add a long skip connection to DiT inspired by U-Net. Notably, connecting the hidden layers inside the transformer block as in U-ViT [] (referred to as DiTTo-uvit-skip) or without any skip connection (referred to as DiTTo-no-skip) is less effective in terms of model convergence than a residual connection that links before and after the transformer block (referred to as DiTTo). Training on the latent from EnCodec instead of Mel-VAE (referred to as DiTTo-encodec) results in performance degradation, confirming that Mel-VAE and DiT structures complement each other.

Finally, we confirm that fixing the input/output sequence length (referred to as DiTTo-fixed-length (trimmed)), where we have to manually trim the output speech to remove silence at the end, degrades performance compared to variable-length modeling. Furthermore, the variable-length modeling enables speech rate control by changing the total length of the generated speech latent, as illustrated in Figure 2. Our model design shows the best performance not only in both WER and SIM-r, but also inference time. Note that we can not measure the total inference time of DiTTo-encodec since there is no speech length predictor that bases its predictions on EnCodec audio embeddings.

Figure: Figure 2: Speech rate controllability is illustrated through mel-spectrograms of generated speech at various rates. The speech rate decreases from top to bottom, achieved by adjusting the latent length from 38 to 63. The transcript is "But to those who knew her well, it was a symbol of her unwavering determination and spirit." The $x$-axis represents the length, and the $y$-axis represents the channels of the mel-spectrogram. Speech samples are available at [https://ditto-tts.github.io](https://ditto-tts.github.io).
Refer to caption: x2.png

### 6.2 Noise Schedule Scale and Sampling

Figure: (a) Noise scale search
Refer to caption: x3.png

We conduct a hyperparameter search for three components regarding noise schedule and sampling: (a) the scale-shift of the noise scheduler by training five different models with scales ranging from 0.1 to 0.5 (Figure 3(a)) as visualized in Figure 5 in Appendix 8.6, (b) the classifier-free guidance (CFG) scale during inference (Figure 3(b)), and (c) the number of diffusion steps during inference (Figure 3(c)). In these figures, The blue dashed line with circle marker represents the WER, and the green dotted line with square marker represents the SIM-r. The selected values in each experiment are marked with vertical black dash-dot lines. From the hyperparameter search, we set the noise scale and the CFG scale to 0.3 and 5.0, respectively. Notably, 25 diffusion steps are sufficient to produce high-fidelity outputs, and we observe that performance gaps among different steps diminish with further training.

## 7 Conclusion

We presented DiTTo-TTS, a latent diffusion model for text-to-speech (TTS) that leverages cross-attention and the prediction of the total length of latent speech representations to achieve text-speech alignment. We demonstrated that the proposed method shows exceptional zero-shot performance in naturalness, intelligibility, and speaker similarity, all without relying on domain-specific elements such as phonemes and durations, while simplifying the training process as well. In the process of obtaining this result, we also found that fine-tuning a speech autoencoder with an auxiliary language modeling objective can significantly enhance the text-speech alignment. Moreover, DiTTo-TTS shows effective scalability with respect to data and model sizes.
Our future work includes: 1) exploring various noise schedules to improve output quality and distillation methods to reduce inference times; 2) enhancing pronunciation accuracy by improving character input normalization; and 3) enabling DiTTo to understand and learn from natural language instructions.

## Acknowledgment

The authors would like to express our gratitude to Kangwook Lee for the valuable discussions. We also extend our thanks to Seungjun Chung for his thorough proofreading of the paper, and to Beomsoo Kim and Gibum Seo for their essential support in data handling and verification.

## 8 Appendix

### 8.1 Dataset Details

English We use the following datasets: 1) Multilingual LibriSpeech (MLS) [], which comprises transcribed speech from multiple speakers and languages, sourced from LibriVox audiobooks. 2) GigaSpeech [], containing multi-domain speeches, such as audiobooks, podcasts, and YouTube videos, with human transcriptions. This dataset includes audio from multiple speakers but lacks speaker information. 3) LibriTTS-R [], a restored version of the LibriTTS [] corpus, sharing the same metadata. 4) VCTK [] and 5) LJSpeech [], which are widely used English datasets in the speech synthesis community, with VCTK being multi-speaker and LJSpeech being single-speaker. Additionally, we include the 6) Expresso Dataset [] in DiTTo-en training. This high-quality (48,000 Hz) expressive speech collection features read speech and improvised dialogues from four speakers, totaling 40 hours.

We train a speech length predictor for DiTTo-en using the same datasets, but also include the LibriSpeech [] dataset. This is due to the dependency on text normalization. MLS datasets consist of audio recordings, each ranging from about 10 to 20 seconds in length, paired with normalized text. It affects the speech length predictor, making it output lengths based on whether the input text is normalized. For example, if we input normalized text, the model tends to predict its speech length in the same range as MLS. While the text in LibriSpeech is also normalized, it offers more varied speech lengths, thus reducing the dependency on text normalization when determining speech lengths.

Korean We use the following datasets: 1) AIHub 14(^4^44[https://www.aihub.or.kr/aihubdata/data/view.do?currMenu=115&topMenu=100&aihubDataSe=realm&dataSetSn=542](https://www.aihub.or.kr/aihubdata/data/view.do?currMenu=115&topMenu=100&aihubDataSe=realm&dataSetSn=542)), which features recordings of everyday people reading provided script sentences. 2) AIHub 15(^5^55[https://www.aihub.or.kr/aihubdata/data/view.do?currMenu=115&topMenu=100&aihubDataSe=realm&dataSetSn=466](https://www.aihub.or.kr/aihubdata/data/view.do?currMenu=115&topMenu=100&aihubDataSe=realm&dataSetSn=466)), which has recordings of 50 professional voice actors expressing seven emotions (joy, surprised, sad, angry, scared, hate, neutral), five speaking styles (narrating, reading, news-like, dialogic, broadcasting), and three vocal ages (kid, young, old). 3) KsponSpeech [], which consists of 2,000 speakers, each recording individual free speech on various topics in a quiet environment. The transcription follows specific guidelines regarding laughter, breathing, and more. 4) AIHub-broadcast(^6^66[https://www.aihub.or.kr/aihubdata/data/view.do?currMenu=115&topMenu=100&aihubDataSe=realm&dataSetSn=463](https://www.aihub.or.kr/aihubdata/data/view.do?currMenu=115&topMenu=100&aihubDataSe=realm&dataSetSn=463)), which is designed for speech recognition and includes 10,000 hours of multi-speaker conversations recorded at 16,000 Hz, covering a wide range of 22 categories without annotations for speaker identity or emotional state. We also include 5) AIHub-expressive(^7^77[https://www.aihub.or.kr/aihubdata/data/view.do?currMenu=115&topMenu=100&aihubDataSe=realm&dataSetSn=71349](https://www.aihub.or.kr/aihubdata/data/view.do?currMenu=115&topMenu=100&aihubDataSe=realm&dataSetSn=71349)), a dataset designed for speech synthesis, comprising 1,000 hours of speech recorded at 44,100 Hz. This dataset features 89 speakers and includes 7 speech styles (monologue, dialogue, storytelling, broadcast, friendly, animation, and reading) and 4 emotional tones (happiness, sadness, anger, and neutral).

Other Language We use subsets of MLS including German, Dutch, French, Spanish, Italian, Portuguese, and Polish.

Dataset Preprocessing Audio stream and its metadata are preprocessed and stored into parquet format. This allows fetching data with minimal IO-overhead from Network Attached Storage, allowing faster training. Parquets consisting of 10K pairs of audio and its metadata are compressed into TAR format, which further compresses to minimize data read overhead. While training, TAR files are decompressed, and parquets are deserialized into audio streams and its corresponding metadata in json format.

### 8.2 Experimental Setup Details

For English-only evaluations, we transcript generated audio using the CTC-based HuBERT-Large model(^8^88[https://huggingface.co/facebook/hubert-large-ls960-ft](https://huggingface.co/facebook/hubert-large-ls960-ft)) []. For Multilingual Evaluations, we utilize OpenAI’s Whisper model(^9^99[https://github.com/openai/whisper/blob/main/model-card.md](https://github.com/openai/whisper/blob/main/model-card.md): ”large-v2”) []. Text normalization is conducted using NVIDIA’s NeMo-text-processing(^10^1010[https://github.com/NVIDIA/NeMo-text-processing](https://github.com/NVIDIA/NeMo-text-processing)) [, ]. In both evaluations, WavLM-TDCNN(^11^1111[https://github.com/microsoft/UniSpeech/tree/main/downstreams/speaker_verification](https://github.com/microsoft/UniSpeech/tree/main/downstreams/speaker_verification)) [] is employed to evaluate SIM-o and SIM-r [].

We conduct subjective evaluations using Amazon Mechanical Turk (MTurk) with US-based evaluators. For SMOS, evaluators assess the likeness of samples to the provided speech prompts, considering speaker similarity, style, acoustics, and background disturbances. In CMOS, evaluators compare the overall quality of a synthesized sample to a reference. They use a given scale to judge whether the synthesized version is superior or inferior to the reference. SMOS employs a 1 to 5 integer scale, where 5 signifies top quality. CMOS uses a scale from -3 (indicating the synthesized speech is much worse than the reference) to 3 (indicating it’s much better), with 1-unit intervals. Samples receive 6 and 12 ratings for SMOS and CMOS, respectively.

### 8.3 Baseline Details

The followings are the details of the baselines we used to compare with our model:

- •
VALL-E []: A zero-shot TTS model integrates autoregressive and non-autoregressive models. We show the objective metric scores directly extracted from the paper. We also present subjective evaluation results by comparing our samples with demo samples, which are at sample rates of 22,050 Hz and 16,000 Hz, respectively.
- •
SPEAR-TTS []: A multi-speaker TTS model that treats TTS as a two-step sequence-to-sequence process. It first converts text into high-level semantic tokens, similar to "reading," and then transforms these semantic tokens into low-level acoustic tokens, akin to "speaking." We present the objective metric scores directly from the paper.
- •
CLaM-TTS []: An advanced neural codec language model that uses probabilistic residual vector quantization to compress token length, enabling the simultaneous generation of multiple tokens, in contrast to VALL-E’s cascaded modeling approach. We offer both objective and subjective comparisons between our samples and those generated from the official model checkpoints.
- •
YourTTS []: A conventional TTS model based on the VITS [] framework for zero-shot multi-speaker and multilingual training. We show both objective and subjective comparisons between our samples and those generated from the model checkpoints. We use the official checkpoint (^12^1212[https://github.com/Edresson/YourTTS](https://github.com/Edresson/YourTTS)).
- •
VoiceBox []: A generative model employs a non-autoregressive flow-based architecture to tackle TTS as a text-guided speech infilling task, utilizing a large dataset. We present the objective metric scores directly from the paper. Additionally, we provide subjective evaluation results by comparing our samples with demo samples, recorded at sample rates of 22,050 Hz and 16,000 Hz, respectively.
- •
Simple-TTS []: A Latent Diffusion Model (LDM) based on U-ViT [] serves as an alternative to complex TTS synthesis pipelines, removing the need for phonemizers, forced aligners, or detailed multi-stage processes. We provide both objective and subjective comparisons between our samples and those generated from the model checkpoints. We use the official checkpoint (^13^1313[https://github.com/asappresearch/simple-tts](https://github.com/asappresearch/simple-tts)).
- •
NaturalSpeech 2 []: A non-autoregressive TTS system that incorporates a neural audio codec employing residual vector quantizers to obtain quantized latent vectors. These vectors serve as input for a diffusion model, which generates continuous latent vectors conditioned on text input. Additionally, the system employs a speech prompting mechanism along with duration and pitch predictors. We present subjective evaluation results by comparing our samples with demo samples, which are at sample rates of 22,050 Hz and 16,000 Hz, respectively.
- •
NaturalSpeech 3 []: A non-autoregressive TTS system with factorized diffusion models based on a factorized vector quantization (FVQ) to disentangle speech waveforms into distinct subspaces such as content, prosody, timbre, and acoustic details. We present subjective evaluation results by comparing our samples with demo samples, which are at sample rates of 22,050 Hz and 16,000 Hz, respectively.
- •
Mega-TTS []: A TTS system that decomposes speech into attributes like content, timbre, prosody, and phase, with each attribute modeled by a module using appropriate inductive biases. We show subjective evaluation results by comparing our samples with demo samples, which have sample rates of 22,050 Hz and 16,000 Hz, respectively.
- •
E3-TTS []: A simple and efficient diffusion based TTS model which takes plain text as input. Unlike previous work, it doesn’t rely on intermediary representations like spectrogram features or alignment data. We present subjective evaluation results by comparing our samples with demo samples, which are at sample rates of 22,050 Hz and 24,000 Hz, respectively.

### 8.4 Model Configuration

**Table 6: $d_{c}$ is the embedding dimension of pretrained encoder, and $d_{h}$ is the hidden dimension of DiTTo. SiLU is used as activation function between two linear layers.**
| Configuration | Num Layers | Input Dim | Hidden Dim | Out Dim |
| --- | --- | --- | --- | --- |
| Timestep Embedder | 2 | 256 | $d_{h}$ | $d_{h}$ |
| Text Embedder | 2 | $d_{c}$ | $d_{h}$ | $d_{h}$ |
| Skip Block | 2 | $2\times d_{h}$ | $d_{h}$ | $d_{h}$ |
| Final MLP | 2 | $d_{c}$ | $d_{h}$ | $d_{h}$ |
| AdaLN Modulation | 1 | $d_{h}$ | - | $9\times d_{h}$ |

**Table 7: Model parameters across different versions of DiTTo**
| Configuration | DiTTo-S | DiTTo-B | DiTTo-L | DiTTo-XL |
| --- | --- | --- | --- | --- |
| Num Layers | 12 | 12 | 24 | 28 |
| Hidden Dim ($d_{h}$) | 384 | 768 | 1024 | 1152 |
| Num Heads | 6 | 12 | 16 | 16 |

Our architecture is based on DiT [] with several modifications. We extract text embeddings from a pre-trained text encoder, either from speechT5 [] or ByT5 [], and use these embeddings as hidden states in cross-attention layers. Adaptive layer normalization (AdaLN) layers condition on both time and text embeddings. AdaLN outputs modulate the layers within each transformer block. We apply RoPE [] in self-attention layers. Following [], we use a single global-AdaLN, sharing AdaLN parameters across all layers instead of using separate AdaLN for each layer. The global AdaLN layer conditions on both time and text, which are embedded and transformed with MLP layers. AdaLN generates scaling, shifting, and gating vectors for each self-attention, cross-attention, and MLP layer. Gating vectors modulate the outputs of each layer over the skip connections in transformer blocks. The vectors generated by AdaLN are further shifted with layer-independent learnable vectors. The MLP block in each transformer block adopts the gating mechanism from [], which is used in the first linear layer of the MLP block. Inspired by [], we insert a long skip connection from the input to the output of the last transformer block, differing from [], where U-Net-like [] long skip connections are used between transformer blocks at opposing ends. The detailed configurations are shown in Table 6, and the parameters for the four different model sizes are listed in Table 7.

### 8.5 Training and Inference of Speech Length Predictor

Figure: (a) Training
Refer to caption: extracted/5671366/files/len_pred_training.png

#### Training

We train two speech length predictors for DiTTo-en and DiTTo-multi. Each was trained using the same text encoder as the respective DiTTo models. Essentially, we followed the same configuration as mentioned above, but without gradient accumulation and with a maximum token size of 10,240 for 600K steps. The number of trainable parameters for the speech length predictors is 33.18M for DiTTo-en and 33.58M for DiTTo-multi. Figure 4 presents a visualization of length sampling during training.

#### Inference

After training the speech length predictor, we can perform inference by either calculating the expected value over all possible length values using its softmax logits or by sampling from a multinomial distribution based on the softmax logits. In both cases, we apply top-$k$ sampling [] with $K=20$. For human subjective evaluation, we set the expected length of the test samples.

### 8.6 Noise Schedule and Classifier-Free Guidance

Figure: Figure 5: Visualization of our noise scheduler with various shift-scale factors. The $x$-axis represents the diffusion time steps, while the $y$-axis shows the cumulative alphas [] of the noise scheduler.
Refer to caption: x6.png

#### Noise Scheduler

We use the cosine noise schedule [] with scale-shift [, ]. Since the impact of noise schedules can vary depending on the resolution of the input, it may be necessary to shift the schedule to appropriately add noise across time steps. As discussed in Section 6.2, we test five different values of scale-shift and visualize them in Figure 5.

#### Classfier-Free Guidance

To leverage classifier-free guidance [] for its efficacy in diffusion-based generative modeling, we train both a conditional and an unconditional diffusion model simultaneously. This is achieved by omitting the text input with a probability of $p=0.1$. In the cross-attention layers of DiTTo, we concatenate a learnable null embedding with the text features along the sequence dimension, following the approach of []. We eliminate the conditioning information by masking out the text embeddings during the cross-attention computation and setting the mean-pooled text embedding to zero.

### 8.7 Mel-VAE++

**Table 8: Performances of English-only cross-sentence task for Mel-VAE++. The boldface indicates the best result.**
| DiTTo Text Encoder | Mel-VAE++ | WER $\downarrow$ | CER $\downarrow$ | SIM-o $\uparrow$ | SIM-r $\uparrow$ |
| --- | --- | --- | --- | --- | --- |
| SpeechT5 | - | 3.07 | 1.15 | 0.5423 | 0.5858 |
| HuBERT | 5.59 | 3.18 | 0.5239 | 0.5742 |  |
| SpeechT5 | 3.09 | 1.12 | 0.5335 | 0.5940 |  |
| ByT5-base | 2.99 | 1.06 | 0.5364 | 0.5982 |  |
| ByT5-base + HuBERT | 2.93 | 1.04 | 0.5495 | 0.5958 |  |
| ByT5-base | - | 6.22 | 3.82 | 0.5482 | 0.5945 |
| HuBERT | 5.88 | 3.64 | 0.5458 | 0.5938 |  |
| SpeechT5 | 4.10 | 1.95 | 0.5465 | 0.6055 |  |
| ByT5-base | 3.11 | 1.17 | 0.5323 | 0.5965 |  |
| ByT5-base + HuBERT | 3.69 | 1.72 | 0.5631 | 0.6101 |  |
| ByT5-large | - | 3.92 | 1.84 | 0.5292 | 0.5756 |
| HuBERT | 3.04 | 1.11 | 0.5429 | 0.5878 |  |
| SpeechT5 | 3.27 | 1.31 | 0.5412 | 0.6029 |  |
| ByT5-base | 3.17 | 1.18 | 0.5368 | 0.5999 |  |
| ByT5-base + HuBERT | 3.80 | 1.75 | 0.5489 | 0.5975 |  |

Figure: (a) CAD without semantic injection
Refer to caption: extracted/5671366/files/cad_job-vodit-byt5-base-default-200k.png

When Mel-VAE is finetuned with the guidance of a pre-trained language model, consistent improvement in performance is observed, closing the gap with the SpeechT5 encoder, which is trained on both unsupervised speech and language data. To improve our understanding of semantic content injection, we finetune Mel-VAE with various pre-trained text and speech encoder models. As shown in Table 8, the experiments use various combinations of DiTTo text encoders and Mel-VAE text encoders. When SpeechT5 is utilized as the text encoder for DiTTo, semantic content injection enhances the performance metrics; however, its gains, while positive, are not as substantial as those observed in the control, which is trained the same number of steps without auxiliary signal, denoted as ‘-’. When ByT5-base is used as the DiTTo text encoder, using ByT5-base as Mel-VAE semantic content injection model significantly boosts metrics. We additionally observe that HuBERT feature matching, as done in [], helps refine its speech quality, as can be seen in the improved Speaker Similarity (SIM) metrics. When ByT5-base and HuBERT are both used, Mel-VAE benefits from both models.

We also investigate how DiTTo interacts with the text encoder latents as training progresses, in settings with and without semantic content injection. Since the TTS process typically ensures a monotonic alignment between text and speech, the diagonality of cross-attention between the text encoder and DiT block indicates how effectively text latents are used during denoising process. The intensity of the cross-attention map reflects how conducive the information provided by the text encoder is to producing DiTTo’s output. We measure the monotonicity of cross-attention maps using Cumulative Attention Diagonality (CAD) [].

The median CAD values are drawn in solid lines as in Figure 6. The model performs inference on 5 samples. The Interquartile Range (IQR) is represented by the shaded regions surrounding the solid lines in the graph. We set the sampling steps as 25, noise schedule scale shift as 0.3, and classifier-free guidance scale as 5.0. Given that the parameters of text encoders are not tuned during DiTTo training, drastic changes in CAD values across training steps without semantic content injection indicate significant changes in DiTTo parameters are needed to align with text encoders. In contrast, with semantic content injection, changes in CAD values as training progresses show minimal fluctuation. This suggests that semantic content injection enables effective interaction between text encoders and DiT blocks. Based on CAD values across diffusion steps, semantic content injection allows the interaction between the text encoder and DiTTo to predominantly occur during the initial stages of the reverse process. This suggests that semantic content injection boosts the provision of rich semantic information by text encoders, allowing DiT blocks to focus on improving acoustic quality in later steps. Furthermore, the significant deviations across samples, as evidenced by the enlarged shaded regions when semantic injection is not employed, suggest that the interaction between the text encoder and DiT blocks is less effective without semantic content injection.

### 8.8 Human Evaluations on Comparison of Demo Samples

**Table 9: Human evaluations on comparison of demo samples from various SOTA models with DiTTo-en (XL). SMOS scores include a 95% confidence interval. The boldface indicates the best result.**
| Model | SMOS | CMOS |
| --- | --- | --- |
| Voicebox [] | 2.87$\pm$0.45 | 0.14 |
| DiTTo-en | 3.57$\pm$0.39 | 0.0 |
| VALL-E [] | 3.50$\pm$0.46 | -0.94 |
| DiTTo-en | 3.90$\pm$0.38 | 0.0 |
| MegaTTS [] | 3.76$\pm$0.44 | -0.31 |
| DiTTo-en | 3.82$\pm$0.32 | 0.0 |
| E3-TTS [] | 4.25$\pm$0.19 | -0.30 |
| DiTTo-en | 4.28$\pm$0.32 | 0.0 |
| NaturalSpeech 2 [] | 3.99$\pm$0.37 | -0.29 |
| DiTTo-en | 4.01$\pm$0.35 | 0.0 |
| NaturalSpeech 3 [] | 4.42$\pm$0.28 | -0.27 |
| DiTTo-en | 3.92$\pm$0.27 | 0.0 |

Table 9 shows the results of human evaluation on demo cases. DiTTo outperforms all others except for NaturalSpeech 3 in SMOS and Voicebox in CMOS. We hypothesize that the prompts used in Voicebox tends to be unintelligible and noisy, and as shown in SMOS, the model samples did not adhere to them well. In contrast, DiTTo follows the prompts better, although the naturalness of the samples themselves is somewhat lacking. Therefore, in CMOS, when comparing two samples without prompts, it is likely that DiTTo will be judged as less natural. Conversely, in NaturalSpeech 3, as shown in SMOS, the model follows the intonation of the prompts too closely, sometimes resulting in unnaturalness. Thus, when comparing samples without prompts, DiTTo may sound more natural.

### 8.9 Multiligual Continuation Task

In this section, we provide the experimental results of multilingual continual tasks for both DiTTo-multi in Table 10 and CLaM-multi in Table 11.

**Table 10: Performances of DiTTo-multi for the multilingual *continuation* task.**
| Model | WER $\downarrow$ | CER $\downarrow$ | SIM-o $\uparrow$ | SIM-r $\uparrow$ |
| --- | --- | --- | --- | --- |
| English / MLS English | 6.91 | 4.15 | 0.4759 | 0.4986 |
| English (HuBERT) / MLS English | 5.73 | 1.84 | - | - |
| German / MLS German | 6.60 | 2.47 | 0.4917 | 0.5239 |
| Dutch / MLS Dutch | 8.89 | 3.11 | 0.5828 | 0.5971 |
| French / MLS French | 8.03 | 3.09 | 0.5711 | 0.5905 |
| Spanish / MLS Spanish | 2.22 | 0.89 | 0.5483 | 0.5776 |
| Italian / MLS Italian | 13.69 | 2.35 | 0.5637 | 0.5902 |
| Portuguese / MLS Portuguese | 6.07 | 2.11 | 0.5346 | 0.5289 |
| Polish / MLS Polish | 10.88 | 3.17 | 0.5090 | 0.5441 |
| Korean / AIHub 14 | 19.44 | 1.90 | 0.5838 | 0.6095 |
| Korean / AIHub 15 | 13.80 | 2.20 | 0.5260 | 0.5454 |
| Korean / Ksponspeech | 27.22 | 18.30 | 0.5205 | 0.5466 |

**Table 11: Performances of CLaM-multi for the multilingual *continuation* task.**
| Model | WER $\downarrow$ | CER $\downarrow$ | SIM-o $\uparrow$ |
| --- | --- | --- | --- |
| English / MLS English | 8.71 | 5.19 | 0.4000 |
| English (HuBERT) / MLS English | 7.71 | 3.19 | 0.4000 |
| German / MLS German | 9.63 | 4.11 | 0.4219 |
| Dutch / MLS Dutch | 12.25 | 4.97 | 0.5983 |
| French / MLS French | 10.29 | 4.08 | 0.5671 |
| Spanish / MLS Spanish | 4.02 | 1.91 | 0.5292 |
| Italian / MLS Italian | 19.70 | 5.19 | 0.5459 |
| Portuguese / MLS Portuguese | 9.66 | 3.72 | 0.5658 |
| Polish / MLS Polish | 14.70 | 5.34 | 0.5519 |
| Korean / AIHub 14 | 20.21 | 1.80 | 0.5423 |
| Korean / AIHub 15 | 13.08 | 2.35 | 0.5280 |
| Korean / Ksponspeech | 30.24 | 20.02 | 0.4488 |

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. Regarding successfully applying joint latent diffusion model for TTS,  the comparison between the autoregressive (AR) model like a language speech model (CLaM in this paper) and diffusion model would be attracting as they are the current two major methodologies as diffusion vs. AR model.  According to the result, the diffusion model archives better similarity and quality from subjective metrics, and also better CER. However, as the total training data is not large enough (82K), would it be enough to be convincing that diffusion model could achieve superior result rather than AR model while language model may work better on larger dataset? It would be a fantastic topic to discuss. 
2. The framework design a latent representation for diffusion model to generate, then use Mel-VAE to decode latent into mel-spectrogram, and get the synthesized speech via BigVGAN. This is a little confusing why not just model mel-spectrogram for diffusion model? 
3. The Table 3 shows the subjective result with SMOS and CMOS. How many speakers and how many test utterances in this test set? It would be related to the diversity of the model  capability.
soundness: 3
strengths: This paper clearly describes the work for an end-to-end fully diffusion model for zero-shot TTS. The strengths lie on the following aspects
1) Rich experiments have done to investigate the different model size of DiTTo model (small, base, large and Xlarge), data scalability; comparison between U-Net and DiT module, and comparison between speech length model with variable and fixed and etc. 
2) A lot of state-of-art TTS system comparison is provided. Although not all the system have compared all the metrics but several metrics are showed for readers to compare and get a relative complete picture on the metrics of different popular system include AR model like VALL-E, CLaM-TTS, NAR model like YourTTS, and SimpleTTS,  etc.
3) Ablation study and demo page quit clear.
summary: This paper introduces a latent diffusion model based zero-shot TTS as DiTTo-TTS.  The author have done rigorous investigations to improve LDM based TTS by replacing U-Net with DiT, remove dependency of domain-specific factors like phoneme and duration with several experiments on the model size and datasets. The experiment results shows the good performance of naturalness (CMOS), intelligibility (CER/WER) and similarity (SMOS and SIM).

The major achievements prove that the LDM TTS model could work as end-to-end in a joint modeling way for the text and speech representation, and it is scalable to different model size and dataset.
weaknesses: The major weakness of this paper is the novelty. The most contribution as the author also mentioned is removing the domain-specific factors like phoneme and duration, and just giving a total duration estimation is good enough for the task which is even better. However, this is not new, and similar work has been done at SeedTTS https://arxiv.org/pdf/2406.02430#page=6&zoom=100,144,548. 

SeedTTS-DiT has no-dependence on the phoneme duration by just giving the total duration for the diffusion model. It already shows that this fully diffusion based speech generation model could achieve superior performance. This integration is just a quite natural way regarding the highly flexibility of noise iteration with DiT structure for a diffusion or flowmatching model. I would believe many folks/peers have applied it in speech generation before or after Seed-TTS-DiT and it indeed works as expected. According to these information, I would give the question on the novelty of this paper.

---

confidence: 5
contribution: 3
initial_rating: 8
presentation: 3
questions: - Table 7:
  - Are the DiTTo-Encodec and DiTTo-DAC models also fine-tuned with an LM objective? If not, please include DiTTo-MELVAE for comparison.
  - Please report sim-o when comparing different neural codecs.
- In general, reporting sim-o for other ablations would be beneficial for the community to facilitate comparisons with any results in the paper.
- Page 7, line 518: "We cannot measure the total inference time due to the lack of a speech length predictor for their embeddings, but even with ground truth lengths, their 7-8 times longer latents lead to significantly slower generation."
  - What is the feature rate for the MelVAE model? Encodec operates at 75Hz, so does this mean that MelVAE operates at approximately 10Hz? The original MelVAE in CLAM-TTS mentions a coderate of 100Hz. It would be helpful to include this information in the paper to make it more self-contained.
- How does speech editing, particularly for the infilling task, work? Do you only provide the prefix as a prompt to determine the duration of the edited segment? Specifically, could you describe the process for editing: 

  **<prefix> <old text> <postfix> to <prefix> <new text> <postfix>**?  
  - My guess is that this might require an alignment model to identify the start and end timings for <old text>, followed by predicting the duration of <prefix> <new text> conditioned on prefix speech tokens.
  - What happens if <prefix> is null? How does the model handle durations in this case? I assume with an alignment tool one could use any partial text and corresponding tokens for prompting.
- Ablation suggestion: Could you explore the impact of the number of training steps on performance? 
- Minor:
  - Typo on page 1: "honemes".
soundness: 4
strengths: - The paper is clear and generally well-written. Detailed ablations highlight the importance of each factor in DiTTo-TTS.
- The contributions related to MelVAE fine-tuning and the auto-regressive speech length predictor are novel. While DiT has been explored previously in audio, the detailed ablations and contributions, such as long-skip connections, provide valuable insights for the field.
- The paper performs an exhaustive comparison against a number of baselines to demonstrate performance differences and establish SoTA results.
summary: This work proposes DiTTo-TTS, a DiT-based architecture for zero-shot text-to-speech (TTS) synthesis applications. The proposed method does not rely on explicit alignments or phoneme units, thereby simplifying the TTS pipeline. For the neural codec, the paper proposes fine-tuning MelVAE to better align with pre-trained text encoders. This work includes exhaustive ablations on data and model scaling, architectural choices such as skip connections, choice of text encoders, and variable vs. fixed-length modeling to understand better the factors contributing to DiTTo-TTS's results. The authors also introduce a novel text and speech-token conditional speech length predictor based on auto-regressive training for variable-length modeling. Experiments on multilingual and English setups provide a detailed understanding of each component and demonstrate that DiTTo-TTS achieves SOTA results for zero-shot TTS without relying on domain-specific factors.
weaknesses: - No major weaknesses, there are some minor details which could be added. See comments section.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. Can you provide confidence intervals or p-values for the comparisons in your result tables? Without these, it is impossible to tell if the result is significantly better than state-of-the-art baselines.

2. In Appendix A3, the instruction for comparative mean opinion score (CMOS) shows that you only ask participants to compare the quality of the synthesized audio and reference, but in the main text you mention that DiTTo-en significantly outperforms the baseline models in naturalness, quality, and intelligibility (as assessed by CMOS). This raises questions about how the evaluations of naturalness and intelligibility for the synthesized audio were conducted in the experiments.
soundness: 3
strengths: 1. Develop a diffusion-based TTS model, DiTTo-TTS, that avoids reliance on domain-specific factors while maintaining competitive quality. 
2. DiTTo-TTS is shown to achieve better performance compared to state-of-the-art baselines with significantly reduced inference time.
summary: This paper introduces DiTTo-TTS, a scalable TTS model based on diffusion transformers that avoids reliance on domain-specific factors like phonemes or durations, achieving high-quality results and efficiency improvements over conventional TTS methods.
weaknesses: **Dependency on Domain-specific factors** DiTTo-TTS is claimed to avoid reliance on domain-specific factors like phonemes or durations; however it needs the pre-trained Text Encoder (ByT5, SpeechT5), which aligns the textual and acoustic information into a unified semantic space, and a pre-trained neural audio codec which aligns the speech latents with the linguistic latents of the pre-trained language model during the autoencoding process. Moreover, DiTTo-TTS also needs a Speech Length Predictor for managing duration. Given these dependencies, it’s unclear how the model avoids domain-specific factors.  Could you clarify your definition of "domain-specific factors" and explain how your approach differs from traditional TTS systems in terms of reliance on phonemes and durations, even with the use of pre-trained components? 

**Limited Novelty of the Proposed Approach** Simple-TTS and NaturalSpeech 2 has already proposed to use a pre-trained text encoder and neural audio codec to align the speech with text through cross-attention and training only the weight of latent diffusion model. Could you provide a more detailed comparison between DiTTo-TTS and Simple-TTS/NaturalSpeech 2 and highlight the key innovations or improvements in DiTTo-TTS that differentiate it from these prior works?

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. Are you plan to release the code and checkpoints of DiTTo-TTS? 

2. Regarding the content in Appendix A.13, I am interested in whether the model performs well in smaller data scenarios, such as a single-speaker setup with less data than LibriTTS. The scale-up trend in performance relative to data quantity seems less pronounced compared to other zero-shot TTS models (e.g., Voicebox Section B.2). I would like to understand the authors' perspective on why DiTTo-TTS remains robust even with limited data.
soundness: 3
strengths: The most notable strength is that the authors conducted comparisons with various baselines and performed extensive ablation studies and analyses on numerous elements. In addition, the paper is well-written, with a high level of detail in the experiments, making it easy to understand the study.
summary: This study presents a model for personalized speech synthesis based on an LDM structure. Personalization is achieved by modeling z obtained through Mel-VAE using an infilling approach. Instead of explicitly calculating text-speech alignment and providing text information accordingly, as is common in previous research, the model allows the text and speech representations to learn alignment autonomously via cross-attention. Additionally, a variable-length predictor is introduced to predict the total speech length during inference. To validate the proposed structure and methodology, the authors conducted comparisons with numerous strong baselines across various languages and benchmarks, as well as extensive ablation studies on proposed elements, data quantity, hyperparameters, and other factors.
weaknesses: Personally, I feel that the novelty is somewhat limited, as LDM-based structures in zero-shot TTS are not entirely new (e.g., NaturalSpeech 2, 3, as you already mentioned in your paper.), and the approach of allowing the model to learn speech-text alignment with cross-attention has been seen in previous works, such as Simple-TTS. Nonetheless, the systematic analysis of each component and the guidance provided to readers on existing design choices and their limitations are indeed valuable contributions.

Meta-review: The paper includes extensive experiments to explore different model sizes (small, base, large, Xlarge), data scalability, and comparisons between U-Net and DiT modules, as well as variable and fixed speech length models.  It provides comparisons with various state-of-the-art TTS systems, including AR models like VALL-E and CLaM-TTS, and NAR models like YourTTS and Simple-TTS, across several metrics. The ablation studies and demo page are well-presented, highlighting the importance of each factor in DiTTo-TTS. The paper is clear and generally well-written, making it easy to understand the detailed experiments and findings.

Decision: Accept (Poster)