NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers
Authors: Kai Shen
Zeqian Ju
Xu Tan
Eric Liu
Yichong Leng
Lei He
Tao Qin
sheng zhao
Jiang Bian
Venue: ICLR 2024 spotlight (2024)
Keywords: text-to-speech
large-scale corpus
non-autoregressive
diffusion
Primary area: representation learning for computer vision, audio, language, and other modalities
TL;DR: A large-scale text-to-speech and singing voice synthesis system with latent diffusion models.

Abstract: Scaling text-to-speech (TTS) to large-scale, multi-speaker, and in-the-wild datasets is important to capture the diversity in human speech such as speaker identities, prosodies, and styles (e.g., singing). Current large TTS systems usually quantize speech into discrete tokens and use language models to generate these tokens one by one, which suffer from unstable prosody, word skipping/repeating issue, and poor voice quality. In this paper, we develop NaturalSpeech 2, a TTS system that leverages a neural audio codec with residual vector quantizers to get the quantized latent vectors and uses a diffusion model to generate these latent vectors conditioned on text input. To enhance the zero-shot capability that is important to achieve diverse speech synthesis, we design a speech prompting mechanism to facilitate in-context learning in the diffusion model and the duration/pitch predictor. We scale NaturalSpeech 2 to large-scale datasets with 44K hours of speech and singing data and evaluate its voice quality on unseen speakers. NaturalSpeech 2 outperforms previous TTS systems by a large margin in terms of prosody/timbre similarity, robustness, and voice quality in a zero-shot setting, and performs novel zero-shot singing synthesis with only a speech prompt. Audio samples are available at https://naturalspeech2.github.io/.

Published as a conference paper at ICLR 2024
NaturalSpeech 2: LATENT DIFFUSION MODELS ARE NATURAL AND ZERO-SHOT SPEECH AND SINGING SYN- THESIZERS
Kai Shen, Zeqian Ju Tan; Yanqing Liu, Yichong Leng, Lei He Tao Qin, Sheng Zhao, Jiang Bian Zhejiang University Microsoft Research Asia & Microsoft Azure Speech University of Science and Technology of China
ABSTRACT
Scaling text-to-speech (TTS) to large-seale, multi mpeakes,eaker, and in in-the-wild datasets is important to capture the diversity in human speech such speaker identities, prosodies, and styles (c.g., singing). Current large TTS systems usually quantize speech into discrete tokens and use language models to generate these tokens by one, which suffer from unstable prosody, word skipping/repeating issue, and poor voice quality. In this paper, we develop NaturalSpeech 2, TTS system that leverages a neural audio codec with residual vector quantizers to get the quantized latent vectors and uses diffusion model to generate these latent vectors conditioned on text input. To enhance the zero-shot capability that is important achieve diverse speech synthesis, we design speech prompting mechanism facilitate in-context learning in the diffusion model and the duration/pitch predictor. We scale NaturalSpeech singing data and evaluate its voice quality unseen speakers. NaturalSpeech 2 outperforms previous TTS systems by a large margin in terms of prosody/timbre similarity, robustness, and voice quality in zero-shot setting, and performs novel zero-shot singing synthesis with only a speech prompt. Audio samples are available at htttps://speecheeseeeae github. 10/ natural speech2
large-scale datasets with 44K hours of speech and
Only Training
Speech x
Training& Inference
In-Context Learning
Codec Decoder
Condition C
IC
Diffusion Model
Latent z
IC
Duration/Pitch Predictor
Codec Encoder
Phoneme Encoder
Speech x
Text
Figure The overview of NaturalSpeech 2, with audio codec encoder/decoder and latent diffusion model conditioned on a prior (a phoneme encoder and a duration/pitch predictor). The details of in-context learning in the duration/pitch predictor and diffusion model are shown in Figure
"The first three authors contributed equally to this work, and their names are listed in random order. Corresponding author: Xu Tan, xuta @mi crosoft.

---

Published as a conference paper at ICLR 2024
INTRODUCTION
Human speech is full of diversity, with different speaker identities (e.g.. gender, accent, timbre), prosodies, styles (e.g., speaking, singing), etc. Text-to-speech (TTS) [Taylor] 2009] Tann al., (021) aims to synthesize natural and human-like speech with both good quality and diversity. With the
(0099)
development of neural networks and deep learning, TTS systems (Wang al. 2017: Shen al.
2018: Li et al. [019] Ren et al.) 2019] 2021a Liu et al., [202]] 2022b] Kim et al. [202]] Tan et al. 2022) have achieved good voice quality in terms intelligibility and naturalness, and some systems (e.g., NaturalSpeech (Tan et al., 2022 Jia et al. 202T) even achieves human-level voice quality on single-speaker recording-studee
benchmarking datasets (e.g.. I.Speecch (Ito 2017)). Given the great achievements in speech intelligibility and naturalness made by the whole TTS community, now we enter a new era of TTS where speech diversity becomes more and more important in order to synthesize natural and human-like speech.
Previous speaker-limited recording-studit datasets are not enough to capture the diverse speaker identities, prosodies, and styles in human speech due to limited data diversity. Instead, TTS models on a large-scale corpus to learn these diversities, and as a by by-product, these trained models can generalize the unlimited unseen scenarios with few-shot zero-shot technologies. Current large-scale TTS systems (Wang et al. 2023) Kharitonov et al., [023) Xue et al. 2023) usually quantize the speech waveform into discrete tokens and model these tokens with autoregressive language models. This pipeline suffers from several limitations: 1) The speech (discrete token) sequence is usually very long (a 10s speech usually has thousands of discrete tokens) and the autoregressive models suffer from error propagation and thus unstable speech outputs. 2) There is a dilemma between the codec and language model: on the one hand, the codee with token quantization (VQ- VQ-VAE (van den Oord et al. 2017: Razavi et al. 2019) or VQ-GAN (Esser et al. (02T)) usually has low bitrate token sequence, which, although eases the language model generation, incurs information loss on the high-frequenn fine-graid traiiee acoustic details; on the other hand, some improving methods (Zeghidour ict al. 2021 Défossez et al.) (022) use multiple residual discrete tokens to represent a speech frame, which increases the length of the token sequence multiple times if flattened and incurs difficulty in language modeling.
train
In this paper, propose NaturalSpeech 2, a TTS system with latent diffusion models to achieve expressive prosody, good robustness, and most importantly strong zero-shot ability for speech synthesis. As shown in [ we first train neural audio codec that converts speech waveform into a sequence of latent vectors with a codec encoder, and reconstructs the speech waveform from these latent vectors with codec decoder. After training the audio codec, we use the codec encoder to extract the latent vectors from the speech in the training set and use them as the target of the latent diffusion model, which is conditioned on prior vectors obtained from phoneme encoder, a duration predictor, and pitch predictor. During inference, we first generate the latent vectors from the text/phoneme sequence using the latent diffusion model and then generate the speech waveform from these latent vectors using the codec decoder.
Table 1: The comparison between NaturalSpeech 2 and previous large-scale TTS systems (Wang al. 2023 Kharitonov et al. 2023
Xue et al. 2023).
Methods
Previous Large-Scale Systems
NaturalSpecch 2
Representations
Discrete Tokens
Continuous Vectors
Generative Models
Autoregressive Models
Non-AAuooegreesvie//ffffe Only Speech is Needed
In-Context Learning
Both Text and Speech are Needed
Robbuity/Robustnesss?
X
One Acoustic Model?
X
Beyond TTS (c.g., Singing)?
X
We elaborate on some design choices in NaturalSpeech 2 (shown in Table ! as follows.
Continuous vectors instead of discrete tokens. To ensure the speech reconstruction quality of the neural codec, previous works usually quantize speech with multiple residual quantizers. As a result, the obtained discrete token sequence is very long (e.g., if using 8 residual quantizers for each speech frame, the resulting flattened token sequence will be 8 times longer), and puts much pressure the acoustic model (autoregressive language model). Therefore, we use continuous
2

---

Published as a conference paper at ICLR 2024
vectors instead of discrete tokens, which can reduce the sequence length and increase the amount of information for fine-gre lineggrained speech reconstruction (see Section B.I).
Diffussio models instead of autoregressive models. We leverage diffusion models to learn the complex distributions of continuous vectors in non-autoreerressive manner and avoid error propagation in autoregressive models (see Section 3.2).
Speech prompting for in-context learning. To encourage the model to follow the speech prompt characteristics and enhance the zero-shot capability, we design speech prompting mechanisms to facilitate in-context learning in the diffusion model and pitch/duration predictors (see Section 3.3)
Benefiting from these designs, NaturalSpeech is more stable and robust than previous autoregressive models, and only needs one acoustic model (the diffusion model) instead of two-stage token prediction as in (Borsos et al. 2022; Wang et al. 2023), ) and can extend the styles beyond TTS (e.g., singing voice) due to the duration/pitch prediction and non-autoregresssvve generation.
We scale NaturalSpeech 2 to 400M model parameters and 44K hours of speech data, and generate speech with diverse speaker identities, prosody, and styles (c.g., singing) in zero-shot scenarios (given only a few seconds of speech prompt). Experiment results show that NaturalSpeech 2 can generate natural speech in zero-shot scenarios and outperform the previous strong TTS systems. Specifically, 1) it achieves more similar prosody with both the speech prompt and ground-truth speech; 2) it achieves comparable or better naturalness (in terms of CMOS) than the ground-truth speech on LibriSpeech and VCTK test sets: 3) it can generate singing voices in a novel timbre either with a short singing prompt, or interestingly with only a speech prompt, which unlocks the truly zero-shot singing synthesis (without a singing prompt). Audio samples can be found in github. io /oonaturaa speech2
2 BACKGROUND
We present the background of NaturalSpeech 2, encompassing the pursuit of high-quality, natural voice in text-to-speech synthesis, neural audio codec models, and generative audio synthesis models.
TTS for Natural Voice. Text-to-speech systems Tan et al. 2021 Wang et al. 2017 Li et al.
2019
Ren al.
2019 Liu al. 2021: 2022b a Jiang et al. [022 Ye al.) 2023 Kim al.
2021 Tan et al.)
2022) aim to generate natural voice with both high quality and diversity. Since
TTS systems have achieved good voice quality, recent works attempt scale the TTS systems to large-scale, multi-speaker, and in-the-wild datasets to pursuc diversity (Betker [2033] Borsos et al.
2023 Zhang et al. 2023). Some works (Jiang al., 2023b Le et al. 2023 Li et al.) (023) propose
to generate mel-spectrogram by flow-mater dow-matching (Lipman et al. 2022 or GAN-based (Goodfellow
ct al. (2014) generation models NAR framework. Since the -pe-steetrogram pre-designed and intuitively less conducive to learning for neural networks, we leverage learnable latent by neural codec as the training objective. In parallel, some works (Borsos et al., 2022 Wang et al,, 2023
Kharitonov al. 2023 Xue et al., 2023 Huang al. (023) usually leverage neural codec to convert speech waveform into discrete token sequence and an autoregressive language model to generate discrete tokens from text, which suffers from a dilemma that: 1) Quantizing each frame into one token with vector-quantize (VQ) (van den Oord et al. 2017 Razavi et al.) 2019 Esser et al.) 202I) simplifies token generation but compromises waveform quality due to high compression. 2) Quantifying each frame into multiple tokens with residual vector-quantizer (RVQ) (Zeghioour et al.) 202T Défossez et al. 2022) ensures high-fidelity waveform reconstruction but hinders autoregressive model generation due to longer token sequences, causing errors and robustness challenges. Thus, previous works, such as AudioLM (Borsos al., 2022). leverage three stage language models first predict semantic tokens autoregressively, followed by generating coarse -raiind tokens per frame and ultimately producing remaining fine-grained tokens. VALL-- tackles this problem using an AR model for the first codec layer tokens and NAR model for the remaining layer tokens. These methods are complicated and incur cascaded errors. To avoid the above dilemma, we leverage a neural codec with continuous vectors and a latent diffusion model with non-autoregressive generation.
Neural Audio Codec. Neural audio codec (Oord et al. 2016 Valin Skoglund 2019 Zeghidour et al. 2021 Défossez et al. 2022) refers to kind of neural network model that converts audio wave- form into compact representations with a codec encoder and reconstructs audio waveform from these representations with codec decoder. SoundStream (Zeghidour al. 2021) and Encodec (Défosse7)
3

---

Published as a conference paper at ICLR 2024
et al. 2022) leverage vector-quantized variational auto-encoders (VQ-VAE) with multiple residual Dector-quantizerr to compress speech into multiple tokens, and have been used as the intermediate
representations for speech/audio generation (Borsos al.. 2022 Kreuk et al. 2022 Wang et al. 2023
Kharitonov et al. 2023 Xue et al. 2023). Residual vector quantizers provide good reconstruction
quality and low bitrate but may not be ideal for speech/audio generation due their long discrete token sequences (Rt times longer if It residual quantizers are used), which makes prediction tasks more challenging and may lead to errors such as word skipping, repetition, or speech collapse. In this paper, we design a neural audio codec that converts waveforms into continuous vectors, retaining fineggrineed details for accurate waveform reconstruction without increasing sequence length.
Generative Models for Speech Synthesis. Neural TTS systems aim to synthesize high-quality voice.
Generative models such as language models (Li et al. 2019 Shen et al.
2018
Wu et al. 2023).
VAE (Ren et al. 2021b Lee et al. 2022). Normalization flow (Kim et al.
2021 Miao et al.
2020
Kim et al. 2020p. GAN (Kim et al. 2021 [iing et al.) 2023a), diffusion model (Kong al., 2021
Jeong et al. 2021 Chen al. 202 Popoy al. 202 Chen et al. 2021b), self-supervised learning
methods (Siuzdak et al. 2022; Du et al., 2022) and speech representation learning methods Hsu et al. (2021); Schhniier et al. (2019) achieve great success. Among these, autoregressive language models and diffusion models are the two most prominent methods. Although both models are based on iterative computation (following the left-to-right process or the denoising process), autoregressive models are more sensitive to sequence length and error propagation, which cause unstable prosody and robustness issues (e.g., word skipping. repeating, and collapse). Considering text-to-speech has a strict monotonic alignment and strong source-target dependency, we leverage diffusion models enhanced with duration prediction and length expansion, which are free from robust issues.
3 NATURALSPEECH 2
In this section, we introduce NaturalSpeech 2, a TTS system for natural and zero-shot voice synthesis with high fiddittysprresivenss/rrbssss and styles). As shown in Figure [ NaturalSpeech 2 consists of neural audio codec and diffusion model with a prior model (a phoneme encoder and a duration/pitcl predictor). Since speech waveform complex and high-dimensional, following the paradigm of regeneration learning (Tan et al. 2023), we utilize post-quantized latent vectors to represent waveform x Next, we employ a prior model to encode text input y into a prior c, and diffusion model to predict the latent vectors 2 conditioned on prior Finally, latent vectors are imput into the audio codec decoder to reconstruct the waveform We introduce the detailed designs of neural audio codec in Section [.] and the latent diffusion model in Section 3.2 as well as the speech prompting mechanism for in-context learning in Section 3.3
on diverse scenarios (various speaker identities, prosodies,
3.1 NEURAL AUDIO CODEC WITH CONTINUOUS VECTORS
We use a neural audio codec to convert speech waveform into continuous vectors instead of discrete tokens, as analyzed in Section 2 Audio codec with continuous vectors enjoys several benefits: 1) Continuous vectors have lower compression rate and higher bitrate than discrete token which can ensure high-quality audio reconstruction. 2) Each audio frame only has one vector instead of multiple tokens as in discrete quantization, which will not increase the length of the hidden sequence.
We employ the SoundStream(Zeeghddou et al. 2021 architecture as our neural audio codec, which comprises an audio encoder, residual rector-quantizar (RVQ), and an audio decoder. The residual vector-quantizer cascades Rt layers vector-quantizer (VQ) and transforms the output of the audio encoder into quantized latent vectors, which serve as the training target of the diffusion model. More details about codec are provided in Appendix A
Actually, to obtain continuous vectors, do not need vector quantizers, but just an autoencoder or variational autoencoder. However, for regularization and efficiency purposes, we use residual vector quantizers with a very large number of quantizers and codebook tokens approximate the continuous vectors. This provides two benefits: 1) Reduced dataset storage by storing codebook embeddings and quantized token IDs instead of high- nigh-dimensional continuous vectors, and the regularization loss discrete classification based on quantized token IDs (see Lce-rva in Section 3.2)
Since our task is compression hut synthesis, do not need high compression low hitrate.
4

---

Published as a conference paper at ICLR 2024
3.2 LATENT DIFFUSION MODEL WITH NON-AUTOREGRESSIVE GENERATION
We leverage a prior model, comprising a phoneme encoder, a duration predictor, and a pitch predictor, process the text input and provide more informative hidden vector Subsequently, the diffusion model predicts the quantized latent vector = conditioned on hidden vector c.
Diffusion Formulation. We formulate the diffusion (forward) process and denoising (reverse) process (Liptser & Shiriiaev, 1977, Sohl-Dickstein et al. 2015; Ho et al. (0200) stochastic
differential equation (SDE) (Song & Ermon| 2019] 2020. Song et al.) 2020), respectively. The forward SDE transforms the latent vectors Zg obtained by the neural codec (i.e., 2) into Gaussian
noises Popov et al. 2021:
1 , dt + v B dwr, E [0, 1],
da 2
(I)
where w, is the standard Brownian motion, E [0. 1], and 3, is non-negative noise schedule function. Then the solution is given by:
2 = e So d.ds + S'" B à Sa S. dw..
(2)
By properties of Ito's integral, the conditional distribution of 2 given 2o is Gaussian: p(ze|zo) ~ (((2zo t), E,), where p(20. t) Jo 8.dx 20 and E, I e Jo 9.ds
The reverse SDE transforms the Gaussian noise back to data 20 with the following process:
1 (5 V loggr(z)))&& di + VB, dw,, I L [0,
dæ.
where wi is the reverse-time Brownian motion. Moreover, we can consider ordinary differential equation (ODE) (Song et al. 2020) in the reverse process:
dæ
2 (z+ Vlog dt, t [0,
(4)
We can train neural network 88 to estimate the score V Peg (the gradient of the log-density of noisy data), and then we can sample data 2g by starting from Gaussian noise 21 ~ N/00, 1) and numerically solving the SDE in Equation 33 or ODE in Equation In our formulation, the neural network so s0(2,, 4, c) is based on WaveNet (Oord et al. 2016), which takes the current noisy vector 2t, the time step and the condition information which we found results in better speech quality. Thus, lg sæ (Zz, t, c). The loss function to train the diffusion model is as follows.
input, and predicts the data ån instead of the score,
Lai Es, 2l 2ol ||, (p(2o,t) 2)) - V (z))( + Nce-rooCoe-rvgl.
(5)
where the first term is the data loss, the second term is the score loss, and the predicted score is calculated by 2, (p(,o,t) z,), which is also used for reverse sampling based on Equation or in inference. The third term Lee-rve is a novel cross-entropy (CE) loss based on residual vector- quantizer (RVQ). Specifically, for each residual quantizer j E [1, R, we first get the residual vector 20 -E e,, where e; is the ground -ruth quantized embedding in the i-th residual quantizer. Then calculate the I.2 distance between the residual vector with each codebook embedding in quantizer j and get probability distribution with a softmax function, and then calculate the cross-entropy loss between the ID of the ground-truth quantized embedding ej and this probability distribution. Lee-rve is the mean of the cross-entropy loss in all It residual quantizers, and Ace-rvq is set to 0.1. Please refer Appendix C.3 for more details of Lee-rvg-
Phoneme Encoder and Duration/Pitch Predictor. The phoneme encoder consists of 6 Transformer blocks (Vaswani et al., 2017: Ren et al. 2019). where the standard feed-forward network is modified convolutional network to capture the local dependency in phoneme sequence. The duration and pitch predictors utilize similar model structure, consisting of several convolutional blocks. The ground-trufe -trth duration and pitch information is used as the learning target train the duration and pitch predictors, with an LI duration loss Laur and pitch loss Cpitch. During training, the ground-truth duration is used to expand the hidden sequence from the phoneme encoder obtain
Prior Model:
5

---

Published as a conference paper at ICLR 2024
the frame-level hidden sequence, and then the ground-trutt pitch information is added the frame- level hidden sequence to get the final condition information c. During inference, the corresponding predicted duration and pitch are used.
The total loss function for the diffusion model as follows:
=
Lait + Laur + Lpitch.
(6)
Duration/Pitch Predictor
Diffusion Model
Zo
Duration/Pitch
FILM
K/V
K/V Q-K-V Attention
K/V
Q-K-V Attention
Q-K-V Attention
Q
a
Q
Speech Prompt Encoder
Convolution
WaveNet
x
N'
Phoneme Hidden
Figure 2: The speech prompting mechanism the duration/pitch predictor and the diffusion model for in-context learning. During training, we use random segment 2 of the target speech & as the speech prompt 2" and use the diffusion model to only predict 2 2Auar During inference, we use reference speech of specific speaker as the speech prompt >". Note that the prompt is the speech latent obtained by the codec encoder instead of the speech waveform.
3.3 SPEECH PROMPTING FOR IN- CONTEXT LEARNING
To facilitate in-context learning for better zero-shot generation, we design speech prompting mechanism to encourage the duration/pitch predictor and the diffusion model to follow the diverse information (e.g., speaker identities) in the speech prompt. For speech latent sequence 2, randomly cut off a segment 2 zu with frame index from u to e as the speech prompt, and concatenate the remaining speech segments 8 1:u and z to form a new sequence z'uur as the learning target of trunsformerbaseee prompt encoder process
the diffusion model. As shown in Figure 2 we use the speech prompt 200 (z" in the figure) to get a hidden sequence. To leverage this hidden sequence have two different strategies for the duration/pitch predictor and the diffusion model: 1) For the duration and pitch predictors, we insert a Q-K-Y V attention layer in the convolution layer, where the query is the hidden sequence of the convolution layer, and the key and value the hidden sequence from the prompt encoder. 2) For the diffusion model, instead of directly attending to the hidden sequence from the prompt encoder that exposes too many details to the diffusion model and may harm the generation. we design two attention blocks: in the first attention block, we use m randomly initialized embeddings as the query sequence to attend to the prompt hidden sequence, and get hidden sequence with length of m as the attention results (Wang et al. 2016 2018 Yin ctt al., 2022);; in the second attention block, we leverage the hidden sequence in the WaveNet layer as the query and the m-length attention results as the key and value. We use the attention results of the second attention block as the conditional information of a FiLM layer (Perez et al., 2018) to perform affine transform the hidden sequence of the WaveNet in the diffusion model. Please refer to Appendix for the details of WaveNet architecture used the diffusion model.
as the prompt,
4 EXPERIMENTS AND RESULTS
4.1 EXPERIMENTAL SETTINGS
In this section, we introduce experimental settings to train and evaluate NaturalSpeech 2, including the dataset, baselines, and evaluation metrics. Please refer to Appendix Crrennix(ffor.. the model configuration and training and inference details.
To train the neural audio codec and the diffusion model, we use the English subset of Multilingual LibriSpeech (MLS) (Pratap et al., 22220), comprising 44K hours transcribed audiobook data. It contains 2742 male and 2748 female distinct speakers. We evaluate using two benchmark datasets: 1) LibriSpeech test-clean (Panayotov et al. 2015), with 40 distinct speakers and 5.4 hours of annotated speech; 2) VCTK dataset (Veaux al. 2016), with 108 distinct speakers. We sample
Datasets:

---

Published as a conference paper at ICLR 2024
15 and 5 utterances per speaker for LibriSpeech and VCTK, resulting in 600 and 540 evaluation utterances, respectively. For synthesis, a different same-speaker utterance is cropped into a o-second audio segment as prompt. All speakers in two datasets unseen during training. Singing datasets follow a similar process, detailed in Section 4.4 See Appendix for data processing details.
Model Comparison: We compare NaturalSpeech 2 with baselines including: 1) YourTTS (Casanoval et al. 20226). 2) FastSpeech 2 (Ren et al. (0211a) We adapt it by adding cross-attention on speech prompts for zero-shot synthesis. Furthermore, we also change the prediction target from the mel- spectrogram to the latent representation. 3) VALL-E (Wang et al. (023). 4) Foundation (Xue) al. 2023). 5) Voicebox (Le et al., 2023). 6) MegaTTS (Jiang et al.) 2023b). For YouTTTS, we use the official code and pre-trained checkpoing For FastSpeech 2. VALL-E, and Foundation we implement them according to the papers. We scale them to 400M parameters and use the same dataset for fair comparison. In addition, for Voicebox, VALL-E, and MegaTTS, since there are no official implementations. we download the audio samples from their demo page and compare them with NaturalSpeech 2 individually. Please refer to Appendix D for more details.
We use both objective and subjective metrics to evaluate the zero-shot synthesis ability of NaturalSpeech 2 and compare it with baselines. Please refer to Appendix F for a more detailed metric description
Evaluation Metrics:
1) Prosody Similarity with Prompt. Following the practice (Zaidi et al. 202 we compare the difference in mean, standard deviation, skewness, and kurtosis of the duration/pitch to assess prosody similarity between synthesized and prompt speech. 2) Word Error Rate. We employ an ASR model to transcribe the generated speech and calculate the word error rate (WER).
Objective Metrics:
Subjective Metrics: 1) Intelligibilitt Score. To test the robustness, following the practice in (Ren et al. 2019), we use the 50 particularly difficult sentences (see Appendix G.2 and conduct an intelligibility test. 2) CMOS and SMOS. We evaluate naturalness using comparative mean option score (CMOS), and speaker similarity using similarity mean option score (SMOS).
4.2 EXPERIMENTAL RESULTS ON NATURAL AND ZERO-SHIO SYNTHESIS
In this section, we conduct experiments comparing the NaturalSpeech 2 with the baselines in terms of: 1) Generation Quality, by evaluating the naturalness of the synthesized audio; 2) Generation Similarity, by evaluating how well the TTS system follows prompts; 3) Robustness, by calculating the WER and an additional intelligibility test. 4) Generation Latency, by evaluating the trade-off between the inference efficiency and generation quality.
Generation Quality. We conduct CMOS test to evaluate the generation quality (i.e., naturalness). We randomly select 20 utterances from the LibriSpeech and VCTK tests and crop the prompt speech to 3s. To ensure high-quality generation, we use a speech scoring model (Chen et al., (022) to filter the multiple samples generated by the diffusion model with different starting Gaussian noises 21. Table 2 shows a comparison of NaturalSpeech 2 against baselines and the ground truth. We have several observations: 1) NaturalSpeech is comparable to the ground-truth recording in LibriSpeech (+0.04 is regarded as par) and achieves much better quality on VCTK datasets (-0.21 is large gap), which demonstrates the naturalness of the speech generated by NaturalSpeech 2 is high enough. 2) NaturalSpeech 2 outperforms all the baselines by large margin in both datasets. Specifically, for VALL-E, Nateca 2 shows 0.29 and 0.31 CMOS gain in LibriSpeech and VCTK, respectively. It demonstrates that the speech generated by NaturalSpeech is much more natural and of higher quality. 3) Using the cases from demo pages, we find NaturalSpeech 2 surpasses the state-of-the-art large-scale TTS systems, which shows the superiority of NaturalSpeech 2.
Generation Similarity, We use two metrics to evaluate the speech similarity: 1) prosody similarity between the synthesized and prompt speech. 2) SMOS test. To evaluate the prosody similarity, we randomly sample one sentence for each speaker for both LibriSpeech test-clean and VCTK dataset to form the test sets. Specifically, to synthesize each sample, we randomly and independently sample the prompt speech with o = 3 seconds. Note that Your has seen 97 speakers in VCTK in training, but we still compare NaturalSpeech 2 with YourTTS on all the speakers in VCTK (i.e., the 97 speakers are seen to YTU but unseen to NaturalSpeech 2).
7

---

Published as a conference paper at ICLR 2024
Table 2:
The CMOS, SMOS and WER results on LibriSpeech and VCTK with 95% confidence intervals. means the results from official demo page. "_" denotes the results are not available. Note that the comparison with demo cases involves pairwise comparisons between NaturalSpeech 2 and baselines across various test cases, rendering the baseline scores in this comparison non-pairwise.
Dataset
LibriSpeech
VCTK
Setting
CMOSt
SMOST
WERL
CMOST
SMOST
WER
Ground Truth
+0.04
4.27
1.94
0..1
1.05_0.11
9.49
YouTTSS (Casanova et al 2022b)
-0.65
2..0
7.10
-0.58
3.39_0.0s
14.80
FastSpeech 2 |Ren et al. 2021a
-0.33
3. 3.45
2.10
-0.64
3.22
8.26
Foundation TTS (Xue ct al.
2023)
0..2
3.81 10.12
4.63
-0.39
3.42=0.13
12.55
VALL-E (Wang al. 2023
-0.29
3.92
5.72
-0.31
3.50
14.68
NaturalSpeech
0.00
2.0
2.01
0.00
3.62
6.72
Comparison with demo cases
VALL-E (Wang al. NaturalSpeech
2023)*
-0.27
3.98
-0.34
3.59_0.13
0.00
4.11
0.00
3.71
MegaTTS (Tiang et NaturalSpecch
2023b
0.20
3.96
0.28
3.63-g.0s
0.00
4.10
0.00
3.74
Voicehox Le al.
2023
0.11
3.75
NaturalSpeech2
0.00
3.86
Table The prosody similarity between synthesized and prompt speech in terms of the difference in mean (Mean), standard deviation (Std), skewness (Skew), and kurtosis (Kurt) of pitch and duration.
Pitch
Duration
LibriSpeech
Mean
Stdl.
Skew
Kurt
Mean
Std.j
Skew
Kurt
YourTTS
10.52
7.62
0.59
1.18
0.84
0.66
0.75
3.70
FastSpeech 2
14.61
9.31
1.83
3.15
0.67
0.71
0.77
3.60
Foundation
10.34
7.04
0.62
1.51
0.67
0.72
0.70
3.38
VALL-E
10,23
6.19
0.54
1.09
0.62
0.67
0.64
3.22
NaturalSpeech 10.11
6.18
0.50
1.01
0.65
0.70
0.60
2.99
We apply the aligmment tool to obtain phoneme-leve duration and pitch and calculate the prosody similarity metrics between the synthesized speech and the prompt speech described in Section 4.] We report the results on LibriSpeech in Tad on VCTK in appendix[H.]] We have the following observations: 1) NaturalSpeech 2 consistently outperforms all the baselines in both LibriSpeech and VCTK on most metrics, which demonstrates that our proposed NaturalSpeech can mimic the prosody of prompt speech much better. 2) Although YourTTS has seen 97 from 108 speakers in VCTK dataset, our model can still outperform it by a large margin. Furthermore, we also compare prosody similarity between synthesized and ground-truth speech Appendix H.2
We further evaluate speaker similarity using SMOS test. We randomly select 10 utterances from LibriSpeech and VCTK datasets respectively, following the setting in the CMOS test. The prompt speech length is set to 3s. The results are shown in Table We find that NaturalSpeech 2 outperforms all the baselines in datasets. Specifically, NaturalSpeech outperforms the state-of-the-ar method VALL-E by 0.14 4 and 0.12 SMOS scores for LibriSpeech and VCTK, respectively. It demonstrates that NaturalSpeech is significantly better in speaker similarity.
Robustness. We use the full test set of LibriSpeech and VCTK as described in Section 4. to synthesize the speech and compute the word eтror rate (WER) between the transcribed text and ground-truth text. To synthesize each sample, we use 3-second prompt by randomly cropping the whole prompt speech. The results are shown in Table 2 significantly outperforms all the baselines in LibriSpeech and VCTK, indicating better synthesis of high-quality and robust speech. 2) Our synthesized speech is comparable to the ground-truth speech in LibriSpeech and surpasses that in VCTK. The higher WER results in VCTK may stem from noisy environment and the lack of ASR model fine-tuning in that dataset.
We observe that: 1) NaturalSpeech
In addition, conduct intelligibility test on 50 particularly hard sentences from FastSpeech (Ren)
ct al. 2019) to evaluate speech robustness. NaturalSpeech 2 demonstrates robustness in these cases without any intelligibility issues. Please refer Appendix G. for the detailed results.
8

---

Published as a conference paper at ICLR 2024
Generation Latency. We conduct comparison of both the latency and generation quality with varying diffusion step (f20, 50, 100, 150}). The comparison also incorporates a NAR baseline (FastSpeech 2) and an AR model (VALL-E). As detailed in Table T2 in Appendix [K] the diffusion step of 150 strikes a balance between quality (with a 0.53 CMOS gain over FastSpeech2 and a 0.29 CMOS gain over VALL-E) and lateney (12.35 times faster than VALL--E. 2) As the diffusion step decreases, the inference speed increases while there is no noticeable degradation in performance. Please refer Appendix K for more details.
4.3 ABLATION STUDY
In this section, we perform ablation experiments. 1) To study the effect of the speech prompt, we remove the Q-K-V attention layers in the diffusion (abbr. w/o. diff prompt), and the duration and pitch predictors (abbr. w/o. dur/pitch prompt), respectively. 2) To study the effect of the cross-entropy (CE) loss Lov based on RVQ, we disable the CE loss by setting Ace-ruo to 0 (abbr. w/o. CE loss). 3) To study the effectiveness of two Q-K-V attention in speech prompting for diffusion in Section [.3] we remove the first attention that adopts m randomly initialized query sequence to attend to the prompt hidden and directly use one Q-K-V attention to attend the prompt hidden (abbr. w/o. query attn). We report CMOS and WER results in Table 4
More detailed results are in Appendix
We have the following observations: 1) When we disable speech prompt in diffusion, the model can not converge, which highlights its importance for high-quality TTS synthesis. 2) Disabling speech prompt in duration/pitch predictor significantly degrades audio quality (i.e., 0.45 CMOS degradation). In practice, we find that without speech prompt, can pronounce the words correctly but with poor prosody, which causes CMOS degradation. 3) Disabling CE loss worsens both CMOS and WER performance. It shows that regularization is important for high-quality synthesis and robustness. 4) Disabling the query attention also degrades both CMOS and WER performance. In practice, we find that applying cross- attention to prompt hidden will leak details and thus mislead generation.
Table 4: The ablation study of Natural-
Speech 2, measured by CMOS and WER. "_" denotes the model can not converge.
CMOS
WER
NaturalSpeech
0.00
2.01
w/o. diff prompt
w/o. dur/pitch prompt
-0.45
2.23
w/o. CE loss
-0.25
3.03
w/o. query atin
-0.13
2.65
4.4 ZERO-SHOE SHOT SINGING SYNTHESIS
In this section, we explore NaturalSpeech 2 to synthesize singing voice in a zero-shot setting, either given a singing prompt or only speech prompt. We use speech and singing data together to train NaturalSpeech 2 with a 5e - 5 learning rate. In inference, we set the diffusion steps to 1000 for better performance. To synthesize a singing voice, we use the ground-truth pitch and duration, and use various singing prompts to generate singing voices with different singer timbres. Interestingly, we find that NaturalSpeech 2 can generate a novel singing voice using speech as the prompt. See the demo page for zero-shot singing synthesis with either singing speech the prompt. Please refer to Appendix E for more details. Furthermore, we extend NaturalSpeech 2 to support more tasks such voice conversion and speech enhancement. Please refer to Appendix D for more details.
5 CONCLUSION
In this paper, we develop NaturalSpeech 2, a TTS system that leverages neural audio codec with continuous latent vectors and a latent diffusion model with non-autoregressive generation to enable natural and zero-shot text-to-speech synthesis. To facilitate in-context learning for zero-shot synthesis, we design a speech prompting mechanism in the duration/pitch predictor and the diffusion model. By scaling NaturalSpecch 2 to 400M model parameters, 44K hours of speech, and 5K speakers, it can synthesize speech with high expressiveness, robustness and strong zero-shot ability, outperforming previous TTS systems. For future work, we will explore efficient strategies such as (Song et al. 2023) to speed up. and explore large-scale speaking and singing voice training to enable more powerful mixed speaking/singing capability. We include our limitation in Appendix M
https:://ppecchesserrhhh httee///geeeheeearrrggggggghio/u speech2

---

Published as a conference paper at ICLR 2024
Broader Impacts: Since NaturalSpeech 2 could synthesize speech that maintains speaker identity, it may carry potential risks in misuse of the model, such as spoofing voice identification or impersonating specific speaker. We conduct experiments under the assumption that the user agrees to be the target speaker in speech synthesis. If the model generalizes to unseen speakers in the real world, it should include protocol ensure that the speaker approves the use of their voice and synthesized speech detection model.
REFERENCES
James Betker. Better speech synthesis through scaling, arXiv preprint arXiv:2305.07243 2023.
Zalán Borsos, Raphaël Marinier, Damien Vincent, Eugene Kharitonov, Olivier Pietquin, Matt Sharifi, Olivier Teboul, David Grangier, Marco Tagliasacchi, and Neil Zeghidour. Audiolm: a language modeling approach to audio generation. arXiv preprint arXiv:2209.00 03 143, 2022.
Zalán Borsos, Matt Sharifi, Damien Vincent, Eugene Kharitonov, Neil Zeghidour, and Marco Tagliasacchi. Soundstorm: Efficient parallel audio generation. arXiv preprint arXiv:2305. 09636, 2023.
Edresson Casanova, Arnaldo Candido Junior, Christopher Shulby, Frederico Santos de Oliveira, João Paulo Teixeira, Moacir Antonelli Ponti, and Sandra Aluísio. Tts-portuguese corpus: a corpus for speech synthesis in brazilian portuguese. Language Resources and Evaluation, 56(3): 1043-1055, 2022a.
Edresson Casanova, Julian Weber, Christopher D Shulby, Arnaldo Candido Junior, Eren Gölge, and Moacir A Ponti. Yourtts: Towards zero-shot multi-speaker and zero-shot voice conversion for everyone. In International Conference on Machine Learning, pp. 2709-2720 PMLR, 2022b.
Nanxin Chen, Yu Zhang, Heiga Zen, Ron J Weiss, Mohammad Norouzi, and William Chan. WaveG- rad: Estimating gradients for waveform generation. In ICLR, 2021a.
Nanxin Chen, Yu Zhang, Heiga Zen, Ron Weiss, Mohammad Norouzi, Najim Dehak, and William Chan. WaveGrad 2: Iterative refinement for text-to-speech synthesis. arXiv preprint arXiv:2106.09660, 2021b.
Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, al. Wavlm: Large-scale self-supervised pre-training for full stack speech processing. IEEE Journal of Selected Topics in Signal Processing, 16(6); 1505-1518, 2022.
Alexandre Défossez. Hybrid spectrogram and waveform source separation. arXiv preprint arXiv:2111.03600, 2021.
Alexandre Défossez, Jade Copet, Gabriel Synnaeve, and Yossi Adi. High fidelity neural audio compression. arXiv preprint arXiv:2210.13438, 2022.
Chenpeng Du, Yiwei Guo, Xie Chen, and Kai Yu. VQTTS: High-fidelit text-to -ot-to--peech synthesis with supervupe VQ acoustic feature. arXiv preprint arXiv:2204.00768 2022.
Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 12873-12883 2021.
Ian J Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aäron Courville, and Yoshua Bengio. Generative adversarial nets. In NIPS, 2014.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. arXiv preprint arXiv:2006. 1239, 2020.
Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM Transactions on Audio, Speech, and Language Processing. 29:3451-3460, 2021.
10

---

Published as a conference paper at ICLR 2024
Rongji Huang, Chunlei Zhang, Yongqi Wang, Dongchao Yang, Luping Liu, Zhenhui Ye, Ziyue Jiang, Chao Weng, Zhou Zhao, and Dong Yu. representation. arXiv preprint arXiv:2305.19269, 2023.
Unified voice synthesis with discrete
Keith Ito. The LJ speech dataset. htttp:://eithito..com Spapa://L---SSeeee--------
2017.
Myeonghun Jeong, Hyeongju Kim, Sung Jun Cheon, Byoung Jin Choi, and Nam Soo Kim. Diff-TTS: A denoising diffusion model for text-to-speech. arXiv preprini arXiv:2104.01409, 2021.
Ye Jia, Heiga Zen, Jonathan Shen, Yu Zhang, and Yonghui Wu. PnG BERT: Augmented BERT on phonemes and graphemes for neural TTS. arXiv preprint 2rX 2021.
Ziyue Jiang, Yi Ren, Ming Lei, and Zhou Zhao. Fedspeech: Federated text-to-speech with continual learning. arXiv preprint 10.0022 2021.
Ziyue Jiang, Jinglin Liu, Yi Ren, Jinzheng He, Zhenhui Ye, Shengpeng Ji, Qian Yang, Chen Zhang, Pengfei Wei, Chunfeng Wang, et al. Boosting prompting mechanisms for zero-shot speech synthesis. In The Twelfth International Conference on Learning Representations, 2023a.
Ziyue Jiang, Yi Ren, Zhenhui Ye, Jinglin Liu, Chen Zhang, Qian Yang. Shengpeng Ji, Rongjie Huang, Chunfeng Wang, Xiang Yin, et al. Mega-tts: Zero-shot text-to-spechh at scale with intrinsic inductive bias. arXiv preprint arxivv2306.03330
2023b.
Jacob Kahn, Morgane Riviere, Weiyi Zheng, Evgeny Kharitonov, Qiantong Xu,
Mazaré, Julien Karadayi, Vitaliy Liptchinsky, Ronan Collobert, Christian Fuegen, et al. Libri-light: A benchmark for asr with limited or supervision. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 7669-7673. IEEE, 2020.
Eugene Kharitonov, Damien Vincent, Zalán Borsos, Raphaël Marinier, Sertan Girgin, Olivier Pietquin, Matt Sharifi, Marco Tagliasacchi, and Neil Zeghidour. Speak, read and prompt: High-fidelity text-to-speech with minimal supervision. arXiv preprint arXiv:2302.035400 2023.
Jachyeon Kim, Sungwon Kim, Jungil Kong, and Sungroh Yoon. Glow-TTS: A generative flow for text-to-speech via monotonic alignment search. Advances in Neural Information Processing Systems, 33, 2020.
Jaehyeon Kim, Jungil Kong, and Juhee Son. Conditional variational autoencoder with adversarial learning for end-to-end text-to-speech. arXiv preprint arXiv:2106.06 106.0060,, 2021.
Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. DiffWave: A versatile
difffusinn model for audio synthesis. In ICLR, 2021.
Felix Kreuk, Gabriel Synnaeve, Adam Polyak, Uriel Singer, Alexandre Défossez, Jade Copet, Devi Parikh, Yaniv Taigman, and Yossi Adi. Audiogen: Textually guided audio generation. arXiv preprint 220i /5352, 2022.
Matthew Le, Apoory Vyas, Bowen Shi, Brian Karrer, Leda Sari, Rashel Moritz, Mary Williamson, Vimal Manohar, Yossi Adi, Jay Mahadeokar, al. Voicebox: Text-guided multilingual universal speech generation at scale. arXiv preprint arXiv:2306.1563) 2023.
Yoonhyung Lee, Joongbo Shin, and Kyomin Jung. Bidirectional variational inference for non- autoregressive text-to-speech. In International conference on learning representations, 2022.
Naihan Li, Shujie Liu, Yanqing Liu, Sheng Zhao, and Ming Liu. Neural speech synthesis with Transformer network. Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 6706-6713, 2019.
Yinghao Aaron Li, Cong Han, Vinay S Raghavan, Gavin Mischler, and Nima Mesgarani. Styletts 2: Towards human- level text-to-speech through style diffusion and adversarial training with large speech language models. arXiv preprint vrXiv:2306.0769/ 2023.
Yaron Lipman, Ricky TQ Chen, Heli Ben- Gen-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint arXiv:2210.027477 2022.

---

Published as a conference paper at ICLR 2024
Robert Shevilevich Liptser and Albert Nikolaevich Shiriiaev. Statistics of random processes: General theory, volume 394. Springer, 1977.
Jinglin Liu, Chengxi Li, Yi Ren, Feiyang Chen, and Zhou Zhao. DiffSinger: Singing voice synthesis via shallow diffusion mechanism. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 11020- 1028, 2022a.
Yanqing Liu, Zhihang Xu, Gang Wang, Kuan Chen, Bohan Li, Xu Tan, Jinzhu Li, Lei He, and Sheng Zhao. DelightfulTTS: The Microsoft speech synthesis system for Blizzard challenge 2021. arXiv preprint arXiv:21 10.110022000 /2612, 2021.
Yanqing Liu, Ruiqing Xue, Lei He, Xu Tan, and Sheng Zhao. DelightfulTTS 2: End-to-end speech synthesis with adversarial vector-quantized auto-encoders. arXiv preprint arXiv:2207.04mm 2022b.
Chenfeng Miao, Shuang Liang, Minchuan Chen, Jun Ma, Shaojun Wang, and Jing Xiao. Flow-TTS: A non-autoregressive network for text to speech based flow. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 7209-7213. IEEE, 2020.
Aäron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. WaveNet: A generative model for raw audio. arXiv preprint 1609.0346 2016.
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. LibriSpeech: an ASR corpus based on public domain audio books. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pp- 5206-5206500 5210. IEEE, 2015.
Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reasoning with a general conditioning layer. Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
Vadim Popov, Ivan Vovk, Vladimir Gogoryan, Tasnima Sadekova, and Mikhail Kudinov. Grad-TTS: A diffusion probabilistic model for text-to-speech. arXiv preprint arXiv:2105.06337 2021.
Vineel Pratap, Qiantong Xu, Anuroop Sriram, Gabriel Synnaeve, and Ronan Collobert. MLS:
large-scale multilingual dataset for speech research. Proc. Interspeech 2020, pp. 2757-2761, 2020.
Konpat Preechakul, Nattanat Chatthee, Suttisak Wizadwongsa, and Supasorn Suwajanakorn. Dif- fusion autoencoders: Toward meaningful and decodable representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10619-10629, 2022.
Ali Razavi, Aaron van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with VQE-2 Advances in neural information processing systems, pp. 14866-14876, 2019.
Yi Ren, Yangjun Ruan, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Yin Liu. FastSpeech: Fast, robust and controllable text to speech. In NeurlPS, 2019.
Yi Ren, Chenxu Hu, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie-Yan Liu. FastSpeech 2: Fast and high-quality end-to-end text to speech. In International Conference on Learning Representations, 2021a. URL
net htt://rrrmmmmmpmpmeemeeeee
Yi Ren, Jinglin Liu, and Zhou Zhao. Portaspeech: Portable and high-quality generative text-to-speech. Advances in Neural Information Processing Systems,
2021b.
Steffen Schneider, Alexei Baevski, Ronan Collobert, and Michael Auli. wav2vec: Unsupervised 2019.
pre-training for speech recognition. Proc. Interspeech 2019, pp.
Jonathan Shen, Ruoming Pang, Ron J Weiss, Mike Schuster, Navdeep Jaitly, Zongheng Yang, Zhifeng Chen, Yu Zhang, Yuxuan Wang, RJ SkenyyRyan, et al. Natural TTS synthesis by conditioning WaveNet on mel spectrogram predictions. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 4779 IEFF, 2018.
12

---

Published as a conference paper at ICLR 2024
Hubert Siuzdak, Piotr Dura, Pol van Rijn, and Nori Jacoby. Wavthruvee: Latent speech representation as intermediate features for neural speech synthesis. arXiv preprint arXiv:2203. 2022.
Jascha Sohl- Sohl-Dickstein. Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning. pp. 2266-2265. PMLR, 2015.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in Neural Information Processing Systems, 32, 2019.
Yang Song and Stefano Ermon. Improved techniques for training score-based generative models. Advances in neural information processing systems, 33:1 13:1124881222 12448, 2020.
Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben
Poole. score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2020.
Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. arXiv preprint arXiv:2303.01469, 2023.
Hao Sun, Xu Tan, Jun- Wei Gan, Hongzhi Liu, Sheng Zhao, Tao Qin, and Tie- Yan Liu. Token-level ensemble distillation for grapheme--oophooneee conversion. In INTERSPEECH, 2019.
Xu Tan, Tao Qin, Frank Soong, and Tie- Yan Liu. A survey on neural speech synthesis. arXiv preprint arXiv:2 106. 15561, 2021.
Xu Tan, Jiawei Chen, Haohe Liu, Jian Cong, Chen Zhang, Yanqing Liu, Xi Wang, Yichong Leng, Yuanhao Yi, Lei He, et al. NaturalSpeech: End-to-end text to speech synthesis with human-level quality. arXiv preprint arXiv:2205.04421, 2022.
Xu Tan, Tao Qin, Jiang Bian, Tie- Yan Liu, and Yoshua Bengio. Regeneration learning: A learning paradigm for data generation. arXiv preprint arXiv:230I. 08846, 2023.
Paul Taylor. Vext-to-speech synthesis. Cambridge university press, 2009.
Jean-Mare Valin and Jan Skoglund. LPCNet: Improving neural speech synthesis through linear prediction. In ICASSP 2019-2019 JEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5891-5895. IEEE, 2019.
Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learning. In Proceedings the 3 International Conference on Neural Information Processing Systems, pp. 6309-6318, 6318, 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, pp. 5998-6008, 2017.
Christophe Veaux, Junichi Yamagishi. Kirsten MacDonald, al. Superseded-CSTN VCTK corpus: English multi-speaker corpus for CSTK voice cloning toolkit. 2016.
Chengyi Wang. Sanyuan Chen, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang. Jinyu Li, et al. Neural codec language models are zero-shot text to speech synthesizers. arXiv preprint arXiv:2301.0211
2023.
Yequan Wang, Minlie Huang, Xiaoyan Zhu, and Li Zhao. Attention-based Istm for aspect-level sentiment classification. In Proceedings of the 2016 conference on empirical methods in natural language processing, pp. 606 615, 2016.
Yuxuan Wang, RJ Syan,y-Ran, Daisy Stanton, Yonghui Wu, Ron J Weiss, Navdeep Jaitly, Zongheng Yang, Ying Xiao, Zhifeng Chen, Samy Bengio, et al. Tacotron: Towards end-to-end speech synthesis. Proc. Interspeech 2017, pp. 4010-4010, 2017.
13

---

Published as a conference paper at ICLR 2024
Yuxuan Wang, Daisy Stanton, Yu Zhang. RJ Skerry-Ryan, Eric Battenberg, Joel Shor, Ying Xiao, Ye Jia, Fei Ren, and Rif A Saurous. Style tokens: Unsupervised style modeling, control and transfer in end-to-end speech synthesis. In International Conference Machine Learning, pp. 5180-5189. PMLR, 2018.
Yiquan Wu, Siying Zhou, Yifei Liu, Weiming Lu, Xiaozhong Liu, Yating Zhang, Changlong Sun, Fei Wu, and Kun Kuang. Precedent -eeeanen-enhaaeed legal judgment prediction with Ilm and domain-model collaboration, arXiv preprint arXiv:2310 2at::3310.0900 09241, 2023.
Ruiqing Xue, Yanqing Liu, Lei He, Xu Tan, Linquan Liu, Edward Lin, and Sheng Zhao. Foun- dationtts: Text-t--specch for asr customization with generative language model. arXiv preprint arXiv:2303.02939, 2023.
Zhenhui Ye, Rongjie Huang, Yi Ren, Ziyue Jiang, Jinglin Liu, Jinzheng He, Xiang Yin, and Zhou Zhao. Clapspeech: Learning prosody from text context with contrastive language-audio pre- training. arXiv preprini arXiv:2305. 10763, 2023.
Dacheng Yin, Chuanxin Tang, Yanqing Liu, Xiaoqiang Wang, Zhiyuan Zhao, Yucheng Zhao, Zhiwei Xiong, Sheng Zhao, and Chong Luo. Retrievertts: Modeling decomposed factors for text- ext-bbaed speech insertion. arXiv preprint arXiv:2206. 13865, 2022.
Julian Zaïdi, Hugo Seut'e, Benjamin van Nickerk, and Marc-André Carbonneau. Daft-exprt: Cross- speaker prosody transfer any text for expressive speech synthesis. In Interspeech, 2021. URL
https://apl. www.ntisssccii..e.ee.........e.e ar. www..orrr.......aaaoaaa. ID: :::p::corrpuu1:::2999999
Neil Zeghidour, Alejandro Luebs, Ahmed Omran. Jan Skoglund, and Marco Tagliasacchi. Sound- Stream: An end-to-end neural audio codee. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2021.
Heiga Zen, Viet Dang, Rob Clark, Yu Zhang, Ron J Weiss, Ye Jia, Zhifeng Chen, and Yonghui Wu. LibriTTS: A corpus derived from librispeech for text-to-speech. Proc. Interspeech 2019, pp. 1526-1530, 2019.
Ziqiang Zhang, Long Zhou, Chengyi Wang. Sanyuan Chen, Yu Wu, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, et al. Speak foreign languages with your own voice: Cross-lingual neural codec language modeling. arXiv preprint arXiv:2303.03922 2023.
14

---

Published as a conference paper at ICLR 2024
A NEURAL AUDIO CODEC
our neural audio codec consists of an audio encoder, a residual vector-quantizer (RVQ), and an audio decoder: 1) The audio encoder consists of several convolutional blocks with total downsampling rate of 200 for 16KHz audio, i.e., each frame corresponds to 12.5ms speech segment. 2) The residual vector-quantizer converts the output the audio encoder into multiple residual vectors following (Zeghidour et al. 202TT. The sum of these residual vectors is taken as the quantized vectors, which are used as the training target the diffusion model. 3) The audio decoder mirrors the structure of the audio encoder, which generates the audio waveform from the quantized vectors. The working flow the neural audio codec as follows.
As shown in Figure B
Audio Encoder : h (ene
n
Residual Vector Quantizer: {e}} Srva(h'), z'= Σ
ej 2 =
j=1
Audio Decoder I Sdec(z);
where fenc. fevq. and faec denote the audio encoder, residual vector quantizer, and audio decoder. x is the speech waveform, h is the hidden sequence obtained by the audio encoder with frame length of n, and 2 the quantized vector sequence with the same length as h. i the index of the speech frame, j is the index of the residual quantizer and R is the total number of residual quantizers, and ej is the embedding vector of the codebook ID obtained by the j-th residual quantizer on the i-th hidden frame (i.e., h²), The training of the neural codec follows the loss function in ZZghidour et al. 2021
Encoder
Residual Vector Quantizer
Decoder
Audio Codec
Quantiee
11001000
Latent Vector
To/From Diffusion Model
Only Training
Training & Inference
Residual Quantizers V:# Codebook Tokens
The neural audio codec consists of an encoder, a residual vector-quantizer (RVQ), and a decoder. The encoder extracts the frame-level speech representations from the audio waveform, the RVQ leverages multiple codebooks to quantize the frame-level representations, and the decoder takes the quantized vectors input and reconstructs the audio waveform. The quantized vectors also serve as the training target of the latent diffusion model.
Figure 3:
B THE DETAILS OF WAVENET ARCHITECTURE IN THE DIFFUSION MODEL
As shown in Figure the WaveNet consists of blocks. Each block consists of 1) dilated CNN with kernel size 3 and dilation 2, 2) Q-K-V attention, and 3) FiLM layer. In detail, we use Q-K-V attention to attend to the key/value obtained from the first Q-K-V attention module (from the speech prompt encoder) shown in Figure 2 Then, we use the attention results generate the scale and bias terms, which are used as the conditional information of the FiLM layer. Finally, we average the skip output results of each layer and calculate the final WaveNet output.
C THE IMPLEMENTATION DETAILS
C.1 MODEL CONHIGURATION DETAILS
The phoneme encoder is 6-layer Transformer (Vaswani al.) (017)) with 8 attention heads, 512 embedding dimensions, 2048 ID convolution filter size, 9 convolution ID kernel size, and 0.1 dropout rate. The pitch and duration predictor share the same architecture of 30-layer 1D convolution with
15

---

Published as a conference paper at ICLR 2024
Layer k's output
Layers' outputs
Layer Layer
Gating
Fanh
scale
FiLM
scale
bias
Linear
bias
X
Convid
Condition
WaveNet output
Dilated Conv
Q-K-V Attention
a
K/V from Prompt
Timestep
Layer k's input
Figure 4: Overview of the WaveNet architecture in the diffusion model.
ReLU activation and layer normalization, 10 Q-K-V attention layers for in-context learning, which have 512 hidden dimensions and attention heads and placed every 3 1D convolution layers. We set the dropout to 0.5 in both duration and pitch predictors. The ground-truth pitch is quantized in the log scale and converted into pitch embedding, which is added to the expanded hidden sequence. The pitch is standardized for the learning target. For the speech prompt encoder, we use 6-layer Transformer with 512 hidden size. which has the same architecture as the phoneme encoder. As for the m query tokens in the first Q-K-V attention in the prompting mechanism in the diffusion model (as shown in Figure 2). we set the token number m to 32 and the hidden dimension to 512.
The diffusion model contains 40 WaveNet layers (Oord et al. 2016). which consist of 1D dilated convolution layers with kernel size, 1024 filter size, and dilation size. Specifically, we use FiLM layer (Perez et al. 2018) at every 3 WaveNet layers to fuse the condition information processed by the second Q-K-V attention in the prompting mechanism in the diffusion model. The hidden size in WaveNet is 512, and the dropout rate is 0.2.
We show the more detailed configuration in Table 5
C.2 MODEL TRAINING AND INFERENCE
We first train the audio codec using 8 NVIDIA TESLA V100 16GB GPUs with batch size of 200 audios per GPU for 440K steps. We follow the implementation and experimental setting of SoundStream (Zeghidour et al. 2021) and adopt Adam optimizer with 2e learning rate. Then use the trained codec extract the quantized latent vectors for each audio to train the diffusion model in NaturalSpeech
The diffusion model in NaturalSpeech 2 is trained using 16 NVIDIA TESLA V100 32GB GPUs with a batch size of 6K frames of latent vectors per GPU for 300K steps. We optimize the models with the AdamW W optimizer with 5e learning rate, 32K warmup steps following the inverse square
16

---

Published as a conference paper at ICLR 2024
Table The detailed model configurations of NaturalSpeech 2.
Module
Configuration
Value
#Parameters
Number of Residual VQ Blocks
16
Codebook size
1024
Audio Codec
Codebook Dimension
256
27M
Hop Size
200
Similarity Metric
L22
Transformer Layer
6
Attention Heads
8
Hidden Size
512
Phoneme Encoder
72M
Conv ID Filter Size
2048
Conv ID Kernel Size
9
Dropout
0.2
Conv ID Layers
30
ConviD Kernel Size
3
Attention Layers
10
Duration Predictor
34M
Attention Heads
8
Hidden Size
512
Dropout
0.5
CD Layers
30
ConvID Kernel Size
5
Attention Layers
10
Pitch Predictor
50M
Attention Heads
8
Hidden Size
512
Dropout
0.5
Transformer Layer
6
Attention Heads
8
Hidden Size
512
Speech Prompt Encoder
69M
Conv ID Filter Size
2048
Conv ID Kernel Size
9
Dropout
0.2
WaveNet Layer
40
Attention Layers
13
Attention Heads
8
Diffusion Model
Hidden Size
512
183M
Query Tokens
32
Query Token Dimension
512
Dropout
0.2
Total
435M
root learning schedule. We a linear noise schedule function, i.c., 3r Bo + (81 -B) t, where 30 = 0.05 and 3, 20.
During inference, for the diffusion model, find it beneficial to use temperature and sample the terminal condition 21 from (0, - (Popov et al. 2021). We set to 1.22. To balance the generation quality and latency, adopt the Euler ODE solver and set the diffusion steps 150. We quantize the predicted latent vectors and feed them into the audio decoder of the codec to obtain the waveform.
17

---

Published as a conference paper at ICLR 2024
C.3
THE DETAILS OF Lce-rvg
For each residual quantizer j E [1, Rt,, we first get the residual vector:
2j = 20
88)
where eThh is the ground-truth quantized embedding in the m-th residual quantizer. Then calculate the L2 distance between the residual vector with each codebook embedding in quantizer j and get probability distribution as follows:
4 = ||ا 12, 12,,8;
DE e-h
where Nj is the code number of residual quantizer j. and 5; is the probability of code in codebook Finally, we calculate the cross-entropy loss of residual quantizer j given the ground-truth code index which is denoted as Lee.j. The final CE-RVQ loss is shown as follows:
Lee- rug
Loe
(10)
D THE DETAILS OF BASELINE METHODS
We compare NaturalSpeech 2 with the following baselines:
2022b). A powerful zero-shot TTS baseline. We use the official code YourTTS (Casanova al.
and pre-trained checkpoin
which is trained on VCTK Veaux et al. (2016), LibriTTS Zen et al.
(2019) and TTS- PTrtuuuteee Casanova et al. (2022a).
FastSpeech 2 (Ren et al., 2021a). which is classic high-quality TTS system. We adapt it by adding cross-attention on speech prompts for zero-shot synthesis. Furthermore, we also change the prediction target from the mel-spectrogram to the latent representation.
Foundation (Xue et al. (023), which is another strong baseline with neural audio codec for discrete speech token extraction and waveform reconstruction and a L.M for discrete token generation from linguistic (phoneme) tokens. To extend it to the zero-shot TTS scenario, we use an additional Transformer to encode the prompt speech features and temporally average the output to obtain one-dimensional speaker embedding. We scale it to 400M parameters and train it on the same MLS dataset for comparison.
VALL-E (Wang al., 2023), which strong large-scale zero-shot TTS system. uses the audio codec to discretize speech waveforms into tokens and language models to generate them. In our experiment, we implement it with reference to a third part-patyy implementatior Specifically, we use the same neural codec, dataset as used in NaturalSpeech 2 and scale it to 400 parameters for fair comparison.
Voicebox (Le et al. 2023). which is large-scale zero-shot TTS baseline. It uses a flow-
matching (Lipman et al. 2022) model to infill speech mel-spectrogram.
MegaTTS (Tiang al. 2023b), which is GAN-based large-scale zero-shot TTS system. They decompose the mel-spectrogram into different speech attributes such as timbre, and prosody, and model them according to their intrinsic properties.
Since the VALL-E, Voicebox, and Mega are not open source, we download the samples from their demo pages and compare them with NaturalSpeech individually. For VALL-E, we collect samples in LibriSpeech and 8 samples in VCTK For Voicebox. we collect 8 samples in LibriSpeect For MegaTTIS, we collect 4 samples in LibriSpeech and samples in VCTK3
httea///oolcbbxx..ettteee....... htm
18

---

Published as a conference paper at ICLR 2024
E
THE DETAILS OF DATASET
Speech Preprocessing:
The speech data is resampled to 16KHz. The input text sequence is first converted into phoneme sequence using trapheme-to-phoree then aligned with speech using our internal alignment tool to obtain the phoneme-level duration. The frame-level pitch sequence is extracted from the speech using Py World
conversion (Sun al. 2019) and
Singing Preprocessing: We collect approximately 30 hours of songs in waveform format and their corresponding lyrics from the Web, with each songs containing singing, accompaniment, backing vocals, et al. To remove them, we employ a demucs (Défossez) [021) model for music source separation. We apply the same duration and pitch extraction method as we apply for speech data. During the training process, we mix the singing and speech data samples. Our experimental results indicate that training the model with mix of large-scale speech data proves to be more advantageous for enhancing performance, as opposed to fine-tuning it on small-scale, singing-only dataset.
F EVALUATION METRICS
We use both objective and subjective metries to evaluate the zero-shot synthesis ability of Natural- Speech 2 and compare it with baselines.
Objective Metries We evaluate the TTS systems with the following objective metrics:
Prosody Similarity with Prompt. Following the practice (Zidi et al. 2021), , we evaluate the prosody similarity (in terms of pitch and duration) between the generated speech and the prompt speech, which measures how well the TTS model follows the prosody in speech prompt in zero-shot synthesis. We calculate the prosody similarity with the following steps: 1) we extract phoneme- level duration and pitch from the prompt and the synthesized speech; 2) we calculate the mean, standard deviation, skewness, and kurtosis (Ren al.) 202 of the pitch and duration in each speech sequence; 3) we calculate the difference of the mean, standard deviation, skewness, and kurtosis between each paired prompt and synthesized speech and average the differences among the whole test set.
Prosody Similarity with Ground Truth. We evaluate the prosody similarity (in terms of pitch and duration) between the generated speech and the ground-truth speech, which measures how well the TTS model matches the prosody in the ground truth. Since there is correspondence between two speech sequences, we calculate the Pearson correlation and RMSE of the pitch/duration between the generated and ground -trtt speech, and average them on the whole test set.
Word Error Rate. We employ ASR model to transcribe the generated speech and calculate the word error rate (WER). The ASR model is CTC-based HuBERT (Hsu et al. 2021) pre-trained on Librilight (Kahn al. 2020) and fine-tuned on the 960 hours training set of LibriSpeech. We use the official code and checkpoin
Subjective Metrics We conduct human evaluation and use the intelligibility score and mean opinion score the subjective metrics:
Intelligibility Score. Neural TTS models often suffer from the robustness issues such as word
skipping, repeating, and collapse issues, especially for autoregressive models. To demonstrate the robustness of NaturalSpeech 2, following the practice in (Ren et al., 2019). we use the 50 particularly hard sentences (see Appendix (..2) and conduct an intelligibility test. We measure the number of repeating words, skipping words, and error sentences as the intelligibility score.
CMOS and SMOS. Since synthesizing natural voices is one of the main goals of NaturalSpeech 2, we measure naturalness using comparative mean option score (CMOS) with 12 native speakers as the judges. We also use the similarity mean option score (SMOS) between the synthesized and prompt speech to measure the speaker similarity, with native speakers as the judges. We calculate the CMOS and SMOS by third-party commercial evaluation platform.
-ovoo http://hggggggaeoooo/////o////// ft
19

---

Published as a conference paper at ICLR 2024
G INTELLIGIBILTY TEST
Table 6: The robustness of NaturalSpeech 2 and other autoregressivvnoon-atttrrrrrggee 50 particularly hard sentences. We conduct an intelligibility test on these sentences and measure the number of word repeating, word skipping, and error sentences. Each kind of word error is counted at once per sentence.
models on
AR/NAR
Model
Repeats
Skips
Error Sentences
Error Rate
Tacotron Wang et al. 2017)
4
12
24%
Transformer I TTS L1 al.
2019
7
15
17
34%
AR
VALL-E (Wang et al. 2023
8
22
25
50%
Foundation Xue at.
2023
2
16
16
32%
FastSpeech Ren et al. 2019)
0
0
0%
NAR
NaturalSpeech TTn et al.
2022)
0
0
0%
NAR
NaturalSpeech 2
0
0
0%%
G.1 INTELLIGIBILTY TEST/ROBUSTNESS TEST
Autoregressive TTS models often suffer from alignment mismatch between phoneme and speech, resulting in severe word repeating and skipping. To further evaluate the robustness of the diffusion- based TTS model, we adopt the 50 particularly hard sentences in FastSpeech (Ren et al. 2019) to evaluate the robustness of the TTS systems. We can find that the non-autoregressive models such as FastSpeech (Ren et al. (211)), NaturalSpeech (Tan et al. 2022), and also NaturalSpeech 2 are robust for the 50 hard cases, without any intelligibility issues.
a comparison, the autoregressive models such as Tacotron (Wang et al., 2017). Transformer TTS (Li et al. 2019). Foundation TTS (Xue et al. 2023), and VAI.L-E (Wang et al., 2023) will have a high error rate on these hard sentences. The comparison results are provided in Table
G.2 THE 50 PARTICULARLY HARD SENTENCES
The 50 particularly hard sentences used in Section G. are listed below:
01. a
02. b
03. c
04. H
05.1 1
06. J
07.
08.
09. 22222222 hello 22222222
zero Cancelled fifty nine three two 10. S D S D Pass zero Fail zero to
sixty four Total fifty nine to three two
11. S D Pass zero zero zero zero Fail zero zero zero zero Cancelled four hundred and sixteen seventy six
12. zero one one two Cancelled zero zero zero zero Total two hundred and eighty six nineteen seven
13. forty one to five three hundred and eleven Fail one one zero two Cancelled zero zero to zero zero Total -
14. zero zero MS03 zero twenty five. MS03 zero thirty two MS03 zero thirty nine seven ole32
15. 1b204928
20

---

Published as a conference paper at ICLR 2024
16. zero zero
zero zero two seven nine eight three forty zero six four two eight zero one eight
17. с five eight zero three three nine a zero bf eight FALSE zero zero zero bba3add2 - c229 4cdb-
18. Calendaring agent failed with emoor code 0x80070005 while saving appointment.
19. Exit process break Id Load module output ud Unload module ignore ser System enror ignore ibp- Initial breakpoint-
20. Common DB connectors include the DB nine, DB fifteen, DB nineteen, DB twenty five,
DB thirty seven, and DB fifty connectors.
21. To deliver interfaces that are significantly better suited to create and process RFC eight twenty one RFC eight twenty two RFC nine seventy seven, and MIME content.
22. intl int2, int3, int4, int5 int6, int7 int9,
23, seven _even_ctl00 ct00 ct104 ct101 ct100 et100
24. Hup0XX, XX Hup2XX,, Hμр3xX,
25. config file must contain A B B, Fnd G.
26. mondo debug mondo ship motif debug motif ship sts debug sts ship Comparing local files to checkpoint files
27. Rusbvts. dll Dsaccessbvts.. dll Exchmembvt. dll Draino. dll Im trying to deploy a new topology
and keep getting this error
28. You can call me directly at four two five seven zero three seven three four four or my cell four two five four four four seven four seven four send me meeting request with all the appropriate information..
29. Failed zero point zero zero percent < one zero zero one zero zero zero zero Internal Exchange. ContentFilter BVT ContentFilter BVT_log xml Error Filename specified
30. C colon backslash o one two f p a y backslash e one two backslash oasys backslash legacy backslash web backslash HELP
31. src backslash mapi backslash backslash x five
I'de dot dot backslash backslash z aaar one
32. copy backslash backslash j o h f a n four backslash scratch backslash M Microsoft dot S a o dot
33. Take a look at h tp colon slash slash w w w dot granite dot b dot c a slash access slash email dot
34. backslash bin backslash premium backslash forms backslash r e g o poiioonnn dot a dot c Raj. DJ
35. Anuraag backslash backslash a five backslash e b g dot one eight zero nine underscore R two dot s s contains
36. p a right bracket backslash left bracket f v right bracket backslash s t u p dot
37. backslash x eight six backslash Ship backslash zero backslash A d o k dot Con a
A d dres
38. Mine is here backslash backslash g b h hyphen m h backslash v underscore
fi c eevrr
39. h t t p colon slash slash teams slash sites slash T A G slash default dot aspx As always any feedback, comments.
40. two thousand and five t colon slash slash news dot com dot com slash slash n e slash f slash two zero zero three slash f'd
41. backslash n
dot x с a n e dot m n a g e m e dot y S e m m anage 42. think Rich's post highlights that we could have been more strategic about how the sum total of XBOX three hundred and sixtys distributed.
43. 64X64 8K one hundred and eighty four ASSEMBLY, DIGITAL VIDEO DISK DRIVE. INTERNAL. 8X
44. So are back to Extended MAPI and C++ because. Extended MAPI does not have dual interface VB or VB .Net can read.
45. Thanks Borge Trongmo Hi gurus Could you help us E2K ASP guys with the following issue ?
21

---

Published as a conference paper at ICLR 2024
Table 7: The prosody similarity between synthesized and prompt speech in terms of the difference in mean (Mean), standard deviation (Std), skewness (Skew), and kurtosis (Kurt) of pitch and duration on VCTK.
Pitch
Duration
VCTK
Mean
Std.j
Skew
Kurt
Mean
Std
Skew
Kurt
YourTTS
13.67
6.63
0.72
1.54
0.72
0.85
0.84
3.31
FastSpeech 2
18.17
9.87
2.04
3.67
0.81
0.79
0.86
3.12
Foundation
13.41
6,59
0.76
1.68
0.80
0.82
0.80
3.38
VALL-E 13.33
6,44
0.73
1.36
0.74
0.79
0.82
2.91
NaturalSpeech 13.29
6.41
0.68
1.27
0.79
0.76
0.76
2.65
46. Thanks J RGR Are you using the LDDM driver for this system the in the build XDDM driver ?
47. Btw, you might remember me from our discussion about OWA automation and OWA readiness day year ago
48. empidtool.
creates HKEY CURRENT USER Software Microsoft Office Common QMPersNum the registry, queries AD, and the populate the registry with MS employment ID) if available else
code is logged.
49. Thursday, via a joint press release and Microsoft Al Blog, we will announce Microsoft's continued partnership with Shell leveraging cloud, AI, and collaboration technology to drive industry innovation and transformation.
50. Actress Fan Bingbing attends the screening Ash Purest White (Jiang Hu Er Nv)' during the 71 annual Cannes Film Festival
H PROSODY SIMILARITY RESULTS
H.1 PROSODY SIMILARITY WITH PROMPT SPEECH ON VCTK
In this section, we report the prosody similarity evaluation results on VCTK in Table 7
H.2 PROSODY SIMILARITY WITH GROUND TRUTH
To further investigate the quality of prosody, we follow the generation quality evaluation of prosody similarity between synthesized and prompt speech in Section 4.2) and compare the generated speech with the ground-truth speech. We use Pearson correlation and RMSE to measure the prosody matching between generated and ground-truth speech. The results are shown in Table g We observe that NaturalSpeech 2 outperforms all baselines by a large margin, which shows that our NaturalSpeech much better in prosody similarity.
I EXPERIMENTS ON PROMPT LENGTH
Since the prompt length is an important hyper-parameter for zero-shot TTS, we would like to investigate the effect of the prompt length. We follow the setting of prosody similarity benween vary the prompt length by
synthesized and prompt speech in Section 4.2
Specifically,
53.5,10 10} seconds and report the prosody similarity metrics of NaturalSpeech 2. The results are shown in Table 9 We observe that when the prompt is longer, the similarity between the generated speech and the prompt higher for NaturalSpeech 2. shows that the longer prompt reveals more details of the prosody, which help the TTS model to generate more similar speech.
J ABLATION STUDY
In this section, we present the detailed ablation results by evaluating the prosody similarity between synthesized audio generated by the ablation model and the prompt speech, which are conducted in Section 4.3 The results shown in Table 10
22

---

Published as a conference paper at ICLR 2024
Table 8: The prosody similarity between the synthesized and ground-truth speech in terms of the correlation and RMSE on pitch and duration.
Pitch
Duration
LibriSpeech
Correlation t
RMSE +
Correlation t
RMSE +
YourTTS
0.77
51.78
0.52
3.24
FastSpeech
0.64
60,39
0.63
2.92
Foundation
0.73
52.18
0.61
3.16
VALL-E
0.73
50.80
0.62
2.88
NaturalSpeech 2
0.81
47.72
0.65
2.72
Pitch
Duration
VCTK
Correlation t
RMSE +
Correlation t
RMSE |
YourTTS
0.82
42.63
0.55
2.55
FastSpeech
0.77
47.40
0.60
2.63
FoundationITS TTS
0.81
46.00
0.53
2.64
VALL-E NaturalSpeech 2
0.83
43.27
0.61
2.52
0.87
39.83
0.64
2.50
Table 9: The NaturalSpeech prosody similarity between the synthesized and prompt speech with different lengths in terms of the difference the mean (Mean), standard deviation (Std), skewness (Skew), and kurtosis (Kurt) of pitch and duration.
Pitch
Duration
LibriSpeech
Mean
Std.
Skew
Kurt.
Mean
Std|
Skew
Kurt.
3s
10.11
6.18
0.50
1.01
0.65
0.70
0.60
2.99
5s
6.96
4.29
0.42
0.77
0.69
0.60
0.53
2.52
10s
6.90
4.03
0.48
1.36
0.62
0.45
0.56
2.48
Pitch
Duration
VCTK
Mean.
Std.
Skew..
Kurt,
Mean
Stdl.
Skew,.
Kurt.
13.29
6.41
0.68
1.27
0.79
0.76
0.76
2.65
5s
14.46
5.47
0.63
1.23
0.62
0.67
0.74
3.40
10s
10.28
4.31
0.41
0.87
0.71
0.62
0.76
3.48
Furthermore, we also compare the prosody similarity between audio generated by the ablation model and the ground-trui truth speech in Table II] Similar to the results of comparing the audio generated by the ablation model and prompt speech, we also have the following observations. 1) The speech prompt is most important to the generation quality. 2) The cross-entropy and the query attention strategy are also helpful in high-quality speech synthesis.
K LATENCY STUDY OF NATURALSPEECH 2
In this section, we report the inference latency of NaturalSpeech We vary the diffusion step in {20,50, 50, 100, 150). and report both the latency (RTF) and generation quality (CMOS). We also compare NaturalSpecch with a NAR baseline (FastSpeech 2) and an AR model (VALL-E). The latency tests conducted NVIDIA V100 GPU. The results are shown in Table [2
server with E5-2690 Intel Xeon CPU, 512GB memory, and one
From the results, have several observations. 1) When the diffusion step is 150 (used in paper), NaturalSpeech 2 is 33.3 times slower than the NAR model FastSpeech 2, but achieves 0.53 CMOS gain. Still, it is 12.35 times faster than VALL--E NaturalSpeech 2 has 0.08 CMOS drop and
23

---

Published as a conference paper at ICLR 2024
Table 10: The ablation study of NaturalSpeech The prosody similarity between the synthesized and prompt speech in terms of the difference in the mean (Mean), standard variation (Std), skewness (Skew), and kurtosis (Kurt) of pitch and duration. x_" denotes the model can not converge.
Pitch
Duration
Mean
Std.j
Skew
Kurt|
Mean
Stdf
Skew
Kurth
NaturalSpeech 2
10.11
6.18
0.50
1.01
0.65
0.70
0.60
2.99
w/o. diff prompt
w/o. dur/pitch prompt
21.69
19.38
0.63
1.29
0.77
0.72
0.70
3.70
w/o. CE loss
10.69
6.24
0.55
1.06
0.71
0.72
0.74
3.85
w/o. query atmn
10.78
6.29
0.62
1.37
0.67
0.71
0.69
3.59
Table 11:
The ablation study of NaturalSpeech The prosody similarity between the synthesized and ground-trute truth speech in terms of the correlation and RMSE on pitch and duration. "_" denotes that the model can not converge.
Pitch
Duration
Correlation f
RMSE
Correlation t
RMSE +
NaturalSpeech 2
0.81
47.72
0.65
2.72
wlo. diff prompt
w/o. dur/pitch prompt
0.80
55.00
0.59
2.76
w/o. CE loss
0.79
50.69
0.63
2.73
w/o. query attn
0.79
50.65
0.63
2.73
2.95 times speedup (compared with 150 steps) when the diffusion step is 50. The CMOS drops 0.21 while it can achieve 7.31 times speedup (compared with 150 steps) when the diffusion step is 20. Furthermore, since NaturalSpeech 2 is parallel to many diffusion speedup works such as the consistency model [Song et al. (2023), we will explore speeding up the diffusion model while retaining the generation quality in the future.
L VOICE CONVERSION AND SPEECH ENHANCEMENT
L.1 VOICE CONVERSION
Besides zero-shot text-to-speech and singing synthesis, NaturalSpeech 2 also supports zero-shot voice conversion, which aims to convert the source audio Zsource into the target audio Ztarget using the voice of the prompt audio Zprompt- Technically, we first convert the source audio 2souree into an informative Gaussian noise 21 using source-aware diffusion process and generate the target audio Zarget using target-aware denoising process, shown follows.
Source-Aware Diffusion Process In voice conversion, it is helpful to provide some necessary information from source audio for target audio in order to ease the generation process. Thus, instead of directly diffusing the source audio with some Gaussian noise, we diffuse the source audio into a starting point that still maintains some information the source audio. Specifically, inspired by the stochastic encoding process in Diffusion Autoencoder (Preechakul et al. 2022). obtain the starting point 21 from Ssource as follows:
' + (pl 2 dt,
==
(11)
where Σ, c),t) 2,) is the predicted score at t. We can think of this process as the reverse of ODE (Equation in the denoising process.
24

---

Published as a conference paper at ICLR 2024
Table 12: The latency study of NaturalSpeech 2. We report the RTF and CMOS results for different diffusion steps.
Model
Diffusion Step
RTF
CMOS
NaturalSpeech
150
0.366
0.00
NaturalSpeech
100
0.244
-0.02
NaturalSpeech
50
0.124
-0.08
NaturalSpeech
20
0.050
-0.21
FastSpeech 2
0.011
-0.53
VALL-E
4.52
-0.29
Target-Aware Denoising Process Different from the TTS which starts from the random Gaussian noise, the denoising process of voice conversion starts from the 21 obtained from the source-aware diffusion process. We run the standard denoising process as in the TTS setting obtain the final target audio Storget, conditioned on and the prompt audio 2prompt. where is obtained from the phoneme and the duration sequence of the source audio and the predicted pitch sequence.
L.2 SPEECH ENHANCEMENT
NaturalSpeech 2 can be extended to speech enhancement, which is similar the extension of voice conversion. In this setting, we assume that we have the source audio *source which contains background noise (z' denotes the audio with background noise), the prompt with background noise Aprompt for the source-aware diffusion process, and the prompt without background noise 2prompt for target-aware denoising process. Note that ='oorrce and *prompt have the same background noise.
To remove the background noise, firstly, we apply the source-aware diffusion process by 2' z'source and Sprompt and obtain the 21 as in Equation 11 The source audio's 's duration and pitch utilized in this procedure. Secondly, we run the targel-aware denoising process to obtain the clean audio by and the clean prompt Eprompt- Specifically, use the phoneme sequence, duration sequence, and pitch sequence of the source audio in this procedure.
M LIMITATION AND FUTURE WORKS
Despite NaturalSpeech 2 has made great progress, it still suffers from the following issues.
Data coverage. Although the 44K speech data from MLS dataset is large compared to previous works, they can not cover everyone's voice. In audiobooks, most speakers will read the books clearly and fluently, while real world people will speak causally, thus leading degradation when generalizing to real- world scenarios. In the future, we will scale NaturalSpeect 2 to more generalized and larger-scale benchmarks to enhance the zero-shot generation ability.
Inference efficiency. Although NaturalSpeech 2 is a non-autoregressive generation model, it still needs multiple iterations during inference. the future, we will explore efficient strategies such as consistency models speed up the diffusion model.
Singing voice quality. Although NaturalSpeech can synthesize singing voices in a zero-shot manner, the quality is not good the TTS synthesis's ss quality. We think we are limited in two aspects: 1) the scale of singing data and 2) the quality of singing data. For data scale, we only collect 30 hours, which is small compared with the speech data scale. For data quality, it difficult obtain clean human voices from commercial songs, which are a combination of vocals, backing vocals, accompaniment, and other background noises. In the future, will first explore methods to collect more singing data and obtain higher-quality singing voices to enhance the singing voice quality.
efficient
25


Reviews:
[{"review_id": "5TQYisWqJ0", "reviewer": "Reviewer_c9Qd", "summary": "This paper presents NaturalSpeech 2, a non-autoregressive TTS model that employs a diffusion mechanism to generate quantized latent vectors from neural audio codecs. It shows enhanced zero-shot TTS performance relative to the state-of-the-art large-scale neural codec language model. The proposed approach exhibits advancements in sample quality, intelligibility, robustness, speaker similarity, and generation speed when benchmarked against the baseline method. The authors further validate the superiority of their method over other alternatives via comprehensive qualitative and quantitative evaluations.", "questions": "Given the concerns mentioned in the above weaknesses, it would be interesting to see if the proposed method could be adapted or refined to reduce its dependency on additional modules without increasing complexity or compromising sample quality.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* The paper effectively tackles several major challenges inherent to non-autoregressive TTS modeling at scale. \n* The authors have carried out robust and wide-ranging experiments, yielding detailed results.\n* The reference list is both extensive and comprehensive.", "weaknesses": "The proposed method's intricate modeling could hinder its extension to other applications. While the introduced model applies diffusion, it necessitates two additional losses and requires supplementary modules like a pitch predictor, prompt encoder and the second attention block. As an example, the recent state-of-the-art flow-matching based TTS method, VoiceBox [1] consists of rather simple model architecture; the flow-matching based duration predictor and audio model.\n\n[1] Le, Matthew, et al. \"Voicebox: Text-guided multilingual universal speech generation at scale.\" arXiv preprint arXiv:2306.15687 (2023).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RoNRaajMH1", "reviewer": "Reviewer_9oT1", "summary": "This paper describes a TTS model combining a number of modern components these include in-context learning (prompting) a diffusion model to connect conditioning information to latents, and latents defined by an autoencoder for waveform reconstruction.\n\nThe resulting model has many of the zero-shot capabilities of LM based TTS that have been presented in recent years, but by maintaining duration prediction for alignment, the model stays robust to a hallucination and dropping errors that impact other generative models.", "questions": "Introduction “the zero-shot capability that is important to achieve diverse speech synthesis” – why is zero-shot necessary for diverse speech synthesis?  Also, for what contexts, and use-cases is diverse speech synthesis necessary?\n\nIn the introduction – the checklist between NaturalSpeech 2 and “previous systems” is somewhat strange.  Certainly there are previous systems that are non-autoregressive, or use discrete tokens.  I understand that this is not “all previous systems” but those listed. But why compare only to those three related systems? The introduction and related work draw contrast with a variety of alternate TTS models.\n\nWhy use a diffusion model instead of any other NAR model?\n\nWhen presenting the “prior model” in section 3.2 is the phone encoder, duration predictor and pitch predictor pre-trained to some other target? or is there some other notion of a prior model here?\n\nWhat is the units used in the L_pitch loss? Hz? log Hz? something else?\n\nThe variable z is used in a number of different ways, could this be clarified (e.g. in Figure 2 between the prompt, input to diffusion model and output?)\n\nSection 4.1\nPage 6 “Both speakers in the two datasets” are there only 2 speakers in the data sets?\nPage 6 what is value of sigma in the sigma-second audio segment as a prompt?\n\nHow much loss is incurred by filtering the output by a speech scoring model?  E.g.  are 99% of utterances accepted? or 1%?  \n\nNote: VCTK utterances are particularly noisy making is a poor comparison for CMOS, but the comparison to Librispeech is more representative.\n\nSection 4.2 “We apply the alignment tool” – which alignment tool?\n\nWhat is the variance of the prosodic measures – it’s hard to track whether the differences in Table 3 are significant or not.\n\n“When we disable the speech prompt in diffusion, the model cannot converge” – this seems remarkable.  Why does the model require a speech prompt to learn?\n\nBroader Impacts: What would such a protocol to protect users from misuse of this model look like? Presumably this model can generalize to unseen speakers already – so what protections are in place regarding the use of this model as of publication?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The model contains innovative structures in the in context learning for duration and pitch, and in the diffusion model.  Moreover the overall structuring of these components is novel.\n\nThe quality of the model is quite high and provides some important balancing between zero-shot capabilities and robustness compared to alternate models", "weaknesses": "The paper is sometimes unclear with regards to what the model components represent and how the components fit together.  For example, the use of SoundStream and wavenet is not obvious.  These are previously published approaches, that are used in novel ways here.  It took multiple readings to understand how they are being used in this paper, and even still i’m not 100% sure that my understanding is correct.  Broadly, the paper relies too heavily on Figure 1.0 to describe how the model fits together. \n\nThe argumentation around continuous vs discrete tokens is very hard to follow.  It’s not clear why the discrete token sequence must necessarily be longer than a continuous sequence (Introduction).  The first three pages spend a lot of effort describing why a continuous representation is a better fit for this task.  Then in Section 3.1 “However, for regularization and efficiency purposes we use residual vector quantizers with a very large number of quantizers and codebook tokens to approximate the continuous vectors.  This provides two benefits…” This is a particularly surprising turn of the argument to then go on to describe why discrete tokens are useful here.\n\nThe diffusion formulation is too compact to be clearly followed.  Page 5. The following sentence includes a number of ambiguities.  “Then we calculate the L2 distance between the residual vector with each codebook embedding in quantizer j and get a probability distribution with a softmax function, and then calculate the cross-entropy loss between the ID of the ground-truth quantized embedding ej and this probability distribution. Lce−rvq is the mean of the cross-entropy loss in all R residual quantizers, and λce−rvq is set to 0.1”  I’d recommend including an appendix entry or describing each clause separately in place.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kFgLdFRMxI", "reviewer": "Reviewer_69Md", "summary": "The paper proposes a new TTS model that is capable of generating speech with diverse speaker identities, prosody, and styles, in zero-shot scenarios and it can also sing. It outperforms the current SOTA methods in both objective and subjective metrics. The way it works is the following. First the neural audio codec that converts a speech waveform into a sequence of latent vectors with a codec encoder, and reconstructs the speech waveform from these latent vectors with a codec decoder. Then the codec encoder extracts the latent vectors from the speech and uses them as the target of the latent diffusion model which is conditioned on prior vectors. During inference it generates the latent vectors from text using the diffusion model and then generate the speech waveform using the codec decoder.", "questions": "-Why did the authors not include any experiments with single speaker data like LJSpeech.\n-It would be interesting to hear some audio samples with people that have an accent. This has not been explored in the community.\n-As an ablation what would be the shortest prompt in seconds that you can give for zero-speech synthesis?\n-After the phoneme Encoder you have a Duration and Pitch predictor. Why didn't you also include an Energy Predictor like the authors did in FastSpeech2 since the idea seems to be derived form there?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "-Paper is very well written and provides good intuition and justification for all model choices that the authors have made. These choices are intuitive to make generated speech more natural and to overcome past bottlenecks in previous methods.\n-The new TTS algorithm has many capabilities such as generating diverse speech (different speakers, prosody, style) and in zero-shot scenarios. Singing is a bonus in this case.\n-NaturalSpeech2 beats current SOTA methods in both objective and subjective metrics.\n-Related work section is quite extensive.\n-In the end I believe that this work is a good contribution to the community.", "weaknesses": "-One can hear in the more strenuous experiments that the audio samples have some kind of weird pitch or pace of speaking.\n-Paper might not be a very good fit in this venue. Although it has to do with learning representations, NaturalSpeech2 is more fit for a Speech venue such as InterSpeech or ICASSP.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kTpU8lwTfD", "reviewer": "Reviewer_8Wev", "summary": "This work proposes a latent diffusion-based speech synthesis framework for high-quality zero-shot speech synthesis. They utilize an audio codec as a latent representation and a conditional latent diffusion model could generate a latent representation. Then, the codec decoder generates a waveform audio. The zero-shot results show a better performance than the codec-based TTS model and YourTTS. Moreover, the audio quality is good.", "questions": "1. This work utilizes a quantized latent vector for latent representation. In my experience, the quality of the model with the continuous latent representation before quantization showed a better performance in latent diffusion model for singing voice synthesis. Have you tried to train your model with the pre- or post-quantized representation for latent representation?\n\n2. The details of singing voice synthesis are missing. It would be better if you could add the details for pre-processing of musical scores. How do you extract the duration of phonemes in this work?\n\n3. How do you extract the pitch information? This significantly affects the performance so the details should be included. (about F0 min, F0 max, resolution, and pitch extraction algorithm).\n\n4. The authors may train the audio codec with their speech dataset. I think it is important to utilize a high-quality speech codec for high-quality speech synthesis. In this regard, I hope that the authors will mention about this part by comparing your model with the same model utilizing an official Soundstream codec as a latent representation.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "They propose the latent diffusion-based speech synthesis model. This work may be the first successful implementation of a latent diffusion model for speech synthesis. Although recently large-language model (LLM) -based speech synthesis models have been investigated, they have too many problems for speech synthesis resulting from the auto-regressive generative manner. However, this work adopts a parallel synthesis framework with latent diffusion, and successfully shows their generative performance by several speech tasks.\n\nRecent papers only compare their work with YourTTS but I do not think YourTTS is a good zero-shot TTS model. The audio quality of YourTTS is too bad. However, although recent models do not provide an official implementation, the authors tried to compare their model with many other works.", "weaknesses": "1. They also conducted an ablation study well. However, it would be better if the authors could add the results according to the dataset and model size. The model size of NaturalSpeech 2 is too complex compared to VITS. In my personal experience, VITS with speaker prompt could achieve significantly better performance than YourTTS. \n \n2. For inference speed, NaturalSpeech 2 still has a low latency for its iterative generation. Although this discussion is included in the Appendix, it would be better if the authors could add the discussion of inference speed in the main text. This is just a limitation of diffusion models so I acknowledge the trade-off between quality and inference speed. Furthermore, I hope to know other metrics of NaturaSpeech 2 according to Diffusion Steps (WER or Similarity metric). Recently, Flow matching using optimal transport is utilized for fast speech synthesis. This could be adopted to this work. \n\n3. Some details are missing. Please see the questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)