Mega-TTS 2: Boosting Prompting Mechanisms for Zero-Shot Speech Synthesis
Authors: Ziyue Jiang
Jinglin Liu
Yi Ren
Jinzheng He
Zhenhui Ye
Shengpeng Ji
Qian Yang
Chen Zhang
Pengfei Wei
Chunfeng Wang
Xiang Yin
Zejun MA
Zhou Zhao
Venue: ICLR 2024 poster (2024)
Keywords: Prompting Mechanisms
Zero-Shot Text-to-Speech
Multi-Sentence Prompts
Primary area: generative models
TL;DR: We propose a novel prompting mechanism for zero-shot text-to-speech, which consistently surpasses the fine-tuning method when the volume of data ranges from 3 seconds to 5 minutes.

Abstract: Zero-shot text-to-speech (TTS) aims to synthesize voices with unseen speech prompts, which significantly reduces the data and computation requirements for voice cloning by skipping the fine-tuning process. However, the prompting mechanisms of zero-shot TTS still face challenges in the following aspects: 1) previous works of zero-shot TTS are typically trained with single-sentence prompts, which significantly restricts their performance when the data is relatively sufficient during the inference stage. 2) The prosodic information in prompts is highly coupled with timbre, making it untransferable to each other.
This paper introduces Mega-TTS 2, a generic prompting mechanism for zero-shot TTS, to tackle the aforementioned challenges. Specifically, we design a powerful acoustic autoencoder that separately encodes the prosody and timbre information into the compressed latent space while providing high-quality reconstructions. Then, we propose a multi-reference timbre encoder and a prosody latent language model (P-LLM) to extract useful information from multi-sentence prompts. We further leverage the probabilities derived from multiple P-LLM outputs to produce transferable and controllable prosody. 
Experimental results demonstrate that Mega-TTS 2 could not only synthesize identity-preserving speech with a short prompt of an unseen speaker from arbitrary sources but consistently outperform the fine-tuning method when the volume of data ranges from 10 seconds to 5 minutes. Furthermore, our method enables to transfer various speaking styles to the target timbre in a fine-grained and controlled manner. Audio samples can be found in https://boostprompt.github.io/boostprompt/.

Published as a conference paper at ICLR 2024
MEGA- 2: BOOSTING PROMPTING MECHANISMS FOR ZERO-SHOT SPEECH SYNTHESIS
Ziyue Jiang
Jinglin Liu + Yi Ren +
Jinzheng He +f
Zhenhui Ye +
Shengpeng Ji Qian Yang Chen Zhang &
Pengfei Wei 9
Chunfeng Wang' 9
Xiang Yin V Zejun Ma v Zhou Zhao a
Zhejiang University & ByteDance
z zhaozhou] @ @zju. edu. cn,
liu. jinglin, ren. yin.yi, yinxiang. stephen]
ABSTRACT
Zero-shot text-to-speech (TTS) aims to synthesize voices with unseen speech prompts, which significantly reduces the data and computation requirements for voice cloning by skipping the fine-tuning process. However, the prompting mecha- nisms of zero-shot TTS still face challenges in the following aspects: 1) previous works of zero-shot TTS are typically trained with sentlessentence prompts, which significantly restricts their performance when the data is relatively sufficient during the inference stage. 2) The prosodic information in prompts highly coupled with timbre, making it untransferable to each other. This paper introduces Mega-TTh TTS 2, generic prompting mechanism for zero-shot TTS, to tackle the aforemen- tioned challenges. Specifically, we design a powerful acoustic autoencoder that separately encodes the prosody and timbre information into the compressed la- tent space while providing high-quality reconstructions. Then, propose multi-reference timbre encoder and a prosody latent language model (P-LLM) extract useful information from multi-sentence prompts. We further leverage the probabilities derived from multiple P-LLM outputs to produce transferable and controllable prosody. Experimental results demonstrate that Mega-TTS 2 could not only synthesize adentity--reserring speech with a short prompt of an unseen speaker from arbitrary sources but consistently outperform the fine-tuning method when the volume of data ranges from 10 seconds to 5 minutes. Fur- thermore, our method enables to transfer various speaking styles to the target timbre in fine-grained and controlled manner. Audio samples can be found in htttts::/bosspproogggg gi wtt.hh.......hss.eeeeeeooe hctboostproooot//
INTRODUCTION
In recent years, there has been remarkable progress in the development of text-to-speech (TTS)
technology (Sen al. 2018 Jia et al. 2018 Li et al. 2019 Kim et al.) 2020 Ren et al. 2019
2020
Kim et al. 2021 2022a). Among them, adaptive TTS systems (Chen et al. 2021: Min et al. 2021
Kim et al. 2022b are capable of cloning personalized voices given few minutes of speech data.
However, the performance of these systems relies heavily on the quality and quantity of the data utilized during the fine-tuning phases (Tan et al. 2021). Insufficient data during the fine-tuning stages can lead to diminished audio naturalness or speech intelligibility (Kang et al. 2023). Moreover, the computational demands also constrain its application for cloning everyone's voice.
To reduce such a reliance, existing works leverage generative models to perform zero-shot TTS (Cooper et al. 2020a Casanova al., 2022 Huang al.] 2022a; Kang al. 2023 Kharitonov et al. 2023 Wang et al. 2023) Shen et al. 2023bb Matthew et al. 2023). These powerful models can effectively synthesize speech given only single speech prompt, eliminating the need for data
'Equal contribution.
1 Interns at ByteDance
1 *oorrsponniig author.

---

Published as a conference paper at ICLR 2024
preparation and the computational requirements for fine-tuning methods. However, the prompting mechanisms of current solutions still face two primary challenges:
Lack of multi-sentence prompting strategies. Previous works of zero-shot TTS typically employ single-sentence speech prompts during training (Wang et al., 2023 Shen al. | Matthew et al.] (023). In inference, the information in the single-sentenes speech prompt is insufficient to guide the zero-shot TTS systems to imitate the voice variability of natural person perfectly From another perspective, the performance of fine-tuning methods can be further improved by increasing the amount of data, while zero-shot TTS systems lack an appropriate strategy to extract useful information from multi-sentence speech prompts.
Lack of specialized prompting mechanism for prosodic information. Current solutions for zero-shot TTS primarily concentrate on improving the similarity of timbre and prosody between the generated speech and the prompts. However, they neglect to express various unseen prosodic styles in controlled manner while also preserving the unique timbre of the given one-sentence prompt. In order control the prosodic styles, it is necessary to disentangle the prosody information from speech prompts.
We address the above challenges by decomposing speech into content, timbre, and prosody, Intuitively, representing speeches for numerous speakers requires substantial number of codebook entries for timbre modeling (Défossez et al.) 2022. Yang et al., 2023). Through the decoupling of prosody information, highly compact codebook for prosody modeling be obtained, which enables our model to effectively handle extremely long prompts and have flexible control over prosodic styles. Therefore, this work proposes Mega-TTS 2, generic framework that boosts the prompting mechanisms for zero-shot TTS systems. Specifically, we begin by designing an acoustic autoencoder that can effectively decompose speech into prosody and timbre representations and represent them in compact latent space. Then, we design multi-reference timbre encoder (MRTE) and a prosody latent language model (P-LLM) to extract useful information from multi-sentence prompts. In addition to the multi-sentence prompting mechanism, we propose prosody interpolation technique to control the generation process of prosody codes by utilizing prosody prompts from multiple speakers while maintaining the target speaker's 's timbre. By utilizing the probabilities derived from both the prosodic prompts of the target speaker and the auxiliary speaker, the prosodic styles of speech can be generated in controlled manner.
Experiments on LibriSpeech test-clean (Panayotov et al. 2015) and ESD (Zhou et al. 2021 datasets show that Mega-TTr TTS 2 outperforms other state-of-the-art fine-tuning and zero-shot TTS models in terms of speaker similarity and speech naturalness. Notably, when the length of the prompt is further extended, our method surpasses the fine-tuning baseline model in the objective and subjective evaluations. The extensive studies on adaptive prosody transfer further highlight the superiority of our proposed prompting mechanisms. The main contributions of this work are summarized follows:
We design an acoustic autoencoder that separately compresses the prosody and timbre information into the latent space, which seconds in length effectively.
allows our model to process prompts of up to 300
We propose multi-reference timbre encoder and an auto-regressive prosody language model to extract fine- tne-grained information from multiple reference speeches, which bridges the speaker similarity gap between zero-shot methods and fine-tuning methods.
Experimental results also reveal that the performance of Mega-TTSs TTS surpasses the powerful fine-tuning baseline when we have 10 seconds to 5 minutes of data for each unseen speaker, indicating the superiority of our proposed prompting mechanisms.
The proposed prosody interpolation technique ensures the controllability of prosody and is capable of transferring various speaking styles to the desired timbre. For instance, we can transform a voice with a sad tone into a happier one with the auxiliary prosody prompt from another speaker.
Althoug the performance of these systems can be further improved by concatenating multiple sentences into long prompt, the gap hetween training and inference still restricts their performance. (See Section 4.2)
2

---

Published as a conference paper at ICLR 2024
2 BACKGROUND
Adaptive TTS Adaptive TTS (Arik et al. 2018 Kons et al. 2019; Moss et al. 2020 Chien et al. 2021] focuses on synthesizing personalized voice for any with few data. During the adaptation process, a TTS model pre-trained on a multi-speaker speech dataset typically fine-tuned with few adaptation data for the target voice (Tan et al. 2021). Chen al. (2018) design independent learned embeddings for each speaker, which requires few data at deployment time to adapt to new speakers rapidly. AdaSpeech (Chen et al. 202T) proposes an acoustic-condition modeling method for high-quality and efficient customization of new voices. There are also some works leveraging the meta-learning approach (Chen et al., 2018 Min et al.] 202T Huang et al.) 2022b) and data augmentation (Cooper et al., [020b] Yang & He 2020) for speaker adaptation. However, although
some works are data-efficient (Min et al. [021]] Huang et al. 2022b) and parameter-efficien (Chen)
et al. 2021), these systems still suffer from audio quality issues when data size is small, as well as computational cost issues due to hundreds of fine-tuning steps.
Zero-shot TTS Zero-shot adaptation (Jia et al. 2018 Arik al., 2018 Cooper et al., 2020a Casanova et al. | 2021] Wu et al. [2022] Huang et al. 2022b a Casanova et al. 2022) aims to synthesize
unseen voices with speaker encoder that extracts speaker embeddings from the reference audio. This scenario is highly attractive because does not require any adaptation data or parameters (Kang et al.
2022). The attention-based adaptation method (Choi al.) 20200] Zhou al. 2022 Yin al. 2022 Lin et al. 2021) utilizes attention mechanisms to extract fine-grained speech features from reference audios. Among them, Attentron (Choi al. 2020) proposes extracts useful style information from arbitrary number of reference audios. However, they do not separately model the timbre and prosody information, lacking controllability over timbre and prosody. Most recently. some works (Kharitonov) er al. 2023|ZZang et al. (023) are proposed to use in-context learning methods (Dong et al.] [002) to efficiently extract speaker information from acoustic prompts and have achieved remarkable results in zero-shot TTS. VALL-E (Wang et al., 2023) proposes the neural codec language model that exhibits strong in-context learning capability for zero-shot speech generation. NaturalSpeech (Shen et al. 2023b introduces in-context learning latent diffusion model (Rombach et al.) 2022). which is achieved by partitioning a speech clip into the prompt and target regions. Voice (Matthew et al., 2023) solves text-guide's ppeech-infilliig task with sarge-scale data to learn from context information.
However, these methods are trained with single-sentence prompts, lacking an appropriate strategy to extract fineinrained information from multi-sentence speech prompts.
Prosody Transfer for Speech Synthesis Prosody transfer (Lee & Kim 2019 Klimkov et al. 2019
Gururani al., 2019 Pan & He| 202T Karlapati et al. 2022) aims transfer the prosody from reference utterance to the synthesized target speech. which is essential for producing natural and expressive speech in a controlled manner (Wagner & Watson 2010). Skerry-Ryaan et al. (2018) first integrate prosody reference encoder into TTS system based on Tacotron (Wang et al.) [00 7),
which is capable of performing similar-text prosody transfer. Recent works try to transfer prosody in different-text and different-speakee settings (Karlapati al. 20200; Zaıdi et al. (021) with the bottleneck of the prosody encoder. Among them, Daft-Exprt (Zandi et al. 2021 uses a gradient reversal layer to penalize the prosody encoder if its output contains information about the speaker identity from the reference utterance, which enhances the target speaker fidelity for cross-speaker prosody transfer. However, as pointed out by /Sigurgeirsson & King (2023). current solutions do not learn a transferable representation of prosody, but rather an utterance-level representation that is relatively dependent on both the reference speaker and reference text.
3 METHOD
This section introduces Mega-TTS- 2.
To begin with, we provide an intuitive illustration of how Mega-TTS 2 decomposes the timbre and prosody information from speech. Next, we provide detailed explanations of our prompting mechanisms and the two-stage training process of the proposed model.
3.1 DECOMPOSITION FOR PROSODY AND TIMBRE
Problem Formulation Denote H (X) as the Shannon entropy of X and Denote I(Y; X)) as the mutual information. We assume that the mel-speetrogram y can be reconstructed through the
3

---

Published as a conference paper at ICLR 2024
Compressive Acoustic Autoencoder
Multi- Reference
Timbre Encoder
E,
Attend
Reconstruction Loes,
GAN Loss
Decoder
Phooeme
Content Encoder
Eg
Vector Quantised Frecddr
Eg
Ep
Ap1
Sym
Prosody Latent
Language Model
Figure The overall architecture of Mega-TTS O. O. k denotes the concatenation, upsampling. and repeating operations, respectively. is concatenated along the time axis.
"Attend" means the attention operation.
following generative process: y (ze: Fpd, Re, gt where and denote the fine-grained content and timbre hidden states. 9 denotes the global style information that contains timbre and prosody. We assume that 2pd (Zp, za) contains the fine-gre ine-grained prosodic style information of pitch and energy Zp and duration 2d- 2d = Aligner(y) can be obtained by the external alignment tools (McAuliffc et al. 2017) and disentangled from 2pd- Denote D) the mel-spectrogram decoder. Our goal is to construct an autoencoder- based model to disentangle speech components.
Decomposition via Corpus Partition Denote Y (yi.... :yn as the speech corpus for certain speaker S. In training, we partition Y into the target mel-spectrogram Ys and the other mel-spectrograms ÿ. Here, we make an important assumption that the mutual information between y and ÿ only contains timbre information H (2x) and global style information H of yt, i.c.
T Twu: 9) (zx) + H(g)
First, based on the assumption, 2; and can be extracted through Eū(ū), and there is no way for (t) to obtain the 2p and 2.. Second, we only feed phoneme sequence to E,, Eg can only pass all the content information ze. Third, since the information and encoder Ep will prioritize removing the fine- fine-grained content and timbre information it is forced to lose some information by information bottleneck B--) (Qian et al. 2019). The bottleneck forces (y)) ) to pass only the fine-gratd fine-grained prosodic style Ep that other encoders cannot supply. hence achieving the decomposition. We provide a detailed explanation of how ensure the validity of Equation T in Appendix A.8] After the decomposition, we describe the detailed designs of our prompting mechanisms in the following subsections.
available now, the prosody
3.2 COMPRESSIVE ACOUSTIC AUTOENCODER
Note that to store timbre information for thousands of speakers, need large number of codebook entries. However, since the prosody and timbre have been decomposed, the prosodic information 2p be compressed into a highly compact codebook, and the timbre information 2; can be extracted via a powerful speaker encoder. The decomposition strategy not only allows our model to accommodate extremely long prosody prompts but also enables our model control the prosodic styles of generated speeches. As shown in Figure D we design the vector quantised (VQ) encoder as Ep. the multi- reference timbre encoder as E,, and the content encoder Eg, Since Ep mainly captures the prosodic variance information, a GAN- frequency details in spectrograms, which ensures perceptually high-quality reconstructions. Overall, the first-stage training loss can be formulated L - Lree + Lvq + Advv where Lree V| - Dill the reconstruction loss, LvQ is the VQ codebook loss (Van Den Oord et al. (017)) and Laav is the LSGAN-styled adversarial loss (Mao al., 2017) whose objective is to minimize the distribution distance between the predicted mel mel-spectrograms and the ground truth mel seeesssctrogramss Among the proposed three encoders, the content encoder is composed of several feed-forward Transformer layers following common practice in non-autoregreesivve TTS systems (Ren et al.) 2019). In the following paragraphs, we describe the details of the prosody and timbre encoders, respectively.
mel- mel-spectroogam decoder D is adopted to model the high-
4

---

Published as a conference paper at ICLR 2024
Vector Quantised Encoder The vector quantised encoder Ep consists of two convolution stacks and a vector quantization bottleneck. The first convolution stacks compress -ee-stpects into hidden states by a factor of r in length, and the second stacks capture the correlation of features. After that, the vector quantization layer utilizes these hidden states to obtain prosody codes u = { {u1, u2 Un and hidden states Ep. The information bottleneck B(-) of the VQ encoder is composed of the temporal compression and the vector quantization layer. The detailed instructions for ensuring an appropriate information bottleneck B-) can be found in Appendix F
Multi-Reference Timbre Encoder Our objective is to extract fine-graines timbre information from multi-sentence speech prompts. Since speakers can change their timbre by using different speaking techniques according to their speaking habits or desired semantic meanings (McAdams) 2013), the timbre encoder needs to extract fine-grained timbre information from multiple prompts that can represent the speakers' habits. Here, we introduce multi-referencee timbre encoder (MRTE) to achieve this objective. First, we concatenate the reference mel-spectrograms y/ that belong to the target speaker but are different from the target mel-spectrogram. The mel encoder then compresses the concatenated mel-spectrogram into acoustic hidden states 2 by a factor of in length. Subsequently, to extract semantically relevant timbre information from speech prompts, we introduce timbre-to- content attention module. This module takes 2e as the query and ž as both the key and the value. Finally, we upsample the output the mo-ooett attention module to match the length of the target mel-spectrogram using the length regulator (Ren et al. 2019).
3.3 PROSODY LATENT LANGUAGE MODEL
Unlike previous models that are trained with single-sentence prompts, our prosody latent language model (P-LLM) aims to capture the speaker's 's prosodic patterns from multi-sentence prompts effec- tively. During the -tage training process, we first extract the compressed prosody hidden states {Zp1, 2p2, , 2pm } and the content hidden states {zet, Ze2. ..., zen) from multiple speech clips (2,, ... } of the target speaker using the proposed compressive acoustic autoencoder. We then concatenate them along the time axis to construct a/, = Concat(Zp1, "yp.... : :pp)) and 2 Concat( (Ze1, Zc2. zen). In order to match the lengths of 2 and z/. in the temporal dimen- sion, expand z' to the frame level with duration information za and compress it times with max pooling layer. After that, we transform z/ to prosody code u' and then feed u' and zz into the P-LLM, which predicts the prosody code in an regressgee manner:
L
p (u' 0):
II p (ul u'gi, 2':1)2::)) =l:0), 1=0
(2)
where f is the parameters of P-LLM and L is the length of the concatenated prosody code u'. In training, we set batch size as 1 to increase the maximum number m of prosody codes in each batch as much as possible. If the total number of speech frames from a single speaker is less than m r, we will include speech samples from other speakers in this batch and incorporate speaker-level attention masks into P-LLM. We do not specifically define the speech prompt; instead, we train the language model directly using the concatenated speech samples through the teacher-foreing technique with the cross-entropy loss. To avoid the transition area problems caused by directly concatenating the prompts, we assign the start token and end token each sentence, which guides P-I.LM to continue writing the current sentence and extract useful information from previous sentences. This training strategy enables the model capture the useful prosody-level information contained in the multi-sentence prompts. Therefore, in the inference stage, users can flexibly improve the generation quality by extending the length of prompts by concatenating the reference speech clips. For duration modeling, we propose a phoneme-leve duration modeling by leveraging the powerful in-context learning capabilities of ruto-regressive models. The overall architecture of the auto-regressive duration model remains the same as P-L.MM, but we use mean squared error (MSF) loss instead.
regressise duration model. This model enhances the
3.4 PROSODY INTERPOLATION
Here, propose a prosody interpolation technique to control replace the prosodic style of the target speaker in the discrete space while ensuring the quality of timbre reconstruction. We achieve this objective by interpolating the probabilities from multiple P-LLM outputs, which come from
5

---

Published as a conference paper at ICLR 2024
multiple speakers. For example, target speaker has relatively sad speaking tone, but want to generate speeches that sound happier for him while preserving his timbre. The solution to 1) extract prosody latent u, from speeches happy tone of other speakers and the sad prosody latent u, from the target speech prompt; 2) utilize two language models to separately decode the target prosody parameters. In
code ú with the prosodic prompt u, and ug- These language models share the every step t of the decoding process, the probability distributions of the two language models are interpolated with the weight y, which can be formulated as follows:
T (ύ) II (I-(1))) 2nD
úca. ub, Concat(#ab, &c); + (xiz Uee,, Uay Concat( Ec):0)), (3)
where Reb, and Zea are the content information from speech clips 85 and Sa- &. is the content infor- mation of the target sentence. With our prosody interpolation technique, users can freely control the prosodic style of the generated speech in the inference stage. Moreover, the proposed prosody interpolation algorithm utilizes the autoregressive probability distribution of the language model for prosody transfer. Compared with directly substi- tuting the time-averaged prosody representation u. with u, (Karlapati et al., 22200; Zaudi et al. 2021). the prosody latent language model is able to mix u, and un in a soft and fineggrained manner in the autoregressive generation process.
Interpolate
P-LLM
Sample
P-LLM
Figure 2: Prosody interpolation.
EXPERIMENTS
4.1 EXPERIMENTAL SETUP
Training Datasets. We train Mega-TTS 2 and all baselines on LibriLight (Kahn et al. 2020), which contains 60K hours of unlabelled speech derived from LibriVor audiobooks. The sample rate 16KHz for all speech data. We transcribe the speech with the hybrid DNN-HMM ASR model pre-trained on 960 hours labeled LibriSpeech following VALL-E (Wang et al. 2023). We align the phoneme sequence with speech using the external alignment tool (McAuliffe al. (017).
Model Configuration. We provide model configuration in Appendix A.4 and detailed hyper- parameter settings in T
Training and Inference. In the first training stage, we train the first-stage model on 4 NVIDIA A100 GPUs, with batch size of 48 sentences on each GPU. In the second stage, we train the P-LLM and duration model on 8 NVIDIA A100 GPUs, with a batch size of 4,000 tokens on each GPU. means that our model supports 4,000 8 frames of prompts theoretically. We use the Adam optimizer with 3 0.9, 32 = 0.999, e 10 and follow the same learning rate schedule in Vaswani et al. (0177. It takes 600k steps for the first stage model's training and 300K steps for the second stage model's 's training until convergence. The predicted mel-spectrograms are transformed into audio samples using pre-trained HiFi-GAN VI (Kong et al. 2020).
Objective Metries. For zero-shot TTS, we evaluate the word error rate (WER). speaker similarity (SIM), and average dynamic time warping (DTW) (Müller (007) distance of the pitch for the ground- truth speech and synthesized speech. In terms of the cosine speaker similarity, we use the WavLM model (Chen et al. (202)) fine-tuned for speaker verificatiorf compute the cosine speaker similarity score between the ground-truth speech and the synthesized speech. The similarity score is in the range of [-1, 1], where a larger value indicates higher similarity of input samples. We also evaluate the word error rate (WER) for cross-lingual TTS. We use the released HuBERT-Lary model (Hsu et al., 2021) fine-tuned on the LibriSpeech 960h dataset to transcribe the generated speech into text. Then, the WER between the transcribed text and the original target text is measured. We use all samples in
https://huggingtte www.c/..c.o.oooooooo co/mi www.:iicccccc.o.oooooo

---

Published as a conference paper at ICLR 2024
Table Zero-shot TTS results on LibriSpeech test-clean set. 5_" means the result not available. "- 10s" means that there are 10 seconds of data from each speaker available for fine-tuning or prompting. #Params. denotes the number of parameters (including the vocoder or codec model). The evaluation is conducted with NVIDIA V100 GPU and batch size 1. RTF denotes the real-time factor.
Model
WERL
SIM
DTW
QMOST
SMOST
RTF
#Params.
Method
GT
1.98%
4.43 + 0.09
4.26 +: 0.111
Baseline-10s
4.74%
0.895
35.12
3.97 0.08
3.76 +
Baseline-60s
4.18%
0.923
31.08
4.01 =: 0.09
3,92 1:
0.089
467M
Fine-tune
Baseline-300s
3.11%
0.934
29.80
4.08 +: 0.07
4.03 + 0.08
VALL-E-3s
5.83%
0.885
36.59
3.89 + 0.30
3.70
1,471
VALL-E-IOS
5.54%
0.893
34.10
3,92 1. 0.11
3.74 1 0.10
1.775
478M
Zero-shot
VALI-E-20s
8.77%
0.805
43.02
3.41 + 0.12
3.25 = 0.14
2.104
Ours-3s
2.46%
0.898
34.39
3.99 = 0.06
3.75 =
0.302
Ours-10s
2.28%
0,905
32.30
4.05 + 0.08
3.79 + 0.09
0.334
473M
Zero-shot
Ours-60s
2.24%
0.926
30.55
4.11 = 0.00
3.95 1 0.09
0.413
Ours-300s
2.23% %
0.932
29.95
4.12 +: 0.30
4.01 + 0.09
0.923
the test set for the objective evaluation. For prosody transfer, we evaluate the WER, SIM, duration emoor (DE), and the moments (standard deviation (o), skewness (7) and kurtosis (x)) (Andreeva et al. 2014 Niebuhr & Skarnitz] 2019) of the pitch distribution.
Subjective Metrics. We conduct the MOS (mean opinion score) evaluation on the test set to measure the audio naturalness via Amazon Mechanical Turk. We keep the text content and prompt speech consistent among different models to exclude other interference factors. We randomly choose 50 samples from the test set of each dataset for the subjective evaluation, and each audio listened to by at least 20 testers. We analyze the MOS in two aspects: QMOS (Quality, clarity, naturalness, and high-frequency details) and SMOS (Speaker similarity in terms of timbre reconstruction and prosodic pattern). We also analyze the CMOS in terms of audio quality and speaker similarity. We tell the testers to focus on one corresponding aspect and ignore the other aspect when scoring.
4.2 RESULTS OF ZERO-SHOP SPEECH SYNTHESIS
In this subsection, we evaluate our model with various lengths of speech prompts and compare our model with zero-shot and fine-tuning baselines to demonstrate the effectiveness of the multi-sentence prompting mechanism. We randomly choose 20 speakers from the LibriSpeech test-clean set and randomly choose 400 seconds of speeches for each of them. We split the 400 seconds of speech into a 300-second prompt set and a 100-second target set. We keep the prompts consistent among different models to exclude other interference factors. We compare the zero-shot speech synthesis performance Mega-TTS 2 with two systems, including: 1) VALL-E (zero-shot) (Wang et al., 2023), large-scale zero-shot TTS model using large language models to generate discrete speech codes. Since VALL-E has not been open-sourced yet, we carefully implement for optimal performance; 2) Bascline (fine-tune), model that incorporates the GAN used in our Mega-TTr TTS 2 to the FastSpeech backbone (Ren et al. 2020). To make the baseline support adaptive scenarios, we use the powerful speaker encoder from Meta-StyleSpeecd (Min et al. 2021) to extract timbre information. We carefully fine-tune the baseline system for 2,000 steps to reach an optimal balance between WER and SIM. Note that all of the systems this experiment are pre-trained on the LibriLight dataset. We provide further explanation for the selection of the baseline systems in Appendix A.7
Analysis As shown in Table as the amount of adaptation data increases, the performance of Mega- TTS 2 continues to improve. Although the performance of VALL-E improves as the data volume increases from seconds to 10 seconds, the performance significantly drops in the 20-second setting due to the single-sentence prompting mechanisms in training. Moreover, since the compression rate of the Encodec model restricts the length of prompts, VALL-E fails to generate reasonable speeches with prompts longer than 20 seconds in experiments. From another perspective, when have 10 seconds or 60 seconds of speeches for each speaker, our Mega-TTF S 2 surpasses the fine-tuning baseline in terms of speech naturalness and speaker similarity. Additionally, when have 300
7

---

Published as a conference paper at ICLR 2024
0.94
Durs-60s
0.93
Baselino- 10s
0.92
WER (9)
0.91
ascelino-305
Ours-10s
0.90
Ours-60s
0.89-
Ours-300
Baseline-10
0.88
Baseline 605
Basaline-
0.87
1x102
1x10
5x10²
2x10*
1x10²
1x10² 5xi0²
2x10
Fine-tuning Steps
Fine-tuning Steps
Figure 3: Word error rate (WER) and speaker similarity score (SIM) comparisons in the few-shot fine-tuning process.
0.016
0.016
Ubrrippecc- 2300
LibrTTSS 2300 FSD 250-0011_500 0011_Surpiis Ours Before
Density
0.012
FSD 001001155000 Surprise Daft-Exprt- Before
Density
0.012
0.008
Daft Exprt After
0.008
After
0.004
0.004
0.000
0.000
150
250
350
50
250
350
(Hz)
FD (HZ)
Figure 4: The visualizations of pitch distribution before and after the prosody transfer. Here, transfer the prosodic styles in a surprised tone to speaker "2300" the LibriSpeech dataset.
seconds of speeches per speaker, Mega-TTS 2 still outperforms the baseline system in terms of WER and achieves comparable performance with it in terms speaker similarity. We also visualize the WER and SIM in the fine-tuning process and compare the baseline system with Mega-TTS in Figure 3] Our approach can enhance speaker similarity by utilizing more data like fine-tuning baseline, while maintaining relatively low word error rate.
4.3 RESULTS OF PROSODY TRANSFER
In this subsection, we evaluate the prosody transfer performance of our model by transferring the emotional styles from the ESD dataset (Zhou et al. 202I) to speakers the LibriSpeech test-clean dataset. We randomly choose 20 speakers from the LibriSpeech test-clean set and choose 50 sentences for each of them. Then, we randomly select an emotional speech clip from the ESD dataset for each of the sentences in the LibriSpeech test-clean and the selected emotional speech the prosodic reference, We keep the reference speeches consistent among different models to exclude other interference factors. We compare the prosody transfer performance Mega-TTS with two systems, including: 1) CopyCat (Karlapati et al., 2020), model that utilizes a reference encoder architecture capable of capturing temporal prosodic representations; 2) Daft-Exprt (Zaıdi et al. [021]. model disentangles identity and prosodic information through an adversarial training strategy that enables accurate prosody transfer across speakers. make fair comparisons, we incorporate the techniques for prosody transfer from CopyCat and Daft-Exprt to the baseline system proposed in the previous subsection and scale up the model capacity to ensure that all models have a comparable number of parameters. All of the systems in this experiment are pre-trained the LibriLight dataset.
Analysis Table demonstrates that compared with CapyCat and Daft-Exprt, the moments (o, 7, and of the generated speeches of Megs-TTS are closer to the ground-truth audio and the DE is lower than other methods, demonstrating the effectiveness of the proposed prosody interpolation techniques. Besides, maintain a high audio quality. We also visualize the prosody distribution before and after the prosody transfer process and compare the baseline system with Mega-TTS 2 in Figure 4
observe that method efficiently preserve the original timbre and
8

---

Published as a conference paper at ICLR 2024
Table 2: Prosody transfer experiments on LibriSpeech test-clean (A) and ESD (B) datasets. SIM-AB means the SIM after we transfer the prosodic styles from B to A. DE denotes the averaged absolute duration error in microseconds. o, 7, and R. denotes the moments of the pitch distribution.
Model
WER
SIM-AB
DE
QMOS
SMOS-AB
GT (ESD)
4,38%
74.91
0.707
0.024
4.18 + 0.08
4.22/2.39
CopyCat
5.29%
0.843/0.740
37.2
59.74
0.889
0.859
1.11
3.53/3.19
Daft-Exprt
4.89%
0.901/0.633
36.5
67.20
0.851
0.427
3.90
3.81/2.90
Ours
4.82%
0.920/0.513
32.8
72.62
0.664
0.197
3.92
3.87/2.64
4.4 ABLATION STUDIES
Prosody and Timbre Prompts We evaluate different lengths of prompts for the MRTE and P-LLM separately. In Ta- bleß the SIM score and the speech qual- ity increase with longer timbre prompts while the DTW distance almost remains unchanged. When we increase the length of prosody prompts, the DTW distance decreases while the speaker similarity re- mains the same level. It be seen that the proposed timbre and prosody prompting mechanisms boost the subjec- tive speaker similarity in terms of timbre and prosody modeling separately.
Table 3: Ablation studies of the timbre and prosody
prompts on zero-shot TTS. "w/ 60s T." denotes that use 60 seconds of timbre prompts, and "w/ 60s P." denotes that we use 60 seconds of prosody prompts.
Setting
WER
SIM1
DTW
CMOS-Q
CMOS-S
Ours- 10s
2.28%
0.905
32.30
0.000
0.000
w/ 60s
2.26%
0.922
32.23
+0.128
+0,241
2.25%
0.930
32.08
+0.162
60s
2.27%
0.906
30.74
+0.014
+0.154
w/ 300s P.
2.24%
0.908
30.25
+0.017
+0.196
VQ Encoder and MRTE We test the following four settings: 1) w/o MRTE, which removes the MRTE from our model and does not disentangle the prosody and timbre; 2) w/ VAE, which uses VAE to perform generative prosody modeling; 3) w/ VDE+ which uses VAE and latent diffusion model (LDM) (Rombach al. 2022 to perform generative prosody model- ing. The architecture and prompting mechanism of LDM is based on Natu-
Table 4: Ablation studies of the proposed VQ encoder and MRTE on zero-shot TTS.
Setting
WER
SIM
DTW,
CMOS-Q
CMOS-S
Ours- 10s
2.28%
0.905
32.30
0.000
0.000
Ours-300s
2.23%
0.932
29.95
+0.144
+0.493
w/o MRTE
5.57%
0.841
36.07
-0.458
-0.619
VAE VAE+LDM
2.31%
0.896
35.18
-0.038
-0.127
2.25%
0.907
32.98
+0.007
-0.005
ralSpeech 2 (Shen al. (0236)) All baselines use 10 seconds of prompts. The results are shown
in Table For setting 1), it can be observed that the removal of MRTE significantly affects both audio quality and speaker similarity. This is because the timbre information is absorbed by the VQ codebook and puts great pressure on the P-LLM, which demonstrates the effectiveness of decomposing timbre and prosody information. For setting 3), substituting the VQ encoder and P-LLM with VAE and LDM results in similar performance compared Ours 10s. However, the performance of w/ VAE+LM still much inferior to Ours-300s, indicating the superiority of the proposed multi-sentence prompting mechanism.
5 CONCLUSIONS
In this paper, we present Mega- TTS 2, framework that boosts the prompting mechanisms for zero- shot TTS systems. With the proposed multi-sentence prompting strategy, our approach outperforms the fine-tuning baseline when 10 seconds to 5 minutes of adaptation data is available for each speaker. Furthermore, our method utilizes a prosody interpolation technique to successfully transfer various prosodic styles to the target speaker while preserving the target speaker's timbre. Experimental results demonstrate that our method exhibits superior performance in terms audio naturalness and speaker similarity. Due to space limitations, we include additional discussions in the appendix.

---

Published as a conference paper at ICLR 2024
6 ACKNOWLEDGMENTS
This work was supported in part by the National Natural Science Foundation of China under Grant No. 62222211 and National Key R&D Program of China under Grant No.220222D01620000.
REFERENCES
Bistra Andreeva, Gražyna Demenko, Bernd Möbius, Frank Zimmerer, Jeanin Jügler, and Magdalena Oleskowicz-Popiel. Differences of pitch profiles in germanic and slavic languages. In Fifteenth Annual Conference of the International Speech Communication Association, 2014.
Sercan Arik, Jitong Chen, Kainan Peng, Wei Ping, and Yanqi Zhou, Neural voice cloning with a few samples. Advances neural information processing systems, 31, 2018.
Max Bain, Jaesung Huh, Tengda Han, and Andrew Zisserman. Whisperx: Time-accurate speech transcription of long- form audio. arXiv preprint arX1v:2303.00747, 2023.
Edresson Casanova, Christopher Shulby, Eren Gölge, Nicolas Michael Müller, Frederico Santos de Oliveira, Arnaldo Candido Junior, Anderson da Silva Soares, Sandra Maria Aluisio, and Moacir Antonelli Ponti. Sc-glowtts: an efficient zero-shot multi-speaker text-to-speech model. arXiv preprint arXiv:2104.05553 2021.
Edresson Casanova, Julian Weber, Christopher Shulby, Arnaldo Candido Junior, Eren Gölge, and Moacir A Ponti. Yourtts: Towards zero-shot multi-speaker tts and zero-shot voice conversion for everyone. International Conference on Machine Learning, pp. 2709 2720.. PMLR, 2022.
Jiawei Chen, Xu Tan, Jian Luan, Tao Qin, and Tie- Yan Liu. Hifisinger: Towards high-fidelity neural singing voice synthesis. arXiv preprint arXiv:2009. 01777. 2020.
Mingjian Chen, Xu Tan, Bohan Li, Yanqing Liu, Tao Qin, Sheng Zhao, and Tie- Yan Liu. Adaspeech: 2021.
Adaptive text to speech for custom voice. arXiv preprint
Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, et al. Wavlm: Large-scale self-supervised pre-training for full stack speech processing. IEEE Journal of Selected Topics in Signal Processing, 16(6); 1518, 2022.
Yutian Chen, Yannis Assael, Brendan Shillingford, David Budden, Scott Reed, Heiga Zen, Quan Wang, Luis Cobo, Andrew Trask, Ben Laurie, et al. Sample efficient adaptive text-to-speech. arXiv preprint arXiv: 1809. /0460, 2018
Chung-Ming Chien, Jheng- Lin, Chien-yu Huang, Po-chun Hsu, and Hung-yi Lee. Investigating incorporating pretrained and learnable speaker representations for multi-speaker multi-style text-to-speech. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 8588 8592.. IEEE, 2021.
Seungwoo Choi, Seungju Han, Dongyoung Kim, and Sungjoo Ha. Attentron: Few-shot text-to-speech utilizing attention- based variable-length embedding. arXiv preprint arXiv:2005.08484 2020.
Erica Cooper, Cheng-I Lai, Yusuke Yasuda, Fuming Fang, Xin Wang, Nanxin Chen, and Junichi Yamagishi. Zero-shot multi-speaker text-to-speech with state-of-the-arr neural speaker embeddings. In ICASSP 2020-2020 IEEE International Conference on Acousties, Speech and Signal Processing (ICASSP), pp. 6184-6188. IEEE, 2020a.
Erica Cooper, Cheng-I I Lai, Yusuke Yasuda, and Junichi Yamagishi. Can speaker augmentation improve multi-speaker end-to-end tts? arXiv preprint arXiv:2005.01245, 2020b.
Alexandre Défossez, Jade Copet, Gabriel Synnaeve, and Yossi Adi. High fidelity neural audio compression. arXiv preprint arXiv:2210. 2022.
Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. A survey for in-context learning. arXiv preprint arXiv:2301.002.: 2022.
10

---

Published as a conference paper at ICLR 2024
Siddharth Gururani, Kilol Gupta, Dhaval Shah, Zahra Shakeri, and Jervis Pinto. Prosody transfer in neural text to speech using global pitch and loudness features. arXiv preprint arXiv:191..09644 2019.
Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEF/ACM Transactions on Audio, Speech, and Language Processing, 29:3451-3460, 2021.
Rongjie Huang, Yi Ren, Jinglin Liu, Chenye Cui, and Zhou Zhao. Generspeech: Towards style transfer for generalizable out-of-domain text-to-speech. Advances in Neural Information Processing Systems, 15:10970-10983, 2022a.
Sung-Feng Huang, Chyi-Jiunn Lin, Da-Rong Liu, Yi-Chen Chen, and Hung-yi Lee. Meta-tts: Meta- learning for few-shot speaker adaptive text-to-speech. JEEE/ACM Transactions on Audio, Speech, and Language Processing, 30: 1558-1511,, 2022b.
Ye Jia, Yu Zhang, Ron Weiss, Quan Wang, Jonathan Shen, Fei Ren, Patrick Nguyen, Ruoming Pang, Ignacio Lopez Moreno, Yonghui Wu, al. Transfer learning from speaker verification to multispeaker text-to-speech synthesis. Advances in neural information processing systems, 31, 2018.
Jacob Kahn, Morgane Rivière, Weiyi Zheng, Evgeny Kharitonov, Qiantong Xu,
Mazaré, Julien Karadayi, Vitaliy Liptchinsky, Ronan Collobert, Christian Fuegen, et al. Libri-light: benchmark for asr with limited or no supervision. ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 7669-7673. IEEE, 2020.
Minki Kang, Dongchan Min, and Sung Ju Hwang, Any-speaker adaptive text-to- text-to-speech synthesis with diffusion models. arXiv preprini 1Xiv:2221.003355 2022.
Minki Kang, Dongchan Min, and Sung Ju Hwang. Grad strad-stylespeehh: Any-speaker adaptive text-to- speech synthesis with diffusion models. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5, IEEE, 2023.
Sri Karlapati, Alexis Moinet, Arnaud Joly, Viacheslav Klimkov, Daniel Sáez-Triguens and Thomas Drugman. Copycat: Many-to-many fine-grained prosody transfer for neural text-to-speech. arXiv preprint arXiv:2004. 14617, 2020.
Sri Karlapati, Penny Karanasou, Mateusz Lajszezak, Ammar Abbas, Alexis Moinet, Peter Makarov, Ray Li, Arent van Korlaar, Simon Slangen, and Thomas Drugman. Copycat2: A single model for multi-speaker tts and many-to-many fine-grained prosody transfer. arXiv preprint 234443 2022.
Eugene Kharitonov, Damien Vincent, Zalán Borsos, Raphaël Marinier, Sertan Girgin, Olivier Pietquin, Matt Sharifi, Marco Tagliasacchi, and Neil Zeghidour. Speak, read and prompt: High-fidelity text-to-speech with minimal supervision. arXiv preprint arXiv:2302.035400 2023.
Heeseung Kim, Sungwon Kim, and Sungroh Yoon. Guided-tts: A diffusion model for text-to-speech via classifier guidance. In International Conference on Machine Learning, pp. 11119-11133. PMLR, 2022a.
Jachyeon Kim, Sungwon Kim, Jungil Kong, and Sungroh Yoon. Glow-tts: A generative flow for text-to-speech via monotonic aligmment search. Advances in Neural Information Processing Systems, 33:8067-8077,, 2020.
Jaehyeon Kim, Jungil Kong, and Juhee Son. Conditional variational autoencoder with adversarial learning for end-to-end text-to-speech. In International Conference on Machine Learning, pp. 5530-5540. PMLR, 2021.
Sungwon Kim, Heeseung Kim, and Sungroh Yoon. Guided-tts 2: A diffusion model for high-quality adaptive text-to-speech with untranscribed data. arXiv preprint 23Xiv:2205.15370, 2022b.
Viacheslav Klimkov, Srikanth Ronanki, Jonas Rohnke, and Thomas Drugman. Fine-grained robust prosody transfer for single-speaker neural text-to-speech. arXiv preprint 19X 2019.
11

---

Published as a conference paper at ICLR 2024
Jungil Kong, Jachyeon Kim, and Jackyoung Bae. Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis. Advances in Neural Information Processing Systems, 33:17022-170333 2020.
Zvi Kons, Slava Shechtman, Alex Sorin, Carmel Rabinovitz, and Ron Hoory. High quality, lightweight and adaptable uts using Ipenet. arXiv preprint arXiv:1905.00590 2019.
Younggun Lee and Taesu Kim. Robust and fine-grained prosody control of end-to-end speech synthesis. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5911-5915. IEEE, 2019.
Yichong Leng, Zhifang Guo, Kai Shen, Xu Tan, Zeqian Ju, Yanging Liu. Yufei Liu, Dongchao Yang, Leying Zhang, Kaitao Song, al. Prompttts 2: Describing and generating voices with text prompt. arXiv preprint arXiv:2309.022285 2023.
Naihan Li, Shujie Liu, Yanqing Liu, Sheng Zhao, and Ming Liu. Neural speech synthesis with transformer network. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pp. 6706-6713, 2019.
Yist Y Lin, Chung-Ming Chien, Jheng-Hao Lin, Hung-yi Lee, and Lin-shan Lee. Fragmentve: Any-to-any voice conversion by end-to-end extracting and fusing fine-graines voice fragments with attention. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5939-5943. IEEE, 2021.
Philipos C Loizou. Speech quality assessment. In Mullimedia analysis, processing and communica- fions, pp. 623-654. Springer, 2011.
Xudong Mao, Qing Li, Haoran Xie, Raymond YK Lau, Zhen Wang, and Stephen Paul Smolley. Least squares generative adversarial networks. In Proceedings of the IEEE international conference on computer vision, pp. 20922722222 27022 2017.
Le Matthew, Vyas Apoorv, Shi Bowen, Karrer Brian, Sari Leda, Moritz Rashel, Williamson Mary, Manohar Vimal, Adi Yossi, Mahadeokar Jay, and Hsu Wei-Ning. Voicebox: Text-guided multilin- gual universal speech generation at scale, 2023. URL comm
Stephen McAdams. Musical timbre perception. The psychology of music, pp. 35-67,
Michael McAuliffe, Michaela Socolof, Sarah Mihuc, Michael Wagner, and Morgan Sonderegger. Montreal forced aligner: Trainable text-speech alignment using kaldi. In Interspeech, volume 2017, pp. 498-502, 2017.
Dongchan Min, Dong Bok Lee, Eunho Yang, and Sung Ju Hwang. Meta-stylespeech: Multi-speaker adaptive text-to-speech generation. In International Conference on Machine Learning, pp. 7748- 7759. PMLR, 2021.
Henry B Moss, Vatsal Aggarwal, Nishant Prateek, Javier González, and Roberto Barra-Chicol Boffin uts: Few-shot speaker adaptation by bayesian optimization. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 7639-7643. IEEE, 2020.
Meinard Müller. Dynamic time warping. Information retrieval for music and motion, pp. 69 69-84, 2007.
Oliver Niebuhr and Radek Skarnitzl. Measuring a speaker's acoustic correlates of pitch- but which? contrastive analysis based on perceived speaker charisma. In Proceedings of 19th International Congress of Phonetic Sciences, 2019.
Shifeng Pan and Lei He. Cross-speaker style transfer with prosody bottleneck in neural speech synthesis. arXiv preprint wtXiv:2107.125622 2021.
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: an corpus based on public domain audio books. 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp. 5206-5210. IEEE, 2015.
12

---

Published as a conference paper at ICLR 2024
Kaizhi Qian, Yang Zhang, Shiyu Chang, Xuesong Yang, and Mark Basegaww-Johnson Autove: Zero-shot voice style transfer with only autoencoder loss. In International Conference on Machine Learning, pp. 5219-5219. PMLR, 2019.
Yi Ren, Yangjun Ruan, Xu Tan. Tao Qin, Sheng Zhao, Zhou Zhao, and Tie- Yan Liu. Fastspeech: Fast, robust and controllable text to speech. Advances in neural information processing systems, 32, 2019.
Yi Ren, Chenxu Hu, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tic- Yan Liu. Fastspeech 2: Fast and high-quality end-to-end text to speech. arXiv preprint 2006:22006045558 2020.
Yi Ren, Ming Lei, Zhiying Huang, Shiliang Zhang, Qian Chen, Zhijie Yan, and Zhou Zhao, Prosospeech: Enhancing prosody with quantized vector pre-training text-to-specch. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 7577-7581. IEEE, 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pp. 10684-10695, 2022.
Jonathan Shen, Ruoming Pang, Ron J Weiss, Mike Schuster, Navdeep Jaitly, Zongheng Yang. Zhifeng Chen, Yu Zhang, Yuxuan Wang. Rj Skerrv-Ryan. et al. Natural tts synthesis by conditioning wavenet on mel spectrogram predictions. In 2018 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp. 477944799449 4783. IEEE, 2018
Kai Shen, Junliang Guo, Xu Tan, Siliang Tang, Rui Wang, and Jiang Bian. A study on relu and softmax in transformer. arXiv preprint arXiv:2302.0646.1 2023a.
Kai Shen, Zeqian Ju. Xu Tan, Yanqing Liu, Yichong Leng, Lei He, Tao Qin, Sheng Zhao, and Jiang Bian. Naturalspeech 2: Latent diffusion models are natural and zero-shot speech and singing synthesizers. arXiv preprint arXiv:2 20t::2304.999 09/ 16, 2023b.
Atli Thor Sigurgeirsson and Simon King. Do prosody transfer models transfer prosodyf. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2023.
RJ Skerry-Ryan, Eric Battenberg, Ying Xiao, Yuxuan Wang, Daisy Stanton, Joel Shor, Ron Weiss, Rob Clark, and Rif A Saurous. Towards end-to-end prosody transfer for expressive speech synthesis with tacotron. In international conference on machine learning, pp. 4693-470000 PMLR, 2018.
Yuhta Takida, Takashi Shibuya, WeiHsiang Liao, Chieh-Hsin Lai, Junki Ohmura, Toshimitsu Uesaka, Naoki Murata, Shusuke Takahashi, Toshiyuki Kumakura, and Yuki Mitsufuji. Sq-vae: Varia- tional bayes on discrete representation with self-annealed stochastic quantization. arXiv preprint 20X5v:2205.07547, 2022.
Xu Tan, Tao Qin, Frank Soong, and Tie- Yan Liu. A survey on neural speech synthesis. arXiv preprint NKKV:22006.15611 2021.
Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Michael Wagner and Duane G Watson. Experimental and theoretical advances in prosody: A review. Language and cognitive processex, 2507-9):905-945, 2010.
Chengyi Wang, Sanyuan Chen, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu. Huaming Wang, Jinyu Li, et al. Neural codec language models are zero-shot text to speech synthesizers. arXiv preprint arXiv:2301.02//1 11, 2023.
13

---

Published as a conference paper at ICLR 2024
Yuxuan Wang, RJ Skemr--yann Daisy Stanton, Yonghui Wu, Ron J Weiss, Navdeep Jaitly, Zongheng Yang, Ying Xiao, Zhifeng Chen, Samy Bengio, et al. Tacotron: Towards end-to-end speech synthesis. arXiv preprint arXi::1703.10133 2017.
Yihan Wu, Xu Tan, Bohan Li, Lei He, Sheng Zhao, Ruihua Song, Tao Qin, and Tie- Yan Liu. Adaspeech 4: Adaptive text to speech in zero-shot scenarios. arXiv preprint arXiv:2204.00430 2022.
Dongchao Yang, Songxiang Liu, Rongjie Huang, Jinchuan Tian, Chao Weng, and Yuexian Zou. Hifi-codec: Group-residual vector quantization for high fidelity audio codec. arXiv preprini arXiv:2305.02765, 2023.
Jingzhou Yang and Lei He. Towards universal text-to-speech. In Interspeech, pp. 3171-3175, 2020.
Dacheng Yin, Chuanxin Tang, Yanqing Liu, Xiaoqiang Wang, Zhiyuan Zhao, Yucheng Zhao, Zhiwei Xiong, Sheng Zhao, and Chong Luo. Retrievertts: Modeling decomposed factors for text-based speech insertion. arXiv preprint arXiv:2206. 13865, 2022.
Julian Zaddi, Hugo Seuté, BV Niekerk, and M Carbonneau. Daft-exprt: Robust prosody transfer across speakers for expressive speech synthesis. arXiv preprint (xXiv:2108.0227/ 2021.
Ziqiang Zhang, Long Zhou, Chengyi Wang, Sanyuan Chen, Yu Wu, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, et al. Speak foreign languages with your own voice: Cross-lingual neural codec language modeling. arXiv preprint arXiv:2303.03924 2023.
Chuanxia Zheng and Andrea Vedaldi. Online clustered codebook. arXiv preprint arXiv:2307.15139 2023.
Kun Zhou, Berrak Sisman, Rui Liu, and Haizhou Li. Seen and unseen emotional style transfer for voice conversion with a new emotional speech dataset. In ICASSP 202 1-221 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 920-924 IEEE, 2021.
Yixuan Zhou, Changhe Song, Xiang Li, Luwen Zhang, Zhiyong Wu, Yanyao Bian, Dan Su, and Helen Meng. Content-dependent fine mne-grained speaker embedding for zero-shot speaker adaptation in text-to-speech synthesis. arXiv preprint arXiv:2204.00990 2022.
A DETAILED EXPERIMENTAL SETTINGS
A. DETAILS IN OBJECTIVE EVALUATIONS
Here, we provide details of the model used in objective evaluations.
Speaker Similarity Model To measure the speaker similarity, we use the WavLM (Chen et al. 2022) model fine-tuned for speaker verification from htttps://uuginnge
co/ mi crosoft/ to extract the speaker embedding. Then the cosine similarity between the synthesized speech 's speaker embedding and the ground-truth speech's speaker embedding is calculated as the speaker similarity score. The WavLM model is pre-trained on 94,000 hours of speech data and fine-tuned on the VoxCelebl dataset using X-Vector head with Additive Margin Softmax loss, which achieves 0.84%, 0.928%, and 1.758% EER (Equal Error Rate) on the Vox I-O, VoxI-E, and Vox 1- trial lists.
davim-bass--puse
To measure the audio quality and speech intelligibility, we evaluate the word error rate (WER) metric. We use the fine-tuned HuBERT-Large model to transcribe the synthesized speech into text and calculate the WER between the transcribed text and the original target text. The HuBERT. cc. wc//accbook/////ooooooooooooo hubert- Large- 1s000e--11900-f-
ASR Model
Large model from https: /tt:://huqquu http///gggggggggggeccc fine-tuned on 960h of Librispeech and achieves 1.5%, 3.0%, 1.9%, and 3.3% WER on the dev-clean, dev-other, test-clean, and test-other set of Librispeech.
14

---

Published as a conference paper at ICLR 2024
A.2 DETAILS IN SUBJECTIVE EVALUATIONS
We perform the audio quality and speaker similarity evaluations on Amazon Mechanical Turk (MTurk). For each dataset, we randomly select 50 samples from the test and use the TTS systems to generate the audio samples. Each audio has been listened to by at least 20 listeners. For MOS, each tester is asked evaluate the subjective score of sentence on a 1-5 Likert scale. For CMOS, listeners are asked to compare pairs of audio generated by systems A and B following Loizou (201T), indicating which of the two audio they prefer. For audio quality evaluation (QMOS and CMOS-Q), we tell listeners to "Please focus on the speech quality in terms of clarity, naturalness, and high-frequency details, and ignore other factors". For speaker similarity evaluations (MOS- we tell listeners to "Please focus only the similarity of the speaker the reference of the timbre and prosodic patterns, and ignore the differences of content, grammar, audio quality, or other factors.". ". We paid $15 participants hourly and totally spent about $ $1200 on participant compensation. We tell the participants that the data will used in scientific research.
in terms
Caev II
GELL
X2
IX5
Cony D GELII
Dowsssmpl: (16x)
Pooling (8s)
GELAI
MRTE
(b) VQ Encoder
Figure 5: The multi-reference timbre encoder (MRTE) and vector quantised (VQ) encoder.
A.3 DETAILED NETWORK STRUCTURE
As shown in Figure 5 the proposed MRTE is composed of two convolution stacks and downsampling block. To reduce the computational requirements while maintaining the quality of timbre reconstruction, we downsample the timbre hidden states by factor of d = 16 in length. In training, we randomly sample 2,000 frames from ÿ for training efficiency.
MRTE
VQ Encoder The bottleneck of our VQ Encoder is composed of a max pooling layer with a stride of 8 and a vector quantised layer. In our experiments, with a compression rate of r = 8 yields superior results compared to phoneme-level compression. We have tried different compression rates (2, 4, 8, 16, 32) and found that r = 8 reached an optimal balance between the reconstruction performance and compression. On the other hand, in the training process, we also found that the vanilla VQ-VAE suffers from codebook collapse (Takida et al. 2022). which means only small portion of codebook vectors are optimized. It restricts the expressive capacity of the codebook and affects the convergence of the training process. To solve the codebook collapse issue, we adopt dynamical initialization strategy based on CVQ-VAE (Zheng & Vedaldi 2023) during training, which ensures the code vectors that are used or unused to be modified more than frequently used ones.
found that compressing the mel-spectrsgrams
15

---

Published as a conference paper at ICLR 2024
A.4 MODEL CONFIGURATION
Our Mega-TTS TTS 2 consists of three encoders, a prosody latent language model, mel decoder, and discriminator. The prosody encoder, timbre encoder, and decoder consist of 5 convolutional blocks with 512 hidden size and 5 kernel size. The content encoder is an 8-layer Transformer (Vaswani et al., 2017, Shen al. 2023a) with 512 hidden size. The GAN discriminator follows the architecture of ML-GAN proposed in Chen al. (2020). The P-LLM model is decoder-only architecture that contains 12 Transformer layers with 1024 hidden size, which has 151M parameters. The duration predictor is an 8-layer decoder-only Transformer model with 512 hidden size. The codebook embedding size 1024, and the hidden size the codebook vector 256. The compression rate r and d is set 8 and 16, respectively. For prosody transfer experiments, 7 is set 0.8. We provide detailed hyper-past hyper-parameter settings about the model configuration in Table
Table 5: Hyperparameters Mega-TTSS models.
Syper-parameter
Value
Encoder Layers
3 2
Hidden Size
384
VQ Prosody Encoder
Conv ID Kernel
VQ Embedding Size
1024
VQ Embedding Channel
256
Phoneme Embedding Size
512
Encoder Layers
Content Encoder
Hidden Size
512
Kernel Size
Filter Size
1024
Encoder Layers
5
Query Encoder Hidden Size
512
MRTE
Key Encoder Hidden Size
256
Key Encoder Stride
16
ConvlD Kernel
Decoder Layers
Mel Decoder
Hidden Size
512
Conv ID Kernel
Decoder Layers
12
Hidden Size
1024
P-LLM
Prosody Code Embedding Size
1026
Attention Headss
16
Number of Discriminators
Window Size
32, 64, 128
Multi-Length Discriminator
Conv2D Layers
Hidden Size
192
Total Number of Parameters
367M
A.5 ERROR BARS AND RANDOM SEEDS
For the subjective evaluations, report confidence intervals of the results of MOS tests. For the objective evaluations, we ran the experiments 10 times with 10 different random seeds (1234, 1111, 2222, 3333, 4444, 5555, 6666, 7777, 8888, 9999) and obtained the averaged results.
16

---

Published as a conference paper at ICLR 2024
A.6 SAMPLING STRATEGY FOR P-LLM
In all of our experiments, we utilize the top-k sampling strategy for P-LLM, where is set to 10. The sampling-basee method, when used with appropriate k, enhances the output diversity compared to greedy decoding.
A.7 ABOUT THE SELECTION OF BASELINES
VALL-E (Wang al. 2023). NaturalSpeech 2 (Shen al. 2023b Leng al., 2023). and Voice-
Box (Matthew et al. 2023) are the state-of-the-art zero-shot TTS models. In the experiments of zero-shot TTS, we have tried carefully reproduce their works but failed reproduce NaturalSpeech 2 and VoiceBox. Since all of them do not provide the pre-trained models and source code, we only compare Mega-TTS 2 with VALL-E in our experiments.
A.8 DETAILED DECOMPOSITION STRATEGY
The prosody encoder E aims to capture fine-graine and local prosodic style 2p. For local prosodic style Zp, we assume that psd(.) a perfect local prosody extractor, and we can obtain the following equation: 7 ([psd(v)), psd(ū) = 0. The content information &e is also local and fine- fine-grained like Zp. On the other hand, the global prosodic information like the averaged volume and pitch can not be captured by E., intuitively. And since have designed an information bottleneck B(-) for Ep. the global prosodic information will be prioritized by the timbre encoder E, and stored in H(z). Now that both the local and global prosodic information is appropriately extracted, the validity of Equation and our disentanglement strategy can ensured.
B ABOUT SCALING UP DATASET SIZE
Scaling up dataset size is crucial for the practical application of zero-shot TTS. Therefore, we crawled 200K hours audiobook recordings from YouTube and novelfi The crawled corpus contains both labelled and unlabelled speeches, and most of them do not have speaker information. To transcribe the unlabelled speech in the wild, we use a powerful ASR model called WhisperX (Bain et al. 2023). And to obtain the speaker informa- tion, we use a released automatic speaker diarization model called syannote.auliq"" which achieves DER=11.24% the VoxConverse dataset and DER=14.09% on the AISHELL-4 dataset. In this exper- iment, we do not change the hyper- parameter settings of our model. The results are shown in T It can be seen that increasing the dataset size can improve the speaker similarity of the generated speeches.
Table 6: The results of zero-shot TTS when we scale up dataset size. Ours-10s (60K) means that use 60K hours of speeches from Librilight to train our model and use 10 seconds of speech prompts.
Sctting
WERL
SIM
DTW1
CMOS-Q
CMOS-S
Ours-10s (60K)
2.28%
0.905
32.30
0.000
0.000
Ours-10s (200K)
2.15%
0.922
32.05
+0.010
+0,215
C ABOUT THE DEFINITION OF ADAPTIVE TTS
The concept of adaptive TTS encompasses many aspects like the adaption for different voices, languages, styles, and domains (Tan et al. 202T). It is also known as various terms in academia and industry, such as voice adaptation (Chen et 2018), voice cloning (Arik et al.) 2018). custom voice (Chen et al. 2021), etc. In this paper, we primarily focus on adaptive TTS for different voices.
com/
17

---

Published as a conference paper at ICLR 2024
D VISUALIZATION OF ATTENTION MATRICES
To further verify the proposed P-LLM and multi-sentence prompting mechanism, we visualize the attention matrices averaged across all layers of P-LLM in Figure In this experiment, we separately conduct short-sentence generation and long-sentence generation. For short-sentence generation, randomly selected two sentences that shorter than seconds from speaker "908" in the LibriSpeech test-clean set and concatenated them together. The target texts for Figure O (a) and (b) are both about 15 words length. For long-sentence generation, that are longer than 15 seconds from speaker "908" in the I LibriSpeech test-clean set and concatenated them together. The target texts for Figure (c) and (d) are both about 100 words length. It can be seen that our P-LLM can capture both short-term and long-term information, demonstrating the effectiveness of the P-LLM's training strategy and the multi-sentence prompting mechanism.
randomly selected two sentences
(a)
(b)
(c)
Figure 6: Visualization of Attention Matrices. (a) and (b) are the attention matrices in short-sentence generation; (c) and (d) are the attention matrices in long-sentence generation.
LIMITATIONS AND ETHICS IMPACTS
In this section, we begin by discussing the limitations of the proposed method and outlining our strategies for addressing them in future research. Subsequently, we discuss the ethical impacts that might be brought by zero-shot TTS and our measures address these concerns.
E. LIMITATIONS AND FUTURE WORK
Firstly, model is trained English dataset and does not support multilingual TTS. We plan to address this problem by introducing more multilingual training data. Secondly, the speech quality be improved by introducing more high-fidelity training data. Thirdly, well-designed attention window may further enhance the in-context-learning capability of our P-LLM.
E.2 ETHICS IMPACTS
Mega-TTV TTS 2 improves the quality and efficiency of zero-shot speech synthesis, which makes easier for people to synthesize personalized speeches. Under appropriate and legal usage, this technique could facilitate applications like movies, games, podcasts, and other services, making human life more convenient. However, zero-shot TTS may be misused in deepfake-related usages, such as spoofing voices. To handle this, potential solutions like building a corresponding deepfake detection model should be considered. We also plan add watermarks to the synthesized speeches that the public can easily tell whether the speeches are synthesized or not. Additionally, restrictions will be included in the license of our project to prevent the misuse of the model.
F DESCRIPTION OF INFORMATION BOTTLENECK
The settings of the information bottleneck B(-) are crucial for the performance of disentanglement of our proposed method. Intuitively, there are four crucial variables for ensuring an appropriate informa- tion bottleneck: the number of codebook embedding, the codebook channel size, the compression
18

---

Published as a conference paper at ICLR 2024
rate of the VQ Encoder, and the downsampling rate d of the MRTE. However, the search space of these hyperparameters is too large. Since the settings of r and d also influence the reconstruction first consider r and d, fix
quality, the burden of P-LLM, and the computational requirements, them, and find the best setting for the hyperparameters of the codebook.
The Compression Rate We have conducted evaluations for different compression rates r of the VQ Encoder. In the experiments, found that lower compression rate would result in better reconstruction performance for the compressive acoustic autoencoder, but it would impose heavier burden on P-LLM since the token sequence is longer. As shown in Table | although r achieves the highest objective similarity score, the subjective speech quality and similarity significantly decrease, which means the final quality of generation is affected. Therefore, we use
r = 8 to reach an optimal balance between the reconstruction performance and compression.
Table 7: Ablation studies of the compression rate r. The length of speech prompts is 3 seconds.
Setting
WER
SIM
DTW
CMOS-Q
CMOS-S
2
5.53%
0.905
42.35
-0.301
-0.149
4.01%
0.901
36.93
-0.148
+0.053
8
2.46%
0.898
34.39
0,000
0.000
== 16
2.62%
0.889
35.18
-0.088
-0.141
32
3.57%
0.880
40.98
-0.288
-0,252
The Downsampling Rate d We have conducted evaluations for different downsampling rates of the MRTE. The results are shown in Figure [0] of the MRTE is low, the mel-spectrogram sequence can provide more information to the timbre encoder, resulting in better reconstruction. However, low downsampling ratio puts a significant computational burden on the attention operation in MRTE. To reduce the computational requirements while maintaining the quality of timbre reconstruction, we choose d 16 for our Mega-TTS
be seen that when the downsampling rate
The Information Bottleneck with Different Amount of Data Intuitively, the performance of the information bottleneck might be very sensitive the size of the dataset. Therefore, we conduct experiments analyzing the relationship between dataset size and the hyperparameters. The results are presented in Appendix G Although the hyperparameters do not change across these experiments, find that the model consistently performs well scenarios with varying amounts of available data.
G SCALING WITH DIFFERENT SIZES OF TRAINING DATA
Here evaluated the performance of Mega-TTS 2 scale with varying amounts of available data. In this experiment, all of the systems use 3 seconds of speech prompts. The results are shown in the following table. We can see that Mega-TTSS 2 performs well with different sizes of training data, while VALL-E fails to obtain satisfying results when the data is insufficient. We also scale our Mega-TTS 2 with 200K hours of speeches and the results can be found in Appendix
H THE STRATEGY OF PROSODY MODELING
In this section, we conduct experiments to verify the performance of the phoneme-level, word-level, and stride-8-leved prosody modeling. Stride-8 means that the stride of the pooling layer inside the VQ encoder is set to 8. It is worth noting that ProsoSpeech (Ren et al., 2022) utilizes word-level prosody modeling. Both of use the auto-regressive Transformer for prosody modeling. However, ProsoSPeech aims to improve the naturalness of prosody modeling. Compared with it, our P-LLM aims at improving the similarity speaker-relevant prosodic patterns, which extracts fine-grained prosodic information from latent prosodic prompts by leveraging the powerful in-context learning capability of LLM. The experimental results are shown in the following table. It can be seen that the stride-8-Ien 8eveel prosody modeling achieves the best performance. Intuitively speaking, the phoneme- level prosody modeling provides finer mner-eraneed information for better reconstruction while word-leved prosody modeling provides more semantic information. Both of these methods would be easily
19

---

Published as a conference paper at ICLR 2024
Table 8: Ablation studies the compression rate d. The length speech prompts 3 seconds.
Sctting
WERI
SIM
DTW4
CMOS-Q
CMOS-S
2.39%
0,900
34.27
+0.025
+0.057
16
2.46%
0.898
34.39
0.000
0.000
32
2.98%
0.880
34.85
-0.112
-0,223
Table 9: The performance of Mega-TTS scale with different sizes of the training data. All of the systems use seconds of speech prompts.
Model
Dataset Usage
WER.
SIM
DTW
QMOS
SMOS
LibriLight (IK hours)
15.73%
0.782
54.05
3.63
3.52iam0
VALL-E
LihriLight (10k hours)
8.29%
0.870
38.93
3.84
3.65
LibriLight (60k hours)
5.83%
0.885
36.59
3,90
3.71
LihriLight (IK hours)
4.69%
0.861
43.07
3.88
3.61
Mcga-TTSS 2
LigriLight (10k hours)
2.91%
0.882
35.76
3.95+au1
3.70
Lihril light (60k hours)
2.46%
0.898
34.39
3.98
3.77
affiected by the alignment accuracy of the speeches in training and inference stages. order to enhance the stability and performance of the proposed model, we use stride---eevl prosody modeling.
Table 10: Ablation studies of the strategy of prosody modeling. The length of speech prompts is 3 seconds.
Setting
WER
SIM
DTWJ
CMOS-Q
CMOS-S
Stride-8-leve
2.46%
0.898
34.39
0.000
0.000
Phoneme-level
2.94%
0.897
35.78
-0,107
-0,052
Word-level
3.12%
0.892
34.42
-0.120
-0.078
I SPECIAL CASES FOR ASSUMPTION
1
In practical scenarios, there is special case for Assumption 1 the timbre of speaker may vary significantly over different time periods. address this special case, we select ý randomly from regions near Ye much possible, ensuring that the timbre information of is close to that of yt-
J DIFFERENT LENGTHS OF CONTEXT DURING TRAINING
Here make ablation studies for different lengths of context during model's training process. We separately train P-LLM with different numbers of the contextual VQ code tokens and train our compressive acoustic autoencoder with different numbers of the contextual mel-spectrogram frames for MRTE. The results are shown in Figure 7 It can be seen that when we increase the length of context, the performance of the model during training significantly improves, demonstrating the effectiveness of our multi-reference training strategy.
K EXPLANATIONS ABOUT MORE CASES OF ROBUST SPEECH S SYNTHESIS
In our Mega-TTS TTS 2, we employ language model only for prosody modeling, enabling our model to benefit from the advantages of in-context learning provided by the LLM model. This approach also helps to address the robustness issues (word skipping or repeating) associated with the autoregressive TTS model. Therefore, we make explanations about more cases of robust speech synthesis, to demonstrate our method's necessity. In commercial scenarios, news reporting, and other formal scenarios, robustness a crucial factor. Just a few repeating or skipping words can have significant negative impacts. These situations better suited for models with duration models that ensure robustness, such as FastSpeech [4], Glow-TTS [5], and our Mega-TTS However, for models like tacotron [6], word omissions repetitions can significantly affect the listening experience. On
20

---

Published as a conference paper at ICLR 2024
Reconstuction Qualit
Cross entrapy Loss
0.402
0.375
4.2
PULN 4030
0.359
3.8
11
0.325
3.5
3.2
9259
0.255
50000
100000
150000
250000 300000 350203 Stops
200000
100000 250000
300000
405020
500000
Stops
(a)
(b)
Figure 7: Visualization of training loss curves with different lengths of context. In subfigure (a), "MRTE w/ 500 CF" means we set the number of the contextual mel-spectrogram frames for MRTE to 500. In subfigure (b), "P-LLM w/ 2000 CT" means we the number of the contextual VQ code tokens for P-LLM to 2000.
the other hand, in some scenarios, robustness is relatively less important. For example, occasional missing words repetitions in dialogue scenes also be natural.
L RESULTS WITH NOISY REFERENCE PROMPTS
To verify model's robustness against noisy reference prompts, conduct experiments LibriSpeech test-other set. The experimental setup for this experiment is consistent with the one described in Section 4.2 The results are shown in Table [1 It can be seen that Mega-TTS maintains excellent performance with noisy reference prompts.
Table 11: Results of zero-shot TTS on the LibriSpeech test-other set.
Model
WERL
SIM
DTW1
QMOS
SMOS
GT
3.17%
3,95
3.98
VALL-E-3s
6.21%
0.889
36.13
3.64
3,70aum1
Ours-3s
2.73%
0.904
32.50
3.77
3.80
21


Reviews:
[{"review_id": "SuIVOUvfT4", "reviewer": "Reviewer_pM8F", "summary": "The paper presents Mega-TTS, a novel framework for zero-shot text-to-speech (TTS) systems. The primary aim of Mega-TTS is to synthesize voices with unseen speech prompts, thereby reducing the data and computational requirements associated with voice cloning. The authors address two main challenges faced by existing zero-shot TTS systems: the lack of multi-sentence prompting strategies and the absence of specialized prompting mechanisms for prosodic information. By decomposing speech into content, timbre, and prosody, they propose a system that effectively handles long prompts and offers flexible control over prosodic styles. Experimental results suggest that Mega-TTS outperforms other state-of-the-art models in terms of speaker similarity and speech naturalness.", "questions": "- How does Mega-TTS handle non-English languages or different dialects? Exploring this could help assess the generalizability of the model across various linguistic contexts.\n- What are the specific computational requirements of Mega-TTS compared to other models? This information is crucial for understanding the trade-offs involved in using Mega-TTS.\n- How does the performance of Mega-TTS scale with the size of the training data? Understanding this can provide insights into how well the model might perform in scenarios with varying amounts of available data.\n- How does Mega-TTS handle non-standard speech patterns such as shouting, laughing, or other forms of emotional expression? This question could illuminate the model's ability to accurately capture and reproduce a wider range of human speech nuances.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The idea seems technically solid and well-motivated, and the demo audio examples clearly show the difference.\n\n- The authors introduce a novel approach to decompose speech into content, timbre, and prosody. This method allows for more effective handling of long prompts and provides greater control over prosodic styles. This is an innovative contribution that sets the groundwork for future research in this area.\n\n- Superior Performance: The paper presents experimental results showing that Mega-TTS outperforms other state-of-the-art models regarding speaker similarity and speech naturalness. This is a significant strength as it demonstrates the practical effectiveness of the proposed method.", "weaknesses": "Unclear Performance Across Languages: The experiments presented in the paper only use English datasets. Therefore, it's unclear how well the system performs with different languages or dialects. This limits the generalizability of the findings and may hinder the application of the system in diverse linguistic contexts. While the authors acknowledge some limitations of their approach, a more extensive exploration and testing of these constraints could have provided a more comprehensive understanding of the model. This additional analysis could guide future research addressing these limitations and further refining the Mega-TTS model.\n\nThe authors have not provided any information about the inference times of Mega-TTS compared to other models. This omission makes it difficult to evaluate the model's performance in real-world scenarios where speed may be as important as accuracy.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pyQ88raur8", "reviewer": "Reviewer_6vRj", "summary": "This paper introduces Mega-TTS, a zero-shot TTS framework designed to enhance multi-sentence prompts by decomposing them into timbre and prosody information. Mega-TTS utilizes an acoustic auto-encoder to independently encode prosody, content, and timbre information. The model integrates a multi-reference timbre encoder and a prosody latent language model (P-LLM) for efficient extraction of information from multi-sentence prompts. This design facilitates the generation of transferable and controllable prosody by leveraging probabilities derived from P-LLM outputs. The paper demonstrates that the synergy between the multi-reference timbre encoder and the prosody interpolation enabled by P-LLM results in fine-grained and controllable prosody transfer. The proposed outperforms Vall-e and the fine-tuning baseline when speech prompts ranging from 10 seconds to 5 minutes are used.", "questions": "- What would be the performance outcome if we generate the prosody latent for the target speaker using only the prosody information from another speaker, instead of using interpolation?\n- It is unclear in which dimension the concatenation is performed in the Multi-Reference Timbre Encoder. While Figure 1 seems to suggest that concatenation occurs along the hidden dimension axis, the description in Section 3.2 of the timbre-to-content attention module implies that it happens along the time axis.\n- In the comparison experiments with VALL-E, why was a different model chosen to measure speaker similarity (SIM) instead of using WavLM the one utilized [1]?\n\n[1] Wang, Chengyi, et al. \"Neural codec language models are zero-shot text to speech synthesizers.\" arXiv preprint arXiv:2301.02111 (2023).", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The proposed method adeptly combines the advantages of non-autoregressive (non-AR) modeling, such as robustness and controllability, with the powerful expressiveness of auto-regressive (AR) modeling, achieving this by decomposing speech into prosody and timbre using an information bottleneck.\n- The proposed model exhibits the capability to independently prompt prosody and timbre within a zero-shot setting.\n- The proposed method empirically shows improved zero-shot performance compared to fine-tuning approaches and outperforms existing state-of-the-art models, and the prosody transfer by using a prosody interpolation technique.", "weaknesses": "- While we understand the issue of having a limited number of available baselines, the absence of comparisons with NaturalSpeech2 and VoiceBox makes it challenging to ascertain the proposed model's superiority. At the very least, it appears necessary to replicate the experimental conditions described in the baseline papers and evaluate the proposed model, comparing its performance using the metrics reported in each paper, especially for prompts that are 3 seconds long. It raises the question: How does the proposed method perform with 3-second prompts?\n- In the case of datasets like LibriVox, which comprises audiobooks, it is commonly observed that there is not a significant variation in speaking style across different utterances by the same speaker. In this context, it becomes difficult to uphold assumption (1). As a result, the timbre encoder may capture prosody information as well, and the experiment does not conclusively demonstrate the complete separation of these two elements.\n- The description of the information bottleneck, a crucial component of the proposed method, is lacking in detail. Specifically, there is an absence of clear guidelines and processes for setting variables such as $r$, $d$, and the hidden dimensions to ensure an appropriate information bottleneck.\n- From the perspective of a practitioner, the prerequisite of using Montreal Forced Aligner (MFA) to extract alignments beforehand could be seen as a cumbersome step.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SUFraa9fmP", "reviewer": "Reviewer_azu9", "summary": "This paper proposes a novel zero-shot TTS framework that can effectively disentangle and control prosody and timbre with extremely long prompts. Specifically, a multi-reference timbre encoder is proposed to extract the timbre information, and a P-LLM is proposed to generate prosody with multiple reference context. A prosody transferring technique is proposed to control the generated prosody with context. Extensive experiments are done to show the superior performance of the proposed method.", "questions": "1. Is there a justification of the assumption in section 3.1 \"the mutual information between $y^t$ and $\\tilde{y}$ only contains timbre information\"? This assumption is not very obvious, since in the audiobook dataset, some performing skills may change the timbre largely in different sentences. Additionally, the average prosody style information can also be shared by different utterances.\n2. Is it possible to show the baseline methods with 300s context?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "1. The proposed TTS framework can separately control both timbre and prosody. Especially the zero-shot prosody control, which is one of the most challenging topic in TTS area.\n2. The proposed method can scale the in-context learning to very long prompts, like 300s, and the performance does not saturate when prompt is longer than 20s, which is promising.\n3. Although verifying the superiority of a zero-shot TTS system is hard given that most of them are closed-sourced, the authors reimplemented the baseline methods and do the comparison with controlled variables like parameter number, and training datasets.", "weaknesses": "1. Some of the zero-shot TTS categories are missing from the related works. One of the most related is the attention-base adaptation method. Early in year 2020, Attentron [1] is proposed, which can adapt to unseen speakers with multiple reference utterances, just like the MRTE in MegaTTS. Such strategy is also used in zero-shot voice conversion domain [2]. Later, methods like [3] introduces cross-attention based model and compress the reference into a fixed-length sequence before decoder attention, which is more close to the one in NaturalSpeech2.\n2. Some ablation studies are missing. Since the primary design of the training strategy is using multiple reference instead of only one utterance, it is important to show the performance difference between using long context vs short context during training to verify the necessity of this training strategy.\n\n[1] Choi, Seungwoo, et al. \"Attentron: Few-shot text-to-speech utilizing attention-based variable-length embedding.\" Interspeech 2020.\n\n[2] Lin, Yist Y., et al. \"Fragmentvc: Any-to-any voice conversion by end-to-end extracting and fusing fine-grained voice fragments with attention.\" ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2021.\n\n[3] Yin, Dacheng, et al. \"RetrieverTTS: Modeling decomposed factors for text-based speech insertion.\" Interspeech 2022", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yN1CZ6YnrW", "reviewer": "Reviewer_WuFJ", "summary": "This work proposed a zero-shot Text-to-speech model with a prosody large language model (p-LLM). They utilize a multi-reference timbre encoder to extract a timbre from multiple references. Moreover, they introduce an autoregressive duration predictor for prosody modeling. The results show that they can transfer the prosody and timbre respectively.", "questions": "1.\tI have a question about the Baseline model. This model might be a FastSpeech 2 with GAN training and style encoder of meta-stylespeech. Why do you fine-tune this model? I just wonder about the performance of zero-shot TTS of baseline. \n\n2.\tAre there any failure cases for the auto-regressive duration predictor? In my opinion, there are many components of this work. I think AR duration predictor is one of the important changes and it could be utilized for other TTS model easily so I hope you to analyze this predictor with additional ablation studies. Do not need to train the model additionally. I just suggest to infer the baseline TTS model with the predicted duration of your model. \n\n3.\tDoes the model synthesize the speech robustly with the noisy reference prompt? I would be better if you could add the results on test-other set. \n\nAlthough recently methods introduce codec-based speech synthesis, it is also important to utilize conventional acoustic representation such as Mel-spectrogram so I like the concepts of this paper adopting the language model for prosody modeling. However, I hope the authors include additional experiments\n\n1. Results on conditioning noisy reference prompts\n\n2. Comparisons with other prosody modeling methods such as Prosody-TTS and CLAPSpeech\n\n3. Comparison with word-level Prosody modeling (ProsoSpeech)", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "They organize the parallel TTS pipeline with autoregressive representation modeling. They only utilize a large language model in prosody modeling so they could enjoy the advantage of in-context learning of the LLM model and may prevent a problem from the autoregressive TTS model which could synthesize a speech with repeating and skipping. It would be better if the authors could explain more cases of robust speech synthesis without repeating and skipping..\n\nIn my opinion, autoregressive duration predictor could significantly improve the prosody transfer performance because duration influences prosody. It would be grate if you could add additional ablation study for autoregressive and non-autoregressive duration predictor.\n\nMeanwhile, an adversarial duration predictor was proposed in VITS2. This could improve the performance of duration modeling in this work.", "weaknesses": "1.\tI just wonder why the authors do not state ProsoSpeech which has the same structure as P-LLM without word-level prosody modeling. It would be better if the authors add the ablation study for phoneme and word-level prosody modeling. \n\n2.\tThe authors should have conducted more experiments on prosody modeling. There are recently prosody modeling works, Prosody-TTS and CLAPSpeech. Although recently large language model has been investigated, I hope that the author could add an additional experiment with them. All of the papers including this work might be from the same research group but they did not state anything. I hope the authors address this issue.\n\n3.\tMulti-reference Style Transfer methods were already utilized in many speech papers such as Attentron [S. Choi, 2020]. \n\n[S. Choi, 2020] S Choi, “Attentron: Few-Shot Text-to-Speech Utilizing Attention-Based Variable-Length Embedding,” Interspeech 2020", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)