PromptTTS 2: Describing and Generating Voices with Text Prompt
Authors: Yichong Leng
Zhifang Guo
Kai Shen
Zeqian Ju
Xu Tan
Eric Liu
Yufei Liu
Dongchao Yang
leying zhang
Kaitao Song
Lei He
Xiangyang Li
sheng zhao
Tao Qin
Jiang Bian
Venue: ICLR 2024 poster (2024)
Keywords: Text-to-Speech
Voice Generation
Prompt
Variation Network
Primary area: representation learning for computer vision, audio, language, and other modalities
TL;DR: We propose PromptTTS 2 to generate voices conditioned on the text description with 1) variation network to model voice details not covered by text descriptions and 2) prompt generation pipeline to automatically generate text description for speech.

Abstract: Speech conveys more information than text, as the same word can be uttered in various voices to convey diverse information. Compared to traditional text-to-speech (TTS) methods relying on speech prompts (reference speech) for voice variability, using text prompts (descriptions) is more user-friendly since speech prompts can be hard to find or may not exist at all. TTS approaches based on the text prompt face two main challenges: 1) the one-to-many problem, where not all details about voice variability can be described in the text prompt, and 2) the limited availability of text prompt datasets, where vendors and large cost of data labeling are required to write text prompts for speech. In this work, we introduce PromptTTS 2 to address these challenges with a variation network to provide variability information of voice not captured by text prompts, and a prompt generation pipeline to utilize the large language models (LLM) to compose high quality text prompts. Specifically, the variation network predicts the representation extracted from the reference speech (which contains full information about voice variability) based on the text prompt representation. For the prompt generation pipeline, it generates text prompts for speech with a speech language understanding model to recognize voice attributes (e.g., gender, speed) from speech and a large language model to formulate text prompts based on the recognition results. Experiments on a large-scale (44K hours) speech dataset demonstrate that compared to the previous works, PromptTTS 2 generates voices more consistent with text prompts and supports the sampling of diverse voice variability, thereby offering users more choices on voice generation. Additionally, the prompt generation pipeline produces high-quality text prompts, eliminating the large labeling cost. The demo page of PromptTTS 2 is available (https://speechresearch.github.io/prompttts2).

Published as a conference paper at ICLR 2024
PROMPTTTS 2: DESCRIBING AND GENERATING VOICES WITH TEXT PROMPT
Yichong Leng*1. Zhifang Guo", Kai Shen, Zeqian Ju, Xu Tan, Yanqing Liu, Yufei Liu Dongchao Yang, Leying Zhang, Kaitao Song, Lei He, Xiang-Yann Li, Sheng Zhao Tao Qin, Jiang Bian
MOE-Microsoft Key Laboratory of Multimedia Computing and Communication, University of Science and Technology of China Microsoft
ABSTRACT
Speech conveys more information than text, as the same word can be uttered in var- ious voices convey diverse information. Compared traditional text-to-speech (TTS) methods relying on speech prompts (reference speech) for voice variability, using text prompts (descriptions) is more user-friendly since speech prompts can be hard to find or may not exist at all. TTS approaches based on the text prompt face two main challenges: 1) the one-to-many problem, where not all details about voice variability can be described in the text prompt, and 2) the limited availability of text prompt datasets, where vendors and large cost of data labeling are required to write text prompts for speech. In this work, we introduce PrompiTTS 2 to address these challenges with variation network to provide variability information of voice not captured by text prompts, and prompt generation pipeline to utilize the large language models (LLM) to compose high quality text prompts. Specifically, the variation network predicts the representation extracted from the reference speech (which contains full information about voice variability) based on the text prompt representation. For the prompt generation pipeline, generates text prompts for speech with a speech language understanding model to recognize voice attributes (e.g., gender, speed) from speech and large language model to formulate text prompts based on the recognition results. Experiments on a large-scale (44K hours) speech dataset demonstrate that compared to the previous works, PromptTTS 2 generates voices more consistent with text prompts and supports the sampling of diverse voice variability, thereby offering users more choices on voice generation. Additionally, the prompt generation pipeline produces high-quality text prompts. eliminating the large labeling cost. The demo page of PromptTTS is available"
INTRODUCTION
In recent years, there have been significant advancements in text-to-speech (TTS) systems (Wang et al. 2017 Popoy et al. 2021b Chen et al. 2022b), which have resulted in enhanced intelligibility and naturalness of synthesized speech (Tan et al. 2021). Some TTS systems have achieved level of quality comparable to that of single-speaker recording (Tan et 2022), and large-scale TTS systems have been developed for multi-speaker scenarios (Wang et al. 20023] Le et al., 2023). Despite these improvements, modeling voice variability remains a challenge, as the same word can be delivered in various ways such as emotion and tone convey different information. Conventional TTS methods often rely on speaker information (e.g.. speaker ID) (Gibiansky et al., (017) or speech prompts (reference speech) (Casanova et al. 2022) to model the voice variability, which are not user-friendly, as the speaker ID is pre-defined and the suitable speech prompt is hard to find or even does not exist (in voice creation scenario). Given that natural language is a convenient interface for users to express
* 'Equal contribution. lycl23go mai..ustc.edu.coo zhifangguo9 gmail.com This work was conducted at Microsoft. Corresponding author: Xu Tan, xuta @ miim@@mocc...commm https://ppeechreeeachh. wtwphhhhhhhhh..oomptttitt

---

Published as a conference paper at ICLR 2024
their intentions voice generation, more promising direction for modeling voice variability is to employ text prompts (Guo et (003) Ramesh et al., [022]] Brown et al., 2020b) that describe voice characteristics. This approach enables easy voice creation through text prompt writing.
In general, TTS systems based on text prompts are trained with a text prompt dataset, consisting of speech and its corresponding text prompt. Voice is generated by model conditioned on the text content to be synthesized and the text prompt describing the variability or style of the voice. Two primary challenges persist in text prompt TTS systems:
One-to-Many Challenge: Speech contains voice variability in detail, making it impossible for text prompts to fully capture all characteristics in speech. So different speech samples can correspond to the same text prompt This one-to-many mapping increases the difficulty of TTS model training, leading to over-fitting or mode collapse. To the best of our knowledge, no mechanisms have been specifically designed to mitigate the one-to-many issue in TTS systems based on text prompts.
Dataset of text prompts describing the voice is hard to construct since the text prompt is rare on the internet. So venders are engaged to compose text prompts, which is both costly and laborious. Consequently, the text prompt datasets tend to be relatively small (approximately 20K sentences) (Guo et al. 2023) or not openly accessible (Yang et al. (023). posing an obstacle for the future research on text prompt based TTS systems.
Data-Scale Challenge:
To address the aforementioned challenges, in our work, we introduce PromptTTS 2 that proposes a variation network to model the voice variability information of speech not captured by the text prompts and utilizes a prompt generation pipeline generate high-quality text prompts:
For the one-to-many challenge, we propose a variation network to predict the missing information of voice variability from the text prompt. The variation network is trained with the help of reference speech, which is regarded to contain all information about voice variability (Wang et al. 2023 Shen al. 2023). Generally, the TTS model in PrompiTTS 2 consists of text prompt encoder for text prompts, reference speech encoder for reference speech, and a TTS module to synthesize speech based on the representations extracted by text prompt encoder and reference speech encoder. Variation network is trained to predict the reference representation from reference speech encoder based on the prompt representation from text prompt encoder? 3 By employing the diffusion model (Song et al.. 2020) in the variation network, we can sample different information about voice variability from Gaussian noise conditioned on text prompts to control the characteristics of synthesized speech, and thus offering users greater flexibility in generating voices and alleviating the one-to-many issue.
For the data-scale challenge, we propose a pipeline to automatically generate text prompts for speech with speech language understanding (SLU) model to recognize voice attributes (e.g.. gender, speed) from speech and large language model (LLM) to compose text prompts based on the recognition results. Specifically, we employ SLU model describe the voice from many attributes (e.g., emotion, gender) by recognizing the attribute values for each speech sample within a speech dataset. Subsequently, sentences are written describe each attribute individually, and the text prompt is constructed by combining these sentences. In contrast to previous work (Guo et al. 2023), which relies on vendors to write and combine sentences, PromptTTS 2 capitalizes on the capabilities of LLM (Brown et al.) 2020a Chowdhery et al., (022) that have demonstrated human-level performance in various tasks (Bubeck et al.] 20233 Touvron et al. (023). We instruct LLM to write high- qual-tualtty sentences describing the attributes and combine the sentences into comprehensive text prompt. This fully automated pipeline eliminates the need for human intervention in text prompt writing.
The contributions of this paper are summarized as follows:
We design a diffusion- based variation network to model the voice variability not covered by the text prompt, alleviating the one-to-many issue in text prompt based TTS systems. During inference, voice variability can be controlled by sampling from Gaussian noise conditioned on the text prompt.
We construct and release a text prompt dataset generated by LLM, equipped with a pipeline for text prompt generation. The pipeline produces high quality text prompts and reduces the reliance on vendors to write text prompts.
For instance, the text prompt "Please generate voice of a boy shouting out" can descrihe numerous shouting voices from boys that differ in details such as timbre. It is worth noting that reference speech is only used in training variation network but used in inference.
2

---

Published as a conference paper at ICLR 2024
We evaluate PromptTTS 2 on a large-scale speech dataset consisting of 44K hours speech data. Ex- perimental results demonstrate that PromptTTS 2 outperforms previous works in generating voices that correspond more accurately to the text prompt while supports controlling voice variability through sampling from Gaussian noise.
BACKGROUND
How to model voice variability has long been a crucial direction in text-to-speech (TTS) re- search (Wang et al. 2018: Bae al., 2020 Bak et al., 202T). In the early stage, TTS systems primarily focus on single-speaker scenarios (Wang et al.,
2017 Anık et al., 2017 Ren et al. 2019).
where voice information is implicitly incorporated into neural networks. Subsequently, the need for modeling diverse voices emerges, leading to the advancement of multi-speaker TTS systems (Gibian sky et al., 2017 Chen et al.] 2020; Popov et al., (221a). which voice variability is controlled but limited in speakers in the dataset. To adapt multi- -uuli--peakkr TTS systems to new speakers, few-shot adaptive TTS approaches (Chen et al., 2021; Yan et al. 2021 Huang et al. 2022) have been employed, which involve fine-tuning the multi-speaker TTS model on limited amount of target speaker data. In contrast, zero-shot adaptive TTS models utilize in in-context learning to generate new voices by exclusively modeling speaker characteristics from speech prompt (i.c., reference speech) (Wu et al.)
2022
Wang et al. 2023
Shen et al. 2023 Li et al. 2023 Le et al. 2023).
Since finding reference speech be cumbersome and the speech data of target speaker is hard to collect or even does not exist (in the voice creation scenario), above methods on modeling voice variability is not user-friendly and scenario-limited. To achieve voice generation and general manner, text prompt based methods have been proposed (Shimizu et al. 2023 Liu et al., 2023a), which create voices using text descriptions and require human-annotated text prompt datasets for speech. However, conttacccnttteeedd datasets are often limited in scale (Guo et al. 2023) or publicly inaccessible (Yang et al.) 2023) due to the associated costs. In this work, we propose pipeline that employs [.LM to generate text prompts, thereby reducing the reliance on human labor.
more natural
Given that is impossible to comprehensively deseribe speech with fine amee-griied details (Yang et al.) 2022; Qian et al. 2019 2020) using text prompts alone, there exists the one-to-many problem in the text prompt based TTS system. Different with previous works that try to construct text prompts with more details (Guo et al. 2023 Shimizu et al. (023), we propose the variation network to alleviate the one- one-to-menn to-many problem by predicting the missing information about voice variability conditioned on the text prompt with generative (diffusion) model.
3 PROMPTTTS 2
In this section, we firstly give an overview the TTS system in PromptTTS 2. Then introduce the variation network that predicts the missing information about voice variability in the text prompt. Finally, we describe our pipeline leverage the LLM to write the text prompt dataset.
3.1 OVERVIEW OF TTS SYSTEM
Figure T and Db present an overview of the TTS system in PrompiTTS 2. Figure depicts a TTS module for synthesizing speech, with its characteristics controlled by style module. Figure Th skips the details for TTS module because the TTS module can be any backbone capable of synthesizing speech from phonemes. We adopt TTS backbone from Shen et al. (2023). described in Appendix
Figure Tp illustrates the details of the style module. During training, in line with previous works (Guo al. 2023), we employ a BERT-based model as text prompt encoder to extract prompt hidden. To alleviate the one-to-many mapping problem (introduced in Section [T] we utilize reference speech encoder to model the information about voice variability not covered by the text prompt, which takes reference speech as input and outputs a reference hidden (Shen et al.) 2023 Wang et al. 2023). Since both the text prompt and reference speech can have varying lengths, we extract a fixed-length representation using
attention (Vaswani al. 2017) with fixed number of query tokens for both text prompt and reference speech. More specifically. the (text) prompt representations (Pı,... Pœr) extracted by learnable query tokens (Qr,,... Q Qpxp) and the reference (speech)
3

---

Published as a conference paper at ICLR 2024
Output Speech
Cross Attcation
TTS Backbone
FiLM
Phoneme Emhodding
Dropout + Add
TTS Module
Feed Forward
R
R
FILM
Variation Network
Dropout Add
R,
Multi Head Attention
Q-K-V Attention
Q-K-V Attention
k/V
k/v
RLM
Prompt Hlidlenn
Reference Hiddlen
Text Prompt Encoder
Reference Spezch Encoder
Encoding
Text Prompt
Reference Speech
RC
Step
(b) Style Module
Variation Network
The overview of TTS system in PromptTTS Subfigure (a) is TTS module to synthesize speech, whose characteristics are controlled by style module. Subfigure (b) shows the style module which takes the text prompt and reference speech as imput and extracts prompt representation (Pi..., PM)) and reference representation (I,,..., Rxx). Since the reference speech is not available in inference, we further propose a diffusion- based variation network (Subfigure (c)) to predict the reference representation based on the prompt representation.
Figure 1:
representations (R,, R extracted by learnable query tokens (Qn...... Quy). M and N represent the fixed lengths prompt and reference representations, respectively.
During inference, only the text prompt is available, and the reference speech is not accessible, so we train a variation network to predict the reference representation (Ri, Rx) conditioned on the prompt representation (P,,.., PM),, and thus the inference can be conducted with the text prompt only. The variation network is introduced in detail in the next section.
3.2 VARIATION NETWORK
The variation network aims to predict the reference representation (R1, Ry) conditioned on the prompt representation (PI....,PP) PMr). To model the reference representation, our variation network employs the diffusion model (Ho et al.) 2020). which has demonstrated robust capability in modeling multimodal distributions and complex data spaces (Kim et al. 2022 Ho et al.) 2022 Nichol & Dhariwal, 2021 Leng al. [022). The diffusion model also enables variation network to sample different voice variability from Gaussian noise. Specifically, the diffusion model consists of diffusion process and denoising process:
For the dissusion process, given the reference representation 20. the forward diffusion process transforms it into Gaussian noise under the noise schedule 3 as follows:
dæ 3 dt v B dwi,
te [0, 1].
(1)
For the denoising process, the denoising process aims to transform the noisy representation 2 the reference representation zo by the following formulation (Song et al. 2020):
dæ
(z+ (log dt. [0. 1).
(2)
Variation network trained to estimate the gradients of log- density of noisy data (V log Pe (2x) by predicting the origin reference representation 20 (Song al. [0200] (hen et al. (023), conditioned on the prompt representation, noised reference representation, and diffusion step that indicates the degree of noise in diffusion model.
4

---

Published as a conference paper at ICLR 2024
SLU Part
LLM Part
Ammibute
Sentence
Serience
Recognition
Attribute
Cemeration
Sentence
Comboration
Attribute
Sentenee
Speech
Text Prompt
Attribute N
Sentence
The overview of our prompt generation pipeline. We first recognize attributes from speech with the SLU model. Then LLM is instructed to generate sentences describing each attribute and combine the sentences of each attribute to formulate text prompts.
Figure 2:
Figure I presents the detailed architecture of variation network, which is based on the Transformer Encoder Vasswani et al. 2017). The input of variation network comprises the prompt representation (Pi...., Pas), noised reference representation
P(g), and diffusion step t. The output of variation network is the hidden representation corresponding to the noised reference representation.
optimized using L.1 loss. To enhance the model's awareness of the diffusion step, we use FiLM (Perez al. 2018) in each layer of the Transformer Encoder (Liu et al. 2023b).
In summary, during inference, we initially extract prompt representation from the text prompt using the style module. Subsequently, variation network predicts the reference representation conditioned on the prompt representation by denoising from Gaussian noise. Finally, the prompt representations are concatenated with the reference representation to guide the TTS module through cross attention.
3.3 TEXT PROMPT GENERATION WITH LLM
In this section, we introduce the prompt generation pipeline to build the text prompt dataset. As illustrated in Figure 2 the pipeline consists of SLU (speech language understanding) part and LLM (large language model) part. Given a speech, the SLU part involves tagging some labels with the speech language understanding models by recognizing attributes (e.g., gender, emotion, age) from speech; and the I.IM part involves instructing large language model to write text prompts based on the labels (i.e., recognition results).
As there exist many SLU models (Baevski et al., 2020; Arora et al. 2022) to recognize attributes from speech, we focus on the LLM part for the text prompt writing based the recognition results of SLU model. It is worth noting that text prompts written by LLM part can be reused for multiple speech with the same label step by step to compose text prompts with high diversity in vocabulary and sentence format. The detail about LLM part is shown in Figure 3 and introduced as follows:
order improve the quality of text prompts, the LLM is instructed
Keyword Construction The SLU models recognize attributes that can describe speech characteris- ties. For each attribute, the SLU model recognizes several classes representing the values of the attributes. Subsequently, LLM is instructed to generate several keywords describing each class for every attribute. In the stage of Figure 3 utilize four attributes, including gender, pitch, speed, and volume. The "gender" attribute comprises two classes: male and female. The keywords generated by LLM for the male class are "man". "me"" and so on.
Sentence Construction In addition the variance keywords, we also require variance in sentences. Therefore, we instruct LLM to generate multiple sentences for each attribute. A placeholder for the attribute used by L.M when composing these sentences (e.g., word "[Gender]" is the placeholder for "gender" attribute in the stage 2 of Figure 3 The design of the placeholder offers two advantages: 1) emphasizes the attribute for LLM, ensuring that the attribute is not omitted in the output sentence, and 2) the output sentence serves for an attribute, emabling the generation of diverse text prompts by filling the placeholder with different keywords. In the provided example, the stage 2 of Figure illustrates several sentences composed by LLM that describe different attributes.
general template for all classes
Sentence Combination Since text prompts can describe more than one attribute, we perform sentence combination based the sentences generated in the stage 2. L.IM is instructed to combine sentences describing different attributes into a new sentence, allowing us to obtain text
*Since the recognition results of SLUU models in pre-defined label
5

---

Published as a conference paper at ICLR 2024
Stage #1: Keyword Generation
Stage #2: Sentence Generation
LLM
LLM
Generate wonds to discribe speech
Generote sentenees to deserie speechs
[Ciender]
(Puch]
Plesse generatz voice tuat sounds like Genday
be, boy
deep,
I'm need of (Cender) voice. possible.
Femule: woman, shc, girl
Meliimm nuemal standanl...
[Spcail speed.
dilghh shorp, geble...
with [Spred] speaking
The spesier's
was (Volumes
[Spoed]
[Volume]
The voltme the spenier's voicr was [Volume))
How daww liisuee...
Lime soft quiel
The spenker uses (Picch piic..
Medium:; moderats,, repular Highht fixt, quick
(Medium: common, general.... Hlight land, sonemous..
This speech has a (Mich) pitch
require a voice with (Picc)) pitch level, thank you
Stage #3: Sentence Combination
Stage #4: Dataset Instantiation
Sentence
Could you generate a distisetly (ender)-sondingg ssanndnng voice for me with (Nolmee) velume lovel, (Pitch) pitch, and speekingg at Speed] pace?
Micoor give me a vorse that low (Pach) prihh and spolis on (Spoal) spood.
A voice with (olume) votume, (Picc) piihh, and [Speed] specking rate is needed.
Sooking voice with (Piach tune and [Volumae] sound.
Simple
Sampla
Sanel:
Keyword 1
LEM
Dacavet
Combine
phrase
Please give me a voice that has low pitch and speaks at fast speed Sasking a viice with lise time and lonal saund.
Text prompt generation using LLM: In Stage 1, LLM generates keywords for each attribute (gender, pitch, speed, and volume). In Stage 2, LLM composes sentences for each attribute, integrating placeholders for the corresponding attributes. In Stage 3, L.L combines the sentences from Stage 2 to create sentence that simultancously describes multiple attributes. In Stage 4, the dataset is instantiated by initially sampling a combined sentence and subsequently sampling keywords to replace the placeholders within the sentence.
Figure 3:
prompts representing various combinations of attributes. It is worth noting that the sentences generated by LLM are always complete and free of grammatical errors. In contrast, users of text prompt based TTS systems may not always describe voices in formal manner. Consequently, also instruct LLM to write phrases to enhance the diversity of constructed sentences. In the stage 3 we present some example combination sentences and phrases generated by LLM.
of Figure 3
Dataset Instantiation The results generated from the previously described three stages form the final text prompt dataset, which is employed alongside a speech dataset. For each instance of speech S within the speech dataset, we tag a class label on every attribute with SLU models. Following this, we select sentence that encompasses all the attributes of speech S. Next, we obtain keyword for each attribute of speech S based on its corresponding class label. The ultimate text prompt is instantiated by substituting all placeholders in the sentence with their corresponding keywords. In the stage 4 of Figure 3 we provide examples illustrating the finalized text prompts. The speech S and the corresponding finalized text prompt formulate speech- proompt paired data.
We provide an example of our pipeline in Appendix A which shows the dialogue process with LLM. More discussion about the scalability of our pipeline can be found in Appendix B
EXPERIMENT CONFIGURATION
In this section, we present the experimental configurations, including the datasets, TTS backbone, baseline systems and experiment details.
Datasets For the speech dataset, we employ the English subset of the Multilingual LibriSpeech (MLS) dataset (Pratap et al. 2020), which comprises 44K hours of transcribed speech data from LibriVox audiobooks. For the text prompt data, we utilize PromptSpeech (Guo et al.) 2023) that contains 20K text prompts written by human describing speech from four attributes including pitch, gender, volume, and speed. We also utilize our prompt generation pipeline to write 20K text prompts with the help of LLM 1GPT-3.5-TURBO contains 1305 text prompts. For the SLU model attribute recognition, we identify gender using an
The test set of PromptSpeech is used as test data, which
6

---

Published as a conference paper at ICLR 2024
Table 1:
The accuracy (%) of synthesized speech on the attribute control of PromptTTS 2 and baselines.
Model
Gender
Speed
Volume
Pitch
Mean
PrompiTTS
Guo et al.
2023
98.01
89.66
92,49
85.98 86.82
91.54
InstructTTS
Yang 2023
97.24
90.57
91.26
91.47
PromptTTS 2
98.23
92.64
92.56
89.89
93.33
open-source model and the other attributes (i.e., pitch, volume, and speed) are recognized using digital signal processing tools
TTS Backhone In general, PrompiTT' extracts fixed-dimension representation control the characteristics of synthesized speech. This approach can be incorporated into any TTS backbone by integrating the representations into the TTS backbone with cross attention. We adopt TTS backbone from SOTA TTS system, NaturalSpeech 2 (Shen et al. 2023), whose details are in Appendix
Baseline Systems We compare PromptTTSS 2 with current SOTA systems of text prompt based TTS, PrompiTTS (Guo al., 2023) and InstrucITTS (Yang et al. 2023). To ensure fair comparison, we modify the backbone in baseline systems to the latent diffusion backbone used in PromptTTS 2.
Experiment Details The number of layers in the reference speech encoder and variation network is 6 and 12, respectively, with a hidden size of 512. The query number M, N in style module is both set to 8. Concerning the TTS backbone and the text prompt encoder, NaturalSpeech 2 (Shen et al. (2023) and PrompiTTS (Guo et al. (002)). respectively. The training configuration also derived from NaturalSpeech 2 (Shen al. 2023
adhere to the settings in
RESULT
In this section, we evaluate the effectiveness of PromptTTS 2. Firstly, We compare the accuracy of attribute control and the speech quality between PromprTTS and baseline systems in Section 1 In Section 5.2 demonstrate that the variation network successfully captures the information about voice variability. In Section 5.3 we compare the text prompts generated by our pipeline with those written by human or other LLM based method. Finally, conduct an analysis the style module in Section 5.4 and perform an extension on face-to-voice (Face2 Voice) generation in Section 5.5
5.1 EFFECTIVENESS OF PROMPTITS 2
We evaluate the effectiveness of PromptTTS 2 from the perspective of attribute control and speech quality. First, we compare the accuracy of attribute control between PrompiTTS 2 and baseline systems, presented in Table I The results demonstrate that PrompiTTS can synthesize speech with higher accuracy across all attributes compared to baseline systems, achieving an average improvement of 1.79%. In Table [ the experiments use the text prompts from our pipeline, and more results on different text prompts comparative MOS (CMOS) test to evaluate the speech quality of PromprTTS 2 and baseline systems, as shown in Table The results of MOS and CMOS show that PrompiTTS achieves higher speech quality than the baseline systems.
be found in Appendix D Then conduct mean-of-score (MOS) and
5.2 STUDY OF VARIATION NETWORK
We examine the information of voice variability learned by variation network. Due to the one-to-many problem between the text prompt and the voice variability in speech, the model might implicitly incorporate voice variability information into specific aspects. Consequently, the model could synthesize varying voices even when presented with identical text prompts (or text prompts with equivalent meanings). For the baseline systems, PromptTTS and InstructTTSS, these aspects include
http:://gghhn..oooo//mmmmyy//C/y ChaapeeaCaCCayyaaaaareeeeeoooooo Wrapper-for forldoor---WW--o-- Vocoder World-Voeoder
7

---

Published as a conference paper at ICLR 2024
Table 2: The results of speech quality with 95% confidence intervals. GT stands for the recording.
Codec reconstruction stands for that the waveform is encoded to latent representation first and then reversed waveform by the decoder of codec.
Setting
MOS
CMOS (vs. PromptTTS 2)
GT GT (Codec Reconstruction)
4.38 + 0.08
4.30 +: 0.07
PromptTTS
Guo al.
2023
3.77 + 0.09
-0.191
InstructTTS
Yang et al. 2023
3.80 + 0.07
-0.157
PrompITTS 2
3.88 + 0.08
0.0
Table 3: The average speech similarity of systems when synthesizing speech with the same intention in text prompts but different text prompts, text contents, sampling results of TTS backbone and sampling results of variation network. The similarity score is in range of [0, 1J.
Model
Text Prompt
Text Content
TTS Backbone
Variation Network
PromptTTS
0.766
0.662
0.799
Instruc(TTS 0.773
0.718
0.796
PromptTTS
0.775
0.873
0,914
0.355
the text prompt (with the same meaning), text content, and TTS backbone (with latent diffusion), as the voice of synthesized speech may differ depending on the text prompt, text content, and TTS backbone. In PromptTTS 2, additional aspect, variation network, is introduced, as the voice of synthesized speech may also vary based on different sampling results of the variation network.
We use WavLM-TDNN model (Chen et al. 2022a) assess the similarity of two speech in range of [0, 1J. where the higher speech similarity, the less voice variability. For each aspect mentioned above, generate speech and calculate the average similarity of the speech. The results are shown in Table From the table, we have the following observation: 1) baseline systems implicitly acquire small amount of voice variability information in the aspect of the text prompt, text content, and TTS backbone, which is undesired as we aim for style to be controlled exclusively by the intention in markedly lower than
text prompt: 2) the speech similarity of variation network in PromptTTS other aspects, showing that the variation network effectively models voice variability information not encompassed by the text prompts (i.e., different sampling results leads to different timbre); 3) for PromprTTS 2, the voice variability acquired in aspects apart from variation network is less than those of baseline systems whose similarity are higher. This indicates that when the variation network successfully captures voice variability, the model is inclined to learn less voice variability information in other aspects. We strongly encourage readers to listen to the samples on our demo page, which offer an intuitive comprehension of the voice variability information present in each dimension.
Besides the WayLM-TDNN model, we evaluate the speech similarity by human experts. The conclusions of subjective test similar with those of WavL.M-TDNN model, shown in Appendix E
5.3 PROMPT GENERATION QUALITY
We analyze the quality of text prompts generated by our pipeline through whether the text prompts can reflect the values of attributes. Specifically, we train a classifier to recognize the intention of text prompts on four attributes. The training data for the classifier is text prompts authored by human (i.e.. the training set of PromptSpeech (Guo et al. (023). 2) TextrolSpeech (Ji et al. 2023) whose text prompts are written by LLM (GPT-3.5-TURBOO with multi-stage prompt programming approach (but without the placeholder sentence combination mechanism in our pipeline), 3) text prompts written by our pipeline. We display the average accuracy of classification on the test set of PromptSpeech in Table The classifier trained on text prompts generated by our pipeline has a higher accuracy compared to the classifier trained on text prompts authored by human or TextrolSpeech. This result indicates that the text prompts generated by our pipeline exhibit higher quality than previous works, verifying the effectiveness of our prompt generation pipeline. More ablation studies on our prompt generation pipeline can be found in Appendix F
8

---

Published as a conference paper at ICLR 2024
Table 4: The accuracy (%) of intention classification four attributes with text prompts from PromptSpeech, TextrolSpeech, and our prompt generation pipeline.
Training Set
Gender
Speed
Volume
Pitch
Mean
PromptSpeech (Guo et al. 2023 TextrolSpeech (Ji et al.
100.00
96.85
89.58
84.51
92.74
2023
98.77
94.18
93.10
92.80
94.71
Our Prompt Generation Pipeline
99.08
97.47
94.48
94.48
96.38
Figure 4: The PCA results of the representation extracted by the reference speech encoder in style module. Each point stands for a speech and the speech with the same speaker (left figure) or the same same emotion (right figure) has the same color.
5.4 FURTHER ANALYSIS
In this section, we study the reference representation extracted from reference speech encoder in style module, which is a high-dimensional vector. To visualize the vector, we employed Principal Component Analysis (PCA) to reduce the dimensionality of the vector and map
2D vector, which is plotted in Figure 4 Each point in figure stands for a speech and the speech with the same speaker the same emotion Zhou et al. 202 2022 has the same color. We observe that the speech samples belonging to the same speaker or the same emotion tend to cluster together in the figure. This observation suggests that the reference representations effectively learn the voice variability uncovered by text prompts (such as speaker emotion). Therefore, given a text prompt, the variation network can sample different voice variability corresponding to the text prompt, which offers users more flexibility generating voices. More ablations on PrompiTTS are in Appendix G and H
5.5 EXTENSION ON FACE2VOICE
PromptTT'S 2 involves modeling voice information utilizing sequence of predictable tokens, enabling its extension many other scenarios involving predicting voices from other modalities. We conduct a preliminary experiment on the Face2 extension, with a objective of predicting voices based on speaker's facial images. More details about Face2Voice extension can be found in Appendix [ which shows that PromptTTS 2 generates voices corresponding more closely to the facial images compared with the baseline method (Weng al. 2023). Furthermore, our findings show that PrompiTTS is general method for generating voices conditioned on text prompts, facial images, or other information. Samples of facial images and generated voices can also be found on our demo page.
CONCLUSION
In this work, we propose PromptTTS& 2 to address the one-to-many and data-scale issues in text prompt based TTS systems, which implements a variation network to model the voice variability information not captured by text prompts and uses LLM for high-quality text prompt generation. The variation network facilitates more detailed voice control by sampling from Gaussian noise. The --seed prompt generation pipeline eliminates the reliance on vendors and provides scalability for easily incorporating new attributes. Experimental results indicate that the proposed variation network assists the TTS model in synthesizing speech more closely corresponding to the text prompt and diverse in voice variability. Our pipeline generates text prompts with higher quality than human--uthonne ones. For future work, we plan to extract additional attributes from large-scale speech data to increase the diversity of voice generation system and apply our method more modalities for voice generation.
9

---

Published as a conference paper at ICLR 2024
REFERENCES
Sercan Ô Arık, Mike Chrzanowski, Adam Coates, Gregory Diamos, Andrew Gibiansky, Yongguo Kang, Xian Li, John Miller, Andrew Ng. Jonathan Raiman, al. Deep voice: Real-time neural text-to-speech. In International conference on machine learning, pp. 195-204, PMLR, 2017.
Siddhant Arora, Siddharth Dalmia, Pavel Denisov, Xuankai Chang, Yushi Ueda, Yifan Peng. Yuekai Zhang, Sujay Kumar, Karthik Ganesan, Brian Yan, et al. Espnet-slu: Advancing spoken language understanding through espnet. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 7 7167-71711 IEEE, 2022.
Jac-Sung Bae, Hanbin Bac, Young-Sun Joo, Jummo Lee, Gyeong- Hoon Lee, et al. Speaking speed control of end-to-end speech synthesis using sentence-level conditioning. 2020.
Alexei Baevski, Henry Zhou, AbdeIrahman Mohamed, and Michael Auli. wav2vec 2.0: A framework for sepf-suue learning of speech representations, 2020.
Taejun Bak, Jae-Sung Bae, Hanbin Bae, Young-Ik Kim, and Hoon-Young Cho. Fastpitchformant: Source-filter based decomposed modeling for speech synthesis. In Conference of the International Speech Communication Association (Interspeech), 2021.
Zalán Borsos, Matt Sharifi, Damien Vincent, Eugene Kharitonov, Neil Zeghidour, and Marco Tagliasacchi. Soundstorm: Efficient parallel audio generation. arXiv preprint A31v:23005096636 2023.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020a
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Dhariwal, et al. Language models are few-shot learners. In Conference and Workshop on Neural Information Processing Systems (NIPS), volume 33, pp. 1877-1901, 2020b.
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Eee Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303. /2711, 2023.
Edresson Casanova, Julian Weber, Christopher Shulby, Arnaldo Candido Junior, Eren Gölge, and Moacir A Ponti. Yourtts: Towards zero-shot multi-speaker tts and zero-shot voice conversion for everyone. In International Conference on Machine Learning, pp. 2709-2720 PMLR, 2022.
Mingjian Chen, Xu Tan, Yi Ren, Jin Xu, Hao Sun, Sheng Zhao, Tao Qin, and Tie- Yan Liu. Multi- speech: Multi-speaker text to speech with transformer. arXiv preprint arXiv:2006. 04664, 2020.
Mingjian Chen, Xu Tan, Bohan Li, Yanqing Liu, Tao Qin, Sheng Zhao, and Tie-Yan Liu. Adaspeech: Adaptive text to speech for custom voice. ICLR, 2021.
Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, et al. Wavlm: Large-scale self--suprvised pre-training for full stack speech processing. IEEE Journal of Selected Topics in Signal Processing, 16(6):
1518, 2022a.
Zehua Chen, Yihan Wu. Yichong Leng, Jiawei Chen, Haohe Liu, Xu Tan, Yang Cui, Ke Wang. Lei He, Sheng Zhao, et al. Resgrad: Residual denoising diffusion probabilistic models for text to speech. arXiv preprint 12Xiv:2212.14:: 14518, 2022b.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung. Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.023111 2022.
Andrew Gibiansky, Sercan Arik, Gregory Diamos, John Miller, Kainan Peng, Wei Ping, Jonathan Raiman, and Yanqi Zhou. Deep voice 2: Multi-speakerr neural cext-to-speech. Advances in neural information processing systems, 30, 2017.
10

---

Published as a conference paper at ICLR 2024
Zhifang Guo, Yichong Leng. Yihan Wu, Sheng Zhao, and Xu Tan. Prompttts: Controllable text-to- speech with text descriptions. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2023.
J. Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. In NeurIPS, 2020.
J. Ho, C. Saharia, W. Chan, Fleet D. J, M. Norouzi, and T. Salimans. Cascaded diffusion models for high fidelity image generation. Journal of Machine Learning Research, 23(47): 2022.
Sung-Feng Huang, Chyi-Jiunn Lin, Da-Rong Liu, Yi-Chen Chen, and Hung-yi Lee. Meta-tts: Meta- learning for few-shot speaker adaptive rext-to-speech. 1EEE/ACM Transactions on Audio, Speech, and Language Processing, 30: 10511558111 1571, 2022.
Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openclip, July 2021. If you use this software, please cite it as below.
Shengpeng Ji, Jialong Zuo, Minghui Fang, Ziyue Jiang, Feiyang Chen, Xinyu Duan, Baoxing Huai, and Zhou Zhao, Textrolspecch: A text style control speech corpus with codec language text-to-speech models. arXiv preprint arXiv:2308.144: 2023.
G. Kim, T. Kwon, and J. C. Ye. Diffusionclip: Text-guided diffusion models for robust image manipulation. In CVPR, 2022.
Matthew Le, Apoorv Vyas, Bowen Shi, Brian Karrer, Leda Sari, Rashel Moritz, Mary Williamson, Vimal Manohar, Yossi Adi, Jay Mahadeokar, et al. Voicebox: Text-guided multilingual universal speech generation at scale. arXiv preprint arXiv:2306. 15687, 2023.
Yichong Leng, Zehua Chen, Junliang Guo, Haohe Liu, Jiawei Chen, Xu Tan, Danilo Mandic, Lei He, Xiangyang Li, Tao Qin, et al. Binauralgrad: two-stage conditional diffusion probabilistic model for binaural audio synthesis. Advances in Neural Information Processing Systems, 35: 23689-23700, 2022.
Yinghao Aaron Li, Cong Han, Vinay Raghavan, Gavin Mischler, and Nima Mesgarani. Styletts 2: Towards human-level text-to-speech through style diffusion and adversarial training with large speech language models. arXiv preprint arXhv:2306.0661, 2023.
Guanghou Liu, Yongmao Zhang, Yi Lei, Yunlin Chen, Rui Wang, Zhifei Li, and Lei Xic. Promptstyle: Controllable style transfer for text-to-speech with natural language descriptions. arXiv preprint arXiv:2305. 19522, 2023a.
Haohe Liu, Zehua Chen, Yuan, Xinhao Mei, Xubo Liu, Danilo Mandic, Wenwu Wang, and Mark D Plumbley. Audioldm: Text-to-audio generation with latent diffusion models. arXiv preprint arXiv:2301.12500 2023b.
A. Nichol and Dhariwal. Improved denoising diffusion probabilistic models. In ICML, 2021.
Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reasoning with general conditioning layer. In Proceedings of the AAAI Conference Artificial Intelligence, volume 32, 2018.
V. Popov, I. Vovk, Gogoryan. T. Sadekova, and M.A. Kudinov. Grad-tts: A diffusion probabilistic model for text-to-speech. In ICML, 2021a.
Vadim Popov, Ivan Vovk, Vladimir Gogoryan, Tasnima Sadekova, and Mikhail Kudinov. Grad-tts: A diffusion probabilistic model for text-to-speech. In International Conference on Machine Learning, pp. 8599-86088 PMLR, 2021b.
Vineel Pratap, Qiantong Xu, Anuroop Sriram, Gabriel Synnaeve, and Ronan Collobert. Mls: A large-scale multilingual dataset for speech research. arXiv preprint arXiv:20/2.03411, 2020.
Kaizhi Qian, Yang Zhang, Shiyu Chang, Xuesong Yang, and Mark Josnsgawa-Jhnnon. Autove: Zero-shot voice style transfer with only autoencoder loss. In International Conference on Machine Learning, pp. 5210-5219. PMLR, 2019.
11

---

Published as a conference paper at ICLR 2024
Kaizhi Qian, Yang Zhang, Shiyu Chang, Mark Hasegawa-Johnson, and David Cox. Unsupervised speech decomposition via triple information bottleneck. In International Conference on Machine Learning, pp. 7836-7846. PMLR, 2020.
Alec Radford, Jong Wook Kim, Chris Hallacy, A. Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICML, 2021.
A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint 220Xi:22204001225, 2022.
Yi Ren, Yangjun Ruan, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie-Yan Liu. Fastspeech: Fast, robust and controllable text to speech. Advances in neural information processing systems, 32, 2019.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade W Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa R Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev. LAION-5b: An open large-scale dataset for training next generation image-text models. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022.
Kai Shen, Zeqian Ju, Xu Tan, Yanqing Liu, Yichong Leng, Lei He, Tao Qin, Sheng Zhao, and Jiang Bian. Naturalspeech 2: Latent diffusion models are natural and zero-shot speech and singing synthesizers. arXiv preprint arXiv:230.009166 2023.
Reo Shimizu, Ryuichi Yamamoto, Masaya Kawamura, Yuma Shirahata, Hironori Doi, Tatsuya Komatsu, and Kentaro Tachibana. Promptts++:: Controlling speaker identity in prompt-based text-to-speech using natural language descriptions, 2023.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456, 2020.
Xu Tan, Tao Qin, Frank Soong, and Tie- Tin-Yan Liu. A survey on neural speech synthesis. arXiv preprint Stdi::2/06.156611 2021.
Xu Tan, Jiawei Chen, Haohe Liu, Jian Cong, Chen Zhang, Yanqing Liu, Xi Wang. Yichong Leng, Yuanhao Yi, Lei He, et al. Naturalspeech: End-to-end text to speech synthesis with human-level quality. arXiv preprint 20Xiv:2205.04421, 2022.
Hugo Touvron, Thibaut Lavril, Gautier Izacard. Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv: 13971, 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurlPS, 2017.
Chengyi Wang. Sanyuan Chen, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, et al. Neural codec language models are zero-shot text to speech synthesizers. arXiv preprint arXiv:2301.021 11, 2023.
Yuxuan Wang, RJ Skerry- Ryan, Daisy Stanton, Yonghui Wu, Ron J Weiss, Navdeep Jaitly, Zongheng Yang, Ying Xiao, Zhifeng Chen, Samy Bengio, et al. Tacotron: Towards end-to-end speech synthesis. arXiv preprint arXiv: / 7703. 10135, 2017.
Yuxuan Wang, Daisy Stanton, Yu Zhang, RJ Skerry-Ryan Eric Battenberg, et al. Style tokens: Unsupervised style modeling. control and transfer in end-to-end speech synthesis. In International Conference on Machine Learning (ICML), 2018.
Shao-En Weng, Hong- Han Shuai, and Wen- Cheng. Zero-shot face-based voice conversion: bottleneck-free free speech disentanglement in the real-world scenario. In Proceedings of the AAAI Conference Artificial Intelligence, volume 37, pp. 13716- 2023.
12

---

Published as a conference paper at ICLR 2024
Yihan Wu, Xu Tan, Bohan Li, Lei He, Sheng Zhao, Ruihua Song, Tao Qin, and Tie- Yan Liu. Adaspeech 4: Adaptive text to speech in zero-shot scenarios. arXiv preprint 2rX1::2204.04336, 2022.
Yuzi Yan, Xu Tan, Bohan Li, Tao Qin, Sheng Zhao, Yuan Shen, and Tin-Yan Yan Liu. Adaspeech 2: Adaptive text to speech with untranscribed data. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6613-6617. IEEE, 2021.
Donghhao Yang, Songxiang Liu, Rongjie Huang, Guangzhi Lei, Chao Weng, Helen Meng, and Dong Yu. Instructtts: Modelling expressive tts in discrete latent space with natural language style prompt. arXiv preprint 2rXiv:2301. 2023.
SiCheng Yang, Methawee Tantrawenith, Haolin Zhuang, Zhiyong Wu, Aolan Sun, Jianzong Wang, Ning Cheng, Huaizhen Tang. Xintao Zhao, Jie Wang, et al. Speech representation disentanglement with adversarial mutual information learning for one-shot voice conversion. arXiv preprint arXiv:2208.088 08755 2022.
Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. Sound- stream: An end-to-end neural audio codec. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 30:495-507, 2021.
Zhimeng Zhang, Lincheng Li, Yu Ding, and Changjie Fan. Flow-guided one-shot talking face generation with high-resolution audio-visual dataset. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3661-3670, 2021.
Kun Zhou, Berrak Sisman, Rui Liu, and Haizhou Li. Seen and unseen emotional style transfer for voice conversion with a new emotional speech dataset. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 920-924, IEEE, 2021.
Kun Zhou, Berrak Sisman, Rui Liu, and Haizhou Li. Emotional voice conversion: Theory, databases and esd. Speech Communication, 137:1-18, 2022.
13

---

Published as a conference paper at ICLR 2024
A EXAMPLE OF PROMPT GENERATION PIPELINE
Table shows a detailed example of our prompt generation pipeline including the queries to instruct LLM. In Stage 1, we ask LLM to generate keywords for "male gender". In Stage 2, ask LLM to write sentences describing "pitch" attribute, using "[placeholder]" to stand for the corresponding attribute (i.e., "pitch"). In Stage 3, ask LLM to combine two sentences describing "gender" attribute and "pitch" attribute into one sentence. In Stage 4, we construct text prompt by first sampling a sentence and subsequently sampling keywords to replace the placeholders within the sentence.
Table 5: The example of text prompt generated by LLM with the pipeline.
Stage
Dialogue
Query: Can you list 5 words or phrases that is the synonyms for male gender in lower case? Please generate one word per line and identify it with serial number.
Stage
1. man.
2. boy.
Response: 3, guy.
4. dude.
5. gentleman.
Query: Please generate 5 templates to to ask for generating a voice. These templates can only describe the pitch of the voice and use [placeholder] to indicate where word describe pitch would be inserted. This template should be simple enough to only have a few words.
Stage 2
need [placeholder] voice.
2. Looking for [placcholder] voice.
Response: 3. In search of a [placeholder] voice.
4. Seeking voice with [placeholder] pitch.
5. Require voice, [placeholder] pitch.
Query: I have some sentences, can you combine these sentences into one phrase and keep all of the '[I' to describe the style of speech with the same meaning? You can generate 5 different phrases and callout number. Please reduce to phrases. The sentences are: Please generate a voice that sounds like a [Gender]. Seeking voice with [Pitch] pitch.
Stage 3
1. Generate a voice that sounds like a [Gender] with [Pitch] pitch.
2. Looking for voice that embodies [Gender] with [Pitch] pitch.
Response: 3. Require voice with [Pitch] pitch that represents [Gender].
4. Requesting [Gender] voice at [Pitch] pitch.
5. Desiring [Gender] voice with [Pitch] pitch.
Stage 4 Generate voice that sounds like boy with low pitch.
B SCALABILITY OF PROMPT GENERATION PIPELINE
We present a brief discussion on the scalability of our prompt generation pipeline. With the help of our pipeline, incorporating a new attribute requires only the definition of classes for the new attribute and the tagging of the speech dataset for that attribute using SL.U model (Baevski et al.] 2020 Arora al. 2022). For example, if we intend to introduce a new "age" attribute into the pipeline, we can define three classes corresponding to the "age" attribute, namely "teenager", "adult" and "elder". Subsequently, the pipeline can generate a text prompt dataset for the "age" attribute with the help of LLM and SLU model on "age" attribute to tag the speech dataset. In summary, our pipeline simplifies the process of adding new attributes, allowing for easier expansion and adaptability to diverse speech characteristics.
C DETAILS ON THE TTS BACKBONE
The TTS backbone of PromptTTS 2 is adopted from state-of-the-arr large-scale TTS system, NaturalSpeech (Shen et al. 2023), which consists of 1) a neural audio codec that transforms the audio waveform into latent vectors and reconstructs the latent representation into the waveform, and
14

---

Published as a conference paper at ICLR 2024
Table 6: The accuracy (%) of synthesized speech the attribute control of PromptTTS 2 and baselines when using the text prompt from PromptSpeech (Guo et al. 2023
Model
Gender
Speed
Volume
Pitch
Mean
PromptTTS (uo et al.
2023
98.93
87.43
89.35
85.44
90.29
InstrucITTS
Yang et al.
2023
96.55
86.13
88.43
85.52
89.16
PromptTTS
98.77
90.80
90.57
89.58
92.43
Table 7: In human subjective test, the average speech similarity (%) of baseline systems and
PrompiTTS when synthesizing speech with the same intention in text prompts but different text prompts, text contents, sampling results of 'TS backbone and sampling results variation network.
Model
Text Prompt
Text Content
TTS Backbone
Variation Network
PromptTTS
94.44
79.63
96.30
Instruc(TTS
92.59
85.18
94.44
PromptTTS2 90.74
98.00
98.15
7.41
2) a latent diffusion model with a prior (a duration/pitch predictor and a phoneme encoder). In detail, first encode the audio waveform into latent representation using the residual vector-quantizer (RVQ) (Zeghidour et al., 2.22). Then, the latent diffusion denoises (synthesizes) the latent speech representation from Gaussian noise. The denoised latent representation is subsequently converted back to the waveform by the decoder of the neural audio codec.
D ABLATION ON TTS SYSTEMS WITH DIFFERENT TEXT PROMPT
Since the ablation study on the text prompt shows the superiority of the text prompts from our prompt generation pipeline over those in other baseline methods (as reported in T)ble the results in Table are conducted when all the models using the text data from our prompt generation pipeline. Thus, the results are a fair comparison in terms of text prompts.
To strengthen the conclusions of our paper, we conducted additional experiments on all models (i.e., PromptTTS. InstructTTS, and PromptTTS 2) using the text data in PromptSpeech (Guo et al., 2023).
The results are shown in Table
From the table, we observe that PromptTTS 2 outperforms baseline methods in average on Prompt- Speech text prompt datasets. By further taking the results in using text data from our pipelines improves the quality of all text prompt based TTS models on most of attributes, compared to using the prompts in PromptSpeech.
Table into consideration, we find that
E SUBJECTIVE TEST ON THE VOICE VARIABILITY IN VARIATION NETWORK
Besides the metric by WavLM-TDNN model, we also evaluate the speech similarity from the perspective of human. For each aspect mentioned in Section 5.2 we generate speech and calculate the average similarity of the 5 speech. In human subjective test, the judges are asked judge whether the two synthesized speech are the same style. The speech similarity of each aspect defined as the ratio of speech pair (among the 5 speech) that is regarded in the same style by judges. The conclusions of subjective test (Table 7 are similar with those of WavLM-TDNN model discussed in Section
F ABLATION STUDY ON PROMPT GENERATION PIPELINE
We conduct ablation studies the prompt generation pipeline. First, remove the design of the placeholder from the pipeline. In this case, LLM is required to directly write text prompts for each class in attributes, after which sentence combination is performed. The results are presented as Placeholder" Table g The drop in classification accuracy demonstrates that the placeholder is beneficial for the prompt generation pipeline. Without it, L.MM might miss attributes or
15

---

Published as a conference paper at ICLR 2024
Table 8: The accuracy (%) of intention classification on four attributes in the ablation of our prompt generation pipeline.
Datasets
Gender
Speed
Volume
Pitch
Mean
Our Prompt Generation Pipeline
99.08
97.47
94.48
94.48
96.38
Placeholder
99.08
97.31
89.27
90.50
94.04
Phrase
99.08
97.01
95.55
92.72
96.09
Sentence
99.08
97.47
93.18
94.94
96.16
Table 9: The accuracy (%) of synthesized speech the attribute control of PromptTTS 2 and baselines when using the TTS backbone from SoundStorm [Borsos et al. (023)).
Model
Gender
Speed
Volume
Pitch
Mean
PromptTTS (uoo al.. 2023
97.47
89.96
92,41
85.06
91.23
InstructTTS
Yang et al.
2023
96.55
89.66
92.18
84.14
90.63
PromptTTS
98.92
92.18
93.93
90.73
93.95
alter them during sentence combination, resulting in low-quality text prompts. In addition to the placeholder, we also conduct ablation studies on instructing LLM to write only phrases or sentences by removing sentences ("- Sentence") or phrases ("- Phrase"). The results indicate that variations in format can marginally improve the robustness of the prompt generation pipeline.
G ABLATION ON TTS BACKBONE
Besides the TTS backbone based on latent diffusion (Shen et al. 2023). we further apply PromptTTS 2 (as well as the baseline methods) another TTS backbone based on the token prediction of codec results in SoundStorm (Borsos et al. 2023). The results are in Table
From the result, we observe that PromptTTS 2 consistently outperform baseline methods in the condition that TTS backbone in SoundStorm (Borsos et al. 2023) is leveraged.
H ABLATION ON REPRESENTATION LENGTH
We further conduct ablation study on unique hyper- hyper-paramedte parameter in PrompITTS 2, i.c., the length
of prompt and reference representations. The results are shown in Table 10 The results indicate that representation length is not a highly sensitive prper-parammter in terms of performance, and increasing the length to 16 can lead to slight improvement in model accuracy.
I EXTENSION ON FACE2VOICE
PromprlT' 2 involves modeling voice information utilizing sequence of predictable tokens, enabling its extension many other scenarios involving predicting voice from other modalities.
We conduct preliminary experiment on the Face2Voice extension, with objective of predicting voice based on the facial image speaker. In this experiment, the facial image processed using an
Table 10: The accuracy (%) synthesized speech on the attribute control of PromptTTS when using different lengths of the prompt and reference representations.
Representation Length
Gender
Speed
Volume
Pitch
Mean
98.54
92.49
93.26
88.66
93.24
8 (Default Choice)
98.23
92.64
92.56
89.89
93.33
16
98.77
92.18
93.95
90.27
93.79
98.39
91.80
91.19
90.27
93.09
16

---

Published as a conference paper at ICLR 2024
Table 11: The MOS results (%) whether the voice is in the same style with the facial image GT stands for judging whether the ground-truth voice is in the same style with the corresponding facial image.
Setting
Same
In-between
Different
GT
46.47
42.05
11.47
SP-FaceVC (Weng et al. 2023)
20.17
45.38
34.45
PromptTTS2
31.78
41.17
27.05
Table 12: The CMOS results (%) which voice (synthesized by PrompiTTS or Pe-FaceVC) corresponds more closely with the facial image.
Setting
Former
Tic
Latter
PromptTTS vs. SP-FaceVC (Weng et al.
2023
51.47
29.41
19.12
image encode pretrained in CLIP (Schuhmann et al.
2022
Radford al. 2021 Ilharco al. 2021
to extract image representations. Simultancously. the speech is processed using a reference speech encoder depicted in Figure Dp extract reference representations. Subsequently, variation network (illustrated in Figure TT) is trained to predict reference representations from image representations.
For this preliminary experiment, dataset designed for talking face generation. The dataset includes more than 300 distinct speakers and encompasses 15.8 hours of video. To extract paired data of facial images and speech, we first select an image (video frame) and then extract speech segment with a duration of 5-10 seconds surrounding the chosen frame. We designate 18 speakers for testing and use the remaining speakers for training.
utilize the HDTF dataset (Zhang et al.) 2021). high-resolution
We compare our method with SOTA method on Face2Voice, SP-FaceVC (Weng et al. 2023 with subjective test (MOS). In the MOS test, the judges asked judge whether a facial image and the voice is in the same style (i.c., is natural for the facial image to have that voice), whose results shown in Figure II The results demonstrate that compared with SP-Face Promp(TTS 2 can generate voice corresponding more closely with the facial image (31.78% versus 20.17%) and fewer unsuitable cases (27.05% versus 34.45%).
We also conduct comparative MOS (CMOS) test to directly judge that given a facial image, which voice (synthesized by PromprTTS The results Table 12 show that in 80.88% cases, PromprTTS 2 synthesizes a better or comparable voice than SP-Face Furthermore, our findings demonstrate that Promp(TTS 2 is general method for generating voices conditioned on text prompts, facial images, or other types of information. Samples of facial images and generated voices can also be found on our demo page.
SP- SP-Face corresponds more closely with the facial image.
"hatps://ithub.com/// http://gghhb..ccommmoooommmaoonp http:://ggthh..ommmmiiaaanggggg
17


Reviews:
[{"review_id": "CYUl56DwHw", "reviewer": "Reviewer_g23M", "summary": "This paper studies the problem of text-based voice creation for text-to-speech synthesis (TTS). \nPrior work on zero-shot TTS often relies on using reference voice samples of the target speaker (YourTTS) or target audio style (including both speaker and prosody, such as VALL-E) to prompt the model to generate the desired voice. \nHowever, the authors argue that such prompts may not always be available, and this paradigm is less user friendly. To address it, authors present a model to enable creation of voices through providing descriptions like “a man with a normal voice”, similar to the setup in InstructTTS and PromptTTS.\nThe contribution of the proposed method is two-fold. \nFirst, the authors tackle the one-to-many problem between text description and voice, where the same description, such as “a low pitched female voice”, can be mapped to many different voices. The authors adopt a variation network to sample the reference speech style embeddings given a text description prompt. \nSecond, the authors presented a pipeline to automatically create text prompts to address the data scarcity issue for descriptive texts for speech. The authors consider controlling four aspects of speech: gender, speed, volume, and pitch.\nIn addition, the authors present a face2voice application replacing text description with facial image.", "questions": "See the questions in the Weakness section", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper studies an interesting problem which enables creation of voices through text descriptions. This line of research has great potential of making speech generation more customizable.\n\n2. The authors present a systematic pipeline to produce text describing four aspects of speech, addressing the data scarcity problem. Ablation studying Table 7 shows the benefit of the step-by-step generation process.\n\n3. The variation model tackles the one-to-many problem. The author verified that when changing variation networks introduce speaker variation in Tabel 3.", "weaknesses": "1. I am not certain if the proposed model and the baseline models are trained on the same data, and hence I cannot draw conclusions that whether the proposed model outperforms the baselines because of the additional LLM generated data or because of the introduction of the variational network to address the one-to-many problem. It would be good to show how well the baseline performs with and without LLM-augmented text prmopts\n\n2. Given that the number of attribute combinations is rather small (2 x 3 x 3 x 3 = 54), I am suspicious about how useful it is to increase the number of text prompts. The author could have conducted ablation studies comparing using only the PromptTTS prompts vs those + x LLM-augmented prompts\n\n3. The authors did not give sufficient background on InstructTTS. That model also deploys a diffusion model and in principle would be capable of modeling one-to-many mapping. Why does the VoiceGen model perform better than InstructTTS?\n\n4. Given that the conditional attributes are all categorical, it can be conditioned with a lookup table for each attribute straightforwardly. How does that method compare with the proposed method? The paper does not really showcase the strength of free-form text description for conditioning attributes that are hard to categorize.\n\n5. For step 1 when generating phrases, the authors show in Table 5 that male can be mapped to man/boy/guy/dude/gentlement. However, one would expect quite different voices between boy and man. This shows the weakness of such pipeline where text descriptions are created from underspecified labels.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bKVHKsjuN0", "reviewer": "Reviewer_Ghqv", "summary": "The paper is about text-to-speech (TTS). The TTS model is based on Naturalspeech 2. The TTS model is extended by a style module, which uses a text prompt to describe the style. During training, it additionally also uses the reference speech as input. However, for speech generation, a variation network instead generates the reference speech encoder outputs. This variation network is the core novelty proposed by the authors. It is supposed to add further variation in the speech style which cannot be covered by the text prompt alone. The variation network is a diffusion model using a Transformer encoder to iterate on the speech encoder outputs.\n\nThis proposed model is called VoiceGen.\n\nThe training data is based on the Multilingual LibriSpeech (MLS) dataset with 44K hours of transcribed speech. To generate the text prompt, needed to train the style model, a text prompt dataset generation pipeline is proposed, to extend the given transcribed speech: Based on a speech language understanding (SLU) model, the gender of the speech is identified. Additionally, using digital signal processing tools, pitch, volume, and speed is extracted and put into classes. Those attribute classes are then fed into a large language model (LLM) to generate a text prompt which conveys the style attributes.\n\nThe variability is measured using a WavLM-TDNN model to assess the similarity of two speeches, and it is shown that the introduction of the variation network leads to higher speech variability. At the same time, mean-opinion-score (MOS) on the quality of the proposed VoiceGen model is slightly better than other text-prompt-based TTS models, PromptTTS and InstructTTS specifically. It is also shown that the text prompt indeed works and can generate speech with the requested attributes.", "questions": "> Compared to traditional text-to- speech (TTS) methods relying on speech prompts (reference speech) for voice variability, using text prompts (descriptions) is more user-friendly since speech prompts can be hard to find or may not exist at all.\n\nI don't exactly understand the difference between speech prompts and text prompts.\n\nA text prompt is really like a description of what should be generated, like “Please generate a voice of a boy shouting out”.\n\nA speech prompt is the same but as speech? Or is this the text of the generated speech? Or sth else?\n\n\n> The input of variation network comprises the prompt representation (P1, ..., PM ), noised reference representation (R1t , ..., PMt ), and diffusion step t\n\nHow exactly? P and R are concatenated sequences? But the diffusion process only runs on R?\n\n\n\n\n\n\nClarification on text attributes for text prompt generalization: There is only gender (via the existing SLU model), pitch, volume, and speed, nothing else? I would expect some more attributes, e.g. different emotion categories, etc.\n\n\nSection 5.1, table 1: I don't exactly understand what is measured under what conditions on what data. So, each TTS model (VoiceGen vs the others) generates some speech, then some attributes are given to produce some prompt, and then, for the given fixed SLU models and digital signal processing tools, the accuracy is measured? That uses the generated prompts via LLM as mentioned before? All of them, so 20K text prompts? How many classes are there for each of the attributes?\n\n\nText prompt dataset: So this is released? Where? I did not find it.\n\n> WavLM-TDNN model (Chen et al., 2022a) to assess the similarity of two speech\n\nIs this a pretrained model which you just take? Or where is the model from?\n\nI don't exactly understand Section 5.1, table 3. This is always with the same phoneme sequence as input to the TTS model? But what is the text context here? Why are the results so different between PromptTTS, InstructTTS and VoiceGen? Are they really comparable? Do they have the same TTS backbone? Does the TTS backbone use diffusion in all cases?\n\nWhen you add variability through the variation network, how well are the properties of the text prompt for the speech style actually preserved? I guess this is table 1? But how much variance do you get when you add sampling results of the variation network?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The claims are tested and it seems the proposed model adds quite a bit of variability, as it was intended.", "weaknesses": "So many details are left out, as this would not really be possible to fit into the paper. So without the exact code + recipe to produce all the results, it will be almost impossible to reproduce the results. I think having code + recipe available here is very important.\n\nAll the experiments basically just show that the proposed model works well and solves the outlined problem. However, there is almost no analysis or ablation studies, etc. E.g. how important is it to use a diffusion model here? What about other model types? What about other smaller model details, and hyper parameters, etc?\n\nThere are a few things a bit unclear (see below).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Bu1yZKDcrz", "reviewer": "Reviewer_2wDM", "summary": "In this paper, the authors proposed VoiceGen, a text-to-speech framework that uses a variation network to provide variability information of voice not captured by text prompts, and a prompt generation pipeline to utilize the large language models (LLM) to compose high quality text prompts.\n\nThe variation network predicts the representation extracted from the reference speech based on the text prompt. And the LLM formulates text prompts based on the recognition results from speech language understanding model. Compared to the previous works, VoiceGen generates voices more consistent with text prompts, offering users more choices on voice generation.", "questions": "1. What's the necessity of concatenating both text prompt representation and speech prompt representation for TTS backbone? Will the speech prompt representation itself be enough to guide the TTS backbone through cross attention?\n\n2. What's the impact of making use of a fixed-length representation for text and speech prompt representations? Fixed-length representations may be fine for global attributes such as age and gender, but is that enough for fine-grained or local control in guided speech generation?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The proposed modeling and data labeling pipelines for text-prompt based TTS systems can generate higher-quality speech with more consistent and noticeable control compared to previous systems. The variation network predicts speech representations that are more closely corresponding to the text prompt and more diversity by sampling from Gaussian noise. On the other hand, the LLM-based prompt generation pipeline can produce high-quality text prompts at scale and can easily incorporate new attributes. Overall, the proposed system provides a framework that is beneficial for future text-prompt based TTS research.", "weaknesses": "After listening to the generated voices on the demo page, audio quality is still an issue and further improvements are required, especially for certain text prompts such as \"Please speak at a fast speed, gentleman\". The reason could be missing or few audio samples for corresponding prompts in training datasets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "b4nCkuImiu", "reviewer": "Reviewer_dNEb", "summary": "The paper proposes a small architecture improvement upon PromptTTS and also proposes to use SLU and LLM to generate text prompts.", "questions": "Will the author open source the generated dataset or source code? I would consider the open sourcing effort as a contribution.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. There is some novelty in using SLU and LLM to generate text prompts for voice, as I believe this is the first work investigating this.\n2. A good mount of ablation and analysis.\n3. The paper is in general written clearly and easy to follow.", "weaknesses": "1. The authors claim that the proposed variational method solves the one-to-many problem, but I doubt if this is the case. The proposed module basically learns to predict a melspec encoder's output from text prompts, in addition to conditoning the TTS backbone. I don't see how this solves the one-to-many problem as in the end we are still predicting from text (one) to different variability (many) in the voice. The only difference is that there is an additional auxiliary loss from the melspec encoder. This seems to be in principle similar to InstructTTS's approach to regularize melspec encoder and text encoder output to close in the embedding space.\n2. The proposed prompt generation method, while the first of its kind, seems to be still quite limited by the very limited SLU output set, and the template used for LLM generation. \n3. The proposed method performs only marginally better than PromptTTS and InstructTTS.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)