P-Flow: A Fast and Data-Efficient Zero-Shot TTS through Speech Prompting
Authors: Sungwon Kim
Kevin J. Shih
Rohan Badlani
Joao Felipe Santos
Evelina Bakhturina
Mikyas T. Desta
Rafael Valle
Sungroh Yoon
Bryan Catanzaro
Venue: NeurIPS 2023 poster (2023)
Keywords: text-to-speech
zero-shot TTS
flow matching generative model

Abstract: While recent large-scale neural codec language models have shown significant improvement in zero-shot TTS by training on thousands of hours of data, they suffer from drawbacks such as a lack of robustness, slow sampling speed similar to previous autoregressive TTS methods, and reliance on pre-trained neural codec representations. Our work proposes P-Flow, a fast and data-efficient zero-shot TTS model that uses speech prompts for speaker adaptation. P-Flow comprises a speech-prompted text encoder for speaker adaptation and a flow matching generative decoder for high-quality and fast speech synthesis. Our speech-prompted text encoder uses speech prompts and text input to generate speaker-conditional text representation. The flow matching generative decoder uses the speaker-conditional output to synthesize high-quality personalized speech significantly faster than in real-time. Unlike the neural codec language models, we specifically train P-Flow on LibriTTS dataset using a continuous mel-representation. Through our training method using continuous speech prompts, P-Flow matches the speaker similarity performance of the large-scale zero-shot TTS models with two orders of magnitude less training data and has more than 20$\times$ faster sampling speed. Our results show that P-Flow has better pronunciation and is preferred in human likeness and speaker similarity to its recent state-of-the-art counterparts, thus defining P-Flow as an attractive and desirable alternative. We provide audio samples on our demo page: [https://research.nvidia.com/labs/adlr/projects/pflow](https://research.nvidia.com/labs/adlr/projects/pflow)

P-Flow: A Fast and Data-Effciient Zero-Shot TTS through Speech Prompting
Sungwon Kim 1.2+f Kevin J Shih', Rohan Badlani', Joōo Felipe Santos', Evelina Bhakturina', Mikýas Desta', Rafael Valle' J Yoon Bryan Catanzaro'
NVIDIA
Department of Electrical and Computer Engincering. Scoul National University? Interdisciplinary Program in Artificial Intelligence, Scoul National University
Abstract
While recent large-scale neural codec language models have shown significant improvement in zero-shot TTS by training on thousands of hours of data, they suffer from drawbacks such as lack of robustness, slow sampling speed similar to previous autoregressive TTS methods, and reliance on pre-trained neural codec representations. Our work proposes P-Flow, a fast and data-efficient zero-shot TTS model that uses speech prompts for speaker adaptation. P-Flow comprises speech- prompted text encoder for speaker adaptation and a flow matching generative decoder for high-quality and fast speech synthesis. Our greech-prompted text encoder uses speech prompts and text input to generate speaker-condition text representation. The flow matching generative decoder uses the speaker-conditional output to synthesize high-quality personalized speech significantly faster than real-time. Unlike the neural codec language models, specifically train P-Flow on LibriTTS dataset using a continuous reereeepestnntinnn Through our training method using continuous speech prompts, P-Flow matches the speaker similarity performance of the large-scale zero-shot TTS models with two orders magnitude less training data and has more than 20 faster sampling speed. Our results show that P-Flow has better pronunciation and is preferred in human likeness and speaker similarity to its recent state-of-the-art counterparts, thus defining P-Flow an attractive and desirable alternative. We provide audio samples on our demo page.
1 Introduction
Zero-shot TTS (rext-to-Speech) refers to the goal of generating text-conditioned speech with novel speaker characteristics at inference without the need for additional training. The speaker charac- teristics are provided by short reference audio segment, which in recent works can be short as 3 seconds long [37, 12, 6). Many recent advances have come cost: larger datasets, more complicated training setups, additional quantization steps, additional pretraining tasks, and computa- tionally expensive autoregressive formulations. Our work explores whether more efficient solution exists. By staying within the general framework prompt continuation, as often seen recent large language model works, we demonstrate that similar results can be achieved with simpler training pipelines, significantly less data, and faster inference.
Following recent successes in the zero-shot and few-shot capabilities of large scale language models such GPT [2), has been generally accepted that an analogous approach will yield similar results
Work done research intern at NVIDIA.
Corresponding authors: Sungwon Kim: ksw0306 an.......su....kkkk..kkkkk kr. Rafael Valle: infaelvallennvidia..
Sungroh Yoon: sryoon @nn.ac.kkr
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

in speech synthesis. However, it is not yet clear that recent approaches in large scale representation learning have a significant advantage over existing and cheaper traditional representations such as mel-spectrograms. Of note, recent works such as SPEAR-TTS and VALL-E make use of pretrained quantized neural audio codec models such as Encodec and SoundStream [10, 40]. The costs of using such a model are expensive. Feature extraction is significantly more computationally demanding than extracting sneettpeessss from audio. Encodec in particular contains autoregressive compontents in both the encoding and decoding, which may be an issue where low-latency is necessary. Further, having too many different versions of such a codec inhibit interoperability across models. While is likely that neural codecs may yet have significant advantages in the near future, our work demonstrates competitive results with standard mel-spectrograms, suggesting that the costs of neural codecs in the zero-shot TTS setting is not yet justified by any significant improvements in generalization.
Improving inference speed is the other focus of this work. Our work improves upon the latency of prior approaches by reducing reliance on autoregressive formulations, and by employing recent advances in ODE-based generative modeling such as Flow Matching [23, 24, 35). In addition to autoregressive components within the neural codec architectures, prior works such as SPEAR- ---SSS and VALL-E also employ autoregressive decoders to infer audio token sequences. While autoregressive modeling is a natural choice for sequential data such as audio signals, our work demonstrates that lower-latency non-autoregressive formulation is both viable and competitive. Another common computational bottleneck is the incorporation of probabilistic generative models such as denoising diffusion models in recent works. Such architectures exist in many recent state-of-the speech and audio synthesis models, but come at the cost of requiring numerous function evaluations during inference. Distillation techniques exist to reduce the number of function evaluations, but require additional training stages, further complicating the training pipeline. In contrast, experiment with recently proposed flow- matching algorithms. These models are closely related to diffusion models, but actively encourage simpler and straighter trajectories, which in turn require significantly fewer function evaluations during inference all without the need for additional distillation stages.
Our work proposes P-Flow, high-quality, data-efficient, and fast zero-shot TTS model with emergent in-context learning capabilities for speaker adaptation. Inspired by recent success using language- model prompting achieve zero-shot TTS results, introduce speech-prompted text encoder within a non-an non- aon--utorrerrssive TTS model. P-Flow combines this prompted text encoder with a how-matching generative decoder to efficiently sample high-quality mel-spectrograms. Our speech- prompted text encoder incorporates a 3-second random segment of speech as a prompt along with the text input and learns predict the mel-spectrogram using the speech-prone prompted text input. Given the ppeaker-conditionn
output from the text encoder, our flow-matching decoder is trained to predict a vector field that models the probabilistic distribution of mel-spectrograms conditioned on the text encoder's ' output. Our prompting approach directly utilizes random segments of speech as input, which poses a risk of P-Flow learning trivial solutions such as the identity function for the random segment. We address the potential problem by masking the loss for the random segment during training. Our contributions are as follows:
We propose a speech prompt approach for the non- oon-autoregressive zero-shot TTS model which surpasses the speaker embedding approach and provides in-context learning capabilities for speaker adaptation.
We propose a flow matching generative model for a high-quality and fast zero-shot TTS that significantly improves the synthesis speed and sample quality compared the large-scale autoregressive baseline.
We demonstrate comparable speaker adaptation performance to the large-scale autoregressive baseline using significantly fewer training data and a small transformer- based encoder, highlighting the effectiveness of the proposed speech prompting approach.
*
P-Flow achieves average inference latency of 0.11 seconds on an NVIDIA A100 GPU.
2
Related work
Zero-Shot Speaker Adaptive TTS (Zero-Shot TTS): Instead of specifying speaker identity with discrete embedding vectors as in the case of multi-speaker TTS models, zero-shot TTS formulations assume the ability to extract speaker embedding equivalents on-the-fly from short audio samples. Common approaches either use separate audio mudio-sample-to-mmbcddii encoder to extract speaker
2

---

vectors on the fly [17, 27, 20, 3, 38), or formulate the problem as continuation task with language model architectures and probabilistic generative models. Methods such as VALL-E and SPEAR-TIS [37, 19] employ the prompting paradigm from large transformer--asee they use the reference audio as the prompt, from which the model is expected to decode the most likely continuation. The decoding procedure is further conditioned with the provided text transcript to achieve a text-to-speech formulation. Relatedly, approaches such as [28, 2, 25] use the data inpainting paradigm from generative probabilistic models, treating the data to continue the available data (prompted audio). Just as in the LM-based approach, the inpainting is guided by conditioning the target text sequence. Our work operates in the same prompt- based zero-shot TTS setting as VALL-E and SPEAR-TTS, establishing competitive results with much less data, and additionally providing faster inference speeds via low-matching- baasciing--aeddddd generative models and non-autoregreessvee formulations.
language models. Notably,
as the missing
Perhaps most architecturally related to our work is AT [4], which similarly trains a transformer (A uses Conformer (12]) model to infer text-conditional mel spectrograms with a masked reconstruction loss. However, A uses forced aligner, requires text transcript for the zero-shot audio prompt, and focuses primarily on the pretraining task and audio reconstruction, with limited results in the zero-shot TTS domain. In contrast, our proposed approach uses end-to-end audio-text alignment mechanism, uses short second audio-only prompt, and focuses primarily on the zero-shot TTS task.
Generative Sampling with Fewer Steps: Generative models in the score matching and denoising diffusion model family typically require hundreds of iterative steps. It is common to reduce the number of steps required via approaches such as the DDIM [32] framework and distillation [31]. Notably, distillation will progressively map every two steps to a single step, halving the number of total steps required with every iteration of the algorithm. Recent works such as flow matching [24. 23, 35] and consistency training (from scratch) [33] offer single-stage approach, encouraging approximately straight, causal trajectories in the initial model, thereby reducing the need for further distillation stages. Our work serves as demonstration of the use of such approaches as flow matching in large-scale practical audio applications.
3 Method
Our work aims to provide in-context learning capabilities for zero-shot speaker adaptation in high-quality and fast non-autoregressive TTS model. To avoid the potential bottleneck of speaker embedding approaches to extract speaker information from the reference speech, we adopt a prompt- ing approach that directly utilizes the reference speech as a prompt for speaker adaptation, similar to neural codec language models. In addition to speech prompting method, generative model as our decoder for efficient sampling. We provide an overview of our method in Section 3.1, followed by a detailed explanation of our decoder in Section 3.2, and we describe the remaining details in Section 3.3.
a flow matching
3.1 P-Flow
P-Flow training is similar to that of masked-autoenee autonnodeessss.... Given <text, speech>> paired data, denote the mel-spectrogram of the speech as x, and the text as c. Let m² be a indicator mask on the sequence æ, masking out randomly positioned 3 second segment from z. We define x" (1-m")-a ) We use a p superseript here indicating that this variable will be replaced with arbitrary 3-second prompts during zero-shot inference. The training objective for P-Flow is as follows: reconstruct the given e and some segment x?. Note that even though we expect P-Flow to reconstruct the entirety of x during training, including the provided x?, the model does the exact positioning of x within x.
P-Flow learns to model the conditional probability distribution of speech p(z|c, xP). Unlike many zero-shot TTS models that compress all the information in the speech into fixed-size vector, we introduce a text encoder fene. which takes the text c and the random segment x! as a speech prompt to generate ppeaker-conditionn
text representation. We refer to our text encoder as speech- prompted text encoder. The output of this text-encoder is then mapped to mel-spectrograms using a flow-matching generative model, and finally waveform using audio vocoder.
Speech-Prompted Text Encoder: shown in Fig. Ia, we input the mel-spectrogram a random segment x" speech prompt along with the text input c. We then project both to the same
3

---

L
h
Remaining segment
m=
d
x m72
L
Duration Prodictor
(.
Align
mp
hg
Loss Mask
m
1
0
1
Lineer
Transformet
3-sec Prompt xP
Speech- ppeeph-prompted Text E Encoder
Flow Matching Decoder
Prompt
X
Text
Xy. h, t
ProNitt
XP
Emb
Speech
Prompt
Text
(a) An overview of P-Flow
(b) Speech-promples Text Encoder
The overall diagram of P-Flow. P-Flow is a model composed of a encoder, which outputs representation containing speaker information from the speech prompt, and a flow matching decoder generates high-quality speech significantly faster than real-time.
Figure 1:
text
dimensions to use as inputs to the text encoder. The role of the generate speaker-conditional text representation h = fene(x" ; using the speaker information extracted from the prompt x². Similar to large-scale codec language models, we employ a non- autoregressive transformer architecture that can attend speech prompts arbitrary text positions.
text encoder is to
To train the speech-prompiee text encoder to effectively extract speaker information from the speech encoder loss that directly minimizes the distance between the text encoder representation and the mel--ppettoggra. In addition to its original purpose in Grad-TTS [30] for reducing sampling steps in diffusion-based single-speaker TTS model, it also encourages the encoder to incorporate speaker-related details into the generated text representation.
prompt,
As
encoder output he and the mel- mel-specttogram x have different lengths, we align the text encoder output with the mel -peccrooror using the monotonic alignment search (MAS) algorithm proposed in Glow-TTS [21]. By applying MAS, we derive alignment A M AS(he, x) that minimizes the overall L.2 distance between the aligned text encoder output and the mel-spectrogram. Based on the alignment A, we determine the duration d for each text token and expand the encoder output he by duplicating encoder representations according to the duration of each text token. This alignment process results in text encoder output h that aligns with mel-specttroramm The fairly straightforward reconstruction loss is written as Lene = (S,,h,,))
In practice even though the model is not given the exact positioning of x" within x during training, we found the model still collapse to a trivial copy-pastinn of x². To avoid this, we simply mask out the reconstruction loss for the segment corresponding to x². Despite this, the final model is still capable of inferring a continuous mel-spectrogram sequence. We define the masked encoder loss Le by using m for the random segment x2 in the mel-spectrogram a::
L MSE m², m -m²)
(1)
By minimizing this loss, the encoder is trained to extract speaker information as much possible from the speech prompt in order to generate an aligned output that closely resembles the given speech x, which results in enhancing in-context capabilities for speaker adaptation.
To perform high-quality and fast zero-shot TTS, generative model as the decoder for modeling the probability distribution p(alc,x²) = (hr Our flow-matching decoder models the conditional vector field vz (h) of Continuous Normalizing Flows (CNF), which represents the conditional mapping from standard normal distribution to data distribution. The decoder trained using a flow-matching loss L that also applies mask m2 for the random segment in LPP - More details will be provided in Section 3.2.
Flow Matching Decoders:
flow-matching
Duration Predictor: To reproduce text-token durations during inference where MAS is unavailable, use a duration predictor trained in a manner similar to [21]] We use the hidden representation

---

of the speech-promplee prompted text encoder as its input without additional speaker conditioning, given this representation already contains speaker information. It is trained simultaneously with the rest of the model with detached inputs to avoid affecting the text-encoder training. The duration predictor estimates the log-scale duration log d for each text token, and the training objective for the duration predictor Leur is to minimize the mean squared error with respect to log-scale duration log d obtained via MAS during training.
The overall training loss for P-Flow is L L + L'ffm + Ldur. Zero-shot inference uses random chunk from a reference sample as the speech prompt along with the desired transcript as inputs to the snrech-prompless text encoder to obtain l.c We do not provide a transcript corresponding to the speech prompt. Then, using the predicted duration d from the duration predictor, we expand the encoder output he to obtain h and generate the personalized speech using the flow-matching decoder.
3.2 Flow Matching Decoder
We use Flow Matching to model the mel- spectrogram decoder task's s conditional distribu- tion: p(z]h). We first provide a brief overview of flow matching, followed by describing our sampling procedure and additional qualitative improvements through guidance-related tech- nique.
SDE trajectory www h w
X1
Conditional Flow Matching ODE trajectory
X0
Flow Matching Overview: Flow Matching [23, 35, 24] is method for fitting the time- dependent probability path between our data density (zr) and our simpler sampling density (o( (assumed to be the standard normal). It closely related to Continuous Normalizing Flows, but is trained much more efficiently in simulation- free fashion, much like the typical setup for Diffusion and Score Matching mod- els [13, 16. 34). We adopt Conditional Flow Matching specified in [23] as their formu- lation encourages simpler and often straighter trajectories between source and target distributions. Simpler trajectories allow for test-time sampling in fewer steps without the need for additional distillation. We will ignore the conditional variable for notational simplicity in this overview.
Pdata
N(O.1)
Figure 2: Conditional Flow Matching [23] can be supervised with linear conditional trajectories dur- ing training when mapping between the complex Paata and the sampling Gaussian, leading to simpler marginal trajectories during inference. Simpler tra- jectories require fewer function evaluations.
Following Lipman al. [23], we define the flow Φ [0, 1] x Rd + Rd the mapping between our two density functions using the following ODE:
d (z) = ve(ee(x)); do (x =
(2)
dt
Here, (x( is the time- dependenn vector field and specifying the trajectory of the probability flow through time. (z) is also our learnable component, henceforth denoted as (z((()): 0). To sample from the distribution, we sample from the sampling distribution Po as our initial condition f and solve the ODE in Eq. (2). Notably, the formulation [23] encourages straighter trajectories, ultimately allowing us to cheaply approximate the ODE solution with 10 Fuler steps with minimal loss in quality.
It turns out that determining the marginal flow (x(x) is difficult in practice. Lipman thus formulates it as marginalizing over multiple conditional flows 6r.,, (x) as follows:
St.z, (z) = (=(x))= + (z(x1)
(3)
Here, (21) and (x1) are time-conditions affine transformations for parameterization the trans- formation between Gaussian distributions pl and po- Finally, let q(x1) be the true but likely non- Gaussian distribution over our data. We define Pi as a miiture-of-Gaassiee perturbing individual samples with small amounts of white noise with miin (empirically set to 0.01). We can specify trajectories without complications from stochasticity in SDE formulations.
approximation of by
5

---

Taking advantage of this, Lipman et al. recommend simple linear trajectories, yielding the following parameterization for o,
(x(x)) = Læ,, (z(2) = - (1 miin M
Training the vector field is performed using the conditional flow matching objective function:
d
LCFM L
(ro); 0) (
Dr,e, (zo))l
(5)
dt
Plugging Eq. (4) in to Eq. (3) and (5), we get our final CFM objective:
LCFM(0) Bl,g(z)) 1lv (xo); - (x1 (1 min )=o))l
(6)
Masked Flow Matching Loss: Recall that our flow matching decoder models the distribution p(z[h). Because h is the output of the text encoder which provided by the subsegment x" e we found it again necessary to mask out the loss for parts of the output corresponding to x² to prevent trivial solutions. Let the generic us (I23 0) be parameterized in our setup 0g (2,, h f) to account for the conditional. Here, & is represented with a continuous sinusoidal embedding. This gives us the masked CFM objective:
DCrM(0) Ex.g(x)) (.p())))))))))))))))))) ll . (fa(ér, (zo), h, () (x1 min man))
Sampling: The conditional flow matching loss marginalizes over conditional vector fields to achieve the marginal vector field, the latter which is used during sampling. While the linearly interpolated conditional trajectories specified in Eq. (4) do not guarantee the same degree of straightness in the resulting marginal, we still get something fairly close. Within the context of this work, we found the conditional flow matching formulation to result in simple enough trajectories such that is sufficient to use the first-order Euler's s method with around 10 steps to solve the ODE during inference. Sampling with N Euler steps is performed with the following recurrence relation:
To NOO, T); Fetg = =t
N (o
Guided Sampling: We find that pronunciation clarity be further enhanced by applying techniques from classifier-free guidance method [14]. In related work, GLIDE [29] amplifies their text- conditional sampling trajectory by subtracting the trajectory for empty text sequence. We employ similar formulation, guiding our sampling trajectory away from the average feature vector computed from h. denoted as h. h is computed by averaging the expanded representation along the time axis to obtain fixed-size vector and then duplicated along the time axis. Let y be our guidance scale. Our suidaacc-amplifien Euler formulation as follows:
Ft + N (60 1) + (000(1,, h,t) Go (xe, t))))
3.3
Model Details
The high-level model architecture is shown in Fig. 1. Our model comprises 3 main components: the prompt-based text encoder, a duration predictor to recover phoneme durations during inference, and Wavenet-based flow matching decoder. Experiments demonstrate strong zero-shot results despite our text-encoder comprising only a small transformer architecture of 3M parameters. We provide additional architectural details for each component in Section B.
4 Experiments
Training and Inference Settings: P-Flow is trained on single NVIDIA A100 GPU for 800K iterations, using batch size of 64. We utilize the AdamW optimizer [26] with learning rate of 0,0001. A G2P model [5] preprocesses the text into the International Phonetic Alphabet (IPA) format. During inference, we generate mel-spectrograms using 10 Euler steps in the flow matching decoder with guidance scale of 1. Mel-spectrogram to waveform is performed using the pre-trained universal Hifi-GAN, available in the HiFi-GAN repository To be compatible with Hifi-GAN, our audio representation is 22kHz audio represented with an 80-bin mel-spectrogram, with FFT parameters:
HHifi-GAN: https//giihubb.oomjj/s77nfiii

---

window size 1024 and hop length 256. In this setup, 3-second mel-spectrogram is fed as a speech prompt with a length of 3422050
259.
250
Data: We train P-Flow on LibriTTS [41]. LibriTTS training set consists of 580 hours of data from 2,456 speakers. We specifically use data that is longer than 3 seconds for speech prompting. yielding a 256 hours subset. For evaluation, we follow the experiments in [3Z. 19] and use LibriSpeech lest-clean, assuring no overlap exists with our training data. We resample all datasets to 22kHz.
Evaluation: We compare P-Flow with two zero-shot speaker-adaptive TTS models, YouTTTS [7] and VALL-E [37]. YourTTS is VITS-bassed multi-speaker TTS, which performs zero-shot speaker adaptation through speaker embedding extracted from a pre-trained speaker encoder. VALL-E achieves zero-shot speaker adaptation through prompting, resulting in much higher speaker similarity than YourTTS. We compare our model with the two models in the same way as described in [37].
and the objective metrics for each model were directly taken from [37]-
Replicating the experiment [37, 19). we evaluate samples ranging from 4 to 10 seconds from the LibriSpeech test-clean dataset, resulting in a total of 2.2 hours of data. For each paired data (zi, ci), we extract a 3-second reference speech z from a different sample xj spoken by the same speaker and generate the synthesized speech 2:!e for the text Ci.
Objective Metries: We measure the inference latency and two objective metrics from [37], word error rate (WER), and speaker embedding cosine similarity (SECS). Inference latency of P-Flow and the autoregressive baseline, VALL-E are evaluated inference latency for P-Flow, we average the text-to-mel generation time for all the samples used in evaluation for LibriSpeech. Because VALL-E is not open-sourced, use PyTorch transformer* of the same size as used in [37] and feed random tensors repeatedly corresponding to the duration of I; to obtain a proxy for the inference latency."
NVIDIA A100 GPU. To compute
We evaluate pronunciation accuracy with WER, using the HuBERT [15] ASR model to measure transcription errors from generated samples. Speaker similarity is measured with SECS between the generated sample x:!eh and the re-synthesized output of the 3-second reference speech sn as in [37]. We compute the speaker embedding using the pre-trained WavLM-TDNN [8] used in [37]. We also measure the WER and SECS for the re-synthesized output of the ground truth audio as an upper bound, reported as "GT (HIFI-GAN)" in Table 2. Note that the final mapping to waveform is different between models: P-Flow uses Hifi-GAN [22] while VALL-E uses EnCodec [11], necessarily influencing automatic metrics. We partially address this with our user study setup for speaker similarity.
User Study: We measure user preference score between VALL-E and P-Flow. Qualified evaluators must first pass hearing test where they count the number of short sinusoidal audio segments within an audio clip. We obtain evaluation scores for 8 samples from each evaluator with minimum of 30 evaluators per comparison experiment. We evaluate the preference for naturalness, acoustic quality, and human likeness using comparative mean opinion score (CMOS). Preference for speaker similarity is reported using comparative speaker similarity mean opinion score (SMOS). SMOS evaluators are provided with a 3-second reference audio, which the ground truth audio rather than re-synthesized audio to avoid being influenced by vocoder artifacts. Each preference score ranges from -2 to 2, using pairwise comparisons between the LibriSpeech samples from the VALL-E demo page.
4.1 Prompting v.s. Separate Speaker Encoder
This ablation study demonstrates the effectiveness of speaker conditioning through speech prompting. For the baseline, instead of directly inputting the speech prompt into the text encoder, encode a random segment of speech x" to a fixed-size speaker embedding using a speaker encoder with the same architecture as the text encoder. The speaker embedding obtained by averaging the last hidden representations of the speaker encoder along the time axis and normalizing it. This speaker embedding is concatenated with the text input of the text encoder. The speaker encoder is jointly trained with the model and follows the same training objective, except for excluding the loss mask
nn. TransformerrEncoder.hnll
'Following [37]. we measure inference on a 12-layer transformer with a hidden size of 1024 and 16 heads. Given that VALL-E's neural audio codec representation is at 75Hz, we approximate the inference latency of VALL-E by measuring the latency required for 75 n forward passes to generate seconds of audio.
7

---

Table Objective metrics for ablation study. "P-Flow (wlo Prompt)" refers to the baseline condition- ing on a fixed-size speaker embedding.
MODEL
WER
SECS
GT (HIFI-GAN)
2.4
0.64
P-FLOW (W/o PROMPT)
2.9
0.373
P-FLOW
2.6
0.544
for the speech prompt. For evaluation, we also sample using 10 Euler steps with guidance scale of similar P-Flow.
In Table 1, we compare the objective metrics of P-Flow and the baseline for the fixed-size speaker embedding approach. Despite having similar WER values, it can be observed that introducing speech prompt-based speaker conditioning significantly enhances SECS compared to the baseline. This confirms that even without changing other aspects, the use of speech prompts leads to a substantial enhancement in speaker similarity. Through this ablation study, we demonstrate that the speech prompt-bascd approach in P-Flow, which utilizes the speech prompt as direct input and adapts through the self-attention mechanism to optimize the loss for the remaining segment, is more effective for zero-shot TTS compared to encoding all the speaker information into fixed-size vector.
Table 2: The amount of training data and objective metrics for zero-shot TTS models. The SECS of "GT (HIFI-GAN)" is computed between the re-synthesized output of 3-second reference speech and that of ground truth. WER and SECS for the baselines are taken from [37]. The inference latency for VALL-- is approximated using the same size of transformer architecture, as mentioned Section 4.
MODEL
DATA (HOURS)
WERJ
SECS
INFERENCE LATENCY(S)L
GT (HIF--GAN
2.4
0.64
YOURTTS!
500+
0.337
VALI-EE
60,000
5.9
0.580
2.515 + 0.040
VALL-E CONTINUAL.
60,000
0.508
2.515 + 0.040
P-FLOW (PROPOSED)
260
2.6
0.544
0.115 + 0.004
4.2 Model Comparison
Samples for each model can be accessed through the demo page. We highly encourage reviewers listen to the samples.
Pronunciation Accuracy and Sample Quality: Our results in Table 2 show that P-Flow offers similar WER to GT (HIFI-GAN) and demonstrates significantly better pronunciation accuracy compared to other baselines while being trained two orders of magnitude less data. This indicates that with training on the more accurately transcribed LibriTTS dataset, we can achieve pronunciation accuracy close the ground truth.
Table Zero-Shot Speech Synthesis Subjective Metrics. Preference scores > 0 indicate preference for P-Flow.
MODEL
CMOS
SMOS
P-FLOw VS VALL-E
0.27 0.10
0.23 0.13
Speaker Similarity Metrics: Compared to the previous speaker embedding- based zero-shot TTS model, YourTTS, P-Flow demonstrates more effective speaker adaptation through speech prompting using a similar amount training data. Considering our model text-encoder only has 3M parameters, competitive performance with large-scale autoregressive baselines suggests achieving high SECS does not require tens of thousands hours audio data, quantization, neural codec models, or
Demo: http:///eerraaaaaddccooooo///ttt

---

large autoregressive models. Further, combining our speaker conditioning ablation study with these results and the fact that our decoder does not have any additional speaker conditioning beyond what is provided by the text encoder, we believe that prompting is one of the most important factors in obtaining high SECS. Finally, we see higher preference scores for P-Flow than for previous SOTA zero-shot models in the human evaluation of speaker similarity with the ground truth audio in Table text encoder enhances the in-context learning capabilities,
This indicates that our similar to large-scale autoregressive transformers.
CMOS: In Table 3, we can observe the CMOS results of pairwise comparisons between samples from VALL-E and P-Flow, evaluating naturalness, prosody, and audio quality. Table 3 demonstrates that P-Flow exhibits significantly better sample quality compared to VALL-E. This CMOS result shows that generative modeling using continuous representation, which has been widely used in TTS, exhibits superior performance compared to generation using discrete tokens. We attribute this result not only to the performance difference between the models but also to the additional performance degradation introduced during the encoding-decoding process using neural codec representation. highlights the advantages of continuous representation and showcases its effectivvness.
Inference Latency: Defaulting 10 Euler steps for P-Flow's decoder, we measure inference latency for audio samples from LibriSpeech with an average length of 5.6 seconds. Tables 2 and show our model achieves an inference latency 0.1 seconds, more than 20 times faster than VALL-E, while maintaining high speaker similarity, sample quality, and pronunciation accuracy. While the autoregressive baseline has generation time proportional the duration of the audio, the non-autoregressive P-Flow maintains nearly constant latency regardless of the length. A 20-second audio sample would take P-Flow 0.12 seconds in our experimental setup.
Table 4: Effect of guidance scale y and Euler steps N for our flow matching generative decoder. Our experiments find that automatic metric numbers plateau quickly after 5 steps, but we continue to hear qualitative improvements up to 10. Significant improvements are also seen due to the guidance.
MODEL
N
WER
SECSt
INFERENCE LATENCY(S)
P-FLOW (DEFAULT)
10
2.6
0.544
0.115 + 0.004
P-FLOW
10
3.7
0.492
0.115 1 0.004
P-FLOw
10
2.6
0.546
0.115 + 0.004
P-FLOW
1
2.7
0.420
0.028 1 0.004
P-FLOW
2
2.9
0.522
0.037 1 0.004
P-FLOW
5
2.6
0.549
0.067 1 0.004
P-FLOW
20
2.7
0.540
0.210 = 0.005
4.3 Effect of Guidance Scale and Euler Steps
Table 4 demonstrates how the guidance scale and number of Euler steps N affects objective metrics. As shown in Table and 4, P-Flow outperforms YourTTS even without the guidance method, showing the effectiveness of the speech--rompting method for speaker adaptation. Applying the guidance method improves pronunciation accuracy and brings our SECS metric closer to that of state-of-the-are zero-shot TTS model without explicit training for guidance. These results confirm that the guidance method applied to P-Flow further boosts the effectiveness of speech prompting. Both guidance scales [1,2} perform well. (We default to y = 1)
Regarding the Euler steps, P-Flow demonstrates low WER even with very few Euler steps, indicating that the flow matching generative model generates speech that can be accurately recognized by the ASR model regardless of audio quality. We default to 10 Euler steps for external comparisons because SECS values plateau roughly after 5 Euler steps and we notice that good audio quality is achieved at around 10 steps. Table 4, the ASR metrics and SECS metrics not directly represent the acoustic quality of the samples according to the Fuler steps. Therefore, we provide mean opinion scores (MOS) for the acoustic quality based according to the Fuler steps Section A. We further provide audio samples corresponding to default inference settings in our demo page.

---

Discussion and Limitations
P-Flow fast and data-efficient flow matching model for zero-shot TTS that achieves comparable naturalness and speaker adaptation performance to its large-scale and autoregressive counterparts. The core the approach is the use of the prompt-continuatio setup recent LLM applications while avoiding complex and expensive setups from recent works in zero-shot TTS. In some ways, our work serves as strong yet simple baseline for future approaches, as we demonstrate that state of the art performance in this task can be achieved without super large-scale datasets, complex training setups, representation quantization steps, pretraining tasks, and expensive autoregressive formulations.
The focus of this work is primarily on zero-shot capabilities with respect to text-encoding and audio decoding. whereas zero-shot capabilities of the duration predictor remain limited and the subject of future work. Furthermore, high-quality zero-shot TTS can yield negative social impact through applications such voice impersonation of public figures and non-consenting individuals, which raise awareness for as a potential misuse of this technology.
Acknowledgements
This work was partially supported by the BK21 FOUR program of the Education and Research Program for Future ICT Pioneers, Seoul National University in 2023 and Institute of Information Communications Technology Planning & Evaluation (IITP) grant funded by the Korea govemn ment (MSIT) INO.2021-0-01:44 --3,333, Artificial Intelligence Graduate School Program (Scoul National University)].
10

---

References
[1] Adaeze Adigwe, Noé Tits, Kevin El Haddad, Sarah Ostadabbas, and Thierry Dutoit. The emotional voices database: Towards controlling the emotion dimension in voice generation systems. arXiv preprint /806v:8800.0999 14, 2018.
[2] Sercan Arik, Jitong Chen, Kainan Peng. Wei Ping, and Yanqi Zhou. Neural voice cloning with few samples. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
[3] Sercan Arik, Jitong Chen, Kainan Peng, Wei Ping, and Yanqi Zhou. Neural voice cloning with a few samples. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa- Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 31, Curran Associates, Inc., 2018.
[4] He Bai, Renjie Zheng, Junkun Chen, Mingbo Ma, Xintong Li, and Liang Huang. A Aligmment-aware acoustic and text pretraining for speech synthesis and editing. Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, edi- tors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 1399-1411, PMLR, 17-23 Jul 2022.
[5] Mathieu Bernard and Hadrien Titeux. Phonemizer: Text phones transcription for multiple languages in python. Journal of Open Source Software, 6(68):3958, 2021.
[6] James Betker. Tortoise.
2022.
[7] Edresson Casanova, Julian Weber, Christopher D Shulby, Arnaldo Candido Junior, Eren Gölge, and Moacir Ponti. Yourtts: Towards zero-shot muuli--peakkrr and zero-shot voice conversion for everyone. In International Conference on Machine Learning, pages 2709-2720. PMLR, 2022.
[8] Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, et al. Wavlm: Large-scale self-supervised pre- training for full stack speech processing. IEEE Journal of Selected Topics in Signal Processing, 1666):500--1118, 2022.
[9] Hyungjin Chung. Jeongsol Kim, Michael Thompson Mccann, Marc Louis Klasky, and Jong Chul Ye. Diffusion posterior sampling for general noisy inverse problems. In The Eleventh Interna- tional Conference on Learning Representations, 2023.
[10] Alexandre Défossez, Jade Copet, Gabriel Synnaeve, and Yossi Adi. High fidelity neural audio compression. arXiv preprint Stti:2210.134388 2022.
[11] Alexandre Défossez, Jade Copet, Gabriel Synnaeve, and Yossi Adi. High fidelity neural audio compression, 2022.
[12] Anmol Gulati, Chung-Cheng Chiu, James Qin, Jiahui Yu, Niki Parmar, Ruoming Pang, Shibo Wang, Wei Han, Yonghui Wu, Yu Zhang, and Zhengdong Zhang, editors. Conformer: Convolution-augmentee Transformer for Speech Recognition, 2020.
[13] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. arXiv preprint arxiv:2006. 1239, 2020.
[14] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NeurlPS 2021 Workshop on Deep Generative Models and Downstream Applications, 2021.
[15] Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and AbdeIrahman Mohamed. Hubert: self-super speech representation learning by masked prediction of hidden units. IEEE/ACM Transactions on Audio, Speech, and Language Processing. 29:3451-3444 2021.
[16] Aapo Hyvärinen and Peter Dayan. Estimation of non-normalized statistical models by score matching. Journal of Machine Learning Research, 6(4), 2005.

---

[17] Minki Kang, Dongchan Min, and Sung Ju Hwang. Any-speaker adaptive text-to text-to-speech synthesis with diffusion models. arXiv preprint arXiv:2211.09331, 2022.
[18] Tero Karras, Miik Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- generative models. In Proc. NeurIPS, 2022.
[19] Eugene Kharitonov, Damien Vincent, Zalán Borsos, Raphaël Marinier, Sertan Girgin, Olivier Pictquin, Matt Sharifi, Marco Tagliasacchi, and Neil Zeghidour. Speak, read and prompt: High-fidelity text-to-speech with minimal supervision. arXiv preprint arXiv:2332203350, 2023,
[20] Heeseung Kim, Sungwon Kim, and Sungroh Yoon. Guided-tts: Text-to-speech with untran- scribed speech. arXiv preprint arXiv:2/11.1113 2021.
[21] Jachyeon Kim, Sungwon Kim, Jungil Kong, and Sungroh Yoon. Glow-TTS: A Generative Flow for Texe-to-Speech via Monotonic Alignment Search. Advances in Neural Information Processing Systems, 33, 2020.
[22] Jungil Kong. Jachyeon Kim, and Jackyoung Bac. Hifi-gam: Generative adversarial networks for efficient and high fidelity speech synthesis. Advances in Neural Information Processing Systems, 33:17022-117 17033, 2020.
[23] Yaron Lipman, Ricky TQ Chen, Heli Ben- Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint arXiv:2110.02747, 2022.
[24] Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. arXiv preprint 2xXi::220900000333 2022.
[25] Zhijun Liu, Yiwei Guo, and Kai Yu. Diffvoice: Text-to-speech with latent diffusion. In ICASSP 2023 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5, 2023.
[26] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019.
[27] Dongchan Min, Dong Bok Lee, Eunho Yang, and Sung Ju Hwang. Meta-stylespeech: Multi- speaker adaptive text-to-speect generation. In International Conference on Machine Learning. pages
PMLR, 2021.
[28] Eloi Moliner, Jaakko Lehtinen, and Vesa Välimäki. Solving audio inverse problems with diffusion model. In ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5, 2023.
[29] Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob Megrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image genera- tion and editing with text-guided diffusion models. In International Conference on Machine Learning, pages 16784-16804, PMLR, 2022.
[30] Vadim Popov, Ivan Vovk, Vladimir Gogoryan, Tasnima Sadekova, and Mikhail Kudinov. Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speech. In Proceedings of the 38th International Conference on Machine Learning, ICM1, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 8599-8608. PMLR, 2021.
[31] Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. arXiv preprint 2rXiv:2202.005 12, 2022.
[32] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.025022 2020.
[33] Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. arXiv preprint arXivv2303.01466 2023.
[34] Yang Song, Sahaj Garg, Jiaxin Shi, and Stefano Ermon. Sliced score matching: A scalable approach to density and score estimation. In Ryan P. Adams and Vibhav Gogate, editors, Proceedings of The 35th Uncertainty in Artificial Intelligence Conference, volume 115 of Proceedings of Machine Learning Research, pages 574-584. PMLR, 22-25 Jul 2020.
12

---

[35] Alexander Tong, Nikolay Malkin, Guillaume Huguet, Yanlei Zhang, Jarrid Rector- Brooos Kilian Fatras, Guy Wolf, and Yoshua Bengio. Conditional flow matching: Simulation-free dynamic optimal transport. arXiv preprint arXiv: 20..222200042,, 2023.
[36] Aäron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew W. Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. arXiv preprint arXiv: 1609, 00499, 2016.
[37] Chengyi Wang, Sanyuan Chen, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, et al. Neural codec language models are zero-shot text speech synthesizers. arXiv preprint arXiv:2301.0211
2023.
[38] Yuxuan Wang, R. J. Skerry-Ryan, Daisy Stanton, Yonghui Wu, Ron Weiss, Navdeep Jaitly, Zongheng Yang, Ying Xiao, Z. Chen, Samy Bengio, Quoc V. Le, Yannis Agiomyrgiannakis. Robert A. J. Clark, and Rif Saurous. Tacotron: Towards end-to-end speech synthesis. In Interspeech, 2017.
[39] Junichi Yamagishi, Christophe Veaux, and Kirsten MacDonald. Cstr vetk corpus: English multi-speaker corpus for estr voice cloning toolkit (version 0.92). 2019.
[40] Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. Soundstream: An end-to-end neural audio codec. IEEE/ACM Transactions Audio, Speech, and Language Processing, 50:495-507, 2021.
[41] Heiga Zen, Viet Dang, Rob Clark, Yu Zhang, Ron J Weiss, Ye Jia, Zhifeng Chen, and Yonghui Wu. Libritts: A corpus derived from librispeech for text-to-speech, arXiv preprint 19Xiv:194.028800 2019.
13

---

A Additional Results
A.1 Efffet of Euler Steps for Acoustic Quality
We present the objective metrics according to the Fuler steps in the result section of the main paper. Since these objective metries have limitations in representing acoustic quality with respect to the Euler step, we also evaluate the sample quality based on the Euler steps and provide it as an additional metric. We measure the acoustic quality using 5-scale Mean Opinion Scores (MOS). We inquire each human evaluator to assess the acoustic quality of each sample, and we evaluate it with the participation of more than 50 human evaluators.
Table 5 presents MOS along with SECS and inference latency shown in the results section, based on the Fuler step N. The table demonstrates that the number of Fuler steps increases, the acoustic sample quality improves. We choose Fuler step 10 as the default, as it ensures a high speaker similarity while providing good balance between inference latency and sample quality.
Table Mean Opinion Scores (MOS) for the acoustic quality and Objective Metrics according to the Euler steps N.
MODEL
NN
MOS
SECS
INFERENCE LATENCY(S))
1
3.55
0.16
0.420
0.028 + 0.004
2
3.71
+ 0.12
0.522
0.037 + 0.004
P-FLOw
5
4.01
0.10
0.549
0.067 + 0.004
4.08
0.10
0.544
0.115 0.004
20
4.14
0.540
0.210 + 0.005
A.2 Model Comparison VCTK Dataset
Considering the training dataset LibriTTS and the test set of LibriSpeech as in-domain data, it appears important to demonstrate results for out-of-domain prompting data. Accordingly, we compare P-Flow and VALL-E samples using the VCTK dataset [39]. We generate samples using the same prompts and sentences that were used for the VCTK samples on the VALL-E demo page. We provide our generated samples on our demo page.
As in the main section, we use 10 ODE steps and guidance scale of 1. We measure WER and SECS for these VCTK samples and calculate the average values for each metric. To measure WER, we utilize the same ASR model as in Section 4. For SECS measurement, in the interest of fair comparison, unlike what we did in other sections of our paper, we measure the SECS between generated and ground truth samples, rather than using the re-synthesized or encoded and decoded samples. Due long leading and trailing silences in VCTK prompts, we measure SECS with the reference speech trimmed at 20dB.
We provide objective metrics for each model on the VCTK dataset in Table 7. Table 7 demonstrates that P-Flow achieves better WER and SECS compared to VALL-E. These results are similar to the LibriSpeech test clean results Section 4.2.
In addition, we collect comparative mean opinion scores (CMOS) and comparative speaker similarity mean opinion score (SMOS) for VCTK samples, using -2 to +2 scale. More than 50 human raters evaluate all the generated samples given 3-second reference data. Table 6 shows that human raters prefer P-Flow over VALL-E in terms of sample quality including acoustic quality, prosody and human likeness, and speaker similarity. These results show that P-Flow can achieve performance similar to or better than VALL-E on in-domain and out-of-domain data, in addition to providing users with an inference latency of 0.1 seconds using 10 ODE steps.
Table 6: Zero-Shot Speech Synthesis Subjective Metrics for VCTK dataset. Preference scores > 0 indicate preference for P-Flow.
MODEL
CMOS
SMOST
P-Frow VS VALL-E
0.188 + 0.10
0.267 + 0.166
14

---

Table Objective metries for sero TTS evaluated on the VCTK dataset. We compare WER and SECS using all the VCTK samples from the VALL-E demo page.
MODEL
WER
SECST
VALL-E
4.3
0.452
P-FLOW (PROPOSED)
2.4
0.465
A.3 Analysis on ODE sampling methods
We measure the pronunciation accuracy (WER) and speaker similarity (SECS) for two second-order ODE methods: midpoint and Heun's method, detailed [18]. We evaluate these two second-order methods for sampling steps N, which have a similar number of function evaluations as 10 Euler steps. Table 8 shows that the midpoint method with N produces wER- 2.7 7 and SECS= 0.540, not better than the Euler method. On the other hand, Heun's 's method improves upon Euler's method with a similar number function evaluations as described in [18].
Table 8: Objective metrics for P-Flow with different ODE sampling methods.
MODEL
WER
SECS1
P-FLOW (BULER METHOD, N 10)
2.6
0.544
P-FLOw (HEUN'S S METHOD, N = 4)
2.6
0.552
P-FLOw (MIDPOINT METHOD, N = 4)
2.7
0.540
A.4 Zero-shot TTS with Emotional Reference Speech
We provide generated samples using emotional reference samples, where each sample exhibits distinct prosody, as demonstrated [37]. We extract reference speech samples from EmoV-DB [1]. representing five different emotions. From each reference speech, we utilize 3-second segment to perform zero-shot TTS. On our demo page, we present generated samples for the same sentence given the speech prompts for these five emotions. P-Flow, similar to VALL-E, utilizes speech- text encoder composed of an autoregressive transformer, enabling the generation of samples with different prosody based on the reference speech.
B Model Architectures
We provide explanations for each module in this section and detailed hyperparameters and architecture of P-Flow are shown in Table 2.
Speech-prompted Text Encoder Our text-encoder consists of several linear projection layers, pre- network with 3 convolutional layers, and 6-layer transformer with 2 attention heads of 192 hidden dimensions. The input to the text encoder is the speech prompt and text embeddings projected into the same dimensions. For the input of the speech-prompted text encoder, we project the speech prompt and text embeddings into the same dimension and input to the same pre-network. The resulting representation then split into prompt and text parts, which positional encodings are added. We define each positional encoding the sum of absolute positional encoding and learnable fixed-size embedding so that the transformer can differentiate the speech prompt and text through learnable embeddings. The representations of the speech prompt and text are then fed into transformer architecture that allows each text position to attend to the speech prompt.
Duration predictor Our duration predictor is shallow convolution based model used in [21]. Since our text encoder output already provides speaker-condition representation before linear projection to h. as the input of the duration predictor.
hidden representation, we use the hidden
Flow matching Decoder Our flow matching decoder utilizes 18 layers of WaveNet-like architecture [36] with 512 hidden dimensions. We use the global conditioning method in WaveNet for condi- tioning and concatenate the aligned encoder output with the input xi along the channel axis for conditioning the speaker-conditional text representation.
15

---

Table 9: Hyperparameters of P-Flow
Hyperparameter
Phoneme Embedding Dim
192
PreNet Conv Layers
PreNet Hidden Dim
192
PreNet Kernel Size
PreNet Dropout
0.5
Transformer Layers
ppecch-prompped Text Encoder
Transformer Hidden Dim
Transformer Feed-forward Hidden Dim
768
Transformer Attention Heads
2
Transformer Dropout
0.1
Prompt Emhedding Dim
192
Number Parameters
3.37M
Conv Layers
Conv Hidden Dim
256
Duration Predictor
LayerNorm Layers
2
Dropout
0.1
Number of Parameters
0.36M
WaveNet Residual Channel Size
512
WaveNet Residual Blocks
18
Flow Matching Decoder
WaveNet Dilated Layers
WaveNet Dilation Rate
Number of Parameters
40.68M
16


Reviews:
[{"review_id": "nIKYh3mIEh", "reviewer": "Reviewer_9LwJ", "summary": "The paper proposes a zero-shot TTS model, P-Flow, which combines a speech-prompted text encoder and a flow-matching generative decoder. The encoder generates a speaker-conditional text representation using the target speech prompt and text, while the decoder utilizes conditional flow matching to model the conditional distribution of mel-spec.  The evaluation shows that P-Flow achieves comparable speaker adaptation performance to the large-scale autoregressive models (e.g., VALL-E) with significantly fewer training data and more than 20x faster sampling speed. Subjective testing also shows human listeners prefer the audios generated by P-Flow compared to VALL-E due to its better pronunciation accuracy and naturalness.  ", "questions": "1. how to calculate \\hat_{h} in the guided sampling? If it is training dataset dependent, how well it can generalize? \n2. During training, is d or \\hat_d used to expand h_c to h?\n3. How is the model performance with long text prompt?  e.g., 100s of words\n4. How duration model itself performs? Is it a bottleneck for further improving the model's performance?\n5. Compared to neural codec (Encodec), how much gain the hifi-gan vocoder itself can achieve?\n6. How about other ODE-solving algorithms? Is the current solution achieves the best trade-off between time cost and accuracy?", "limitations": "1. The paper discussed a little bit regarding the potential ethical and social implications of P-Flow, such as voice cloning and impersonation. I would suggest the author either develop another model to identify whether one audio is generated by P-Flow or use existing open-sourced fake audio detector to evaluate whether the synthesised audio can be identified from the real audios. \n", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The paper challenges the recent trend that using LM-similar training approach for speech synthesis, instead, it argues that non-autoregressive formulation using the cheaper traditional representations such as mel-spectrograms can make speech synthesis both fast and high-quality.\n2. The paper proposed a novel speech-prompted text encoder. Combining the text and speech prompt, this encoder tries extracting speaker information from the prompt and generating a speaker-conditional text representation. Specifically, the MAS algorithm is utilized to align the text encoder output with the mel-spectrogram, and the training target of the encoder is set to minimize the distance between the text encoder representation and the mel-spec. It not only distills speaker information to the encoder representation but also provides a way for end-to-end training for the whole system.\n3. The paper argues that it is sufficient to use the first-order Euler's method to solve the ODE during inference, instead of depending on the off-the-shelf numerical ODE solvers.\n4. The paper proposes to guide the sampling trajectory away from the average feature vector computed from h.\n5. The paper explains well why masked loss is necessary, i.e., preventing the model from collapsing to a trivial copy-pasting. Other works such as voicebox also utilizes masked loss but they do not provide a good explanation.   \n6. The paper conducts extensive experiments to evaluate the performance of P-Flow in terms of objective metrics (WER, SECS, inference latency) and subjective metrics (CMOS, SMOS). The paper also provides an ablation study to show the effectiveness of speech prompting and guidance methods. \n7. Both the experiment results and demo audios show that P-Flow achieves lower WER, competitive SECS, much smaller inference latency, and much higher human preference scores, compared with VALL-E.", "weaknesses": "1. LM-similar training approach, i.e., modeling the sequential discrete tokens in autogressive way, is supper good at long sequences, which has been demonstrated by the success of ChatGPT with very long input tokens. Since the paper is trying to argue that using non-autoregressive method with mel-spectrum representation is at least as good as the modeling metrology proposed in VALL-E, a very important experiment is to show the model's performance in terms of the very long prompt text, which is missing from this paper.\n2. The paper shows that P-Flow generated audios are preferred by human listeners compared to VALL-E, and it also points out that the difference is not only attributed to the model but also the performance degradation introduced by the neural codec. A very important experiment is to show compared to Encodec, what is performance gain can be achieved by the hifi-gan vocoder. This can help the readers to get a better insight regarding the potential of the model itself.\n3. duration predictor is very important in P-Flow, however the paper does not provide the experiment results of the duration predictor, at least it should be included in the supplemental materials. \n4. The paper uses the first-order Euler's method to solve the ODE during interference. How this simple inference comparing to the off-the-shelf ODE solvers? \n4. There are still some important things which are not clarified well. \n   1) \\hat_{h} is not well explained, how to calculate \\hat_{h}? \n   2) In table 1, the GT achieves higher WER comparing to P-Flow, why?\n   3) during inference, \\hat_{d} is used to expand h_c to h. How about training, do you use d or \\hat_d?\n   4) In table 2, the inference latency of VALL-E is approximated using the size of the transformer. The readers cannot tell how accurate the approximation is and I would suggest the authors to give more details regarding the approximation in the supplemental materials.\n5. There are some typos: Line 147, \"it also serves encourages the\" should be \"it also encourages\"; L176, \"objective (L_{dur})is to minimized\" should be \"objective (L_{dur}) is to minimize\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1 and Q3 - P-Flow with long prompt**\n\nConcerning P-Flow for long prompts, we began by taking inspiration from the VALL-E and SPEAR-TTS, which demonstrated effective voice cloning using only 3 seconds of voice data. Based on this, we fixed our speech prompt length at 3 seconds for all experiments. Our focus was on showing that non-autoregressive models can achieve similar levels of speaker adaptation using the 3-second prompt, as shown in the previous LM-similar approach. Exploring speaker adaptation using longer prompts would require training the model to perform generative modeling on variable-length segments masked during the training process. We leave this approach as future work.\n\n**W2 and Q5 - Gain achieved by using HiFi-GAN** \n\nWe believe that properly ablating on the effect of the vocoder or neural condec in this context is a very challenging task: on the one hand it would require adapting P-Flow, a flow-matching model operating on continuous data, to operate on discrete EnCodec codes; on the other hand, it would require adapting VALL-E, a Autoregressive Transformer operating on discrete codes, to operate on continuous data. We hope the reviewer would agree that each of these tasks is a research project on its own that would go beyond the scope of this rebuttal.\nNonetheless, we agree with reviewer 9LwJ that the choice of neural codec or vocoder is an important design choice. As such, we provide an upperbound in SECS by computing it between different ground truth audio and samples from the same speaker re-synthesized with EnCodec and HiFiGAN:\n\n- SECS (GT 3-sec prompt, GT) = 0.669\n- SECS (GT-EnCodec 3-sec prompt, GT-EnCodec) = 0.622\n- SECS (GT-HiFiGAN 3-sec prompt, GT-HiFiGAN) = 0.643 \n\nIn the case of using EnCodec, the SECS results obtained by resynthesizing with GT are lower compared to HiFi-GAN. This implies that P-Flow gains an advantage by using mel-spectrograms and HiFi-GAN instead of EnCodec.\n\n**W3 and Q4 - Duration predictor**\n\nP-Flow employs a deterministic duration predictor consisting of shallow convolutional layers. It takes the speaker-conditional output from the text encoder to regress phoneme duration values. We conducted model comparisons using the speaker-conditional durations predicted by the duration predictor. The results from CMOS or SMOS, among others, confirm the duration predictor's capability to generate reasonable phoneme durations.\n\nTo demonstrate the performance bound of the duration predictor itself, we employed ground truth audio, similar to the training process, to derive the optimal duration $d$ using the Monotonic Alignment Search (MAS) algorithm. By using this optimal duration $d$ during inference, we established an upper bound for SECS using the same encoder and decoder parameters as P-Flow. When employing the optimal duration, the upper bound SECS value was determined to be 0.572 using 10 Euler steps and guidance scale 1. This finding indicates that by adjusting durations alone, a significant improvement in speaker similarity can be achieved, compared to the 0.544 SECS score for P-Flow in Table 2 of the paper.\n\nIn an attempt to enhance duration prediction, we explored training a transformer-based duration predictor using regression loss. However, our efforts did not result in meaningful improvement over the 0.544 SECS score. We noted that straightforward architectural modifications to the duration predictor within P-Flow did not yield substantial improvements in duration prediction performance. As indicated in the conclusion, further advancements in duration prediction are reserved as a topic for future work in this paper.\n\n**W4 and Q6 - Different ODE samplers**\n\nWe reported only the results obtained using a simple first-order Euler method with a small number of sampling steps, as it provided favorable outcomes. As suggested by Reviewer 9LwJ, showing the results using different ODE sampling methods would greatly benefit the paper. This could have led to achieving an even better SECS, and we will incorporate these results in the appendix.\n\nWe measure the pronunciation accuracy (WER) and speaker similarity (SECS) for two second-order ODE methods: midpoint and Heun's method. We evaluated these two second-order methods for sampling steps N, which have a similar number of function evaluations as 10 Euler steps. \n\nThe midpoint method with N=4 produces WER=2.7 and SECS=0.540, not better than the Euler method. On the other hand, Heun's method improves upon Euler method:\n\n**Table. Objective metrics for P-Flow with different ODE sampling methods.**\n\n| Model        | WER| SECS |\n|---|:----:|:---:|\n| P-Flow (1st-order Euler method, N=10) | 2.6|0.544 |\n| P-Flow (2nd-order Heun's method, N=4) | 2.6|**0.552**|\n| P-Flow (2nd-order midpoint method, N=4) | 2.7|0.540|\n\n**W5 and Q1, 2 - Missing details**\n\n- $\\bar{h}$ is a representation that involves averaging the expanded representation $h$ along the time axis to obtain a fixed-size vector, which is then duplicated along the time axis. This is not a pre-computed value derived from the training dataset; rather, it is a representation computed during the inference process by averaging and duplicating the expanded representation $h$.\n\n- We apologize for the confusion. It was our mistake. WER and SECS values for GT in Table 1 must be the values in Table 2: WER=2.4 and SECS=0.64.\n\n- As in line 153, during training, we use the duration $d$ obtained by MAS to expand $h_c$ to $h$. \n\n- We will provide specific details about measuring the inference latency of VALL-E in the appendix. Following VALL-E's paper, we measure inference on a 12-layer transformer with a hidden size of 1024 and 16 heads. Given that VALL-E's neural audio codec representation is at 75Hz, we approximate the inference latency of VALL-E by measuring the latency required for 75*n + 7 forward passes to generate $n$ seconds of audio. \n\n**W6 - typos**: \nWill be corrected"}, {"review_id": "G2KCq7TcnC", "reviewer": "Reviewer_Bg7K", "summary": "The paper introduces P-Flow, a novel zero-shot text-to-speech (TTS) model that addresses the limitations of existing large-scale neural codec language models. P-Flow utilizes speech prompts for speaker adaptation and consists of a speech-prompted text encoder and a flow-matching generative decoder. The text encoder generates a speaker-conditional text representation using speech prompts and text input, while the generative decoder synthesizes high-quality personalized speech at a significantly faster speed than real-time. P-Flow is trained on the LibriTTS dataset using continuous mel-representation and achieves comparable speaker similarity performance to large-scale zero-shot TTS models but with significantly less training data and more than 20 times faster sampling speed. The results demonstrate that P-Flow offers improved pronunciation and is preferred in terms of human likeness and speaker similarity compared to state-of-the-art alternatives.", "questions": "- Can you provide visualizations of the generated mel-spectrograms to assist us in assessing the quality of the synthesized speech?\n- Why do you prefer using the flow-matching model as the decoder instead of diffusion models like DDPM? What advantages does this model offer in zero-shot TTS tasks? If you're looking to speed up inference, there are also numerous acceleration methods available for diffusion models.\n- Libri-TTS is derived from LibriSpeech and can be considered as in-domain data. Can you provide results using out-of-domain prompting data, such as speech audio from podcast or YouTube, to assess the zero-shot cloning performance in real scenarios?", "limitations": "The authors have adequately addressed the limitations.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- Novel approach: P-Flow introduces a unique methodology for zero-shot TTS that combines speech prompts and text input for speaker adaptation, resulting in improved speech synthesis quality and faster synthesis speed.\n- Data efficiency: P-Flow achieves comparable speaker similarity performance to large-scale models but with only a fraction of the training data, making it more data-efficient.\n- Improved sampling speed: P-Flow demonstrates a significant improvement in sampling speed compared to previous autoregressive TTS methods and large-scale neural codec language models, enabling real-time or near-real-time synthesis.\n- High-quality synthesis: The flow-matching generative decoder in P-Flow produces high-quality personalized speech, enhancing the naturalness and human-likeness of the synthesized audio.\n- Preference over state-of-the-art: P-Flow is preferred over recent state-of-the-art counterparts in terms of pronunciation, human likeness, and speaker similarity, making it an attractive alternative for zero-shot TTS.", "weaknesses": "- The motivation of using the flow-matching-based decoder is not very clear. This mask and prediction training schema can be applied to other generative models such as GAN and DDPM/DDIM. What makes you believe that the flow-matching model is the most appropriate choice?\n- The evaluation of zero-shot performance in real scenarios is not verified. Although this paper assures no overlap exists with our training data, as Libri-TTS (the training set) is derived from LibriSpeech (the test set), the current results are not yet sufficiently convincing to demonstrate its ability and performance in real zero-shot scenarios.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Motivation for using a flow matching decoder**\n\nThe precursor papers on conditional flow matching [1, 2] provided evidence of the advantages of flow matching versus DDPM, especially flow matching's ability to simulate simpler trajectories while retaining good sample quality with fewer ODE steps than DDPM.\n    \nThus, in the initial phase of our experiments, we evaluated sample quality across various sampling steps using flow matching and DDPM decoders on a single-speaker TTS dataset, LJSpeech. Both models consist of the same text encoder (6-layer Transformer), alignment search algorithm (MAS), and decoder (18-layer WaveNet). The model using the DDPM decoder was trained for 400K iterations with an encoder loss + score matching loss, while the model using the flow matching decoder was trained for 400K iterations with encoder loss + flow matching loss. Additionally, we evaluated both models using Euler ODE sampling without classifier-free guidance. In our initial experiments, we did not use speech prompting and masking in P-Flow.\n\nWe uploaded to our demo page (Rebuttal 2 Section) samples from two models to ablate and support our choice in using flow-matching.  The model utilizing the DDPM decoder is akin to Grad-TTS, wherein a zero mean prior is employed to define forward and reverse processes. The second model utilizes the flow matching decoder, and can be seen as a variant of P-Flow without speech prompt and masking. We sample from these models with varying amounts of sampling steps. Consistent with the results from the Grad-TTS demo page, the DDPM's decoder sample quality degrades severely with fewer than 50 steps. In contrast, the flow matching decoder produces high-quality samples with as low as 10 ODE sampling steps. Moreover, we observed that even with very few sampling steps, such as 2 or 4, this model generates reasonably good quality speech.\n\nGiven these results, we decided to develop on top of the strong flow matching baseline instead of finding mechanisms, such as distillation, to improve the DDPM decoder such that it meets the flow matching baseline. Because these initial experiments do not directly relate to zero-shot speaker adaptation, we chose not to include them in the paper at the time of submission. We will revise our draft to include them in the appendix.\n\n**Visualizations of the generated mel-spectrograms**\n\nAs requested by Reviewer Bg7K, we present visualizations of mel-spectrograms generated by P-Flow in the official comment PDF file. These mel-spectrograms are generated using three sentences used for model comparison in the demo.\n\n**OOD prompting data**\n\nAs pointed out by Reviewer Bg7K and KPCd, considering the training dataset LibriTTS and LibriSpeech's test set as in-domain data, it appears important to demonstrate results for out-of-domain prompting data. Accordingly, we compared P-Flow and VALL-E samples using the VCTK dataset. We generated samples using the same prompts and sentences that were used for the VCTK samples on the VALL-E demo page. We have also provided our generated samples in the “Rebuttal 1” section of our demo page. We kindly ask the reviewers to check our VCTK experiments in the updated demo page. \n\nWe generated samples using 10 ODE steps and a guidance scale of 1. We measured WER and SECS for these VCTK samples and calculated the average values for each metric. To measure WER, we utilized the same ASR model as in our paper. For SECS measurement, in the interest of a fair comparison, unlike what we did in other sections of our paper, we measured the SECS between generated and ground truth samples, rather than using the re-synthesized or encodec and decoded samples. Due to long leading and trailing silences in VCTK prompts, we additionally measured SECS with prompts trimmed at 20dB.\n\nThe results below provides objective metrics for this experiment on the VCTK dataset and show that P-Flow achieves better WER and SECS similar to VALL-E when leading silence included. When leading silence is trimmed, P-Flow exhibits better SECS than VALL-E. These results are similar to the LibriSpeech test clean results in Section 4.2.\n\n**Table. Model comparison on VCTK dataset (Objective evaluation)**\n\n| Model        | WER| SECS (w/o trimming)                | SECS (w/ trimming prompt)  |\n|--------------|:--------------------------------------------:|:-------------------------------:|-------:|\n| VALL-E | 4.3|0.446 | 0.452 |\n| P-Flow | 2.4|0.441 | 0.465 | \n\nIn addition, we collected Comparative Mean Opinion Scores (CMOS, ~ 50 human raters) and comparative Speaker similarity Mean Opinion Score (SMOS, ~ 100 human raters) for the above mentioned out-of-distribution samples, using a -2 to +2 scale. The results below show that human raters prefer P-Flow over VALL-E in terms of sample quality (acoustic fidelity, prosody and human-likeness) and speaker similarity.\n\n**Table. Model comparison on VCTK dataset (Subjective evaluation).**\n\n| Model        | CMOS $\\uparrow$| SMOS $\\uparrow$ |\n|--------------|:--------------------------------------------:|:-------------------------------:|\n| P-Flow vs. VALL-E | 0.188 $\\pm$ 0.10|0.267 $\\pm$ 0.166 |\n\nThese results show that P-Flow can achieve performance similar or better than VALL-E on in-domain and out-of-domain data, in addition to providing users with an inference latency of 0.1 seconds using 10 ODE steps.\n\nAdditionally, the results we presented for emotional reference speech in the demo page and Appendix 2.2 section were generated using out-of-domain data, the EmoV-DB dataset.\n\n[1] Lipman et al., \"Flow Matching for Generative Modeling\"\n\n[2] Tong et al., \"Improving and generalizing flow-based generative models with minibatch optimal transport\""}, {"review_id": "BFtpPlvn10", "reviewer": "Reviewer_zUdt", "summary": "This work propose a flow-matching zero-shot TTS model called P-Flow. It is fast and data-efficient in comparison with Vall-E.", "questions": "No", "limitations": "No", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "I like the idea of flow matching, and it seems a new fashion for generative tasks. I do believe flow-matching will benefit the speech generation community.\nThe idea of prompting speaker encoder is novel and potentially effective.\nI believe the clarity and quality of the presentation are good and easy to follow.\nThe samples in the demo page are promising.\nThe major claims are supported with in-depth ablation studies.\n", "weaknesses": "The overall novelty is limited. The main architecture of this model is very similar to recent NAR TTS models such as Glow-TTS, Grad-TTS. For example, all these models leverages MAS to compute alignments, and all these models have a powerful probabilistic decoder that\nallows to learn complex distribution from known latent prior, such as normalizing flow (Glow-TTS), DDPM (Grad-TTS), and flow matching (P-Flow).\n\nFrom my side, the experimentation is far from sufficient. As I said, this work is very similar to NAR TTS models like Glow-TTS and Grad-TTS. A direct comparison of P-Flow with Glow-TTS and Grad-TTS in the high quality TTS synthesis is essential, or the authors could report\nthe comparison of P-Flow with the promoting version of Glow-TTS or Grad-TTS in zero-shot scenarios.\n\nThe vocoder is trained separately.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We agree that our base model architecture is similar to existing Non-AutoRegressive (NAR) TTS methods (Glow-TTS and Grad-TTS). However, we believe our work provides valuable insight into the current state of zero-shot TTS research. The recently proposed language modeling-based zero-shot TTS model, VALL-E, achieved a breakthrough in the zero-shot TTS task, surpassing existing NAR zero-shot TTS models by a significant margin. Our work draws into question whether certain design choices in current SOTA models are truly necessary, and hopefully provides some guidance on promising directions of future research in this topic. Ideally, we would like to maintain the computational efficiency of NAR TTS frameworks while achieving zero-shot TTS capabilities at the lever of large autoregressive models such as VALL-E.\n\nWe consider the reliance on a fixed-size speaker embedding vector for speaker adaptation in these NAR zero-shot TTS models as a factor that hinders zero-shot speaker adaptation compared to VALL-E's speech prompting method. To achieve our goal, we closely followed the base components of existing NAR TTS models and introduced speech prompting-based conditioning for zero-shot speaker adaptation. By introducing speech prompting-based speaker adaptation, we significantly enhance speaker similarity of NAR zero-shot TTS to a level compared to VALL-E, while retaining NAR models' faster inference speeds. Furthermore, our model has the advantage of not requiring a pre-trained speaker encoder, unlike the existing NAR zero-shot TTS models.\n\nReviewer zUdt makes a reasonable request about providing a comparison to Glow-TTS or Grad-TTS. While we did not perform a zero-shot TTS comparison, **we provide a DDPM vs Flow Matching ablation on our demo page (Rebuttal 2)**, also detailed in our response to reviewer Bg7K. This section includes uploaded samples from two models to ablate our choice of using flow-matching. The model utilizing the DDPM decoder is akin to Grad-TTS, wherein a zero mean prior is employed to define forward and reverse processes. The second model utilizes the Flow matching decoder, and can be seen as a variant of P-Flow without speech prompt and masking. We sample from these models with varying amounts of sampling steps. Consistent with the results from the Grad-TTS demo page, the DDPM's decoder sample quality degrades severely with fewer than 50 steps. In contrast, the flow matching decoder produces high-quality samples with as low as 10 ODE sampling steps. Moreover, we observed that even with very few sampling steps, such as 2 or 4, this model generates reasonably good quality speech.\n\nWe did not find it necessary to compare directly against these models because YourTTS [1], based on the superior VITS model already significantly lags behind VALL-E regarding speaker adaptation performance. Further, the extension of Glow-TTS to zero-shot TTS, SC-GlowTTS [2], also shows inferior zero-shot speaker adaptation performance compared to YourTTS. \n\n\n[1] Casanova et al., \"YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for everyone\"\n\n[2] Casanova et al., \"SC-GlowTTS: an Efficient Zero-Shot Multi-Speaker Text-To-Speech Model\""}, {"review_id": "vUWkL4zLzn", "reviewer": "Reviewer_KPCd", "summary": "The paper proposes P-Flow that can achieve high speaker similarity performance and fast inference speed on the zero-shot TTS Task. To improve the speaker similarity score, P-Flow uses  a speech prompted text encoder to generate speaker-conditional text representation for speaker adaptation. To achieve fast inference speed and high audio quality, P-Flow incorporates the flow matching generative decoder with the speaker-conditional text representation. Experimental results demonstrates that P-Flow trained on a relatively small dataset matches the speaker similarity performance of the large-scale zero-shot TTS models and has more than 20× faster sampling speed.", "questions": "1. The audio samples provided by the authors on the demo page exhibit unstable timbre issues. For example, in ''Prompting v.s. Separate Speaker Encoder'', the timbre of the example 4 of P-Flow is unstable. Moreover, in ''Model Comparison'', the timbre of the example 1 of P-Flow is also unstable (e.g., ''...had fulfilled his mission.''). The authors should explain about this phenomenon.\n2. The paper claims that P-Flow ''matches the speaker similarity performance of the large-scale zero-shot TTS models with two orders of magnitude less training data''. But the unstable timbre issue in Point 1 may due to the limited training data. I think the generalization capacity of the model should be further evaluated. For example, the LibriSpeech test-clean dataset (test set) and the LibriTTS dataset (training set) are in the same domain (both derived from read audiobooks from the LibriVox project). Further experiments on the VCTK or other datasets should be conducted to validate the generalization capacity of P-Flow with out-of-domain data.\n3. In Section 4.1, the authors compare the proposed prompt-based speaker conditioning mechanism with the baseline that encodes all the speaker information into a fixed-size vector. After listening to the audio samples in the demo page, I believe the overall objective speaker similarity is improved by the proposed speech prompting approach. But the unstable timbre issue in Point 1 may also greatly influence the subjective results. The authors should conduct subjective evaluations for the ''Prompting v.s. Separate Speaker Encoder'' experiment to make it more convincing.\n4. In Section 1, the paper claims that large scale datasets of tens of thousands of hours of audio data has not been convincingly proven to be necessary. I agree that P-Flow achieves similar results with VALL-E (a well-known large-scale TTS model). However, can the speaker similarity score of P-Flow be further improved by being trained on a relatively larger datasets? More experiments should be conducted to validate it.", "limitations": "Yes, the authors have addressed some of the impacts and limitations in Section 5.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed speech prompt approach for the non-autoregressive zero-shot TTS model is interesting, which surpasses the speaker embedding approach and provides in-context learning capabilities for speaker adaptation.\n2. In experiments, P-Flow shows comparable speaker adaptation performance to the large-scale autoregressive baseline using significantly fewer training data and a small transformer-based encoder.\n3. The flow matching generative decoder enjoys a faster sampling speed than previous neural codec language models while maintaining a good audio quality.", "weaknesses": "After listening to the audio samples in the demo page, my main concerns are mainly related to the unstable timbre issue. Please refer to the following Questions section for details.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1,3. Regarding timbre instability**\n\nAfter careful analysis, we believe that the event observed by the reviewer is a change in the fundamental frequency range. While timbre is defined to be F0 invariant [1], it is possible that a change in fundamental frequency range can be perceived as a change in timbre. Furthermore, given the nature of the data used to train the model (LibriTTS, narration of books), it is possible that the voice actors modify their voices to represent different characters in the story and, as such, it is possible that the model will replicate this behavior. Given our recent CMOS and SMOS results on VCTK and earlier results on LibriSpeech's test set, we believe that these events are either rare or marginally significant for human raters, given their consistent preference for P-Flow over VALL-E.\n\n**Q2. Generalization capacity of P-Flow with OOD data**\n\nAs pointed out by Reviewer Bg7K and KPCd, considering the training dataset LibriTTS and LibriSpeech's test set as in-domain data, it appears important to demonstrate results for out-of-domain prompting data. Accordingly, we compared P-Flow and VALL-E samples using the VCTK dataset. We generated samples using the same prompts and sentences that were used for the VCTK samples on the VALL-E demo page. We have also provided our generated samples in the “Rebuttal 1” section of our demo page. We kindly ask the reviewers to check our VCTK experiments in the updated demo page. \n\nWe generated samples using 10 ODE steps and a guidance scale of 1. We measured WER and SECS for these VCTK samples and calculated the average values for each metric. To measure WER, we utilized the same ASR model as in our paper. For SECS measurement, in the interest of a fair comparison, unlike what we did in other sections of our paper, we measured the SECS between generated and ground truth samples, rather than using the re-synthesized or encodec and decoded samples. Due to long leading and trailing silences in VCTK prompts, we additionally measured SECS with prompts trimmed at 20dB.\n\nThe results below provides objective metrics for this experiment on the VCTK dataset and show that P-Flow achieves better WER and SECS similar to VALL-E when leading silence included. When leading silence is trimmed, P-Flow exhibits better SECS than VALL-E. These results are similar to the LibriSpeech test clean results in Section 4.2.\n\n**Table. Model comparison on VCTK dataset (Objective evaluation)**\n\n| Model        | WER| SECS (w/o trimming)                | SECS (w/ trimming prompt)  |\n|--------------|:--------------------------------------------:|:-------------------------------:|-------:|\n| VALL-E | 4.3|0.446 | 0.452 |\n| P-Flow | 2.4|0.441 | 0.465 | \n\nIn addition, we collected Comparative Mean Opinion Scores (CMOS, ~ 50 human raters) and comparative Speaker similarity Mean Opinion Score (SMOS, ~ 100 human raters) for the above mentioned out-of-distribution samples, using a -2 to +2 scale. The results below show that human raters prefer P-Flow over VALL-E in terms of sample quality (acoustic fidelity, prosody and human-likeness) and speaker similarity.\n\n**Table. Model comparison on VCTK dataset (Subjective evaluation).**\n\n| Model        | CMOS $\\uparrow$| SMOS $\\uparrow$ |\n|--------------|:--------------------------------------------:|:-------------------------------:|\n| P-Flow vs. VALL-E | 0.188 $\\pm$ 0.10|0.267 $\\pm$ 0.166 |\n\nThese results show that P-Flow can achieve performance similar or better than VALL-E on in-domain and out-of-domain data, in addition to providing users with an inference latency of 0.1 seconds using 10 ODE steps.\n\n**Q4. P-Flow with More Data**\n\nOur claim that large-scale datasets are unnecessary was due to poor choice in phrasing, which we thank the reviewer for pointing out. Our intent was only to demonstrate improved data efficiency, and not that large-scale datasets provide no benefit. We will rectify our phrasing in the draft.\n\nTo better support our data efficiency claim, we fine-tuned P-Flow on 5K hours of data for 100K iterations, one order of magnitude more data than our original experiments, and computed WER and SECS on LibriSpeech-Test-Clean. Our results below are very promising, and suggest that with more data we can achieve even better speaker similarity while retaining good pronunciation accuracy (WER). We believe this is a promising direction to further explore, and expect that larger datasets of around 60K hours, similar in scale to what is used in VALL-E, could further improve P-Flow's speaker similarity score.\n\n**Table. Objective metrics for P-Flow trained with LibriTTS and fine-tuned with MLS 5K hours of data**\n\n| Model      | WER                | SECS |\n|--------------|:-------------------------------:|-------:|\n| P-Flow | 2.6 | 0.544 |\n| P-Flow (fine-tuned w/ MLS 5K hours of data, 100K iter) | 2.6 | 0.553 |\n\n[1] https://uen.pressbooks.pub/auralskills/chapter/timbre/\n\n[2] https://www.microsoft.com/en-us/research/project/vall-e-x/vall-e/"}]

Author rebuttal:
Thank you for taking the valuable time to review our paper and providing helpful feedback. All the questions will greatly contribute to improving our paper. 

We have provided responses to the weaknesses and questions mentioned by each reviewer. Additionally, we have included figures and tables for additional results in the attached PDF. In response to Reviewer Bg7K's request, we have added figures in the PDF showing the mel-spectrograms generated using P-Flow.

To summarize the tables presented in the PDF:
1. In response to Reviewer 9LwJ's suggestion about using a different ODE method instead of the Euler method for sampling, we conducted experiments as suggested and included the results in Table 1. We demonstrated that using the 2nd order Heun's method yields better objective metrics for speaker similarity compared to the Euler method.
2. Furthermore, as requested by Reviewers Bg7K and zUdt, we conducted experiments on out-of-domain data using LibriTTS and LibriSpeech. The results of these experiments are shown in Tables 2 and 3. We compared the performance of P-Flow and VALL-E on the out-of-domain VCTK dataset and demonstrated that P-Flow performs well even in an OOD domain it hasn't been trained on. The generated samples of VCTK using VALL-E and P-Flow have been uploaded to the existing demo page.
3. In response to Reviewer zUdt's query about the effects of using a larger dataset, we present the results in Table 4. We showed that fine-tuning the previously trained P-Flow on 5K hours of data can improve SECS.

Lastly, as Reviewer Bg7K asked for an explanation of the motivation behind the use of the flow matching decoder, we have uploaded additional sample results on the demo page. These results are from a single-speaker TTS task, comparing the flow matching decoder trained on LJSpeech with the DDPM decoder from the early stages of our research.

Meta-review: This paper introduces P-Flow, a fast and efficient zero-shot Text-to-Speech (TTS) model that utilizes speech prompts for speaker adaptation. P-Flow is capable of achieving speech sampling that is 20 times faster. The authors’ rebuttals successfully addressed the concerns raised by the reviewers, resulting in three out of four reviewers raising their scores. I recommend that this paper be accepted.
Decision: Accept (poster)