SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The high-level model architecture is shown in Fig. 1. Our model comprises 3 main components: the prompt-based text encoder, a duration predictor to recover phoneme durations during inference, and a Wavenet-based flow matching decoder. Experiments demonstrate strong zero-shot results despite our text-encoder comprising only a small transformer architecture of 3M parameters. We provide additional architectural details for each component in Section B.
Training and Inference Settings: P-Flow is trained on a single NVIDIA A100 GPU for 800K iterations, using a batch size of 64. We utilize the AdamW optimizer [26] with a learning rate of 0.0001. A G2P model [5] preprocesses the text into the International Phonetic Alphabet (IPA) format. During inference, we generate mel-spectrograms using 10 Euler steps in the flow matching decoder with a guidance scale of 1. Mel-spectrogram to waveform is performed using the pre-trained universal Hifi-GAN, available in the HiFi-GAN repository<sup>3</sup>. To be compatible with Hifi-GAN, our audio representation is  $22\mathrm{kHz}$  audio represented with an 80-bin mel-spectrogram, with FFT parameters:
<sup>3</sup>Hifi-GAN: https://github.com/jik876/hifi-gan
window size 1024 and hop length 256. In this setup, a 3-second mel-spectrogram is fed as a speech prompt with a length of  $\left\lceil \frac{3*22050}{256}\right\rceil = 259$
Data: We train P-Flow on LibriTTS [41]. LibriTTS training set consists of 580 hours of data from 2,456 speakers. We specifically use data that is longer than 3 seconds for speech prompting, yielding a 256 hours subset. For evaluation, we follow the experiments in [37, 19] and use LibriSpeech test-clean, assuring no overlap exists with our training data. We resample all datasets to  $22\mathrm{kHz}$ .
Evaluation: We compare P-Flow with two zero-shot speaker-adaptive TTS models, YourTTS [7] and VALL-E [37]. YourTTS is a VITS-based multi-speaker TTS, which performs zero-shot speaker adaptation through speaker embedding extracted from a pre-trained speaker encoder. VALL-E achieves zero-shot speaker adaptation through prompting, resulting in much higher speaker similarity than YourTTS. We compare our model with the two models in the same way as described in [37], and the objective metrics for each model were directly taken from [37].
Replicating the experiment in [37, 19], we evaluate samples ranging from 4 to 10 seconds from the LibriSpeech test-clean dataset, resulting in a total of 2.2 hours of data. For each paired data  $(x_{i},c_{i})$ , we extract a 3-second reference speech  $x_{j}^{p}$  from a different sample  $x_{j}$  spoken by the same speaker and generate the synthesized speech  $x_{i}^{gen}$  for the text  $c_{i}$ .
Objective Metrics: We measure the inference latency and two objective metrics from [37], word error rate (WER), and speaker embedding cosine similarity (SECS). Inference latency of P-Flow and the autoregressive baseline, VALL-E are evaluated on an NVIDIA A100 GPU. To compute inference latency for P-Flow, we average the text-to-mel generation time for all the samples used in our evaluation for LibriSpeech. Because VALL-E is not open-sourced, we use a PyTorch transformer<sup>4</sup> of the same size as used in [37] and feed random tensors repeatedly corresponding to the duration of  $x_{i}$  to obtain a proxy for the inference latency.<sup>5</sup>
We evaluate pronunciation accuracy with WER, using the HuBERT [15] ASR model to measure transcription errors from generated samples. Speaker similarity is measured with SECS between the generated sample  $x_{i}^{gen}$  and the re-synthesized output of the 3-second reference speech  $x_{j}^{p}$  as in [37]. We compute the speaker embedding using the pre-trained WavLM-TDNN [8] used in [37]. We also measure the WER and SECS for the re-synthesized output of the ground truth audio  $x_{i}$  as an upper bound, reported as "GT (HIFI-GAN)" in Table 2. Note that the final mapping to waveform is different between models: P-Flow uses Hifi-GAN [22] while VALL-E uses EnCodec [11], necessarily influencing automatic metrics. We partially address this with our user study setup for speaker similarity.
User Study: We measure user preference score between VALL-E and P-Flow. Qualified evaluators must first pass hearing test where they count the number of short sinusoidal audio segments within an audio clip. We obtain evaluation scores for 8 samples from each evaluator with a minimum of 30 evaluators per comparison experiment. We evaluate the preference for naturalness, acoustic quality, and human likeness using comparative mean opinion score (CMOS). Preference for speaker similarity is reported using comparative speaker similarity mean opinion score (SMOS). SMOS evaluators are provided with a 3-second reference audio, which is the ground truth audio rather than re-synthesized audio to avoid being influenced by vocoder artifacts. Each preference score ranges from -2 to 2, using pairwise comparisons between the LibriSpeech samples from the VALL-E demo page.
4.1 Prompting v.s. Separate Speaker Encoder
This ablation study demonstrates the effectiveness of speaker conditioning through speech prompting. For the baseline, instead of directly inputting the speech prompt into the text encoder, we encode a random segment of speech  $x^{p}$  to a fixed-size speaker embedding using a speaker encoder with the same architecture as the text encoder. The speaker embedding is obtained by averaging the last hidden representations of the speaker encoder along the time axis and normalizing it. This speaker embedding is concatenated with the text input of the text encoder. The speaker encoder is jointly trained with the model and follows the same training objective, except for excluding the loss mask
$^{4}$ https://pytorch.org/docs/stable/generated/torch.nn.TransformerEncoder.html
$^{5}$ Following [37], we measure inference on a 12-layer transformer with a hidden size of 1024 and 16 heads. Given that VALL-E's neural audio codec representation is at  $75\mathrm{Hz}$ , we approximate the inference latency of VALL-E by measuring the latency required for  $75*n + 7$  forward passes to generate  $n$  seconds of audio.
Table 1: Objective metrics for ablation study. "P-Flow (w/o Prompt)" refers to the baseline conditioning on a fixed-size speaker embedding.
MODEL	WER↓	SECS↑
GT (HIFI-GAN)	2.4	0.64
P-FLOW (W/O PROMPT)	2.9	0.373
P-FLOW	2.6	0.544
for the speech prompt. For evaluation, we also sample using 10 Euler steps with a guidance scale of 1, similar to P-Flow.
In Table 1, we compare the objective metrics of P-Flow and the baseline for the fixed-size speaker embedding approach. Despite having similar WER values, it can be observed that introducing speech prompt-based speaker conditioning significantly enhances SECS compared to the baseline. This confirms that even without changing other aspects, the use of speech prompts leads to a substantial enhancement in speaker similarity. Through this ablation study, we demonstrate that the speech prompt-based approach in P-Flow, which utilizes the speech prompt as direct input and adapts it through the self-attention mechanism to optimize the loss for the remaining segment, is more effective for zero-shot TTS compared to encoding all the speaker information into a fixed-size vector.
Table 2: The amount of training data and objective metrics for zero-shot TTS models. The SECS of "GT (HIFI-GAN)" is computed between the re-synthesized output of 3-second reference speech and that of ground truth. WER and SECS for the baselines are taken from [37]. The inference latency for VALL-E is approximated using the same size of transformer architecture, as mentioned in Section 4.
MODEL	DATA (HOURS)	WER↓	SECS↑	INFERENCE LATENCY(S)↓
GT (HIFI-GAN)	2.4	0.64
YOURTTS†	500+	7.7	0.337
VALL-E†	60,000	5.9	0.580	2.515 ± 0.040
VALL-E CONTINUALT	60,000	3.8	0.508	2.515 ± 0.040
P-FLOW (PROPOSED)	260	2.6	0.544	0.115 ± 0.004
4.2 Model Comparison
Samples for each model can be accessed through the demo page. We highly encourage reviewers to listen to the samples.
Pronunciation Accuracy and Sample Quality: Our results in Table 2 show that P-Flow offers similar WER to GT (HIFI-GAN) and demonstrates significantly better pronunciation accuracy compared to other baselines while being trained on two orders of magnitude less data. This indicates that with training on the more accurately transcribed LibriTTS dataset, we can achieve pronunciation accuracy close to the ground truth.
Table 3: Zero-Shot Speech Synthesis Subjective Metrics. Preference scores  $\geq 0$  indicate preference for P-Flow.
MODEL	CMOS↑	SMOS↑
P-FLOW VS VALL-E	0.27 ± 0.10	0.23 ± 0.13
Speaker Similarity Metrics: Compared to the previous speaker embedding-based zero-shot TTS model, YourTTS, P-Flow demonstrates more effective speaker adaptation through speech prompting using a similar amount of training data. Considering our model text-encoder only has 3M parameters, competitive performance with large-scale autoregressive baselines suggests achieving high SECS does not require tens of thousands of hours of audio data, quantization, neural codec models, or
$^{6}$  Demo: https://research.nvidia.com/labs/adlr/projects/pflow
large autoregressive models. Further, combining our speaker conditioning ablation study with these results and the fact that our decoder does not have any additional speaker conditioning beyond what is provided by the text encoder, we believe that prompting is one of the most important factors in obtaining high SECS. Finally, we see higher preference scores for P-Flow than for previous SOTA zero-shot models in the human evaluation of speaker similarity with the ground truth audio in Table 3. This indicates that our speech-prompted text encoder enhances the in-context learning capabilities, similar to large-scale autoregressive transformers.
CMOS: In Table 3, we can observe the CMOS results of pairwise comparisons between samples from VALL-E and P-Flow, evaluating naturalness, prosody, and audio quality. Table 3 demonstrates that P-Flow exhibits significantly better sample quality compared to VALL-E. This CMOS result shows that generative modeling using continuous representation, which has been widely used in TTS, exhibits superior performance compared to generation using discrete tokens. We attribute this result not only to the performance difference between the models but also to the additional performance degradation introduced during the encoding-decoding process using neural codec representation. It highlights the advantages of continuous representation and showcases its effectiveness.
Inference Latency: Defaulting to 10 Euler steps for P-Flow's decoder, we measure inference latency for audio samples from LibriSpeech with an average length of 5.6 seconds. Tables 2 and 3 show our model achieves an inference latency of 0.1 seconds, more than 20 times faster than VALL-E, while maintaining high speaker similarity, sample quality, and pronunciation accuracy. While the autoregressive baseline has a generation time proportional to the duration of the audio, the non-autoregressive P-Flow maintains nearly constant latency regardless of the length. A 20-second audio sample would take P-Flow 0.12 seconds in our experimental setup.
Table 4: Effect of guidance scale  $\gamma$  and Euler steps  $N$  for our flow matching generative decoder. Our experiments find that automatic metric numbers plateau quickly after 5 steps, but we continue to hear qualitative improvements up to 10. Significant improvements are also seen due to the guidance.
MODEL	γ	N	WER↓	SECS↑	INFERENCE LATENCY(S)↓
P-FLOW (DEFAULT)	1	10	2.6	0.544	0.115 ± 0.004
P-FLOW	0	10	3.7	0.492	0.115 ± 0.004
P-FLOW	2	10	2.6	0.546	0.115 ± 0.004
P-FLOW	1	1	2.7	0.420	0.028 ± 0.004
P-FLOW	1	2	2.9	0.522	0.037 ± 0.004
P-FLOW	1	5	2.6	0.549	0.067 ± 0.004
P-FLOW	1	20	2.7	0.540	0.210 ± 0.005
4.3 Effect of Guidance Scale and Euler Steps
Table 4 demonstrates how the guidance scale  $\gamma$  and number of Euler steps  $N$  affects objective metrics. As shown in Table 2 and 4, P-Flow outperforms YourTTS even without the guidance method, showing the effectiveness of the speech-prompting method for speaker adaptation. Applying the guidance method improves pronunciation accuracy and brings our SECS metric closer to that of the state-of-the-art zero-shot TTS model without explicit training for guidance. These results confirm that the guidance method applied to P-Flow further boosts the effectiveness of speech prompting. Both guidance scales  $\gamma = \{1,2\}$  perform well. (We default to  $\gamma = 1$ )
Regarding the Euler steps, P-Flow demonstrates low WER even with very few Euler steps, indicating that the flow matching generative model generates speech that can be accurately recognized by the ASR model regardless of audio quality. We default to 10 Euler steps for external comparisons because SECS values plateau roughly after 5 Euler steps and we notice that good audio quality is achieved at around 10 steps. In Table 4, the ASR metrics and SECS metrics do not directly represent the acoustic quality of the samples according to the Euler steps. Therefore, we provide mean opinion scores (MOS) for the acoustic quality based according to the Euler steps in Section A.1. We further provide audio samples corresponding to our default inference settings in our demo page.
P-Flow is a fast and data-efficient flow matching model for zero-shot TTS that achieves comparable naturalness and speaker adaptation performance to its large-scale and autoregressive counterparts. The core of the approach is the use of the prompt-continuation setup in recent LLM applications while avoiding complex and expensive setups from recent works in zero-shot TTS. In some ways, our work serves as a strong yet simple baseline for future approaches, as we demonstrate that state of the art performance in this task can be achieved without super large-scale datasets, complex training setups, representation quantization steps, pretraining tasks, and expensive autoregressive formulations.
The focus of this work is primarily on zero-shot capabilities with respect to text-encoding and audio decoding, whereas zero-shot capabilities of the duration predictor remain limited and the subject of future work. Furthermore, high-quality zero-shot TTS can yield negative social impact through applications such as voice impersonation of public figures and non-consenting individuals, which we raise awareness for as a potential misuse of this technology.
This work was partially supported by the BK21 FOUR program of the Education and Research Program for Future ICT Pioneers, Seoul National University in 2023 and Institute of Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) [NO.2021-0-01343, Artificial Intelligence Graduate School Program (Seoul National University)].
[35] Alexander Tong, Nikolay Malkin, Guillaume Huguet, Yanlei Zhang, Jarrid Rector-Brooks, Kilian Fatras, Guy Wolf, and Yoshua Bengio. Conditional flow matching: Simulation-free dynamic optimal transport. arXiv preprint arXiv:2302.00482, 2023.
[36] Aäron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew W. Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. arXiv preprint arXiv:1609.03499, 2016.
[37] Chengyi Wang, Sanyuan Chen, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, et al. Neural codec language models are zero-shot text to speech synthesizers. arXiv preprint arXiv:2301.02111, 2023.
[38] Yuxuan Wang, R. J. Skerry-Ryan, Daisy Stanton, Yonghui Wu, Ron J. Weiss, Navdeep Jaitly, Zongheng Yang, Ying Xiao, Z. Chen, Samy Bengio, Quoc V. Le, Yannis Agiomyrgiannakis, Robert A. J. Clark, and Rif A. Saurous. Tacotron: Towards end-to-end speech synthesis. In Interspeech, 2017.
[39] Junichi Yamagishi, Christophe Veaux, and Kirsten MacDonald. Cstr vctk corpus: English multi-speaker corpus for cstr voice cloning toolkit (version 0.92). 2019.
[40] Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. Soundstream: An end-to-end neural audio codec. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 30:495-507, 2021.
[41] Heiga Zen, Viet Dang, Rob Clark, Yu Zhang, Ron J Weiss, Ye Jia, Zhifeng Chen, and Yonghui Wu. Libritts: A corpus derived from librispeech for text-to-speech. arXiv preprint arXiv:1904.02882, 2019.
A.1 Effect of Euler Steps for Acoustic Quality
We present the objective metrics according to the Euler steps in the result section of the main paper. Since these objective metrics have limitations in representing acoustic quality with respect to the Euler step, we also evaluate the sample quality based on the Euler steps and provide it as an additional metric. We measure the acoustic quality using 5-scale Mean Opinion Scores (MOS). We inquire each human evaluator to assess the acoustic quality of each sample, and we evaluate it with the participation of more than 50 human evaluators.
Table 5 presents MOS along with SECS and inference latency shown in the results section, based on the Euler step  $N$ . The table demonstrates that as the number of Euler steps increases, the acoustic sample quality improves. We choose Euler step 10 as the default, as it ensures a high speaker similarity while providing a good balance between inference latency and sample quality.
Table 5: Mean Opinion Scores (MOS) for the acoustic quality and Objective Metrics according to the Euler steps  $N$ .
MODEL	N	MOS↑	SECS	INFERENCE LATENCY(S)↓
P-FLOW	1	3.55 ± 0.16	0.420	0.028 ± 0.004
2	3.71 ± 0.12	0.522	0.037 ± 0.004
5	4.01 ± 0.10	0.549	0.067 ± 0.004
10	4.08 ± 0.10	0.544	0.115 ± 0.004
20	4.14 ± 0.10	0.540	0.210 ± 0.005
A.2 Model Comparison on VCTK Dataset
Considering the training dataset LibriTTS and the test set of LibriSpeech as in-domain data, it appears important to demonstrate results for out-of-domain prompting data. Accordingly, we compare P-Flow and VALL-E samples using the VCTK dataset [39]. We generate samples using the same prompts and sentences that were used for the VCTK samples on the VALL-E demo page. We provide our generated samples on our demo page.
As in the main section, we use 10 ODE steps and a guidance scale of 1. We measure WER and SECS for these VCTK samples and calculate the average values for each metric. To measure WER, we utilize the same ASR model as in Section 4. For SECS measurement, in the interest of a fair comparison, unlike what we did in other sections of our paper, we measure the SECS between generated and ground truth samples, rather than using the re-synthesized or encoded and decoded samples. Due to long leading and trailing silences in VCTK prompts, we measure SECS with the reference speech trimmed at  $20\mathrm{dB}$ .
We provide objective metrics for each model on the VCTK dataset in Table 7. Table 7 demonstrates that P-Flow achieves better WER and SECS compared to VALL-E. These results are similar to the LibriSpeech test clean results in Section 4.2.
In addition, we collect comparative mean opinion scores (CMOS) and comparative speaker similarity mean opinion score (SMOS) for VCTK samples, using a -2 to +2 scale. More than 50 human raters evaluate all the generated samples given 3-second reference data. Table 6 shows that human raters prefer P-Flow over VALL-E in terms of sample quality including acoustic quality, prosody and human likeness, and speaker similarity. These results show that P-Flow can achieve performance similar to or better than VALL-E on in-domain and out-of-domain data, in addition to providing users with an inference latency of 0.1 seconds using 10 ODE steps.
Table 6: Zero-Shot Speech Synthesis Subjective Metrics for VCTK dataset. Preference scores  $\geq 0$  indicate preference for P-Flow.
MODEL	CMOS↑	SMOS↑
P-FLOW VS VALL-E	0.188 ± 0.10	0.267 ± 0.166
Table 7: Objective metrics for zero-shot TTS evaluated on the VCTK dataset. We compare WER and SECS using all the VCTK samples from the VAL-E demo page.
MODEL	WER↓	SECS↑
VALL-E	4.3	0.452
P-FLOW (PROPOSED)	2.4	0.465
A.3 Analysis on ODE sampling methods
We measure the pronunciation accuracy (WER) and speaker similarity (SECS) for two second-order ODE methods: midpoint and Heun's method, detailed in [18]. We evaluate these two second-order methods for sampling steps  $N$ , which have a similar number of function evaluations as 10 Euler steps. Table 8 shows that the midpoint method with  $N = 4$  produces  $\mathrm{WER} = 2.7$  and  $\mathrm{SECS} = 0.540$ , not better than the Euler method. On the other hand, Heun's method improves upon Euler's method with a similar number of function evaluations as described in [18].
Table 8: Objective metrics for P-Flow with different ODE sampling methods.
MODEL	WER↓	SECS↑
P-FLOW (EULER METHOD, N=10)	2.6	0.544
P-FLOW (HEUN'S METHOD, N=4)	2.6	0.552
P-FLOW (MIDPOINT METHOD, N=4)	2.7	0.540
A.4 Zero-shot TTS with Emotional Reference Speech
We provide generated samples using emotional reference samples, where each sample exhibits distinct prosody, as demonstrated in [37]. We extract reference speech samples from EmoV-DB [1], representing five different emotions. From each reference speech, we utilize a 3-second segment to perform zero-shot TTS. On our demo page, we present generated samples for the same sentence given the speech prompts for these five emotions. P-Flow, similar to VALL-E, utilizes a speech-prompted text encoder composed of an autoregressive transformer, enabling the generation of samples with different prosody based on the reference speech.
B Model Architectures
We provide explanations for each module in this section and detailed hyperparameters and architecture of P-Flow are shown in Table 9.
Speech-prompted Text Encoder Our text-encoder consists of several linear projection layers, a pre-network with 3 convolutional layers, and a 6-layer transformer with 2 attention heads of 192 hidden dimensions. The input to the text encoder is the speech prompt and text embeddings projected into the same dimensions. For the input of the speech-prompted text encoder, we project the speech prompt and text embeddings into the same dimension and input to the same pre-network. The resulting representation is then split into prompt and text parts, to which positional encodings are added. We define each positional encoding as the sum of absolute positional encoding and a learnable fixed-size embedding so that the transformer can differentiate the speech prompt and text through learnable embeddings. The representations of the speech prompt and text are then fed into a transformer architecture that allows each text position to attend to the speech prompt.
Duration predictor Our duration predictor is a shallow convolution-based model used in [21]. Since our text encoder output already provides speaker-conditional hidden representation, we use the hidden representation before linear projection to  $h_c$  as the input of the duration predictor.
Flow matching Decoder Our flow matching decoder utilizes 18 layers of WaveNet-like architecture [36] with 512 hidden dimensions. We use the global conditioning method in WaveNet for conditioning  $t$  and concatenate the aligned encoder output  $h$  with the input  $x_{t}$  along the channel axis for conditioning the speaker-conditional text representation.
Table 9: Hyperparameters of P-Flow
Hyperparameter
Speech-prompted Text Encoder	Phoneme Embedding Dim	192
PreNet Conv Layers	3
PreNet Hidden Dim	192
PreNet Kernel Size	5
PreNet Dropout	0.5
Transformer Layers	6
Transformer Hidden Dim	192
Transformer Feed-forward Hidden Dim	768
Transformer Attention Heads	2
Transformer Dropout	0.1
Prompt Embedding Dim	192
Number of Parameters	3.37M
Duration Predictor	Conv Layers	3
Conv Hidden Dim	256
LayerNorm Layers	2
Dropout	0.1
Number of Parameters	0.36M
Flow Matching Decoder	WaveNet Residual Channel Size	512
WaveNet Residual Blocks	18
WaveNet Dilated Layers	3
WaveNet Dilation Rate	2
Number of Parameters	40.68M