SlowGuess/NeurIPS_2023_OCR huggingface dataset:
To test the sensitivity to sample size, we use the second partition to create subsets by sampling  $r\%$  of utterances from each speaker in that partition. This sampling method is denoted as "utt". We computed that on average, each speaker contributed approximately 2.33 sessions, with each session containing around 52.45 utterances. Therefore, the subsets created using the sampling method are expected to have similar audio style distributions to the reference set and the FSD is expected to stay low regardless of the subset size. We consider  $r \in \{1, 5, 10, 25, 50, 100\}$ .
To test the correlation with diversity, we again use the second partition to create subsets by sampling  $r\%$  of speakers and including all the utterances in the partition from those speakers. This sampling method is denoted as "spk" where a smaller  $r$  leads to a subset with fewer speakers and hence lower diversity. Therefore the FSD is expected to increase as  $r$  decreases. The same set of values for  $r$  is considered. For the same  $r$ , the "utt" subset should always have a lower FSD than the "spk" subset.
We compare three different features for computing the FSD score. The first is the supervised WavLM-TDCNN feature used for computing audio similarity (SIM-r and SIM-o). The second is the self-supervised wav2vec 2.0 BASE [2] feature reduced to 128 dimensions using principle component analysis (PCA). The last one is the supervised audio event classification model feature that is used to compute FAD [30] for non-speech audio generation.
Figure C6 first compares using different layers of wav2vec 2.0 features. All of them yield similar desirable results where "utt" stays low and "spk" increases drastically when the sample size reduces and speaker diversity decreases. We then decide to use the middle layer (layer 6) as the default feature for FSD computation.
layer-8
ratio(%)
Figure C6: FSD based on different layers of wav2vec 2.0 BASE. utt: utterance-based sampling, spk: speaker-based sampling. Vertical bars denote standard deviation.
layer-10
ratio(%)
layer-12
ratio(%)
Figure C7 further compares wav2vec 2.0-layer 6 with the two other features. WavLM-TDCNN and wav2vec 2.0-layer 6 present similar trends and both have low variance. Both of them are suitable for measuring diversity, and we decide to use wav2vec 2.0 features as it is self-supervised and would be able to capture more holistic information of speech such as prosody and emotion.
In contrast, FAD score [30] is not appropriate for measuring speech diversity. The score does not increase much between  $r = 25\%$  and  $r = 1\%$  for "spk" sampling method, showing that the score does not reflect the decreasing speaker diversity. On the other hand, "utt" sampling method observes huge FAD score increase when reducing the sample size from  $r = 25\%$  to  $r = 1\%$  where the diversity does not change much as the number of speakers remains the same. Moreover, at  $r = 1\%$  both sampling methods result in similar FAD score while the two subsets exhibit very different levels of diversity. We hypothesize that this is because FAD score is computed based on features extracted from an audio even classifier trained on AudioSet, which learns to distinguish between events like lawn mower, car engine, and human speech, but does not learn to capture the variation within speech, such as different voices.
Figure C7: FSD with different sample size using supervised WavLM-TDCNN, self-supervised wav2vec 2.0, and supervised audio event classifier features.utt: utterance-based sampling, spk: speaker-based sampling. Vertical bars denote standard deviation.
Quality In addition to measuring diversity, Fréchet distance is a commonly used metric for assessing quality in image generation [19]. To show its applicability for speech generation, we evaluate the FSD score of speech utterances with varying levels of quality. The reference set samples are 1K hours of English training data, and the hypothesis set is the Librispeech test-clean split with noise added. We added Gaussian noise at different SNRs, ranging from 0 to 50 dB. Lower SNR values correspond to lower quality. We use the default speech feature extractor (i.e., wav2vec 2.0, layer-6) throughout the experiments.
Our results, summarized in Figure C8, show that a subset with a lower SNR has a higher FSD score. Therefore, a lower FSD score indicates higher acoustic quality for the set of test samples when diversity is fixed.
Figure C8: FSD under different noisy levels. Feature: Wav2vec 2.0 layer-6 feature. Noise is added upon model output from Voicebox under unconditional setting.
C.2 Standalone metrics for duration models
As mentioned in the main text, we can utilize end-to-end metrics of WER, SIM, and FSD to evaluate duration models, but also consider metrics specifically for duration.
First, we consider two metrics aimed at the quality of duration predictions, here denoted  $\hat{l}(l_{ctx}, y)$ . For a regression model, we use  $\hat{l}(l_{ctx}, y) = g(l_{ctx}, y; \theta)$ . For a flow matching model, we set  $\hat{l}$  as the mean over 20 samples, ensuring a fairer comparison.
Duration correctness (MS-MAE) Our first metric, multi-sample mean-absolute error (MS-MAE), is the masked absolute error per-utterance divided by the average number of masked phonemes per-utterance
$$
\frac {\mathbb {E} _ {m , l , y} \| m \odot (l - \hat {l} (l _ {c t x} , y)) \| _ {1}}{\mathbb {E} _ {m , l , y} \| m \| _ {1}} \tag {4}
$$
Speaking rate correlation (MS-Corr) Our next metric, multi-sample correlation (MS-Corr), computes the average masked predicted duration and unmasked duration context per utterance, and computes their correlation across utterances. Comparing MS-Corr with the same correlation computed from the ground truth, we observe to what extent predicted durations capture appropriate correlations with the context.
Duration diversity and quality (FDD) Additionally, we evaluate the quality and diversity of duration samples at the distribution level, similar to our audio evaluation of diversity and quality via FSD. We produce one sample per utterance from a duration model and collect all sampled phoneme durations, possibly many per-utterance, into an empirical distribution. We compare means and variances of this sampled distribution versus the means and variances of the training distribution, labeled  $\mu$ ,  $s$ , and  $\mu'$ ,  $s'$  respectively. We define the Fréchet duration distance (FDD) as the Fréchet distance between the distributions
$$
\left(\mu - \mu^ {\prime}\right) ^ {2} + s + s ^ {\prime} - 2 \sqrt {s s ^ {\prime}}, \tag {5}
$$
treated as though they were Gaussians. FDD depends on the sampled durations accurately reflecting the training distribution of real durations. As for FSD, this metric is specific to unconditional text-to-speech generation.
C.3 Duration model evaluation with standalone metrics
We evaluate three duration model variants. The first and second utilizes flow matching and regression, trained using masked conditional flow matching and regression respectively as described in Section 3.3. The third is a regression model that ignores duration context  $l_{ctx}$  and only uses phonetic transcript  $y$ , referred to as unconditional regression below. This is the duration model used in FastSpeech2 [51], A3T [3] and many other non-autoregressive speech synthesis models.
We evaluate our three duration model variants on the Librispeech test-other split on two tasks. The first is unconditional TTS where we generate all durations from given phonemes (i.e.  $l_{ctx}$  is entirely masked). The second task is infilling the second half of each utterance's durations, where  $l_{ctx}$  are durations from the unmasked half of the utterance. This second infilling task distinguishes between the two regression model variants, since the unconditional regression ignores  $l_{ctx}$ , and hence predicts identical durations for the tasks. Duration metrics are computed for TTS and infilling in Table C8 and C9. The prefix Phn or Sil indicates the associated metric was either computed across all non-silence or all silence phonemes. Start and end silences were not trimmed for these duration metric evaluations.
Starting with prediction quality metrics (MS-MAE and MS-Corr), the duration-conditional regression performs slightly better on MS-MAE overall than the other models. Larger differences are seen on Phn-MS-Corr where the unconditional regression has a correlation substantively below the other models (Phn-MS-Corr of ground truth is 0.47), indicating conditioning on duration context  $l_{ctx}$  is beneficial. Flow-matching shows the largest distinction versus regression on the distributional comparison captured by FDD. The regression models have generally larger FDD because they underestimate the standard deviation in phoneme and silence durations, and hence produce samples with less duration diversity and more regular duration lengths.
Table C8: English TTS duration metrics on LS test-other.
Duration Model	Phn-MS-MAE	Phn-FDD	Sil-MS-MAE	Sil-FDD
Unconditional Regression	2.53	0.72	5.32	2.39
Duration-conditional Regression	2.52	0.76	5.10	8.40
Duration-conditional Flow Matching	2.63	0.61	5.18	2.48
Table C9: English second-half infilling duration metrics on LS test-other.
Duration Model	Phn-MS-MAE	Phn-MS-Corr	Sil-MS-MAE
Unconditional Regression	2.57	0.26	5.44
Duration-conditional Regression	2.45	0.35	5.20
Duration-conditional Flow Matching	2.52	0.41	5.32
C.4 Duration model evaluation with end-to-end metrics
We now present end-to-end metrics for our three duration variants for zero-shot TTS cross-sentence and continuation, as well as diverse speech generation, corresponding to Sections 5.1 and 5.3. Zero-shot TTS cross-sentence and continuation results are shown in Table C11 and diverse speech generation results in Table C10. These results are not comparable with the main text as they utilize the flow-matching model described in Appendix B.3, denoted as VB-En-1K.
Overall, FSD and SIM are similar across duration variants. On the other hand, WER is sensitive to the choice of duration model, where the duration-conditional regression achieves a substantially lower WER. Subjective listening from the duration-conditional regression and flow-matching confirms that the regression model is producing more regular patterns of speech, that may be easier for ASR to recognize, while sacrificing some duration diversity.
Table C10: Diverse speech generation from LS test-other text.
Duration Model with VB-En-1K	WER	FSD (LS-train)
Unconditional Regression	3.8	148.7
Duration-conditional Regression	3.7	148.1
Duration-conditional Flow Matching	5.4	155.1
C.5 MOS instructions
Table C12 shows the instruction presented to the raters for quality mean opinion score study. Table C13 shows the instruction presented to the raters for similarity mean opinion score study.
D Limitation and Broader Impact
Limitation Voicebox models presented in this paper are trained on read speech from audiobooks in up to six written languages. Hence, the current models may not transfer well to conversational speech [15], which is more casual and contains more non-verbal sounds such as laughing and back-channeling (e.g., um-hmm). We plan to tackle the problem by scaling the training data to incorporate more diverse speech.
On the other hand, Voicebox depends on a phonemizer and a forced aligner to produce frame-level phonetic transcript. In addition, many existing phonemizers [40] are word-based, which does not take neighboring words of the target into account when predicting the pronunciation. Such phonemizers cannot accurately predict phonetic transcript given text because pronunciation is context-dependent in many languages (e.g., liaisons in French). In the future, we will explore more end-to-end methods where a model would be able to take raw text with punctuation as input [7], and eliminate the need of phonemizers and forced aligners to improve the performance and increase the language coverage.
Table C11: English zero-shot TTS results on filtered LS test-clean.
Duration Model with VB-En-1K	WER	SIM-o	SIM-r
cross-sentence
Unconditional Regression	3.0	0.538	0.584
Duration-conditional Regression	2.7	0.545	0.591
Duration-conditional Flow Matching	3.4	0.528	0.578
continuation
Unconditional Regression	2.5	0.485	0.524
Duration-conditional Regression	2.2	0.491	0.533
Duration-conditional Flow Matching	2.7	0.481	0.525
Table C12: Quality mean opinion score (QMOS) instruction.
Your task is to evaluate the subjective quality and intelligibility of the speech from short (2-8 second) audio files. Each HIT can be completed in roughly around 120 seconds.
Task Instructions
In this task you will hear samples of speech recordings. The purpose of this test is to evaluate the quality and intelligibility of each file in terms of its overall sound quality and the amount of mumbling and unclear phrases in the recording.
Please keep in mind that speech samples can be distorted and noisy, however these are only specific examples.
Please use a headset for listening and adjust your volume level to your comfort during this training, and do not change later during the experiment.
You should give a score according to the following scale, known as the MOS (mean opinion score) scales:
Score (Quality and Intelligibility of the speech)
5 (Excellent)
4 (Good)
3 (Fair)
2 (Poor)
1 (Bad)
Last but not least, while Voicebox yields impressive results on transferring audio style (voice, speaking style, emotion, and acoustic condition), the model does not allow independent control of each attribute. In other words, one cannot ask the model to generate speech that resembles voice of one sample while resembling the emotion of another sample. We leave disentangled control of attributes through prompting or text description for future work.
Broader impact A high-quality and versatile generalist speech generation model like Voicebox can enable many applications that improve the quality of our life. For example, zero-shot TTS could bring the voice back to people who suffer from diseases or underwent surgeries such as laryngectomy the causes inability to speak. Zero-shot TTS can also be combined with visual speech recognition systems [23] to avoid the need of typing. When paired with speech translation models, cross-lingual zero-shot TTS enables everyone to speak any language in their own voice. Content editing and speech denoising can be productivity tools for users to create content more effortlessly. Diverse speech sampling, as shown in the paper, can significantly reduce the cost of creating data for training speech-input models.
While Voicebox can bring many positive social impacts, it also carries the potential of misuse and unintended harm. To mitigate the risk, we have presented a highly effective classifier showing that the
Task Name
Rate the similarity of the synthesized speech samples to a given prompt.
Your task is to evaluate the similarity of the synthesized speech samples to the given speech prompt. You should focus on the similarity of the speaker, speaking style, acoustic conditions, background noise, etc. You should rank the recordings on the scale between 1-5, where 5 is the best quality and 1 is the worst.
In other words, please rank the recordings according to their acoustic similarity to the given prompt, meaning as if they were recorded in the same place by the same speaker speaking in similar styles. This task typically requires approximately 120 seconds to complete.
model can accurately distinguish between real and synthetic speech. For future work, we also plan to investigate proactive methods for training the generative model such that the synthetic speech can be more easily detected, such as embedding artificial fingerprints [66] that can be trivially detected without hurting the speech quality.
To prevent Voicebox from learning biases, we also need to carefully select its training data. First, if Voicebox is only trained on a smaller number of speakers from a specific group with similar accents, it will not be able to generate diverse speech representing the accents around the globe, and downstream models trained on Voicebox generated speech would perform worse on groups with underrepresented accents. For zero-shot style transfer, the performance would also degrade for underrepresented accents. To mitigate this, we have leveraged in-the-wild speech that includes a wide variety of accents, and will continue investing in collecting diverse speech to avoid such biases.
Second, if Voicebox is trained on data where samples from one ethnic group always have lower audio quality (e.g., more noise) while the other ethnic group always has higher audio quality samples, the model would also learn undesired association. To mitigate this, we want the distribution of audio quality (and other audio attributes) and ethnic group to be less correlated, which is usually the case when we have larger scale data collected from in-the-wild sources. We can further tackle this by leveraging data augmentation to decorrelate the distribution, such as adding noise and enhancing speech to widen the audio quality distribution.
Table C13: Similarity mean opinion score (SMOS) instruction.