SlowGuess/NeurIPS_2023_OCR huggingface dataset:
During inference, we are only given the text transcript, which does not tell us where we should insert silence to. Hence, it is desired to have the duration model not only predict the duration for each phone (SIL included), but also predict the existence of SIL at eligible locations (between words and at the two ends of the utterance). To tackle it, we introduce ghost silence to our phonetic transcript, which are silences in between words with duration of zero frames.
To give an example, suppose the transcript contains three words: “Hey what’s up” with pronunciation “{Hey: [A, B], what’s: [C], up: [D, E, F]}\”, and the frame-level phonetic transcript  $z$  obtained through forced alignment is  $z = (\text{SIL A B B SIL C D D D E E F SIL SIL})$ . The phonetic transcripts become  $y = (\text{SIL A B SIL C SIL D E F SIL})$ , where the ghost silence is highlighted in green. The corresponding duration would be  $l = (1, 1, 2, 1, 1, 0, 3, 2, 1, 2)$ . A ghost silence is inserted between what’s and up during training, and the duration model should predict the duration of it as zero to indicate that there should not be a pause between the two words.
Word-position-dependent phone The possible absence of silence between words in the frame-level phone transcript can make it hard for the audio model to identify word boundaries. To help the audio model identify the word boundary which is important when reading a sentence, we introduce word-position-dependent phones which are commonly used in Hidden Markov Model based acoustic models for speech recognition [47]. This adds a postfix to each phone in the transcript to denote where it is in the corresponding word. There are four postfixes: _B for beginning, _E for end, _I for intermediate, and _S for singleton. The above example becomes “{Hey: [A_B, B_E], what’s: [C_S], up: [D_B, E_I, F_E]}” with frame-level phonetic transcript  $z = (\text{SIL A_B B_E B_E SIL C_S D_D_B D_B E_I E_I F_E SIL SIL})$ .
Phone-level mask In terms of masking, given duration  $l$ , the relationship of phone-level mask  $m'$  and frame-level mask  $m$  can be written as  $m = \mathrm{rep}(m', l)$ . For the applications where a duration model is involved (zero-shot TTS, content editing, diverse speech sampling), the frame-level mask  $m$  is extended such that no phone is partially masked. In other words, all the frames corresponding to a phone is either entirely masked or entirely unmasked. During training, we mask a contiguous chunk of audio, infilling of which is a more challenging task compared to infilling multiple smaller segments. All frames that are aligned to a phone are either entirely masked or unmasked. Note that masking all frames for a phone is not a necessity but was chosen due to ease of implementation.
A.3 Data transformation
The Mel spectrogram is normalized with the global mean (-5.8843) and standard deviation (2.2615) to stabilize training. The statistics are estimated on 30k random training samples from the 1K hours of
English audio. Input and output duration are dequantized  $(x\sim \mathcal{U}[x - 0.5,x + 0.5])$  and transformed with  $\log (1 + x)$  following [51]. Prediction of duration is quantized and clipped such that the minimal duration is greater than or equal to zero.
A.4 Cross-lingual zero-shot TTS test data filtering
We create a test set for each language by selecting samples from the MLS test split which have Whisper transcription WER lower than  $20\%$  (or  $30\%$  for Polish and Portugueses test splits which contains less than 1K samples), because we found MLS test set contains many examples with incomplete transcriptions missing a large portion of the utterance. In addition, a small amount of utterances were excluded due to MFA alignment failure. Table A1 lists the number of samples remained for each language.
Table A1: Number of MLS test samples after filtering.
Language	#samples before filtering	#samples after filtering
English	3769	3535
Spanish	2385	2323
German	3394	3183
French	2426	2284
Polish	520	508
Portuguese	871	838
A.5 Setup for training ASR models with synthetic speech
To train an ASR model in Section 5.3, we extract 80-dimensional log Mel features with a 25ms window and a 10ms frame shift, and then apply global mean-variance normalization. The ASR model is an RNN-T with a Conformer-based encoder [16]. The conformer applies time scale reduction to the input features with stride 6, embeds them into 512-dimensional vectors, passes these vectors through a 20-layer conformer which has 8 attention heads and 2048-dimensional fully-connected layers. The conformer output is further mapped to 1024 dimensions through a linear layer followed by layer normalization before being passed to the joiner. The predictor of the network first embeds wordpiece units into 512 dimensional embeddings, applies layer normalization, a 512-dimensional LSTM, a dropout layer and a linear layer that maps the LSTM output to 1024 dimensions. The joiner adds the encoder and predictor outputs, applies tanh non-linearity and uses a linear layer that maps the 512-dimensional joiner input into wordpiece units. There are 4096 wordpiece units estimated from the LibriSpeech 960hr training text.
We apply SpecAugment [44] in all ASR runs. The models are trained using PyTorch [45] with Adam [33] optimizer for 120 epochs unless otherwise noted. The learning rate follows a tri-stage schedule with a maximum of 0.001. We applied gradient clipping at 10 and a weight decay parameter of 0.1. For the 960hr setting, we used a variable batch size capped at 1K utterances or 30K frames, whichever is smaller. This corresponds to about 45K update steps for 120 epochs. For the 100hr setting, we set the maximum learning rate to 0.0001 and used smaller batch size (capped at 200 utterances or 5K frames). In this case, 120 epochs corresponded to about 120K updates. For decoding, we used n-best decoding with a beam-size of 15, and evaluated the WER on the 1-best path.
A.6 How to format input for different task
Fig. A1 shows how input are formatted to perform diverse speech sampling, content editing, and zero-shot TTS using Voicebox.
A.7 Bi-directional ALiBi Bias
We use a symmetric variant bi-directional variant of ALiBi bias where any query  $Q_{i}$  and key  $K_{j}$  with  $|i - j| = N$  use the same representations. Furthermore, for any query  $Q_{i}$  the bias corresponding
Our implementation is similar to symmetric option.
Figure A1: Detailed diagrams of diverse speech sampling, content editing, and style transfer. Text in red and blocks in orange at the input of a model denote segments to be predicted. Numbers in blue and spectrogram in cyan at the model output denote predicted duration and spectrogram.
to the flow step  $x_{t}$  is set to 0. Similarly the bias from the flow-step  $x_{t}$  to any other query 0. In our experiments, we find ALiBi Bias to improve convergence and extrapolation to longer sequences.
B.1 Detecting generated speech
We recognize the potential risks of a model capable of generating speech in the style of arbitrary users. In an effort to diminish these risks we show that a binary classification model is able to consistently distinguish between real world speech and that which is generated from our model.
Inspired by [29], we train a convolutional binary classification model to distinguish between real and generated speech. The model consists of 6 blocks with hidden dimension sizes: [64, 128, 256, 256, 512, 512]. Each block contains a  $(3 \times 1)$  convolution along the time axis, a  $(1 \times 3)$  convolution along the frequency axis, followed by a ReLU activation and batch normalization. After each block that
increases the hidden dimension size we also apply max pooling with a stride of 2 across both the time and frequency dimensions. Finally, global max pooling is applied and a linear layer projects to a single value that is fed into a binary cross entropy loss. At inference time we create a sliding window with hop length equal to 250ms and run each chunk of audio through the classifier and average the outputs.
The model is tested on the dev-clean split of Librispeech. We then take a 100 hour subset of the 60K hour-English data and set aside 2,703 random utterances (to match the size of dev-clean) which is used as a validation split. The remaining utterances from the 100 hours subset are used as the ground truth utterances for training. For each split we synthesize audio, conditioned on each utterance of the split by masking out frames in the spectrogram corresponding to  $90\%$ ,  $50\%$ , and  $30\%$  of the phonemes of the utterance. All samples are generated using classifier-free guidance with  $w = 0.7$ , midpoint ODE solver (step size  $0.0625 / \mathrm{NFE} = 64$ ), and the regression duration model.
We consider two detection tasks. The first one is to distinguish between original audio and Voicebox-generated audio. The second one is to distinguish resynthesized audio and Voicebox-generated audio. The resynthesized audio is created by extracting the Mel Spectrogram from original audio and then vocoding it with the HiFi-GAN vocoder.
Table B2 presents the results for each setting. The model can trivially distinguish original audio from Voicebox-generated audio. This results from the fact that a model can also trivially distinguish original audio from resynthesized audio, most likely by recognizing artifacts produced by the vocoder. The task of differentiating Voicebox-generated audio from resynthesized audio is much harder. When  $90\%$  of the audio is masked, the model is able to reliably classify the audio as Voicebox-generated. In lower masking regimes this decreases a bit, but this is likely due to a naive inference method of averaging the outputs of all sliding windows. Since the majority of windows are non-synthetic, this leads to mis-classifications.
Table B2: Synthetic speech detection metrics
% Mask	Accuracy	Precision	Recall
Original audio vs Voicebox-generated audio
30%	1.000	1.000	1.000
50%	1.000	1.000	1.000
90%	1.000	1.000	1.000
Resynthesized audio vs Voicebox-generated audio
30%	0.704	0.714	0.680
50%	0.809	0.796	0.831
90%	0.907	0.881	0.942
B.2 How context length affects monolingual and cross-lingual zero-shot TTS
Monolingual: For in-context zero-shot TTS in Section 5.1, we used 3.0 seconds of prompt audio. Here we examine how WER / SIM-r vary with different amounts of prompt audio using duration from regression duration model for the target text. If the desired prompt is longer than the available audio, the shorter audio is used as the prompt. Results are shown in Figure B2. As expected, WER mildly decreases and SIM-r grows quickly flattens with longer audio prompts. Comparing against VALL-E, Voicebox is more efficient at leveraging an audio prompt, achieving the same speaker similarity as VALL-E with roughly two thirds the input audio.
Cross-lingual: Here we examine the effect of increasing the prompt length for the case of cross-lingual zero-shot TTS. As described in 5.1, this setting has a total 36 language transfer directions for each pair of source and target language. For each target text in a given transfer setting, we examine how WER / SIM-o $^5$  vary as the prompt length increases. Similarly, the regression duration model is used for the target text. Fig. B3 and Fig. B4 plot the SIM-o (speaker similarity) and WER trends respectively. When concatenating the prompt to the target for MLS, we find that the samples are quite a bit longer than what the model was trained on (16s max length), because MLS test set samples are in average 15 seconds long. To alleviate this out of domain issue and focus the study
${}^{5}$  Same trend is observed with SIM-r. We present SIM-o to be consistent with Table 4
(a)WER
(b) Speaker Similarity
Figure B2: WER and SIM-r as a function of prompt audio time in seconds for the Zero-shot TTS task 5.1. Audio is generated using classifier-free guidance strength  $(\alpha)$  of 0.7 and midpoint ODE solver with a NFE of 32. The blue line is for Voicebox and the red star is VALLE at 3 seconds. The speaker similarity (SIM-r) remains same for longer prompts (up to 10s).
(d) Portuguese
Figure B3: Each subplot considers one of the six target language and shows SIM-o (speaker similarity) as a function of prompt audio duration in seconds for cross-lingual style transfer from different source language. We set the classifier-free guidance strength  $(\alpha)$  to 1.0 and use midpoint ODE solver with a NFE of 32.
(e) German
(f) Polish
on varying the prompt length, we truncate the target sequences to 4 seconds (at word boundaries). We notice that WERs are higher compared to Table 4, likely because the ASR model struggles with incomplete sentences. Each subplot contains the trend for one of the target languages from all six source languages.
The speaker similarity consistently improves as the prompt length is increased, similar to the monolingual setting. In contrast, we find that WER increases as we increase the prompt length for most directions. The WER increases much more for  $\mathrm{En} \rightarrow$  non-En directions. We hypothesize that this is due to training data imbalance across languages, where English accounts for over  $90\%$  of the multilingual training data. Hence, when transferring from English, the model is more likely to assume that the whole sentence is in English as the prompt length increases and produce incorrect
(d) Portuguese
(f) Polish
Figure B4: Each subplot considers one of the six target language and shows WER as a function of prompt audio duration in seconds for cross-lingual style transfer from different source language. We find WER remain reasonably low for all cases except for "English" to "X" style transfer. We set the classifier-free guidance strength  $(\alpha)$  to 1.0 and use midpoint ODE solver with a NFE of 32.
pronunciation for the non-English target. Note that during the training phase, the model was only exposed to audio samples and phonemes originating from a single language.
B.3 Comparing audio model training objectives
While A3T is considered the regression-based speech infilling baseline, it is trained on a smaller dataset and uses a smaller model compared to Voicebox. Here we present a controlled study comparing the flow-matching and regression objectives, as well as the effectiveness of masked loss.
We consider a reduced setup for this ablation to save the compute. All models were trained on an English audiobook dataset with 1K hours of speech using a smaller model configuration (12 layers, 1024-dimensional Transformer embedding, 2048-dimensional feed-forward layer, 8 attention heads) for 150k steps with an effective batch size of 120k frames. These models are evaluated on the cross-sentence zero-shot TTS setup (Section 5.1) and diverse speech sampling (Section 5.3).
Results in Table B3 show that while regression audio models produce comparable WER, the audio similarity and diversity are significantly worse. Subjective listening also reveals that the audio quality and audio similarity are much worse. On the other hand, masked loss improves audio similarity and diversity while having little impact on intelligibility.
Table B3: Comparison of flow-matching and regression models, trained with loss computed on all frames or only masked frames. Results of the proposed objective is boldfaced.
Method	Loss	Zero-Shot TTS (cross-sentence)	Diverse sampling
WER	SIM-r	WER	FSD
Flow Matching	Masked	2.1	0.597	3.1	242.5
Flow Matching	All	2.0	0.528	3.1	243.1
Regression	Masked	2.0	0.520	2.9	278.8
Regression	All	2.0	0.512	2.9	282.8
B.4 Effectiveness on data scaling
We create four subsets of the 60K hour English data (0.1%, 1%, 10%, 100% in duration). In particular, the  $x\%$  subset would contain roughly  $x\%$  of the speakers from the original set. We train one model on each subset with a reduced setup described in Appendix B.3 and evaluate them on zero-shot TTS (cross-sentence) and diverse sampling. Results show that scaling data constantly improves the zero-shot TTS performance (WER and SIM-r) as well as WER on diverse sampling. For FSD it shows regression when scaling from 6K hour to 60K hour, but this could result from the reference distribution is computed from the 1K hour English audiobook data that has less diverse samples.
Table B4: Experiments on the effect of scaling training data.
Train data (hr)	Zero-Shot TTS	Diverse sampling
WER	SIM-r	WER	FSD
60	2.30	0.151	3.48	280.48
600	2.11	0.417	3.19	205.39
6,000	2.08	0.573	2.96	195.52
60,000	2.05	0.645	2.95	214.38
B.5 Complete results on comparing generative modeling approaches
Table B5 presents the full results of Section 5.5 on all combinations of training steps, inference steps with results on both monolingual zero-shot TTS (Section 5.1) and diverse speech sampling (Section 5.3) for the ablation study presented in Section 5.5. In all settings Flow Matching with OT paths performs strictly better than both of the other approaches.
B.6 Transient noise removal in more conditions
We expand the experiments in Section 5.2 by comparing the models on two noise levels (low noise: 10dB and high noise: -10dB), three overlapping ratios (30%, 50%, 70%), and also two types of noise (speech noise and non-speech noise).
Results are presented in Table B6. Voicebox consistently produces the most intelligible audio at all conditions (indicating the percentage of speech to infill). In terms of audio similarity, Voicebox is constantly better in the high noise condition with gains ranging from 0.265 to 0.324 compared to Demucs, and is on par with Demucs in low noise condition.
B.7 Additional results on inference efficiency versus performance
As explained in Section 5.4, for the cross-sentence setup of Section 5.1, we find that WER remains stable with mean of 2.0 and variance of 0.005. This can be also be observed from Fig. B5a.
(a) NFE vs WER (Zero-shot TTS)
(b) NFE vs WER (Divers speech sampling)
Figure B5: Trade-off between NFE and WER for different classifier-free guidance strengths (a) presents the WER for cross-sentence zero-shot TTS (Section 5.1) and (b) presents the WER for diverse speech sampling (Section 5.3).
Table B5: Comparison of FM w/OT vs. FM w/Diffusion vs. SM.
Method	Train Steps	NFE	ZS-TTS (cross-sentence)	Diverse sampling
WER	SIM-o	SIM-r	WER	FSD
FM w/ OT	50000	4	2.7	0.303	0.362	4.8	276.499
8	2.5	0.353	0.412	4.8	235.958
16	2.4	0.366	0.425	4.7	227.485
32	2.5	0.364	0.424	4.7	225.931
100000	4	2.5	0.347	0.404	4.3	258.358
8	2.2	0.411	0.468	4.2	216.512
16	2.3	0.429	0.483	4.3	206.538
32	2.2	0.431	0.487	4.2	203.792
150000	4	2.4	0.356	0.410	4.0	249.712
8	2.2	0.430	0.481	4.0	208.511
16	2.2	0.453	0.503	4.0	198.040
32	2.1	0.458	0.508	3.9	195.304
FM w/ diff	50000	4	99.9	0.050	0.050	99.8	3478.910
8	99.9	0.047	0.047	99.9	4704.237
16	98.8	0.052	0.048	96.5	5336.591
32	76.0	0.060	0.066	49.5	2485.400
100000	4	98.9	0.048	0.048	96.6	4486.401
8	14.6	0.104	0.137	12.0	669.564
16	4.0	0.210	0.262	7.0	381.891
32	3.1	0.285	0.344	6.3	294.777
150000	4	11.5	0.132	0.171	11.4	692.560
8	3.0	0.305	0.359	5.6	334.237
16	2.7	0.391	0.447	5.4	244.067
32	2.6	0.423	0.478	5.2	224.963
SM w/ diff	50000	4	99.6	0.050	0.048	99.7	2816.083
8	99.3	0.051	0.048	99.6	3079.040
16	97.5	0.052	0.050	98.4	3710.340
32	73.3	0.057	0.062	86.2	3011.030
100000	4	99.4	0.050	0.050	99.3	3474.579
8	97.2	0.049	0.048	97.9	3600.423
16	53.9	0.064	0.071	69.6	2060.892
32	17.4	0.150	0.176	34.4	1071.579
150000	4	94.5	0.055	0.054	79.4	2953.417
8	42.3	0.070	0.076	27.5	1071.010
16	11.5	0.191	0.218	12.8	698.411
32	5.1	0.309	0.349	8.8	519.468
In Fig. B5b, we show the WER for the samples generated on Librispeech test-other text. We find that for  $\alpha = 0$ , WER increases slightly from 2.8 to 3.1 as NFE goes from 2 to 32. For a larger classifier-free guidance strength, WER remains more stable. Subjective listening and FSD reveal that 1) a lower NFE leads to less natural samples with lower diversity, and 2) a higher guidance weight leads to lower diversity. In addition, we observe that ASR can perform well with unnatural samples that contains artifacts, but degrades when the samples are more diverse and expressive (e.g., with whispering voice or with strong reverberation). As a result, we see that the combination of low guidance weight and high NFE leads to a higher WER due to the higher diversity.
B.8 Choice of audio model output features
The performance of our model is upper bounded by how well the chosen acoustic features can be reconstructed to waveform. The reconstruction performance is determined jointly by the encoding
Table B6: Results of transient noise removal with varying overlapping percentage and noise level. "sp" means added noise is speech, and "non-sp" means non-speech.
WER(↓)	SIM(↑)	WER(↓)	SIM(↑)
sp	non-sp	sp	non-sp	sp	non-sp	sp	non-sp
SNR=-10dB, overlap=30%	SNR=10dB, overlap=30%
Noisy speech	26.7	24.9	0.202	0.238	3.7	3.1	0.605	0.603
Demucs	20.5	19.7	0.247	0.247	3.2	2.8	0.570	0.567
A3T	7.5	0.058	same as left
VB-En (α=0.7)	2.2	0.566	same as left
SNR=-10dB, overlap=50%	SNR=10dB, overlap=50%
Noisy speech	43.6	40.8	0.256	0.292	4.5	3.8	0.649	0.649
Demucs	34.3	32.5	0.291	0.288	3.8	3.3	0.616	0.613
A3T	11.5	0.064	same as left
VB-En (α=0.7)	2.0	0.612	same as left
SNR=-10dB, overlap=70%	SNR=10dB, overlap=70%
Noisy speech	60.0	56.0	0.260	0.303	6.3	4.6	0.595	0.592
Demucs	49.5	45.4	0.293	0.294	4.6	3.8	0.572	0.564
A3T	16.6	0.063	same as left
VB-En (α=0.7)	2.0	0.559	same as left
process, as in how much information is lost when encoding waveform into the features, and the decoding process, as in how well the vocoder can translate the encoded information into waveform.
To motivate the choice of the acoustic feature and the vocoder, we compare four combinations: the first one is Mel spectrogram + HiFi-GAN which is what this paper adopts. The second is Mel spectrogram + Parallel WaveGAN [65] that is used by A3T [3]. The third one is Encodec post-quantization dense feature + Encodec decoder, which is analogous to VALL-E's setup. The last one is also Encodec but with pre-quantization dense feature, which we include to study how much information is lost during quantization.
We also note that Mel spectrogram features are 80 dimensional encoded at  $100\mathrm{Hz}$ , which is 8K dimensions per second, while Encodec features are 128 dimensional encoded at  $75\mathrm{Hz}$ , which is 9.6K dimensions per second, higher than the Mel spectrogram features.
Table B7 presents the results evaluated on the Librispeech dev-clean and dev-other splits. All three models have the same WER resynthesizing dev-clean split, but ParallelWaveGAN degrades the most on dev-other. Interestingly Encodec even produces audio of lower WER than the ground truth.
In terms of audio similarity, besides the default audio feature extractor WavLM-TDCNN, we also include results of similarity computed with another speaker encoder ECAPA [13]. Parallel WaveGAN is consistently the worst. However, it is unclear whether HiFi-GAN or Encodec performs better. Encodec prevails with the WavLM-TDCNN embedder and HiFi-GAN wins using ECAPA. It may require subjective MOS test to conclude which one reconstructs the audio better, and we leave exploration of modeling Encodec dense features for future study.
Table B7: Comparison of different audio features and vocoders on audio reconstruction. Librispeech dev-clean (d-c) and dev-other (d-o) are used for evaluation. WER and audio similarity computed withwavLM-TDCNN and ECAPA are reported.
Audio feature / Vocoder	WER	SIM-o (WavLM)	SIM-o (ECAPA)
d-c	d-o	d-c	d-o	d-c	d-o
Ground truth	2.1	4.7	1.000	1.000	1.000	1.000
Mel spectrogram / HiFi-GAN	2.1	4.7	0.915	0.909	0.766	0.762
Mel spectrogram / Parallel WaveGAN	2.1	5.2	0.868	0.847	0.721	0.711
Encode post-quantized feature / Encode decoder	2.1	4.5	0.943	0.944	0.724	0.722
Encode pre-quantized feature / Encode decoder	2.1	4.4	0.943	0.944	0.724	0.722
C Additional Details and Studies on Metrics
C.1 Measuring speech diversity and quality with FSD
Diversity We first validate if FSD reflects the diversity for a set of speech samples and study its sensitivity to sample size. To achieve that, we design controlled experiments to compute FSD on sets of samples with varying diversity and sample sizes. Specifically, we create two partitions from 1K hours of English speech, where each partition has the same set of speakers and the same number of utterances for each speaker. The first partition is considered the reference set.