SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Transient noise removal & content editing: When recording speech, one might misspeak a few words or the recording my be interrupted by unexpected background noise. In these scenarios it is desired to just edit the problematic segment instead re-recording the speech. Voicebox can perform transient noise removal through re-generating the noise corrupted segment given the original frame-level transcript and the surrounding clean audio.
For content editing, Voicebox first samples duration for the new phones given the edited phone transcript and the duration of existing phones to create the edited frame-level phone transcript. Given the new frame-level phone transcript and the audio for existing frames, Voicebox then samples the audio for frames corresponding to the new phones.
Diverse speech sampling & alignment-preserved style shuffling: Voicebox can generate diverse speech samples by infilling the whole utterance. We first use the duration model to sample  $\hat{l}$  given the phone transcript  $\hat{y}$ . We then use the audio model to sample  $\hat{x}$  given  $\hat{z} = \mathsf{rep}(\hat{y},\hat{l})$ . Similar to style transfer, Voicebox can also shuffle the audio style while keeping the alignment by sampling  $\hat{x}$  conditioning on the frame-level transcript  $\bar{z}$  of the target speech clip  $\bar{x}$ .
The common goal of audio-conditioned tasks is to produce realistic speech that is coherent with the context and has the correct textual content. For tasks not conditioned on audio context, it is desired to generate diverse and realistic samples with distribution similar to training data. Prior studies often adopt subjective metrics like mean opinion scores (MOS) [52] which are not comparable across papers, or quantitative metrics like mel cepstral distortion [35] that assume the output is deterministic given input, which is often not realistic [54]. In this paper, we advocate the following reproducible model-based perceptual metrics.
Correctness and intelligibility: We measure it by the word error rate (WER) of the synthesized speech's transcription with respect to the input text, which has been adopted in prior work [62]. Public automatic speech recognition (ASR) models are used for comparability. For English-only setups, we
follow [61] and use HuBERT-L [22] pre-trained on 60K hours of Librilight [26] and fine-tuned on 960 hours of Librispeech [43]. For multilingual setups we use the Whisper large-v2 model [49].
Coherence: This is measured by the similarity between the embedding of generated speech and that of the audio context, where different embedding models would reflect coherence of different attributes. VALL-E proposed to use WavLM-TDCNN speaker embedding model, which maps an audio clip to a fixed dimensional vector, to measure voice similarity. We consider the same model to compare with VALL-E. In particular, VALL-E reports similarity with respect to resynthesized audio context by its vocoder (Encodec-decoder), which we call SIM-resyn (SIM-r). SIM-resyn is not comparable across models using different vocoders. Hence, we advocate for computing similarity against the original audio context, which we call SIM-orig (SIM-o).
Diversity and quality: Fréchet Inception Score (FID) [17] is widely adopted for image generation evaluations, which captures the similarity between generated and real images at the distribution level in some feature space. A shorter distance implies the distributions are more similar and generally reflects both higher sample quality and diversity. We adapt the metric for speech by using self-supervised wav2vec 2.0 feature [2] and refer to it as Fréchet Speech Distance (FSD). We verify its effectiveness in Appendix C.1 along with alternative features.
As supplementary metrics, we include quality MOS (QMOS) for subjective audio quality evaluation, and similarity MOS (SMOS) for subjective audio similarity evaluation given pairs of prompt and system-generated audio clips. Both of which are in the scale of 1 to 5 with 5 being the best. The MOS instructions and standalone metrics for duration models can be found in Appendix C.
Data: We train the English-only model on 60K hours ASR-transcribed English audiobooks and the multilingual model on 50K hours of multilingual audiobooks from six languages: English (En), French (Fr), German (De), Spanish (Es), Polish (Pl) and Portuguese (Pt). The two models are abbreviated as VB-En and VB-Multi. The Montreal Forced Aligner (MFA) [40] is used to phonemize and force align the transcript based on the MFA phone set. Word position postfixes are added. Audio is represented as a 80-dimensional log Mel spectrogram and a HiFi-GAN vocoder trained on the same 60K hours English speech is used to generate waveform. More details about phone representation, data transformation, and vocoder can be found in Appendix A1-A3.
Model: Transformer [59] with convolutional positional embedding [2] and symmetric bi-directional ALiBi self-attention bias [48] are used for both the audio and the duration model. ALiBi bias for the flow step  $x_{t}$  is set to 0. More details in Appendix Appendix A.7. The audio model has 24 layers, 16 attention heads, 1024/4096 embedding/feed-forward network (FFN) dimension, 330M parameters. The duration model has 8 heads, 512/2048 embedding/FFN dimensions, with 8/10 layers for English/multilingual setup (28M/34M parameters in total).
Training: VB-En/VB-Multi audio models are trained for  $500\mathrm{K} / 750\mathrm{K}$  updates with an effective batch size of  $240\mathrm{K}$  frames. For training efficiency, audio length is capped at 1,600 frames and chunked randomly if length exceeds. Duration models are trained for  $600\mathrm{K}$  updates with an effective batch size of  $60\mathrm{K}$  frames. The Adam [33] optimizer is used with a peak learning rate of 1e-4, linearly warmed up for 5K steps and decays over the rest of training. The audio/duration sequence is masked with  $p_{\mathrm{drop}} = 0.3 / 0.2$ , and otherwise a segment of  $r\%$  sequence length is masked, where  $r \sim \mathcal{U}[70,100] / \mathcal{U}[10,100]$ .  $p_{\mathrm{uncond}}$  is set to 0.2 for audio/duration models.
Inference: The torchdiffreq [9] package is used, which implements both fixed and adaptive step ODE solvers. By default, the midpoint solver is used with a step size of 0.0625 (NFE=32). The regression duration model is used by default. Silence at both ends are trimmed to 0.1 second max.
Baselines: We consider three baselines: 1) VALL-E [61], SOTA for English zero-shot TTS trained on Librilight. 2) YourTTS [7], SOTA multilingual zero-shot TTS model trained on VCTK, LibriTTS, TTS-Portugese [8], and M-AILABS French. It is a flow-based model adapted from VITS [32] using a pre-trained speaker embedder. 3) A3T [3], SOTA for NAR speech editing and infilling trained with a regression loss on VCTK. We also consider Demucs [11], a SOTA speech enhancement model trained with regression and adversarial losses for denoising experiments.
5.1 Monolingual and cross-lingual zero-shot TTS
Table 2 presents the zero-shot TTS results of the English model VB-En. Following [61], the test set is constructed by selecting 4 to 10 second long samples from Librispeech test-clean. We consider cross-sentence prompting where a 3 second clip from another sample of the same speaker is used as audio context, and continuation prompting where the first 3 seconds of each utterance is used. Voicebox outperforms all baselines on all metrics in both cases. In particular, Voicebox transfers style much more effectively  $(+0.101 / +0.108$  SIM-r on cross-sentence/continuation) than VAL-E, and the gap is even bigger when compared against raw audio  $(+0.141$  SIM-o on continuation). MOS studies also confirm the quality and similarity of Voicebox are subjectively better than YourTTS.
Table 2: English zero-shot TTS results on filtered LS test-clean. *obtained via personal communication.
Model	WER	SIM-o	SIM-r	QMOS	SMOS
Ground truth	2.2	0.754	n/a	3.98±0.14	4.01±0.09
cross-sentence
A3T	63.3	0.046	0.146	-	-
YourTTS	7.7	0.337	n/a	3.27±0.13	3.19±0.14
VALL-E	5.9	-	0.580	-	-
VB-En	1.9	0.662	0.681	3.78±0.10	3.71±0.11
continuation
A3T	18.7	0.058	0.144	-	-
VALL-E	3.8	0.452*	0.508	-	-
VB-En (α = 0.7)	2.0	0.593	0.616	-	-
Table 3: Transient noise removal where noise overlaps with  $50\%$  of the speech at a -10dB SNR.
Model	WER	SIM-o	QMOS
Clean speech	2.2	0.687	4.07±0.15
Noisy speech	41.2	0.287	2.50±0.15
Demucs	32.5	0.368	2.86±0.17
A3T	11.5	0.148	3.10±0.15
VB-En (α = 0.7)	2.0	0.612	3.87±0.17
Table 4 presents cross-lingual zero-shot TTS results, where the audio context and the target text are in different languages. Note that VB-Multi is not trained on any sample with multiple languages in an utterance spoken by the same speaker. The test set is constructed using filtered MLS test split described in Appendix A.4. For each target text, we sample one 3-second long audio context from each language, which creates 36 language transfer directions in total. Voicebox yields better performance than YourTTS everywhere. Specifically, on En/Fr/Pt which YourTTS supports, Voicebox obtains  $3.1\% /5.9\% /8.1\%$  lower WERs and 0.136/0.141/0.160 higher similarity averaged across audio context in six languages. Addition studies on prompt lengths are presented in Appendix B.2
Table 4: Multilingual zero-shot TTS results on filtered MLS test sets. GT/YT/VB-Multi refers to ground truth/YourTTS/multilingual Voicebox. "Ref" column shows the audio context language.
Ref	De	En	Es	Fr	Pl	Pt
WER	SIM-o	WER	SIM-o	WER	SIM-o	WER	SIM-o	WER	SIM-o	WER	SIM-o
GT	-	5.9	0.725	5.0	0.636	4.1	0.729	5.2	0.714	4.9	0.743	5.8	0.725
YT	De	n/a	n/a	7.3	0.373	n/a	n/a	11.3	0.361	n/a	n/a	13.7	0.263
En	n/a	n/a	7.0	0.403	n/a	n/a	11.4	0.298	n/a	n/a	14.1	0.234
Es	n/a	n/a	7.6	0.327	n/a	n/a	11.6	0.316	n/a	n/a	13.5	0.256
Fr	n/a	n/a	7.6	0.363	n/a	n/a	10.7	0.459	n/a	n/a	13.1	0.299
Pl	n/a	n/a	7.8	0.349	n/a	n/a	11.8	0.370	n/a	n/a	15.1	0.308
Pt	n/a	n/a	7.6	0.322	n/a	n/a	11.8	0.297	n/a	n/a	13.6	0.436
AVG	n/a	n/a	7.5	0.356	n/a	n/a	11.4	0.350	n/a	n/a	13.9	0.299
VB-Multi (α=1.0)	De	4.8	0.632	4.8	0.522	3.6	0.442	5.3	0.489	5.5	0.449	5.4	0.420
En	5.9	0.435	4.2	0.535	4.1	0.423	6.8	0.423	8.3	0.402	7.6	0.385
Es	4.9	0.460	4.3	0.479	3.6	0.613	5.3	0.473	5.2	0.436	5.4	0.435
Fr	4.9	0.476	4.3	0.485	3.7	0.479	5.1	0.602	4.8	0.408	5.4	0.418
Pl	4.7	0.491	3.8	0.503	3.5	0.528	5.1	0.503	4.0	0.641	4.9	0.476
Pt	4.9	0.422	4.6	0.426	3.7	0.476	5.5	0.453	4.8	0.406	5.2	0.620
AVG	5.0	0.486	4.4	0.492	3.7	0.494	5.5	0.491	5.5	0.457	5.7	0.459
5.2 Transient noise removal
We construct a noisy test set by mixing the filtered Librispeech test-clean from Section 5.1 with non-speech noise such that it overlaps with  $50\%$  of the duration at a -10dB signal-to-noise ratio. Additional conditions can be found in Appendix B.4. Table 3 presents the results comparing Voicebox with A3T and Demucs. It should be noted that A3T and Voicebox utilize transcript and location of the noise while Demucs does not. Compared to the baselines, Voicebox generates samples best on all metrics. A3T is better than Demucs on intelligibility and quality, but the infilled speech is not coherent because it is only trained on VCTK.
5.3 Diverse speech sampling and application to ASR data generation
Table 5 compares the ability to generate diverse samples for Librispeech test-other text. We consider English Voicebox (VB-En) with regression (repr) or flow-matching (FM) duration models. VITS-VCTK additionally conditions on a speaker ID, which we randomly sample for each sentence. YourTTS conditions on text and a reference audio, which we draw from the LS train splits. Qualitatively, A3T generates the same robotic voice and VITS-LJ generates high quality but from a single voice, hence both yield high FSD (bad quality or diversity) but VITS-LJ has a low WER. VITS-VCTK improves the voice diversity and FSD and YourTTS further advances it as it is trained on more speakers. Voicebox models (with different duration samplers) outperform the baseline on FSD by large margins, showing Voicebox's ability to produce realistic and diverse samples whose distribution is close to the training data. Among them, the FM duration model creates more varying speaking styles compared to the regression one which ASR may struggle more to recognize.
Table 5: Diverse speech generation from LS test-other text.
Model	WER	FSD
Ground truth	4.3	171.1
require additional input
VITS-VCTK	10.6	306.6
YourTTS (ref=LS train)	9.0	277.9
text-only
A3T	37.9	373.0
VITS-LJ	5.6	344.2
VB-En (α = 0, dur=regr)	3.1	155.7
VB-En (α = 0, dur=FM, α_dur = 0)	5.6	159.8
Table 6: Performance of ASR models trained on real or synthetic speech, tested on real speech and decoded with or without a 4-gram language model.
ASR training data	WER on real data
No LM	4-gram LM
test-c	test-o	test-c	test-o
Real audio (100hr)	9.0	21.5	6.1	16.2
Real audio (960hr)	2.6	6.3	2.2	5.0
VITS-LJ	58.0	81.2	51.6	78.1
VITS-VCTK	33.8	55.5	30.2	53.1
YourTTS (ref=LS train)	25.0	54.6	20.4	51.2
VB-En (α = 0, dur=regr)	7.1	17.6	6.5	14.6
VB-En (α = 0, dur=FM, α_dur = 0)	3.1	8.3	2.6	6.7
We next train an ASR model using only synthetic speech and evaluate it on real speech, which has not been successful before because synthetic data were not realistic and representative enough. Table 6 compares real and synthetic data from Voicebox and three baseline models. Each TTS model generates one sample per text from the Librispeech training set, resulting in 281K utterances per system. For real data, we consider train-960 and train-clean-100. Details about the ASR model and training are in Appendix A.5. The results are highly correlated with the FSD scores for synthetic data. In particular, the ASR model trained on Voicebox data with FM duration model reduces WERs by over  $85\%$  compared to baselines, and only lags behind real data by  $0.4\%$  and  $1.7\%$  absolute.
5.4 Inference efficiency versus performance
We examine the trade-off between the metrics of interest (WER, SIM, FSD) for different settings of guidance strength  $(\alpha)$  and NFE specified by the user. Fig. 3a shows the Voicebox inference time to generate an audio sample of 10 seconds (including vocoding and predicting duration) as NFE varies and compares that to VALL-E. For NFE=2 without CFG, Voicebox takes about 0.31 seconds, about 20 times faster than VALL-E. At NFE=64, Voicebox is only  $4\%$  slower than VALL-E.
Next, we study the cross-sentence setup of Section 5.1 to analyze the impact on WER and SIM-r. We find that for all settings Voicebox has better WER than VALL-E. WER remains stable with mean of 2.0 and variance of 0.005. WER plot can be found in Appendix B.5. As shown in Fig. 3b, in the case of SIM-r, lower classifier guidance strength values ( $\alpha = 0$  or 0.3) produce higher speaker similarity when operating in a lower NFE regime ( $\leq 4$ ). However, starting from NFE=8, a higher classifier guidance strength improves speaker similarity. Finally, in Fig. 3c we examine FSD by generating samples for Librispeech test-other text. We find that lower classifier guidance strength produces lower FSD scores and more diverse samples. Increasing the NFE for each setting improves FSD.
5.5 Ablation on generative modeling approaches
We compare three generative modeling approaches: the proposed flow-matching with the OT path (FM w/ OT), flow-matching with the variance preserving (VP) diffusion path (FM w/ diff), and score-
3 Re-implemented and confirmed with the authors that our re-implementation is faster (6.2 vs 10 seconds).
(a) NFE vs inference time w/o CFG
(b) NFE vs Speaker Similarity
(c) NFE vs FSD
Figure 3: Trade-off between NFE and different metrics. Inference time will be doubled with CFG.
Table 7: Comparing different objectives on training efficiency. 32 NFEs are used for inference. Each model is evaluated on the monolingual zero-shot TTS task.
Method	upd=50K	upd=100K	upd=150K
WER	Sim-o	WER	Sim-o	WER	Sim-o
FM w/ OT (proposed)	2.5	0.424	2.2	0.487	2.1	0.508
FM w/ diff	76.0	0.066	3.1	0.344	2.6	0.478
SM w/ diff	73.3	0.062	17.4	0.176	5.1	0.349
Table 8: Comparing different objectives on inference efficiency. All models are trained for 150K updates. Each model is evaluated on the monolingual zero-shot TTS task.
Method	NFE=4	NFE=8	NFE=16	NFE=32
WER	Sim-o	WER	Sim-o	WER	Sim-o	WER	Sim-o
FM w/ OT (proposed)	2.4	0.410	2.2	0.481	2.2	0.503	2.1	0.508
FM w/ diff	11.5	0.171	3.0	0.359	2.7	0.447	2.6	0.478
SM w/ diff	94.5	0.054	42.3	0.076	11.5	0.218	5.1	0.349
matching with the VP diffusion path (SM w/ diff). A reduced setup described in B.3 is adopted, with a lowered learning rate (1e-4) and the loss in Eq. (1) to ensure convergence for all three objectives.
We vary the number of training and inference steps, and evaluate models on the zero-shot TTS task (Section 5.1). Results in Table 7 shows that FM w/ OT trains significantly faster than the other two objectives, achieving the best performance with 100K training steps, and even outperforms SM w/ diff using only 50K updates. Results in Table 8 shows superior inference efficiency of FM w/ OT, which can produce good results with just 8 NFEs, while FM w/ diff requires at least 8 NFEs and SM w/ diff requires over 32 NFEs. Complete results are in Table B5
This paper presents Voicebox, the most versatile generative model for speech. By learning to solve a text-guided speech infilling task on large scale multilingual datasets with a power model and training objective Voicebox demonstrates impressive task generalization capabilities. Voicebox achieves state-of-the-art performance on mono and cross-lingual zero-shot TTS, speech inpainting, and diverse speech sampling, and can generate speech 20 times faster than the best autoregressive models.
With high fidelity speech generation models like Voicebox, it brings the potential of misuse and unintended harm. To mitigate the risk, we also detail in Appendix B.1 that a highly effective classifier can be built to distinguish between authentic and synthetic speech. Voicebox is now trained only on read speech from audiobooks in six languages, and cannot transfer one attribute (e.g., emotion) from a reference while transferring another attribute (e.g., voice) from another reference. Due to space limit, we expand our discussion of limitation and broader impact in Appendix D. For future work, we would continue scaling the model and the data to include more languages and diverse types of speech such as conversations, and explore disentangled prompting for different attributes.
The authors would like to thank Kristin Lauter and Joelle Pineau for supporting the project, thank Ricky Chen, Yaron Lipman, Alexandre Defossez, Gabriel Synnaeve for the technical discussion, thank Jade Copet and Gabriel Synnaeve for the compute support, thank Eleonora Presani and Jackie Pan for discussing the responsible AI studies, thank William Ngan, Somya Jain, Lydia Baillergeau, Dana Beaty, Chantal Mora, Daniel Duncan, Gopika Jhala, Steph Miles, Josh Terry, Valeryia Aranovich, Ashton Evans, Aly Gill, Andrea Mileskiewicz, Emily Richards, and Aaron Vasquez for developing the visual assets and the website, thank Alyssa Newcomb and Oliver Libaw for developing the posts, thank Peter Gray, Natalie Hereth, Shauna Kelleher, Ashley Gabriel, Seine Kim, Ana Paula Kirschner Moffarej and Aiman Farooq for coordinating the launch, thank Harrison Rudolph, Mallika Malhotra, Carolyn Krol, Lauren Cohen and Mo Metanat for the reviews, and thank Alexandra Gualdino, Ana Paula Kirschner Mofarrej, Benjamin Muller, Chloe Rolland, Daniella Kalfa, Darcie Da Silva, Gabriel Synnaeve, Hunter Goldman, Juan Pino, Karen Ulrich, Kris Sekula, Manuel Ribeiro, Marina Zannoli, Mary Williamson, Rashel Moritz, Stephanie Castillo, Tu Anh Nguyen, Vlad Sobal, Volker Seeker, and anonymous volunteers for sharing speech samples for the demo.
[1] A. Aghajanyan, L. Yu, A. Conneau, W.-N. Hsu, K. Hambardzumyan, S. Zhang, S. Roller, N. Goyal, O. Levy, and L. Zettlemoyer. Scaling laws for generative mixed-modal language models. ArXiv, abs/2301.03728, 2023.
[2] A. Baevski, Y. Zhou, A. Mohamed, and M. Auli. wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in neural information processing systems, 2020.
[3] H. Bai, R. Zheng, J. Chen, X. Li, M. Ma, and L. Huang. A3T: Alignment-aware acoustic and text pretraining for speech synthesis and editing. In International Conference on Machine Learning, 2022.
[4] Z. Borsos, R. Marinier, D. Vincent, E. Kharitonov, O. Pietquin, M. Sharifi, O. Teboul, D. Grangier, M. Tagliasacchi, and N. Zeghidour. AudioLM: a language modeling approach to audio generation. *ArXiv*, abs/2209.03143, 2022.
[5] Z. Borsos, M. Sharifi, and M. Tagliasacchi. SpeechPainter: Text-conditioned speech inpainting. In Interspeech, 2022.
[6] T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. J. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei. Language models are few-shot learners. ArXiv, abs/2005.14165, 2020.
[7] E. Casanova, J. Weber, C. D. Shulby, A. C. Junior, E. Gölge, and M. A. Ponti. YourTTS: Towards zero-shot multi-speaker tts and zero-shot voice conversion for everyone. In International Conference on Machine Learning, 2021.
[8] E. Casanova, A. C. Junior, C. Shulby, F. S. d. Oliveira, J. P. Teixeira, M. A. Ponti, and S. Aluásio. Tts-portuguese corpus: a corpus for speech synthesis in brazilian portuguese. Language Resources and Evaluation, 56(3):1043-1055, 2022.
[9] R. T. Q. Chen. torchdiffeq, 2018. URL https://github.com/rtqichen/torchdiffeq.
[10] R. T. Q. Chen, Y. Rubanova, J. Bettencourt, and D. K. Duvenaud. Neural ordinary differential equations. In Neural Information Processing Systems, 2018.
[11] A. Défossez, G. Synnaeve, and Y. Adi. Real time speech enhancement in the waveform domain. ArXiv, abs/2006.12847, 2020.
[12] A. Défossez, J. Copet, G. Synnaeve, and Y. Adi. High fidelity neural audio compression. ArXiv, abs/2210.13438, 2022.
[65] R. Yamamoto, E. Song, and J.-M. Kim. Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogram. In International Conference on Acoustics, Speech and Signal Processing, 2020.
[66] N. Yu, V. Skripniuk, S. Abdelnabi, and M. Fritz. Artificial fingerprinting for generative models: Rooting deepfake attribution in training data. In Proceedings of the IEEE/CVF International conference on computer vision, pages 14448-14457, 2021.
[67] N. Zeghidour, A. Luebs, A. Omran, J. Skoglund, and M. Tagliasacchi. Soundstream: An end-to-end neural audio codec. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 30:495-507, 2022.
A Additional Details of Experiment Setup
A.1 Vocoder
We adapt the HiFi-GAN V1 configuration to generate  $16\mathrm{kHz}$  audio from 80 dimensional log Mel spectral features sampled at  $100\mathrm{Hz}$ . To compute the log Mel spectrogram, we use a 1024-point short time Fourier transform with a 640-sample (40ms) analysis window, 160-sample (10ms) shift, and the Hann windowing function to compute the amplitude spectrogram, and then apply an 80 dimension Mel filter with a cutoff frequency at  $8\mathrm{kHz}$ . The original HiFi-GAN V1 has four transposed convolution blocks for upsampling. The upsampling factors are  $[8, 8, 2, 2]$  and the corresponding kernel sizes are  $[16, 16, 4, 4]$ . Here we only need a total upsampling factor of 160 instead of 256, and we adjust the upsampling factors to  $[5, 4, 4, 2]$  and kernel sizes to  $[11, 8, 8, 4]$  accordingly. The other parameters are identical to the HiFi-GAN V1 configuration. Total number of parameters is 13M. We train the adapted HiFi-GAN on the 60K hours of English audiobook data for 1.5M updates on 8 GPUs, which takes 7.5 days.
A.2 Phone representation
Ghost silence The frame-level phonetic transcript used for training is obtained through force-aligning speech and phonetic transcript. In particular, a forced aligner may align some frames to a special phone "SIL" for non-speech frames (silence or noise). For most forced aligners, only frames between words and frames at the beginning and at the end of an utterance can be aligned to SIL.