Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale
Authors: Matthew Le
Apoorv Vyas
Bowen Shi
Brian Karrer
Leda Sari
Rashel Moritz
Mary Williamson
Vimal Manohar
Yossi Adi
Jay Mahadeokar
Wei-Ning Hsu
Venue: NeurIPS 2023 poster (2023)
Keywords: speech generation
flow-matching
diffusion
in-context learning
text-to-speech

Abstract: Large-scale generative models such as GPT and DALL-E have revolutionized the research community. These models not only generate high fidelity outputs, but are also generalists which can solve tasks not explicitly taught. In contrast, speech generative models are still primitive in terms of scale and task generalization. In this paper, we present Voicebox, the most versatile text-guided generative model for speech at scale. Voicebox is a non-autoregressive flow-matching model trained to infill speech, given audio context and text, trained on over 50K hours of speech that are not filtered or enhanced. Similar to GPT, Voicebox can perform many different tasks through in-context learning, but is more flexible as it can also condition on future context. Voicebox can be used for mono or cross-lingual zero-shot text-to-speech synthesis, noise removal, content editing, style conversion, and diverse sample generation. In particular, Voicebox outperforms the state-of-the-art zero-shot TTS model VALL-E on both intelligibility (5.9\% vs 1.9\% word error rates) and audio similarity (0.580 vs 0.681) while being up to 20 times faster. Audio samples can be found in \url{https://voicebox.metademolab.com}.

Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale
Matthew Le"
Apoorv Vyas"
Bowen Shi"
Brian Karrer"
Leda Sari Rashel Moritz
Mary Williamson Vimal Manohar
Yossi Adi
Jay Mahadeokar Wei-Ning Hsu"
Fundamental AI Research (FAIR), Meta
Abstract
Large-scale generative models such as GPT and DALL-E have revolutionized the research community. These models not only generate high fidelity outputs, but are also generalists which can solve tasks not explicitly taught. In contrast, speech generative models are still primitive in terms of scale and task generalization. this paper, present Voicebox, the most versatile text-guided generative model for speech at scale. Voicebox is non-autoregressive flow-matching model trained infill speech, given audio context and text, trained on over 50K hours of speech that are not filtered or enhanced. Similar to GPT, Voicebox can perform many different tasks through in-context learning, but is more flexible as it also condition on future context. Voicebox can used for mono or cross- looss-lingual zero-shot text- to-speech synthesis, noise removal, content editing, style conversion. and diverse sample generation. In particular, Voicebox outperforms the state-of-the-ar zero- shot TTS model VALL-E on both intelligibility (5.9% vs 1.9% word error rates) and audio similarity (0.580 vs 0.681) while being up to 20 times faster. Audio samples be found in https: httt::://voiebbsss www.:iiw.........ooeeo....oe www.deeoaaaaaaaammmmammaam com
1 Introduction
Recent advances in large-scale generative models 6 towards building general-purpont models, which can perform many new tasks not explicitly trained on. These generative models learn to predict the missing data given the context. Post training, we can directly input question, optionally with few contextual question-answer examples, instead of fine-tuning with labeled data. While the training objective appears simple, it subsumes many tasks as one can convert them into some form of context. For the model to perform well at every task, implies that the estimation of p pmmissnng data context) needs to be accurate for every context. Hence, scale and diversity are the most crucial factors for building
42 50
have led to major paradigm shift
models 20 IT.
Despite the success of lcrge-scale generative models in other areas. most speech models are still trained datasets the scale of tens hundreds of hours 55 31 32
46
24
58 7) Previous works
consider highly curated datasets such VCTK [64], which contains only clean audio recorded in studio from about 100 speakers with little speaking style and text variation. Such models struggle to synthesize speech with rich variation in emotion, voice, background noise, acoustic condition, and have not been tested on the abilities to generalize to tasks not explicitly trained on.
This paper presents Voicebox, the most versatile text-conditioned speech generative model at scale. Voicebox is trained on a text-guided speech infilling task, where the goal is to generate masked speech
'Equal contribution. Corresponding authors: {mattle, wnhsu) com FAIR & Hebrew University of Jerusalem.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Editing
Style Transfer
Text-Only Sampling
Table 1:
Comparing Voicebox with baselines on task capabilities. "Through infilling, A3T and Voicebox can remove transient noise but not stationary background noise.
Model
Input
MOVI
Model
78 TTS
Denoise
Fdltt
Sampling
YourT
X
X
Jee
VALL-E
/
K
X
/
Model
Output
Demues
x
/
x
x
/
/
Figure I: Task generalization via in-context learning.
given its surrounding audio and text transcript. This can be considered as guided in-context learning problem, where audio style is inferred from the audio context and textual content is specified through transcript. Voicebox does not require any audio style labels (e.g., speaker, emotion, and noise), which differentiates Voicebox from the majority of prior work where such labels are used extensively. Prior work uses labels to make the mapping between input (text and audio style) and output (speech) more deterministic reduce underfitting [60] 46). We show that Voicebox text- exxt-guided speech infilling approach is much more scalable in terms of data while subsuming many common generative tasks.
In terms of modeling, Voicebox is a non-autoregressive (NAR) continuous normalizing flow (CNF) model [O]. Similar to diffusion models [9), CNFs model the transformation from simple distribution to complex data distribution, parameterized by neural network. We train Voicebox with flow-matching (88] recently proposed method that enables efficient and scalable training of CNFs via simple vector field regression loss. In contrast to auto-regressive models, Voicebox can consume context not only in the past but also the future. Moreover, the number of flow steps can be controlled at inference time to flexibly trade off quality and runtime efficiency.
Voicebox is trained on 60K hours English audiobooks and 50K hours multilingual audiobooks in 6 languages for the mono and multilingual setups. Voicebox achieves state-of ofethh-rrt performance on mono- wwaoo-iigaaaooss-inngeegee
zero-shot TTS, speech denoising, speech editing. diverse speech sampling and an application to data creation for speech recognition. To tackle the lack of comparability due to the use of subjective metries, this paper presents a series of metrics using public models to facilitate reproducible comparison and model development for speech generation studies.
Related Work
Generative speech models: Most speech generative models are task-specific and trained on different datasets. One common type of task is audio style conversion, which aims to convert only a specific attribute while keeping other attributes the same. Voice conversion 27 39. emotion conversion [3
34). speech enhancement [63 [ [55) belong this category. Many of these models supervised and trained on pairs of data that only differ in one attribute, for example, emotion [34] is hard to
obtain such data. Moreover, some attributes, such speaking style, are hard to annotate. Hence, these models are often trained on small datasets.
Controllable text-to-speech synthesis (TTS) is another common task, which aims to synthesize speech in target audio style given text. While some styles like voice can be specified through labels [32] or pre-trained embeddings like YourTTS 7 and Jia et al. (25); others like prosody are hard to annotate or embed. Previous studies [62] tried to control them by learning a residual embedding. However, these models encode style in low-dimensional space and impose overly simple distribution of speech given text and residual embedding [51 (6).. They cannot generate realistic noisy speech given low dimensional vector, and performance degrades when conditioned noisy references [2]].
Infilling can be considered as another type of task. It aims to predict speech given context [6 Instead of learning explicit embedding to control style, infilling models predict speech coherent to the context. In other words, these models perform in- context learning similar to Large Language Models (LLM). While this is a step toward building large
and optionally text guidance 3 3 61

---

scale generalist models using little explicit supervision, most prior work using text guidance still assumes a deterministic mapping from text and context to target B1. which only realistic for very short segments. Voicebox is text-guided infilling model, but it leverages the CNF model that can parameterize any distribution. Hence, Voicebox can infill speech of any length and can be trained on in-the-wild datasets with rich variation, and provide general solution that subsumes many tasks in text-guided fashion.
Large scale in-context learning models: With the advancement in neural codec for speech 22 12 67 many recent studies explore token-based language modeling for speech generation. The GSLM-family [36 28 41 are textless language models built upon HuBERT units [22] for speech
continuation without using text. HuBERT units encode mostly content, and the generated speech
does not preserve the voice of the prompt. To tackle this, AudioLM [4] considers a cascaded approach which first generates HuBERT-likke tokens and then predicts SoundStream (67] tokens, a reconstruction based codec that preserves style. These models are not conditioned on text and are evaluated on spoken language modeling tasks.
VALL-E [1]] is most related to Voicebox. It is a text conditioned LM trained on Encodec [12 tokens (similar SoundStream). Encodec encodes each frame with 8 ordered codebooks at 75Hz using a residual quantization layer. VALL-E has two modules. The first is an auto-regressive (AR) model that predicts the first code of each frame given text and the audio prompt. The second is an NAR model that predicts the remaining seven codebooks sequentially.
VALL-E demonstrates state-of-the-art (SOTA) zero-shot TTS performance through in-context learn ing, where speech of the desired style is used as prompt. The model considers the prompt as part of the whole utterance such that generates the rest of the utterance containing the target text in the same audio style. Voicebox has several design advantages compared to this. 1) Voicebox can use context both in the past and future, which is useful for editing where only a segment in the middle needs to be generated. 2) Voicebox can generate speech much faster than VALL-E because fow-matching can produce high quality samples with less than 10 NAR steps, while VALL-E requires AR and 7 NAR steps. 3) Voicebox decouples duration and audio modeling, enabling finer grained alignment control. 4) Voicebox is compatible with any continuous features including Encodec embeddings.
3 Method
3.1 Background: Flow Matching with an optimal transport path
Let Rd be the data space with data points x E Rd drawn from some unknown distribution q(x).. Continuous Normalizing Flows (CNFs) 10 are a family of generative models that learn the transformation from simple prior distribution po (e.g., normal distribution) to the data distri- bution P1 x q. CNFs parameterize a time-dependent vector field U, : [0, 1] x Rd + Rd that is used to construct a flow: Φ; : [0, 1] x Rd + Rd that pushes points from the prior towards the target distribution. The relationship is defined via the ordinary differential equation (ODE) as: d = ve (x)) and (x) = x. For a flow Φr. the probability path (time-dependent proba- bility density function) p : [0, 1] x Rd + R50 can be derived via the change of variables formula: Pe(z) = >000p;(00 po(0r (x)) det [Ø0_ (x)) To sample from p. (x)), we first draw 2g from po and then solve the initial value problem (IVP) for (x0) given d = ve())(x)) and do(x)) = z0. We use 2y, and di (zo) interchangeably.
Let pt be a probability path and u, be the corresponding vector field that generates pt. The vector field (x( 8) parameterized by a neural network @ can be trained with the Flow Matching objective: CFM(9) (0) = (x)) [x) ve I)1 where t ~ [4, 1] and (z))). While the objective appears simple, in practice we do not have the prior knowledge of p. or Ut. and cannot directly compute the loss or its gradient estimator.
Let F1 be a random variable distributed according to data distribution q. Lipman et al. 38 first notes that a probability path Pr (z) can be constructed via a mixture of simpler conditional paths Pe(x xx whose vector field u. (x: x1) can be easily computed. To construct Pt (2)), a conditional
path is defined such that 1) po(a x1) Po(x) and 2) pı (x x1) =Nxxx I o² I), Gaussian
distribution centered at 1r with a sufficiently small a (typically 10 10-5). The marginal path is computed
as f (x(x x1 [x )dz1, which closely approximates ((x1) at t = 1. With that, [8 presents the Conditional Flow Matching (CFM) objective, LCFM (0) = (eellte)...eee))llllll
x1) (u
(2; 2
3

---

It is proven that FM and CFM have identical gradients w.r.t. 8. More importantly. one can easily draw samples from pz 21) and compute u4 21) derive an unbiased gradient estimator.
The next question is how to choose a conditional flow. A flow defines trajectories, which dictates how each point moves between po and P1- Intuitively, simpler trajectory (e.g., ., straight line) be learned faster and the IVP can be solved more accurately and efficcinnll. Lipman et al. [38] presents conditional flow called optimal transport (OT) path, which has the form of pr x
Aa
10 (1- -
3mm))))) 1) and ue (z 21) = (x) (1 min )z) (1 (1 min )t) The flow is
arguably simple because points move with constant speed and direction. We adopt it for Voicebox.
Lipman et al. [38] also presents another flow that recovers the path of diffusion models [57], which more complex than the OT path. We will present ablation studies comparing different paths (OT vs diffusion) and different objectives (CFM vs score-matchigg). Results show the superiority in performance and efficiency of CFM with OT path.
3.2 Problem formulation
Given dataset of transcribed speech (z, y) where and denote an audio sample and its transcript, respectively, the goal is to build a single model that can perform many text-guided speech generation tasks through in-context learning. We propose to train such generative model the text- text-guided speech infilling task, which predicts a segment of speech given its surrounding audio and the complete text transcript. Let m be binary temporal mask which is of the same length x, and Tmis and Fetx = (1 m) @ x be the complementary masked versions of x. The generative model learns p(Imii y, Fetz) ) In other words, y and Feta are the context and Emis is the missing data.
O
Volcebox Audio Model
Fale
Liber
16.44
H
Voicebox Audio Model Inference
008 eup
Sinuso embedter
210.1
crdiions (sitar)
ViCaOO de model
do
Phonemize & Foroed mige
Random span musk ng
Compute 53 malle Bon slop for palls
Fow doeg
Tor prir
(b) Inference is equivalent to solving an ODE with an initial condition zo sampled from the prior, a derivative dai specified by the audio model, and conditional inputs (z, Fetz).
(a) Voicebox audio model. The lower half illustrates how inputs are created during training.
Figure Illustration of Voicebox training and inference.
3.3 Model and Training
Motivated by the need that some applications require fine-grained alignment control between speech and text, we decouple Voicebox into two components: an audio model and duration model. Let (x:1 22..... .... x. be an audio sample of N frames, = (y² .y?
M be a text sequence of M phones, and I (1', 12 ....... 1M) be the per-phone duration where 1 denotes how many audio frames yd correspond to and EØe, = We further define = = rep(y, I) = (z 2,, ,,>) to be the frame-level phone transcript, which repeats each y' by ( times such that z² denotes the phone label of the audio frame z². For a pair of (I., y). l and a can be estimated through forced aligumens using a speech recognition model. The estimation of q(Emis audio model q(Emis 2, &ear) and the duration model g(Imis y. L.tz), where imis and Latz denote I masked by m and 1 m', and m' is downsampled from m based on I, detailed in Appendix A.2.
y. Fetx) is then broken down into the
Audio Model: Given context z and Fat of length N, the distribution of Tmis highly stochastic especially when Eiis has large temporal span. Hence, parameterize it with a CNF and train

---

it using the flow matching objective with the optimal transport path. Audio r is represented as an 30-dimensional log Mel spectrogram (x' I25 extracted at a 100Hz frame rate. The audio context Fet O where m² = 1 and Fet = xf where m² = O. For simpler conditioning, we model the conditional distribution q( 2, Tetx) of all frames x instead of only masked frames Imis- A neural network is used to parameterize the conditional vector field u, Reta: 0) that additionally takes Teta and z as input. Note that 2t is a sample at flow step and x = 21-
Given as input Jetz E NR Ir E RR-XP, phone sequence 2 E [K] with K denoting the number of phone classes, and time step E [0. 1], we employ a Transformer model to parameterize the vector is used to embed the phone sequence 2, resulting in the embedded sequence Zemb E IR Nx fl where eemb L(2") for i I,..., .N. Subsequently, the three sequences (Iz, Retar and zemb) are concatenated frame-by-fin bo-fm--eeeeeeee and projected by employing matrix W E (2(2+H))x thereby obtaining the sequence He E NxD where D represents the embedding dimension of the Transformer model.
field v2. A lookup table, denoted as L E R xxxxxxx H
To embed the flow step, a sinusoidal positional encoding is applied to map t E [0, 1] to h4 E R..
The sequence H. E RN++1)xD which serves as the input to the Transformer model, is derived by concatenating He with the vector hy along the time dimension. Given the Transformer output Ve (e Tmis: 2; 0) E RR. NxF, which is the sub-sequence corresponding to He, the loss is computed as:
Caudio-CFMi = (o)
x) - oe (2 Feta, 0)1
(I)
by reparameterization. During training, given audio sample æ and prior sample To. have Ie = (1 - (1 min Txo + to and (x z) = I - (1 mmin )ay. This function computes the loss on all frames, including those that are not masked and would not be required during inference. To divert the model's 's focus to masked frames, we present a masked version of Laudio
Caudio-CrM-m ( Ex,, Im @ (Me(2z x) (2 Fetar 2; 0)) 112
(2)
where the loss is only computed on masked frames. Appendix B.3 shows it leads to better results
Duration model: We consider two solutions. The first one closely follows the audio model. models q(! y, fetw) via a conditional vector field which swaps (Ir, Tota, z) with (l, lotes y) and accordingly for the flow, where Lete E R Adxx and y E [K] The masked version of the CFM loss is used for training. On the other hand, previous studies have shown that regression duration models can produce reasonable speech [3]] Hence we consider a second solution that regresses the masked duration liii given the context duration letz and phonetic transcript y. The same Transformer model is used, except that there are only two input sequences instead of three, and the time embedding is not used. The model is trained with an L,1 regression loss on masked phones:
Cour-rogrmm (0) Emn.g(t.y)llrr Imm O (Lmis - g(lote y; 0)) I|l,
where 9 denotes the regression- based duration model. This is similar to the duration model used in FastSpeech2 [51], but with additional duration context letz as input.
3.4 Inference
To sample from the the learned audio distribution Pı (x 2, 2, a noise zo is first sampled from Po. and then an ODE solver is used to evaluate di (2)) given dep (x) /dt = vy (z), Fetas 2; f) and the initial condition do (20) = zo. Intuitively, the ODE solver computes σi (zro by evaluating v at multiple to approximate the integration from / = 0 to t = given the initial condition Φo(x0) = zi- The number of function evaluation (NFE) is defined as how many times dg) /dl is evaluated. A higher NFE often leads to a more accurate solution of d1 (mo) at the cost of longer run time. This provides great flexibility for users decide the trade-off between speed and accuracy. Moreover, find that empirically Voicebox can already generate very high quality speech with less than 10 NFEs, making it significantly faster compared to auto-regressive models.
3.5 Classifier-Free Guidance
Classifier guidance (CG) [4 is technique used to trade off mode coverage and sample fidelity for diffusion models post training. It modifies the score estimate of diffusion model to include the gradient of the log likelihood of an auxiliary classifier. Ho and Salimans [8] notes that CG approximates sampling from p(z c)p(c x) where is the conditioner, and this can be simulated
5

---

without a classifier by mixing the score estimate of a conditional model and an unconditional model. The unconditional model can be jointly trained by dropping the conditioner with some probability. and the same model provides score estimates for both p(æ) and p( c).
We extend the idea of classifier free guidance (CFG) to flow-matching models. The conditioner e is equivalent to (z, Jeta) for audio models and (y, lets) for duration models, which is dropped with Puncond during training. During inference, the modified vector field o, for the audio model becomes Ut (w; 0). where or is the strength of the guidance, and 2y [w; 8) obtained by dropping Feta and We use a and Odur for the CFG strengths for the audio and the duration model, selected based on validation. Note that the computation is doubled for
(wr, Tmis 2; 0) = (1+ ax) a:) Uy (r(w, Eeta, 2; 0)
the same NFE when using CFG, because the model forward called twice to compute öt.
3.6 Applications
We demonstrate that Voicebox exhibits in-context learning abilities similar to LLMs by presenting a few examples of how to create context to perform tasks Voicebox was not explicitly trained on. Fig. shows detailed diagram of how inputs are formatted for each task.
Zero-shot TTS & sligmment-preservee style transfer: Given a target text á and a transcribed reference audio (r, zero-shot TTS aims to synthesize speech resembling the possibly unseen audio style of the reference. Voicebox performs the task by treating the reference audio and the target speech as one utterance where the target speech is masked. Let and z be phone duration and frame-level transcript of (z, y). The target duration I is sampled given the duration context / and concatenated phone sequence cat 9). The target speech 2 is then sampled given the context x and concatenated frame-level phones cat(z, rep(ū.1) ()).
Voicebox can also convert the audio style for speech F while preserving its alignment ž. This is useful for editing audio that is synchronized with other modalities such as video. Similar to zero-shot TTS, Voicebox simply perform the task by sampling target speech è given the context and concatenated frame-level phones cat(2,
Transient noise removal & content editing: When recording speech, one might misspeak a few words or the recording my be interrupted by unexpected background noise. In these scenarios it is desired to just edit the problematic segment instead re-recording the speech. Voicebox can perform transient noise removal through re-generating the noise corrupted segment given the original frame-level transcript and the surrounding clean audio.
For content editing, Voicebox first samples duration for the new phones given the edited phone transcript and the duration of existing phones to create the edited frame-level phone transcript. Given the new frame-level phone transeript and the audio for existing frames, Voicebox then samples the audio for frames corresponding to the new phones.
style shuffling: Voicebox can generate diverse speech samples by infilling the whole utterance. We first use the duration model to sample i given the phone transcript ý. We then use the audio model to sample è given & rep(ū. f). Similar to style transfer, Voicebox can also shuffle the audio style while keeping the alignment by sampling ø conditioning on the frame-level transcript of the target speech clip F.
Diverse speech sampling &
Metrics
The common goal of audio-conditioned tasks is to produce realistic speech that is coherent with the context and has the correct textual content. For tasks not conditioned on audio context, it is desired to generate diverse and realistic samples with distribution similar to training data. Prior studies often adopt subjective metrics like mean opinion scores (MOS) [2 which are not comparable across papers, or quantitative metrics like mel cepstral distortion [35 that assume the output is deterministic given input, which is often not realistic [54]. In this paper, advocate the following reproducible model-based perceptual metrics.
Correctness and intelligibility: We measure it by the word error rate (WER) of the synthesized speech's 's transcription with respect to the input text, which has been adopted prior work [62]. Public automatic speech recognition (ASR) models are used for comparability. For English-only setups,

---

follow [61] and use HuBERT-L 22 pre-trained on 60K hours of Librilight [26] and fine-tuned on 960 hours of Librispeech 43 For multilingual setups we use the Whisper large-v2 model 44
This is measured by the similarity between the embedding of generated speech and that of the audio context, where different embedding models would reflect coherence of different attributes. VALL-E proposed to use WayLM-TDCNN speaker embedding model, which maps an audio clip to a fixed dimensional vector, measure voice similarity. We consider the same model to compare with VALL-E. In particular, VALL-E reports similarity with respect to resynthesized audio context by its vocoder Sencodec-decoderi which we call SIM-resyn (SIM-r). SIM-resyn is not comparable across models using different vocoders. Hence, we advocate for computing similarity against the original audio context, which we call SIM-orig (SIM-o).
Coherence:
Fréchet Inception Score (FID) 17 is widely adopted for image generation evaluations, which captures the similarity between generated and real images the distribution level in some feature space. A shorter distance implies the distributions are more similar and generally reflects both higher sample quality and diversity. We adapt the metric for speech by using self- supervised wav2vec 2.0 feature (2) and refer to it as Fréchet Speech Distance (FSD). We verify its effectiveness in Appendix C.I along with alternative features.
Diversity and quality:
As supplementary metrics, we include quality MOS (QMOS) for subjective audio quality evaluation, and similarity MOS (SMOS) for subjective audio similarity evaluation given pairs of prompt and seste--gnneraed audio clips. Both of which are in the scale of I to 5 with 5 being the best. The MOS instructions and standalone metrics for duration models can be found in Appendix C.
5 Experiment
Data: We train the English-only model on 60K hours ASR-transeribed English audiobooks and the multilingual model on 50K hours of multilingual audiobooks from six languages: English (En), French (Fr), German (De). Spanish (Es), Polish (PI) and Portuguese (Pt). The two models are abbreviated as VB-En and VB-Multi. The Montreal Forced Aligner (MFA) [40] is used to phonemize and force align the transcript based on the MFA phone set. Word position postfixes are added. Audio represented as a 80-dimensional log Mel spectrogram and HiFi-GAN vocoder trained on the same 60K hours English speech is used to generate waveform. More details about phone representation, data transformation, and vocoder can be found in Appendix A1-A3.
Model: Transformer 59 with convolutional positional embedding (2 and symmetric bi-directional ALiBi self-attention bias [48] are used for both the audio and the duration model. ALiBi bias for the flow step I₁. is set to O. More details in Appendix Appendix [A.7] The audio model has 24 layers, 16 attention heads, 1024/4096
saabedding/feed-frwwaar network (FFN) dimension, 330M parameters. The duration model has 8 heads, 512/2048 embedding/FFi) dimensions, with 8/10 layers for English/mullilnguall setup (28M/34M parameters in total).
Training: TTB-En/VB-Mull audio models are trained for 500K/750K updates with an effective batch size of 240K frames. For training efficiency, audio length is capped at 1,600 frames and chunked randomly if length exceeds. Duration models are trained for 600K updates with an effective batch size of 60K frames. The Adam (33] optimizer is used with peak learning rate of Ie-4, linearly warmed up for 5K steps and decays over the rest of training. The audio/duration sequence is masked with parop = 0.3/0.2, and otherwise segment of 7% sequence length is masked, where r ~ (70, 100)/44 10, 100). Puncond is set to 0.2 for audio/duration models.
The torchdiffeq 9 package is used, which implements both fixed and adaptive step ODE solvers. By default, the midpoint solver is used with a step size of 0.0625 (NFE=32). The regression duration model is used by default. Silence at both ends are trimmed to 0.1 second max.
Inference:
Baselines: We consider three baselines: 1) VALL-E [6]. SOTA for English zero-shot TTS trained on Librilight. 2) YouTTTS [7]. SOTA multilingual zero-shot TTS model trained on VCTK, LibriTTS, PTS-Portugese R and M-AILABS French. It is flow-based model adapted from VITS [32 using a pre-trained speaker embedder. 3) A3T 3. SOTA for NAR speech editing and infilling trained with a regression loss on VCTK. We also consider Demucs [1] a SOTA speech enhancement model trained with regression and adversarial losses for denoising experiments.
7

---

5.1 Monolingual and cross-lingual zero-shot TTS
Table 2 presents the zero-shot TTS results of the English model VB-En. Following 61], the test set is constructed by selecting 4 to 10 second long samples from Librispeech test-clean. We consider cross-sentence prompting where 3 second clip from another sample of the same speaker used as audio context, and continuation prompting where the first 3 seconds of each utterance is used. Voicebox outperforms all baselines on all metrics both cases. particular, Voicebox transfers style much more effectively (+0. 100./01440.11 108 SIM-r on cros--sentencc//onnnuuaonnn than VALL-E, and the gap is even bigger when compared against raw audio (+0.141 SIM-o on continuation). MOS studies also confirm the quality and similarity of Voicebox are subjectively better than YourTTS.
Table 2: English zero-shot TTS results filtered L. test-clean. "obtained via personal communication.
Table 3: Transient noise removal where noise overlaps with 50% of the speech at a -10BB SNR.
Model
WER
SIM-o
SIM-r
QMOS
SMOS
Ground truth
0.754
3.98
a.0
CRAS--sentece
Model
WER
SIM-o
QMOS
A3T
63.3
0.046
0.146
YOuTTTS
0.337
3:272003
3.19
Clean speech
2.2
0.687
2.07
VALL-E
5,9
0,580
Noisy speech
41.2
0,287
2.50
VB-Ea
1.9
0.662
0.681
a.78
3.71aan
Demues
32,5
0.368
2,86+017
contimuation
A3T
11.5
0.148
3.10
A3T
18.7
0.058
0.144
VB-En (a - 0.7)
2.0
0.612
3,87
VALL-E
0.452
0.508
VB-Ea 0.7)
2.0
0.593
0.616
Table 4 presents cross-lingual zero-shot TTS results, where the audio context and the target text are in different languages. Note that VB-Multi is not trained on any sample with multiple languages in an utterance spoken by the same speaker. The test set is constructed using filtered MLS test split described in Appendix A.4. For each target text, we sample one 3-second long audio context from each language, which creates 36 language transfer directions in total. Voicebox yields better performance than YouTTTS everywhere. Specifically, on En/Fr/Pt which YourTTS supports, Voicebox obtains 3.1% 1%%55%5 1% lower WERs and 0. 13630.0 116.0.14//.... 141/0.160 higher similarity averaged across audio context in six languages. Addition studies on prompt lengths are presented Appendix B.2
Table 4: Multilingual zero-shot TTS results on filtered MLS test sets. GT/YT/VB-Mult refers to ground
Voicebox. "Ref" column shows the audio context language.
Ref
De
En
Es
Fr
Pi
Pt SIM-o
WER
SIM-o
WER
SIM-o
WER
SIM-o
WER
SIM-o
SIM-o
WER
GT
5.9
0.725
5.0
0.636
4.1
0.729
5.2
0.714
4.9
0.743
5,8
0.725
De
1//a
n/a
7.3
0.373
t/a
11.3
0.361
n//a
13.7
0.263
n/a
n/a
Es
n/a
mia
7.6
0.327
n/a
n/a
11.6
0.316
m/a
n/a
13.5
0.256
YT
Fr
11/a
n/a
7.6
0.363
0//
10.7
0.459
13.1
0.299
n/a
n/a
7.8
Pt
n/A
7.6
0.322
13/3
1/3
11.8
0.297
a/a
n/A
13.6
0.436
AVG
n/a
n/a
7.5
0.356
n/a
n/a
11.4
0.350
1//
n/a
13.9
0.299
4.8
En
5.9
0.435
4.2
0.535
4.1
0.423
6.8
0.423
0.402
7.6
0.385
VB-Mult
Es
4.9
0.460
4.3
0.479
3.6
0.613
5.3
0.473
5.2
0.436
5.4
0.435
(o 1.0)
0,476
4.7
0.491
3.8
0,503
3.5
0.528
5.1
0.503
0.641
4.9
0.476
Pt
4.9
0.422
4.6
0.426
3.7
0.476
5.5
0.453
4.8
0.406
5.2
0.620
5.0
0.486
4.4
0.492
0.459
5.2 Transient noise removal
We construct noisy test set by mixing the filtered Librispeech test-clean from Section 5.1 with non-speech noise such that it overlaps with 50% of the duration at -10dB signal-to- ratio. Additional conditions can be found in Appendix B.4 B.4. TableB]prssents the results comparing Voicebox with A3T and Demucs. It should the noise while Demucs does not. Compared to the baselines, Voicebox generates samples best on all metrics. A3T better than Demucs on intelligibilty and quality, but the infilled speech is not coherent because it is only trained on VCTK.
noted that A3T and Voicebox utilize transcript and location of
8

---

5.3 Diverse speech sampling and application to ASR data generation
Table 5 compares the ability to generate diverse samples for Librispeech test-other text. We con- sider English Voicebox (VB-En) with regression (regr) or flow fow-matching (FM) duration models. VITS-VCTK additionally conditions on a speaker ID, which we randomly sample for each sen- tence. YourT'S conditions text and reference audio, which draw from the LS train splits. Qualitatively, A3T generates the same robotic voice and VITS-LJ generates high quality but from single voice, hence both yield high FSD (bad quality diversity) but VITS-LJ has low WER. VITS-VCTK improves the voice diversity and FSD and You further advances it as it is trained on more speakers. Voicebox models (with different duration samplers) outperform the baseline on FSD by large margins, showing Voicebox's 's ability to produce realistic and diverse samples whose distribution is close the training data. Among them, the FM duration model creates more varying speaking styles compared to the regression one which ASR may struggle more to recognize.
Table 5: Diverse speech generation from LS
Table 6: Performance of ASR models trained on real or synthetic speech, tested on real speech and decoded with without 4-gram language model.
test-other text.
Model
WER
FSD
Ground truth
4.3
171.1
WER real data LM
4-gram LM
require additional inpur
ASR training data
test-c
test-c
Iost-o
VITS-VCTK VCTK
10.6
306.6
YouTTSS (ref=LS train)
9.0
277.9
Real audio (960hr)
Real audio (100hr)
2.6
21.5
6.3
2.2
16.2
5.0
fext-only
VITS-
81.2
78.1
A3T
37.9
373.0
VITS VCTK
31.8
55.5
30.2
53.1
VITS-LJ
5.6
344.2
YourTTS (ref=LS train)
25.0
54.6
20.4
51.2
En (a dur=regr)
155.7
VB-En (a O. dur-egr)
7.1
17.6
6.5
14.6
VB-Ea (a dur=FM, Oxter
5.6
159.8
VB-En (a 0, dur 0)
3.1
8.3
2.6
6.7
We next train an ASR model using only synthetic speech and evaluate it on real speech, which has not been successful before because synthetic data were not realistic and representative enough. @ real and synthetic data from Voicebox and three baseline models. Each TTS model generates one sample per text from the Librispeech training set, resulting 281K utterances per system. For real data, we consider train-960 and train-clean-I06 Details about the ASR model and training are in Appendix A.5, The results are highly correlated with the FSD scores for synthetic data. In particular, the ASR model trained on Voicebox data with FM duration model reduces WERs by over 85% compared to baselines, and only lags behind real data by 0.4% and 1.7% absolute.
5.4 Inference efficiency versus performance
We examine the trade-off between the metrics of interest (WER, SIM, FSD) for different settings of guidance strength (ox) and NFE specified by the user. Fig. shows the Voicebox inference time to generate an audio sample of 10 seconds (including vocoding and predicting duration) as NFE varies and compares that VALLEE For NFE-2 without CFG, Voicebox takes about 0.31 seconds, about 20 times faster than VALL-E. At NFE=64, Voicebox is only 4% slower than VALL-E.
Next, we study the cross-sentence setup of Section 5.1 to analyze the impact WER and SIM-r. We find that for all settings Voicebox has better WER than VALL-E. WER remains stable with mean of 2.0 and variance of 0.005. WER plot can be found in Appendix B.5. As shown in Fig, 33b in the case of SIM-r, lower classifier guidance strength values (a = 0 or 0.3) produce higher speaker similarity when operating in lower NFE regime (S 4). However, starting from NFE=8, a higher classifier guidance strength improves speaker similarity. Finally, in Fig. we examine FSD by generating samples for Librispeech test-other text. We find that lower classifier guidance strength produces lower FSD scores and more diverse samples. Increasing the NFE for each setting improves FSD.
5.5 Ablation on generative modeling approaches
We compare three generative modeling approaches: the proposed flow fow-matching with the OT path (FM w/ OT), flow-matching with the variance preserving (VP) diffusion path (FM w/ diff), and score-
Pe "Re-implemented and confirmed with the authors that re-implementation is faster (6.2 vs 10 seconds).

---

Voicebox
a=0
a=0.3
a=0.7
VALL-E
0.675
Inference (1Ds
0.650
200
20.4X
SIM
0.625
0.600
180
0.575
160
2
2
2
2
2
2
NFE
NFE
NFE
(a) NEE inference time w/o CFG
(b) NEE vs Speaker Similarity
(c) NFE FSD
Figure 3: Trade-off between NFE and different metrics. Inference time will be doubled with CFG.
Table 7: Comparing different objectives training efficiency. 32 NFEs are used for inference. Each
model is evaluated on the monolingual zero-shot TTS task.
UPd=50K
upd=100K
upd=150K
Method
WER
Sim-o
WER
Sim-o
WER
Sim-o
FM OT (proposed)
2.5
0.424
0.487
2.1
0,508
FM
76.0
0.066
0.344
0.478
SM w/
73.3
0.062
17.4
0.176
5.1
0.349
Table 8: Comparing different objectives inference efficiency. All models trained for 150K
updates. Each model is evaluated on the monolingual zero-shot TTS task.
NFE=4
NFE=8
NFF=16
NFF=32
Method
WER
Sim-o
WER
Sim-o
WER
Sim-o
WER
Sim-o
FM w/ OT (proposed)
2.4
0.410
2.2
0.481
2.2
0.503
2.1
0.508
w/ diff
11.5
0.171
0.359
2.7
0.447
2.6
0.478
SM w/ diff
94.5
0.054
42.3
0.076
11.5
0.218
5.1
0.349
matching with the VP diffusion path (SM diff). reduced setup described in B.3 adopted, with a lowered learning rate (le-4) and the loss in Eq. (I) to ensure convergence for all three objectives.
We vary the number of training and inference steps, and evaluate models the zero-shot TTS task (Section J.JI). Results Table shows that FM w/ OT trains significantly faster than the other two objectives, achieving the best performance with 100K training steps, and outperforms SM w/ diff using only 50K updates. Results in Table shows superior inference efficiency of FM w/ OT, which can produce good results with just 8 NFEs, while FM w/ diff requires least NFEs and SM w/ diff requires over 32 NFEs. Complete results are in Table
Conclusion
This paper presents Voicebox, the most versatile generative model for speech. By learning to solve text-guided speech infilling task on large scale multilingual datasets with a power model and training objective Voicebox demonstrates impressive task generalization capabilities. Voicebox achieves sate-of--theatt performance on mono and cross longull zero-shot TTS, speech inpainting, and diverse speech sampling, and can generate speech 20 times faster than the best autoregressive models.
With high fidelity speech generation models like Voicebox, it brings the potential of misuse and unintended harm. To mitigate the risk, we also detail in Appendix B. that highly effective classifier can be built distinguish between authentic and synthetic speech. Voicebox now trained only on read speech from audiobooks in six languages, and cannot transfer one attributes (e.g.. emotion) from a reference while transferring another attribute (e.g., voice) from another reference. Due to space limit, we expand our discussion of limitation and broader impact in Appendix D) For future work, we would continue scaling the model and the data to include more languages and diverse types of speech such conversations, and explore disentangled prompting for different attributes.
10

---

Acknowledgment
The authors would like to thank Kristin Lauter and Joelle Pineau for supporting the project, thank Ricky Chen, Yaron Lipman, Alexandre Defossez, Gabriel Synnaeve for the technical discussion, thank Jade Copet and Gabriel Synnaeve for the compute support, thank Eleonora Presani and Jackie Pan for discussing the responsible AI studies, thank William Ngan, Somya Jain, Lydia Baillergeau, Dana Beaty, Chantal Mora, Daniel Duncan, Gopika Jhala, Steph Miles, Josh Terry. Valeryia Aranovich, Ashton Evans, Aly Gill, Andrea Mileskiewicz. Emily Richards, and Aaron Vasquez for developing the visual assets and the website, thank Alyssa Newcomb and Oliver Libaw for developing the posts, thank Peter Gray, Natalic Hereth, Shauna Kelleher, Ashley Gabriel, Seine Kim, Ana Paula Kirschner Moffarej and Aiman Farooq for coordinating the launch, thank Harrison Rudolph, Mallika Malhotra, Carolyn Krol, Lauren Cohen and Mo Metanat for the reviews, and thank Alexandra Gualdino, Ana Paula Kirschner Mofarrej, Benjamin Muller, Chloe Rolland, Daniella Kalfa, Darcie Da Silva, Gabriel Synnaeve, Hunter Goldman, Juan Pino, Karen Ulrich, Kris Sekula, Manuel Ribeiro, Marina Zannoli, Mary Williamson, Rashel Moritz, Stephanie Castillo, Tu Anh Nguyen, Vlad Sobal, Volker Seeker, and anonymous volunteers for sharing speech samples for the demo.
References
[1] A. Aghajanyan, L. Yu, A. Conneau, W.-N. Hsu, K. Hambardzumyan,, S. Zhang. Roller, N. Goyal, O. Levy, and I.. Zettlemoyer. Scaling laws for generative mixed-modal language models. ArXiv, abs/2301 03728, 2023.
[2] A. Baevski, Y. Zhou, A. Mohamed, and M. Auli. wav2vec 2.0: A framework for sule-supe learning of speech representations. Advances neural information processing systems, 2020.
[3] H. Bai, R. Zheng, J. Chen, X. Li, M. Ma, and L. Huang. A3T: Alignment- acoustic and text pretraining for speech synthesis and editing. In International Conference on Machine Learning, 2022.
[4] Z. Borsos, R. Marinier, D. Vincent, E. Kharitonov, O. Pietquin, M. Sharifi, O. Teboul, D. Grang- ier, M. Tagliasacchi, and N. Zeghidour. AudioLM: a language modeling approach to audio generation. ArXiv, abs/2209.03 143,,, 2022.
[5] Z. Borsos, M. Sharifi, and M. Tagliasacchi. SpeechPainter: Text-conditioned speech inpainting. In Interspeech, 2022.
[6] T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. J. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray,
B. Chess, Clark, C. Berner, S. McCandlish, A. Radford. I. Sutskever, and D. Amodei. Language models are few-shot learners. ArXiv, abs/2005, 0t0o/2005511115,,, 2020.
[7] R. Casanova, J. Weber, C. D. Shulby, A. Júnior, F. Gölge, and M. A. Ponti. YouTTSS: Towards zero-shot multi-speaker tts and zero-shot voice conversion for everyone. In International Conference on Machine Learning, 2021.
[8] E. Casanova, A. C. Junior, C. Shulby, F. S. d. Oliveira, J. P. Teixeira, M. A. Ponti, and S. Aluísio. Tts-portuguese corpus: corpus for speech synthesis in brazilian portuguese. Language 2022.
Resources and Evaluation,
[9] R. T. Q. Chen. torchdiffeq. 2018. htl.https::/// hattta///ggsubb...cc//r/c/////dr
[10] R. T. Q. Chen, Y. Rubanova, J. Bettencourt, and D. K. Duvenaud. Neural ordinary differential equations. In Neural Information Processing Systems, 2018.
[11] A. Défossez, G. Synnaeve, and Y. Adi. Real time speech enhancement in the waveform domain. ArXiv, abs/2006. 12847, 2020.
[12) A. Défossez, Copet, G. Synnaeve, and Adi. High fidelity neural audio compression, ArXiv, abs/2210.13438, 2022.
11

---

[13] B. Desplanques, J. Thienpondt, and K. Demuynck. ECAPA- TDNN: Emphasized Channel Attention, propagation and aggregation in TDNN based speaker verification. Interspeech, 2020.
[14] Dhariwal and A. Nichol. Diffusion models beat GANs on image synthesis. Advances in Neural Information Processing Systems, 2021.
[15] Godfrey, E. C. Holliman, and McDaniel. Switchboard: Telephone speech corpus for research and development. In Acoustics, Speech, and Signal Processing, IEEE International Conference on, volume I, pages 517-520. IEEE Computer Society, 1992.
[16] A. Gulati, J. Qin, C.-C. Chiu, N. Parmar, Y. Zhang, J. Yu, W. Han, S. Wang, Z. Zhang, Y. Wu, al. Conformer: Convolution-augmentte transformer for speech recognition. arXiv preprint arXiv:2005.08) 2020.
[17] M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter. GANs trained by a two time-scale update rule converge to local Nash equilibrium. Advances in neural information processing systems, 2017.
[18] J. Ho and T. Salimans. Classifier-free diffusion guidance. arXiv preprint 2022.
[19] J. Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 2020.
[20] J. Hoffmmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. de Las Casas, I.. Hendricks, J. Welbl, A. Clark, Hennigan, E. Noland, Millican, G. van den Driessche, B. Damoc, A. Guy. S. Osindero, K. Simonyan, E. Elsen, J. W. Rae, O. Vinyals, and L. Sifre. Training compute-optimal large language models. ArXiv, abs/2203. 15556, 2022.
[21] W.-N. Hsu, Y. Zhang, R. J. Weiss, H. Zen, Y. Wu, Y. Wang. Y. Cao, Y. Jia, Z. Chen, J. Shen, al. Hierarchical generative modeling for controllable speech synthesis. In International Conference on Learning Representations, 2019.
[22] W.-N. Hsu, B. Bolte, Y.-H. H. Tsai, Lakhotia, R. Salakhutdinov, and A. Mohamed. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. JEEE/ACM Transactions on Audio, Speech, and Language Processing. 29:3451-3460 2021.
[23] W.-N. Hsu, T. Remez, B. Shi, J. Donley, and Y. Adi. Revise: Self-supervised speech resyn- thesis with visual input for universal and generalized speech enhancement. arXiv preprint 1rXiv:2212.11377 2022.
[24] R. Huang, M. W. Y. Lam, J. Wang, D. Su, D. Yu, Y. Ren, and Zhao. FastDiff: fast conditional diffusion model for high-quality speech synthesis. In International Joint Conference on Artificial Intelligence, 2022.
[25] Y. Jia, Y. Zhang. R. Weiss, Q. Wang, J. Shen, F. Ren, P. Nguyen, R. Pang. I. Lopez Moreno, Wu, et al. Transfer learning from speaker verification to multispeaker text-to-speech synthesis. Advances in neural information processing systems, 2018
[26] J. Kahn, M. Rivière, W. Zheng, Kharitonov, Q. Xu, P.-E. Mazar'e e, J. Karadayi, V. Liptchinsky, R. Collobert, C. Fuegen, T. Likhomanenko, G. Synnaeve, A. Joulin, A. rahman Mohamed, and E. Dupoux. Libri-Light: A benchmark for asr with limited or no supervision. International Conference Acoustics, Speech and Signal Processing, 2019.
[27] H. Kameoka, T. Kaneko, K. Tanaka, and Hojo. StarGAN-VC: non-parallel many-to-many voice conversion using star generative adversarial networks. IEEE Spoken Language Technology Workshop, 2018
[28] E. Kharitonov, A. Lee, A. Polyak, Y. Adi, J. Copet, K. Lakhotia, T. Nguyen, M. Rivière, A. rahman Mohamed, E. Dupoux, and W.-N. Hsu. Text-free prosody-aware generative spoken language modeling. In Annual Meeting of the Association for Computational Linguistics, 2021.
12

---

[29] E. Kharitonov, D.
Vincent, Z. Borsos, R. Marinier, S. Girgin, O. Pietquin, M. Sharifi, M. Tagliasacchi, and N. Zeghidour. Speak, read and prompt: High-fidelity text-to-speech with minimal supervision, 2023.
[30] K. Kilgour, M. Zuluaga, D. Roblek, and M. Sharifi. Fréchet audio distance: A reference-free metric for evaluating music enhancement algorithms. In Interspeech, 2019.
[31] J. Kim, S. Kim, J. Kong, and S. Yoon. Glow-TTS: A generative flow for cext-to-speech via monotonic alignment search. Advances in Neural Information Processing Systems, 2020.
[32] J. Kim, J. Kong, and J. Son. Conditional variational autoencoder with adversarial learning for end-to-end text-to-speech. International Conference on Machine Learning, 2021.
[33] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. CoRR, abs/1412.6980, 2014.
[34] F. Kreuk, A. Polyak, J. Copet, E. Kharitonov, T.-A. Nguyen, M. Rivière, W.-N. Hsu, A. Mo- hamed, E. Dupoux, and Y. Adi. Textless speech emotion conversion using decomposed and discrete representations. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, 2022.
[35] R. Kubichek. Mel-cepstral distance measure for objective speech quality assessment. In Pro- ceedings of IEEE pacific rim conference communications computers and signal processing, volume 1, pages 125-128 IEEE, 1993.
[36] K. Lakhotia, E. Kharitonov, W.-N. Hsu, Y. Adi, A. Polyak, B. Bolte, T. Nguyen, J. Copet, A. Baevski, A. B. Mohamed, and E. Dupoux. On generative spoken language modeling from raw audio. Transactions of the Association for Computational Linguistics, 9: 9:1336-1354, 2021.
[37] A. Lańcucki. Fastpitch: Parallel tex--to-speech with pitch prediction. In International Confer- ence on Acoustics, Speech and Signal Processing, 2021.
[38] Y. Lipman, R. T. Q. Chen, H. Ben-Hamu, M. Nickel, and M. Le. Flow matching for generative modeling. In International Conference on Learning Representations, 2023.
[39] J. Lorenzo-Trueha J. Yamagishi, T. Todaa, D. Saito, F. Villavicencio, T. H. Kinnunen, and Z. Ling. The voice conversion challenge 2018: 8: Promoting development of parallel and nonparallel methods. ArXiv, abs/l804.0422) 2018.
[40] M. McAuliffe, M. Socolof, Mihuc, M. Wagner, and M. Sonderegger. Montreal forced aligner: Trainable text-speech alignment using kaldi. In Interspeech, 2017.
[41] T. Nguyen, F. Kharitonov, J. Copet, Y. Adi, W.-N. Hsu, A. M. Elkahky, P. Tomasello, R. Al- gayres, B. Sagot, A. Mohamed, and F. Dupoux. Generative spoken dialogue language modeling. Transactions of the Association for Computational Linguistics, 11 11:250-2666 2022.
[42] A. Nichol, P. Dhariwal, A. Ramesh, P. Shyam, P. Mishkin, B. McGrew, I. Sutskever, and M. Chen. GLIDE: Towards photorealistic image generation and editing with text-guided diffusion models. In International Conference on Machine Learning, 2021.
[43] V. Panayotov, G. Chen, D. Povey, and S. Khudanpur. Librispeech: An asr corpus based on public domain audio books. International Conference on Acoustics, Speech and Signal Processing, 2015.
[44] D. Park, W. Chan, Y. Zhang, C.-C. Chiu, B. Zoph, E. D. Cubuk, and Q. V. Le. SpecAugment: A simple data augmentation method for automatic speech recognition. In Interspeech, 2019.
[45] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L.. Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 2019.
[46] V. Popov, I. Vovk, V. Gogoryan, Sadekova, and M. Kudinov. Grad-TTS: A diffusion probabilistic model for text-to-specch. In International Conference Machine Learning, 2021.
13

---

[47] D. Povey, A. Ghoshal, G. Boulianne, L. Burget, O. Glembek, N. Goel, M. Hannemann, Motlicek, Y. Qian, P. Schwarz, al. The kaldi speech recognition toolkit. In JEEE 2011 workshop automatic speech recognition and understanding, number CONF. IEEE Signal Processing Society, 2011.
[48] O. Press, N. A. Smith, and M. Lewis. Train short, test long: Attention with linear biases enables input length extrapolation. ArXiv, abs/2108. 12409, 2021.
[49] A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever. Robust speech recognition via large-scale weak supervision. ArXiv, abs/2212.04356, 2022.
[50] A. Ramesh, M. Pavlov, G. Goh, S. Gray, C. Voss, A. Radford, M. Chen, and Sutskever. Zero-shot text-to-image generation. ArXiv, abs/2102. 12092, 2021.
[51] Y. Ren, C. Hu, X. Tan, T. Qin, S. Zhao, Z. Zhao, and T.-Y. Liu. Fastspeech 2: Fast and high- quality end-to-end text to speech. In International Conference Learning Representations, 2021.
[52] F. Ribeiro, D. Florêncio, C. Zhang, and M. Seltzer. CrowdMOS: An approach for crowdsourcing mean opinion score studies. In International Conference Acoustics, Speech and Signal Processing, 2011.
[53] C. Robinson, N. Obin, and A. Roebel. soauenncctt--squenee modelling of FO for speech emotion conversion. In International Conference on Acoustics, Speech and Signal Processing, 2019.
[54] C. Saharia, W. Chan, H. Chang, C. Lee, Ho, T. Salimans, D. Fleet, and M. Norouzi. Palette: Image-to-image diffusion models. In ACM SIGGRAPH 2022 Conference Proceedings, 2022.
[55] J. Serrå, S. Pascual, J. Pons, R. O. Araz, and D. Scaini. Universal speech enhancement with score-based diffusion. ArXiv, 0ass22006000655. 2022.
[56] Shen, R. Pang, J. Weiss, M. Schuster, N. Jaitly, Z. Yang. Z. Chen, Y. Zhang, Y. Wang. R. J. Skerry-Ryan, R. A. Saurous, Y. Agiomyrgiannakis, and Y. Wu. Natural TTS synthesis by conditioning wavenet on mel spectrogram predictions. International Conference on Acoustics, Speech and Signal Processing, 2017.
[57] Y. Song and S. Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
[58] X. Tan, Chen, H. Liu, J. Cong, C. Zhang, Y. Liu, X. Wang, Y. Leng, Y. Yi, L. He, K. Soong, T. Qin, S. Zhao, and T.-Y. Liu. NaturalSpeech: End-to-end text to speech synthesis with human-level quality. ArXiv, abs/2205.044a0 .04201. 2022.
[59] A. Vaswani, N. M. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. ArXiv, abs/1706.00 00062, 2017.
[60] C. Wang, W.-N. Hsu, Y. Adi, A. Polyak, A. Lee, P.-J. Chen, J. Gu, and J. M. Pino. fairseq $2: scalable and integrable speech synthesis toolkit. In Conference on Empirical Methods in Natural Language Processing, 2021.
[61] C. Wang, S. Chen, Y. Wu, Z.-H. Zhang, L. Zhou, S. Liu, Z. Chen, Y. Liu, H. Wang. J. Li, L. He, S. Zhao, and F. Wei. Neural codec language models are zero-shor text to speech synthesizers. ArXiv, abs/2301.022 1..3111 11, 2023.
[62] Y. Wang, D. Stanton, Y. Zhang, R. J. Skerry-Ryan, E. Battenberg, J. Shor, Y. Xiao, Ren, Y. Jia, and R. A. Saurous. Style tokens: Unsupervised style modeling, control and transfer in end-to-end speech synthesis. In International Conference on Machine Learning, 2018
[63] Y. Xu, J. Du, L.-R. Dai, and C.-H. Lee. A regression approach to speech enhancement based on deep neural networks. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 23(1):7-19, 2014.
[64] Yamagishi, C. Veaux, and K. MacDonald. Cstr vetk corpus: English multi-speaker corpus for estr voice cloning toolkit (version 0.92). 2019.
14

---

[65] R. Yamamoto, E. Song, and J.-M. Kim. Parallel WaveGAN; A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogram. In International Conference Acoustics, Speech and Signal Processing, 2020.
[66] N. Yu, V. Skripniuk, S. Abdelnabi, and M. Fritz. Artificial fingerprinting for generative models: Rooting deepfake attribution in training data. In Proceedings of the IEEE/CVF International conference on computer vision, pages 11448-14457, 2021.
[67] N. Zeghidour, A. Luebs, A. Omran, J. Skoglund, and M. Tagliasacchi. Soundstream: An end-to-end neural audio codec. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 30:495-507, 2022.
15

---

A Additional Details of Experiment Setup
A.1 Vocoder
We adapt the HiFi-GAN V1 configuration to generate 16kHz audio from 80 dimensional log Mel spectral features sampled at 100Hz. To compute the log Mel spectrogram, we use a 1024-point short time Fourier transform with 640-sample (40ms) analysis window, 160-sample (10ms) shift, and the Hann windowing function to compute the amplitude spectrogram, and then apply an 80 dimension Mel filter with a cutoff frequency at 8kHz. The original HiFi-GAN VI has four transposed convolution blocks for upsampling. The upsampling factors are [8, 8, 2, 2] and the corresponding kemel sizes are [16, 16, 4, 4). Here we only need total upsampling factor of 160 instead of 256, and we adjust the upsampling factors to [5, 4, 4, 2] and kernel sizes to [11, 8, 8, 4] accordingly. The other parameters are identical to the HiFi-GAN VI configuration. Total number of parameters is 13M. We train the adapted HiFi-GAN on the 60K hours of English audiobook data for 1.5M updates on GPUs, which takes 7.5 days.
A.2 Phone representation
The frame-level phonetic transcript used for training is obtained through force- aligning speech and phonetic transcript. In particular, forced aligner may align some frames to special phone "SIL" for non-speech frames (silence or noise). For most forced aligners, only frames between words and frames at the beginning and at the end of an utterance can be aligned SIL.
Ghost silence
During inference, we are only given the text transcript, which does not tell us where we should insert silence to. Hence, it is desired to have the duration model not only predict the duration for each phone (SIL included), but also predict the existence SIL at eligible locations (between words and at the two ends of the utterance). To tackle it, introduce ghost silence to our phonetic transcript, which are silences in between words with duration of zero frames.
To give an example, suppose the transcript contains three words: "Hey what's up" with pronun- ciation "[Hey: [A, what As: [C], up: [D, F]]F" }", and the frame-level phonetic transcript obtained through forced alignment is = (SIL A B B SIL C D D D E E F SIL SIL). The phonetic transcripts becomes y = (SIL A SIL SIL E SIL), where the ghost silence is highlighted in green. The corresponding duration would be I = 11, 1, 1, 0, 2, 1,. A ghost silence is inserted between what a and up during training, and the duration model should predict the duration of it as zero to indicate that there should not be a pause between the two words.
Word- Worddpositionn-de peeetion-dependent phone The possible absence of silence between words in the frame- level phone transcript can make it hard for the audio model to identify word boundaries. To help the audio model identify the word boundary which is important when reading a sentence, we in- troduce pooedd-osiioon-deeedeee
phones which are commonly used in Hidden Markov Model based acoustic models for speech recognition [47]. This adds a postfix to each phone in the transcript denote where it is in the corresponding word. There are four postfixes: _B for be- ginning, _E for end, _I for intermediate, and _S for singleton. The above example becomes "[Hey: (A_B,B_BB E], what's ss [C_S], up: [D B, E_ I, [_E]]" with frame-level phonetic transcript (SIL A_ B_E B_E SIL C_S D_B D_B D_B E E F_E SIL SIL).
== (SIL
Phone-level mask In terms of masking, given duration I, the relationship of phone-level mask m' and frame-level mask m can be written as m rep(m'.1) For the applications where a duration model involved (zero-shot TTS, content editing, diverse speech sampling), the frame-level mask m is extended such that no phone is partially masked. In other words, all the frames corresponding to phone is either entirely masked or entirely unmasked. During training, we mask contiguous chunk of audio, infilling of which is more challenging task compared to infilling multiple smaller segments. All frames that are aligned to phone are either entirely masked or unmasked. Note that masking all frames for a phone is not a necessity but was chosen due to ease of implementation.
A.3 Data transformation
The Mel spectrogram is normalized with the global mean (-5.8843) and standard deviation (2.2615) to stabilize training. The statistics are estimated on 30k random training samples from the IK hours of
16

---

English audio. Input and output duration are dequantized (a ~ Ui 0.5, z + 0.5)) and transformed with log + x) following [5T]. Prediction of duration is quantized and clipped such that the minimal duration is greater than or equal to zero.
A.4 Cross-lingual zero-shot TTS test data filtering
We create a test set for each language by selecting samples from the MLS test split which have Whisper transcription WER lower than 20% (or 30% for Polish and Portugueses test splits which contains less than 1K samples), because we found MLS test set contains many examples with incomplete transcriptions missing large portion of the utterance. In addition, small amount of utterances were excluded due to MFA alignment failure. Table Al lists the number of samples remained for each language.
Table Al: Number of MLS test samples after filtering.
Language
#samples before filtering
#samples after filtering
English
3769
3535
Spanish
2385
2323
German
3394
3183
French
2426
2284
Polish
520
508
Portuguese
871
838
A.5 Setup for training ASR models with synthetic speech
To train ASR model in Section 5.3
extract 80-dimensional log Mel features with 25ms window and a 10ms frame shift, and then apply global mean-variance normalization. The ASR model an RNN-T with Conformer-basedd encoder [16]. The conformer applies time scale reduction to the input features with stride 6, embeds them into 512-dimensional vectors, passes these vectors through 20-layer conformer which has 8 attention heads and -i4--dimensional fully-connected layers. The conformer output is further mapped to 1024 dimensions through a linear layer followed by layer normalization before being passed the joiner. The predictor of the network first embeds wordpiece units into 512 dimensional embeddings, applies layer normalization, a 512-dimensional LSTM, a dropout layer and linear layer that maps the LSTM output to 1024 dimensions. The joiner adds the encoder and predictor outputs, applies tanh non-linearity and uses a linear layer that maps the 512-dimensional joiner input into wordpiece units. There are 4096 wordpiece units estimated from the LibriSpeech 960hr training text.
We apply SpecAugment 44 in all ASR runs. The models trained using PyTorch 45 with
Adam [33] optimizer for 120 epochs unless otherwise noted. The learning rate follows a tri-stage schedule with maximum of 0.001. We applied gradient clipping at 10 and a weight decay parameter 0. For the 960hr setting, we used a variable batch size capped at 1K utterances or 30K frames, whichever is smaller. This corresponds about 45K update steps for 120 epochs. For the 100hr setting, we set the maximum learning rate to 0.0001 and used smaller batch size (capped at 200 utterances or 5K frames). this case, 120 epochs corresponded about 120K updates. For decoding, we used n-best decoding with a beam-size of 15, and evaluated the WER on the 1-best path.
A.6 How to format input for different task
Fig. AT shows how input are formatted to perform diverse speech sampling, content editing, and zero zero-shot TTS using Voicebox.
A.7 Bi-directional ALiBi Bias
We use a symmetric variant bi-directional variant ALiBi bias where any query Qe and key Ky with jl = N use the same representations Furthermore, for any query Q, the bias corresponding
OOr implementation is similar to symmetric option
17

---

Diverse sampling
[a
10
Duration model
Audio model
phonemize
upsample
"Hello world"
[SIL
8IL
W X2R
SIL)
[SIL BE*3 D=1 SIL*10]
[0
Editing / infilling
E8
2
21
15
2
20
Forced-align Replace
Duration model
Audio model
upsample
'Hello world" "Hllo
[SIL*0
28
2
(audio of Hello world')
"world" segment removed,
insert frames of mask (predicted "lam"
Style transfer
1
Duration model
Audio model
Forced-align
upsample
"Hello world"
[SIL
BH
SIL
Ha
BY SIL OM
(SILA9 xy3 S*3 SIL*4]
18
(sudio of "Hello world") mask whose length the sum of predicted "Hey folks
"Hoy folks'
phonemize
Figure A1: Detailed diagrams of diverse speech sampling, content editing, and style transfer. Text in red and blocks in orange at the input of a model denote segments to be predicted. Numbers in blue and spectrogram in cyan at the model output denote predicted duration and spectrogram.
to the flow step 2, is set to 0. Similarly the bias from the flow-step Hę to any other query 0. In our experiments, find ALiBi Bias to improve convergence and extrapolation to longer sequences.
B
Additional Experiments
B.1 Detecting generated speech
We recognize the potential risks of model capable of generating speech in the style of arbitrary users. In an effort to diminish these risks we show that a binary classification model is able to consistently distinguish between real world speech and that which is generated from model.
Inspired by 29), we train convolutional binary classification model to distinguish between real and generated speech. The model consists of 6 blocks with hidden dimension sizes: [64, 128, 256, 256, 512, 512]. Fach block contains the frequeney axis, followed by ReLU activation and batch normalization. After each block that
convolution along the time axis, 3) convolution along
18

---

increases the hidden dimension size we also apply max pooling with stride of 2 across both the time and frequency dimensions. Finally, global max pooling applied and linear layer projects to a single value that is fed into a binary cross entropy loss. At inference time we create sliding window with hop length equal to 250ms and run each chunk of audio through the classifier and average the outputs.
The model is tested on the dev-clean split of Librispeech. We then take 100 hour subset of the 60K hour-Englin Sour-English data and set aside 2,703 random utterances (to match the size of dev-clean) which is used as a validation split. The remaining utterances from the 100 hours subset are used as the ground truth utterances for training. For each split we synthesize audio, conditioned on each utterance of the split by masking out frames in the spectrogram corresponding to 90%, 50%, and 30% of the phonemes of the utterance. All samples are generated using classifier-free guidance with w = 0.7, midpoint ODE solver (step size 0.0625/ / NFF=64), and the regression duration model.
We consider two detection tasks. The first one is to distinguish between original audio and Voicebox- generated audio. The second one is to distinguish resynthesized audio and Voicebox-generann audio. The resynthesized audio is created by extracting the Mel Spectrogram from original audio and then vocoding it with the HiFi-GAN vocoder.
Table B2 presents the results for each setting. The model can trivially distinguish original audio from
audio. This results from the fact that a model can also trivially distinguish original audio from resynthesized audio, most likely by recognizing artifacts produced by the vocoder. The task of differentiating bhicehox-generacee audio from resynthesized audio is much harder. When 90% of the audio is masked, the model is able to reliably classify the audio as Voicebox- -eeeerox-eeeerate.... In lower masking regimes this decreases bit, but this is likely due to naive inference method of averaging the outputs of all sliding windows. Since the majority of windows are non-synthetic, this leads to missclassifications.
Table B2: Synthetic speech detection metrics
% Mask
Accuracy
Precision
Recall
Original audio vs boicebox-generaae audio
30%
1.000
1.000
1.000
50%
1.000
1.000
1.000
90%
1.000
1.000
1.000
Resynthesized audio vs Voicebox-Be seteebox-geeeraeed audio
30%
0.704
0.714
0.680
50%
0.809
0.796
0.831
90%
0.907
0.881
0.942
B.2 How context length affects monolingual and cross-lingual zero-shot TTS
Monolingual: For in-context zero-shot TTS in Section 5.1 we used 3.0 seconds of prompt audio. Here we examine how WER/ SIM-r vary with different amounts of prompt audio using duration from regression duration model for the target text. If the desired prompt is longer than the available audio, the shorter audio is used as the prompt. Results are shown in Figure B2 As expected, WER mildly decreases and SIM-r grows quickly flattens with longer audio prompts. Comparing against VALL-E, Voicebox is more efficient at leveraging an audio prompt, achieving the same speaker similarity as VALL-E with roughly two thirds the input audio.
Cross-lingual: Here we examine the effect of increasing the prompt length for the case of cross- lingual zero-shot TTS. As described in [.1] this setting has total 36 language transfer directions for each pair of source and target language. For each target text a given transfer setting, we examine how WER / SIM-cF] vary as the prompt length increases. Similarly, the regression duration model is used for the target text. Fig. and Fig. [4|plot the SIM-o (speaker similarity) and WER trends respectively. When concatenating the prompt to the target for MLS, we find that the samples are quite bit longer than what the model was trained on (16s max length), because MLS test set samples are in average 15 seconds long. To alleviate this out of domain issue and focus the study
"Same trend is observed with SIM-r. We present SIM-o to be consistent with Table
19

---

0.6
1
WER
SIM-r
0.3
Seconds
Seconds
(a) WER
(b) Speaker Similarity
Figure B2: WER and SIM-r as function of prompt audio time in seconds for the Zero-shot TTS task
5.1 Audio is generated using classifier-free guidance strength (a) of 0.7 and midpoint ODE solver with NFE of 32. The blue line is for Voicebox and the red star is VALLE at seconds. The speaker similarity (SIM-r) remains same for longer prompts (up to 10s).
english
german
polish
portuguese
0.7
0.6
0.6
0.6
0.5
SI1--
SIM-o
0.5
0.4
0.3
0.3
10
Prompt (s)
Prompt (s)
Prompt (s)
(a) English
(b) French
(c) Spanish
0.7
0.7
0.6
SIM-0
0.5
SIM-o
0.5
SIM.o
0.5
0.4
0.4
0.3
0.3
0.3
2
10
2
Prompt
Prampt (s)
Prompt
(d) Portuguese
(e) German
(I) Polish
Figure B3: Each subplot considers one of the six target language and shows SIM-o (speaker similarity) function prompt audio duration seconds for ross-linguaa style transfer from different source language. We set the classifier-free guidance strength (a) to 1.0 and use midpoint ODE solver with
NFE of 32.
on varying the prompt length, we truncate the target sequences to 4 seconds (at word boundaries). We notice that WERs higher compared to Table 4 likely because the ASR model struggles with incomplete sentences. Each subplot contains the trend for one of the target languages from all six source languages.
The speaker similarity consistently improves as the prompt length is increased, similar to the mono- lingual setting. In contrast, we find that WER increases as we increase the prompt length for most directions. The WER increases much more for En non-En directions. is due to training data imbalance across languages, where English accounts for over 90% of the multilingual training data. Hence, when transferring from English, the model is more likely to assume that the whole sentence is in English as the prompt length increases and produce incorrect
hypothesize that this
20

---

engissh
french
german
polish
portugurse
spanish
40
40
8.5
8.0
30
30
WER
WER
7.5
20
20
7.0
10
6.5
10
2
8
10
2
4
10
8
10
Prampt (s)
Prompt (s)
Prompt (5)
(a) English
(b) French
(c) Spanish
40
30
20
10
2
10
Prompt (5)
Prompt
Prompt (s)
(d) Portuguese
(e) German
(I) Polish
Figure B4: Each subplot considers one of the six target language and shows WER function of prompt audio duration in seconds for cross-lingual style transfer from different source language. We find WER remain reasonably low for all cases except for "English" "X" style transfer. We set the classifier-free free guidance strength (a) to 1.0 and use midpoint ODE solver with a NFE of 32.
pronunciation for the non- mon-English target. Note that during the training phase, the model was only exposed to audio samples and phonemes originating from single language.
B.3 Comparing audio model training objectives
While A3T is considered the regression- based speech infilling baseline, it is trained smaller dataset and uses a smaller model compared to Voicebox. Here we present a controlled study comparing the flow-macching and regression objectives, as well the effectiveness of masked loss.
consider a reduced setup for this ablation to save the compute. All models were trained on an English audiobook dataset with IK hours of speech using smaller model configuration (12 layers, 1024-dimensional Transformer embedding. 2048- 1048-diimnssinal feed-forward layer, attention heads) for 150k steps with an effective batch size of 120k frames. These models evaluated the cross-sentence zero-shot TTS setup (Section S.T and diverse speech sampling (Section 5.3).
Results in Table B3 show that while regression audio models produce comparable WER, the audio similarity and diversity are significantly worse. Subjective listening also reveals that the audio quality and audio similarity are much worse. On the other hand, masked loss improves audio similarity and diversity while having little impact on intelligibility.
Table B3: Comparison of moa-man and regression models, trained with loss computed on all frames only masked frames. Results of the proposed objective is boldfaced.
Zero-Shot TTS (cross-sentence)
Diverse sampling WER
Method
Loss
WER
SIM-r
FSD
Flow Matching
Masked
2.1
0.597
3.1
242.5
Flow Matching
All
2.0
0.528
3.1
243.1
Regression
Masked
2.0
0.520
2.9
278.8
Regression
All
2.0
0.512
2.9
282.8
21

---

B.4 Effectiveness on data scaling
We create four subsets the 60K hour English data (0.1%, 1%, 10%, 100% duration), In particular. the 4% subset would contain roughly x% of the speakers from the original set. We train one model on each subset with reduced setup described Appendix [B.3 and evaluate them on zero-shot TTS (cross-sentence) and diverse sampling. Results show that scaling data constantly improves the zero-shot TTS performance (WER and SIM-r) as well as WER on diverse sampling. shows regression when scaling from 6K hour to 60K hour, but this could result from the the reference distribution is computed from the IK hour English audiobook data that has less diverse samples.
FSD
Table B4: Experiments on the effect of scaling training data.
Zero-Shot TTS
Diverse sampling WER
Train data (hr)
WER
SIM-r
FSD
60
2.30
0.151
3.48
280.48
600
2.11
0.417
3.19
205.39
6,000
2.08
0.573
2.96
195.52
60,000
2.05
0.645
2.95
214.38
B.5 Complete results comparing generative modeling approaches
Table presents the full results of Section 5.5 on all combinations of training steps, inference steps with results both monolingual zero-shot TTS (Section [.I and diverse speech sampling (Section 5.3) for the ablation study presented in Section 5.5 In all settings Flow Matching with OT paths performs strictly better than both of the other approaches.
B.6 Transient noise removal in more conditions
We expand the experiments in Section 5.2 comparing the models on two noise levels (low noise: 10dB and high noise: - 10dB), three overlapping ratios (30%, 50%, 70%), and also two types of noise (speech noise and non-speech noise).
Results are presented in Table B6 Voicebox consistently produces the most intelligible audio at all conditions (indicating the percentage of speech infill). In terms of audio similarity, Voicebox is constantly better in the high noise condition with gains ranging from 0.265 to 0.324 compared to Demucs, and is on par with Demucs in low noise condition.
B.7 Additional results on inference efficiency versus performance
As explained in Section 5.4 for the cross-sentence setup of Section 5.1 we find that WER remains stable with mean 2.0 and variance of 0.005. This can also be observed from Fig. B5a
Voicebox
a=0
d=0.3
a=0.7
VALL-E
3.1
WER
3.0
3
2
22
2
22
2
NFE
NFE
(a) NFE WER (Zero-shot TTS)
(b) NFE vs WER (Divers speech sampling)
Figure B5: Trade-off between NFE and WER for different classifier-free guidance strengths (a) presents the WER for cross-sentence zero-shot TTS (Section 5.1 and (b) presents the WER for diverse speech sampling (Section 5.3).
22

---

Table B5: Comparison of FM w/OT vs. FM w/Diffusion vs. SM.
ZS-TTS (cross-sentence)
Diverse sampling
Method
Train Steps
NFE
WER
SIM-o
SIM-r
WER
FSD
4
2.7
0.303
0.362
4.8
276,499
2.5
0.353
0,412
4.8
235.958
50000
16
2.4
0.366
0,425
4.7
227.485
32
2.5
0.364
0.424
225.931 4.7
2.5
0.347
0.404
4.3
258,358
FM w/ OT
8
2.2
0.411
0,468
4.2
216.512
100000
16
2.3
0.429
0.483
4.3
206.538
32
2.2
0.431
0.487
203.792 4.2
4
2.4
0.356
0.410
4.0
249.712
8
2.2
0,430
0.481
4.0
208.511
150000
16
2.2
0.453
0.503
4.0
198.040
32
2.1
0.458
0.508
3.9
195.304
4
99.9
0.050
0,050
99.8
3478.910
99.9
0.047
0.047
99.9
4704.237
50000
16
98.8
0.052
0.048
96.5
5336.591
32
76.0
0.060
0.066
49.5
2485.400
4
98.9
0.048
0.048
96.6
4486.401
FM w/ diff
14.6
0.104
0.137
12.0
669.564
100000
16
4.0
0.210
0.262
7.0
381.891
32
3.1
0.285
0.344
6.3
294.777
11.5
0.132
0.171
11.4
692.560
8
3.0
0.305
0.359
5.6
334.237
150000
16
2.7
0.391
0.447
5.4
244.067
32
2.6
0.423
0,478
5.2
224.963
4
99.6
0.050
0.048
99.7
2816.083
8
99.3
0.051
0.048
99.6
3079.040
50000
16
97.5
0.052
0.050
98.4
3710.340
32
73.3
0.057
0.062
86.2
3011.030
4
99.4
0.050
0.050
99.3
3474.579
SM w/ diff
97.2
0.049
0.048
97.9
3600.423
100000
16
53.9
0.064
0.071
69.6
2060.892
32
17.4
0.150
0.176
34.4
1071.579
94.5
0.055
0.054
79.4
2953.417
8
42.3
0.070
0.076
27.5
1071.010
150000
16
11.5
0.191
0.218
12.8
698.411
32
5.1
0.309
0.349
8.8
519.468
In Fig. B5bb we show the WER for the samples generated on Librispeech test-other text. We find that for a O, WER increases slightly from 2.8 to 3.1 as NEE goes from to 32. For a larger classifier-fres free guidance strength, WER remains more stable. Subjective listening and FSD reveal that 1) a lower NFE leads to less natural samples with lower diversity, and 2) higher guidance weight leads to lower diversity. In addition, observe that ASR perform well with unnatural samples that contains artifacts, but degrades when the samples are more diverse and expressive (e.g., that the combination of low
with whispering voice with strong reverberation). As result, guidance weight and high NFE leads to a higher WER due to the higher diversity.
B.8
Choice of audio model output features
The performance of our model upper bounded by how well the chosen acoustic features can be reconstructed to waveform. The reconstruction performance is determined jointly by the encoding
23

---

Table B6: Results of transient noise removal with varying overlapping percentage and noise level. "sp" means added noise is speech, and "non-sp" means non- speech.
WER(L)
SIM(T)
WER())
SIM(1)
sp
non-sp
sp
non-sp
sp
non-sp
sp
non-sp
SNR
1OdB, overlap=30%
SNR
10dB,
overlap=309
Noisy speech
26.7
24.9
0.202
0.238
3.7
3.1
0.605
0.603
Demues
20.5
19.7
0.247
0.247
3.2
2.8
0.570
0.567
A3T
7.5
0.058
same as left
VB-En (a: 0.7)
2.2
0.566
same as left
3NR=-10dD
overlap=50%
ANR=1OdB,
overlap=500 500%
Noisy speech
43.6
40.8
0.256
0.292
4.5
3.8
0.649
0.649
Demues
34.3
32.5
0.291
0.288
3.8
3.3
0.616
0.613
A3T
11.5
0.064
same as left
VB-En 0.7)
2.0
0.612
same as left
SNR /0dB, overlap=70%
SNR= 10dB, overlap=
Noisy speech
60.0
56.0
0.260
0.303
6.3
4.6
0.595
0.592
Demucs
49.5
45.4
0.293
0.294
4.6
3.8
0.572
0.564
A3T
16.6
0.063
same as left
VB-En (ar = 0.7)
2.0
0.559
same as left
process, in how much information is lost when encoding waveform into the features, and the decoding process, as in how well the vocoder can translate the encoded information into waveform.
To motivate the choice of the acoustic feature and the vocoder, we compare four combinations: the first one is Mel spectrogram + HiFi-GAN which is what this paper adopts. The second is Mel spectrogram Parallel WaveGAN 655 that is used by A3T [3]. The third one is Encodec post- quantization dense feature + Encodec decoder, which is analogous VALL-E's setup. The last one is also Encodec but with nre-quanizzation dense feature, which we include to study how much information lost during quantization.
We also note that Mel spectrogram features are 80 dimensional encoded at 100Hz, which is 8K dimensions per second, while Encodec features are 128 dimensional encoded 75Hz, which is 9.6K dimensions per second, higher than the Mel spectrogram features.
Table B7 presents the results evaluated on the Librispeech dev-clean and dev-other splits. All three models have the same WER resynthesizing dev-clean split, but Paralle/avveGAN degrades the most on dev-other. Interestingly Encodec produces audio of lower WER than the ground truth.
In terms of audio similarity, besides the default audio feature extractor WavLM-TDCND we also include results of similarity computed with another speaker encoder ECAPA [3.. Parallel WaveGAN consistently the worst. However, unclear whether HiFi-GAN or Encodec performs better. Encodec prevails with the WTVLM-TDCNN embedder and HiFi-GAN wins using ECAPA. It may require subjective MOS test to conclude which one reconstructs the audio better, and we leave exploration of modeling Encodec dense features for future study.
Table B7: Comparison different audio features and vocoders on audio reconstruction. Librispeech dev-clean (d-c) and dev-other (d-o) are used for evaluation. WER and audio similarity computed with WTVLM-TDCNN and ECAPA are reported.
WER
SIM-o (WavLM)
SIM-o (ECAPA)
Audio feature Vocoder
d-c
d-o
d-c
d-o
d-c
d-o
Ground truth
2.1
4.7
1.000
1.000
1.000
1.000
Mel spectrogram/ / HiFi-GAN
2.1
4.7
0.915
0,909
0.766
0.762
Mel spectrogram/ / Parallel WaveGAN Encodec post-quantized feature / Encodee decoder
2.1
5.2
0.868
0.847
0.721
0.711
2.1
4.5
0.943
0.944
0.724
0.722
Encodee pre-quantized fcature / Encodee decoder
2.1
4.4
0.943
0.944
0.724
0.722
24

---

C Additional Details and Studies on Metrics
C.1 Measuring speech diversity and quality with FSD
Diversity We first validate FSD reflects the diversity for a set of speech samples and study its sensitivity sample size. To achieve that, design controlled experiments compute FSD sets of samples with varying diversity and sample sizes. Specifically, we create two partitions from 1K hours of English speech, where each partition has the same of speakers and the same number of utterances for each speaker. The first partition is considered the reference set.
To test the sensitivity to sample size, we use the second partition to create subsets by sampling r% of utterances from each speaker in that partition. This sampling method is denoted as "utt". We computed that average, each speaker contributed approximately 2.33 sessions, with each session containing around 52.45 utterances. Therefore, the subsets created using the sampling method are expected to have similar audio style distributions the reference set and the FSD is expected stay low regardless of the subset size. We consider r E (1, 5, 10, 25, 50, 100}.
To test the correlation with diversity, we again use the second partition to create subsets by sampling r% of speakers and including all the utterances the partition from those speakers. This sampling method is denoted as "spk" where smaller y leads to a subset with fewer speakers and hence lower diversity. Therefore the FSD expected to increase as r decreases. The same set of values for considered. For the same r, the "utt" subset should always have a lower FSD than the "spk" subset.
We compare three different features for computing the FSD score. The first is the supervised WavI.M-TDDNN feature used for computing audio similarity (SIM-r and SIM-o). The second is the self-supervised wav2vee 2.0 BASE (2 feature reduced 128 dimensions using principle component analysis (PCA). The last one is the supervised audio event classification model feature that is used to compute FAD 30 for non-speech audio generation.
Figure C6 first compares using different layers of wav2vec 2.0 features. All of them yield similar desirable results where "utt" stays low and "spk" increases drastically when the sample size reduces and speaker diversity decreases. We then decide to use the middle layer (layer 6) as the default feature for FSD computation.
layer-2
layer-4
layer-6
250
250
200
200
150
150
150
Fréchet
100
100
50
50
10a
10a
10a
10a
10
10a
10a
10a
10a
layer-8
layer-10
layer-12
300
150
200
250
200
150
150
Frechat
100
100
50
50
25
ratio(%)
ratio(%)
ratio(%)
Figure C6: FSD based on different layers of wav2vec 2.0 BASE. utt: utterance-based sampling, spk: speaker-based sampling. Vertical bars denote standard deviation.
Figure further compares wav2vee 2.0-layer with the two other features. WTVLMMTTDNN and wav2vec 2.0-layer 6 present similar trends and both have low variance. Both of them are suitable for measuring diversity, and we decide to use wav2vee features is self-supervised and would be able to capture more holistic information of speech such as prosody and emotion.
25

---

In contrast, FAD score 30 is not appropriate for measuring speech diversity. The score does not increase much between r 25% and r 1% for "spk" sampling method. showing that the score does not reflect the decreasing speaker diversity. On the other hand, "utt" sampling method observes huge FAD score increase when reducing the sample size from r r= 25% to 1% where the diversity does not change much the number of speakers remains the same. Moreover, at 1% both sampling methods result in similar FAD score while the two subsets exhibit very different levels of diversity. We hypothesize that this is because FAD score is computed based on features extracted from an audio even classifier trained on AudioSet, which learns distinguish between events like lawn mower, engine, and human speech, but does not learn to capture the variation within speech, such as different voices.
WavLM-TDCNN
Wav2vec 2.0-layer
FAD
400
utt
Frechet Distance
spk
300
2
150
200
50
100
10
10
10
10a
10
10
10"
10A
10
(atio(%)
ratiors)
ratiol%)
Figure C7: FSD with different sample size using supervised WavLM-TDCNNN, self-supervised wav2vec 2.0, and supervised audio event classifier features. utt: utterance-based sampling, spk: speaker-based sampling. Vertical bars denote standard deviation.
Quality In addition measuring diversity, Fréchet distance is commonly used metric for assessing quality in image generation [19]. To show its applicability for speech generation, we evaluate the FSD score of speech utterances with varying levels of quality. The reference set samples are 1K hours of English training data, and the hypothesis set is the Librispeech test-clean split with noise added. We added Gaussian noise at different SNRs, ranging from to 50 dB. Lower SNR values correspond to lower quality. We use the default speech feature extractor (i.e., wav2vec 2.0, layer-6) throughout the experiments.
Our results, summarized in Figure C8 show that subset with a lower SNR has a higher FSD score. Therefore, lower FSD score indicates higher acoustic quality for the of test samples when diversity is fixed.
3500
3000
Fréchet Distance
2500
2000
1500
1000
500
10
30
40
50
SNR (dB)
Figure C8: FSD under different noisy levels. Feature: Wav2vec 2.0 layer-6 feature. Noise is added upon model output from Voicebox under unconditional setting.
26

---

C.2 Standalone metries for duration models
As mentioned in the main text, we can utilize end-to-en metrics of WER, SIM, and FSD to evaluate duration models, but also consider metries specifically for duration.
First, we consider two metrics aimed at the quality of duration predictions, here denoted i(leto: y)- For regression model, we use [(Ietur y) g(1ctar y; 0). For flow matching model, we set as the mean over 20 samples, ensuring a fairer comparison.
Duration correctness (MS-MAE) Our first metric, multi-sample mean-absolute error (MS-MARE the masked absolute error per-utterance divided by the average number of masked phonemes per-utterance
E Em 1|m C (1- i(leter (1-i(lete: 8) l11
(4)
Emn,y |m|la
Speaking rate correlation (MS-Corr) Our next metric, multi-sample correlation (MS-Corr), computes the average masked predicted duration and unmasked duration context per utterance, and computes their correlation across utterances. Comparing MS-Corr with the same correlation computed from the ground truth, we observe to what extent predicted durations capture appropriate correlations with the context.
Duration diversity and quality (FDD) Additionally, we evaluate the quality and diversity of duration samples at the distribution level. similar to our audio evaluation of diversity and quality via FSD. We produce one sample per utterance from duration model and collect all sampled phoneme durations, possibly many per-utterance, into an empirical distribution. We compare means and variances of this sampled distribution versus the means and variances of the training distribution, labeled μr, 8, and μ', respectively. We define the Fréchet duration distance (FDD) as the Fréchet distance between the distributions
(10-1()² 14)2² + 2v
-
(5)
treated though they Gaussians. FDD depends the sampled durations accurately reflecting the training distribution of real durations. As for FSD, this metric is specific to unconditional text-to-speech generation.
C.3 Duration model evaluation with standalone metrics
We evaluate three duration model variants. The first and second utilizes flow matching and re- gression, trained using masked conditional flow matching and regression respectively as described in Section 3.3 The third is a regression model that ignores duration context letz and only uses phonetic transcript y. referred to as unconditional regression below. This is the duration model used in FastSpeech2 [51]. A3T 3 and many other non-autoregresssvve speech synthesis models.
We evaluate our three duration model variants on the Librispeech test-other split on two tasks. The first is unconditional TTS where we generate all durations from given phonemes (i.e. letz is entirely masked). The second task is infilling the second half of each utterance's 's durations, where leta are durations from the unmasked half of the utterance. This second infilling task distinguishes between the two regression model variants, since the unconditional regression ignores letur, and hence predicts identical durations for the tasks. Duration metrics are computed for TTS and infilling Table C8 and C9 The prefix Phn or Sil indicates the associated metric was either computed across all non- silence or all silence phonemes. Start and end silences were not trimmed for these duration metric evaluations.
Starting with prediction quality metrics (MS-MAE and MS-Corr), the duration-cconditonaal regression performs slightly better on MS-MAE overall than the other models. Larger differences are seen on Thn-MS-Cor where the unconditional regression has a correlation substantively below the other models (Phn-MS-Cor) of ground truth is 0.47), indicating conditioning on duration context late beneficial. Flow-matching shows the largest distinction versus regression on the distributional comparison captured by FDD. The regression models have generally larger FDD because they underestimate the standard deviation in phoneme and silence durations, and hence produce samples with less duration diversity and more regular duration lengths.
27

---

Table C8: English TTS duration metrics LS test-other.
Duration Model
Phn-MS-MAE
Phn-FDD
Sil-MS-MAE
Sil-FDD
Unconditional Regression
2.53
0.72
5.32
2.39
Regression
2.52
0.76
5.10
8.40
Duration-conditionaa Flow Matching
2.63
0.61
5.18
2.48
Table C9: English second-half infilling duration metrics on LS test-other.
Duration Model
Phn-MS-MAE Phn-MS-Corr
Sil-MS-MAE
Unconditional Regression
2.57
0.26
5.44
Regression
2.45
0.35
5.20
Duration-conditional Flow Matching
2.52
0.41
5.32
C.4 Duration model evaluation with end-to-end metrics
We now present end-to-end metrics for our three duration variants for zero-shot TTS cross-sentence and continuation, as well as diverse speech generation, corresponding to Sections [5.1 and [5.3] Zero-shot TTS cross-sentence and continuation results are shown in Table CubleCCI]anne and diverse speech generation results Table CIO These results are comparable with the main text as they utilize the flow-matching model described in Appendix B.3 denoted as VB-En-IK.
Overall, FSD and SIM similar across duration variants. On the other hand, WER is sensitive the choice of duration model, where the duration-condittonall regression achieves a substantially lower WER. Subjective listening from the duration-condiitooaal regression and flow-matching confirms that the regression model is producing more regular patterns of speech, that may be easier for ASR to recognize, while sacrificing some duration diversity.
Table C10: Diverse speech generation from LS test-other text.
Duration Model with VB-En-1K
WER
FSD (LS-train)
Unconditional Regression
3.8
148.7
Regression
3.7
148.1
Duration-conditiona Flow Matching
5.4
155.1
C,5 MOS instructions
Table shows the instruction presented the raters for quality mean opinion score study. Table C13 shows the instruction presented to the raters for similarity mean opinion score study.
D Limitation and Broader Impact
Limitation Voicebox models presented in this paper are trained on read speech from audiobooks in to six written languages. Hence, the current models may not transfer well to conversational speech [15], which is more casual and contains more non-verbal sounds such as laughing and back- channeling (c.g., um-hmm). We plan to tackle the problem by scaling the training data to incorporate more diverse speech.
On the other hand, Voicebox depends phonemizer and forced aligner to produce frame-level phonetic transcript. In addition, many existing phonemizers (40) are word-based, which does not take neighboring words of the target into account when predicting the pronunciation. Such phonemizers cannot accurately predict phonetic transcript given text because pronunciation context-dependent in many languages (e.g., liaisons in French). In the future, we will explore more end-to-end methods where model would be able take raw text with punctuation as input (7]. and eliminate the need of phonemizers and forced aligners to improve the performance and increase the language coverage.
28

---

Table C11: English zero-shot TTS results on filtered LS test-clean.
Duration Model with VB-En-1K
WER
SIM-o
SIM-r
cross-senience
Unconditional Regression
3,0
0,538
0.584
Duration ceattionconnitionall Regression
2.7
0.545
0.591
Duration-condine conditional Flow Matching
3,4
0,528
0.578
contimuation
Unconditional Regression
2.5
0,485
0.524
Duration-condittonaa Regression
2.2
0,491
0.533
Duration-conditional Flow Matching
2.7
0.481
0.525
Table C12: Quality mean opinion score (QMOS) instruction.
Introduction
Your task is to evaluate the subjective quality and intelligibility of the speech from short (2-8 second) audio files. Each HIT can be completed in roughly around 120 seconds.
Task Instructions
In this task you will hear samples of speech recordings. The purpose of this test is to evaluate the quality and intelligibility of each file in terms of its overall sound quality and the amount of mumbling and unclear phrases in the recording.
Please keep in mind that speech samples can be distorted and noisy, however these are only specific examples.
Please headset for listening and adjust your volume level your comfort during this training, and do not change later during the experiment.
You should give a score according to the following scale, known as the MOS (mean opinion score) scales:
Score (Quality and Intelligibility of the speech)
5 (Excellent)
4 (Good)
3 (Fair)
(Poor)
(Bad)
Last but not least, while Voicebox yields impressive results on transferring audio style (voice, speaking style, emotion, and acoustic condition). the model does not allow independent control of each attribute. In other words, one cannot ask the model generate speech that resembles voice of one sample while resembling the emotion of another sample. We leave disentangled control of attributes through prompting or text description for future work.
Broader impact A high-quality and versatile generalist speech generation model like Voicebox can enable many applications that improve the quality our life. For example, zero-shot TTS could bring the voice back to people who suffer from discases or underwent surgeries such as laryngectomy the causes inability to speak. Zero-shot TTS can also be combined with visual speech recognition systems 23 to avoid the need of typing. When paired with speech translation models, tross-lingual zero-shot TTS enables everyone to speak any language their own voice. Content editing and speech denoising be productivity tools for to create content more effortlessly. Diverse speech sampling, as shown in the paper, can significantly reduces the cost of creating data for training speech-input models.
While Voicebox can bring many positive social impacts, it also carries the potential of misuse and unintended harm. To mitigate the risk, we have presented highly effective classifier showing that the
29

---

Table C13: Similarity mean opinion score (SMOS) instruction.
Task Name
Rate the similarity of the synthesized speech samples to a given prompt.
Task Instructions
Your task is to evaluate the similarity of the synthesized speech samples to the given speech prompt. You should focus on the similarity of the speaker, speaking style, acoustic conditions, background noise, etc. You should rank the recordings on the scale between 1-5, where is the best quality and is the worst.
In other words, please rank the recordings according to their acoustic similarity to the given prompt, meaning as if they were recorded in the same place by the same speaker speaking in similar styles. This task typically requires approximately 120 seconds to complete.
Please use a headset for listening and adjust your volume level your comfort during this training, and do not change later during the experiment.
model can accurately distinguish between real and synthetic speech. For future work, we also plan to investigate proactive methods for training the generative model such that the synthetic speech can be more easily detected, such as embedding artificial fingerprints [66] that can be trivially detected without hurting the speech quality.
To prevent Voicebox from learning biases, also need carefully select its training data. First, if Voicebox is only trained on a smaller number of speakers from a specific group with similar accents, it will not be able to generate diverse speech representing the accents around the globe, and downstream models trained on Voicebox generated speech would perform worse on groups with underrepresented accents. For zero-shot style transfer, the performance would also degrade for underrepresented accents, To mitigate this, we have leveraged in-the-wild speech that includes a wide variety of accents, and will continue investing collecting diverse speech to avoid such biases.
Second, Voicebox is trained on data where samples from one ethnic group always have lower audio quality (e.g., more noise) while the other ethnic group always has higher audio quality samples, the model would also learn undesired association. To mitigate this, we want the distribution of audio quality (and other audio attributes) and ethnic group to be less correlated, which is usually the case when we have larger scale data collected from in-the-wild sources. We can further tackle this by leveraging data augmentation decorrelate the distribution, such adding noise and enhancing speech to widen the audio quality distribution.
30


Reviews:
[{"review_id": "ubs3Zq8oDY", "reviewer": "Reviewer_F9C6", "summary": "The authors present Voicebox, the text-guided generative model for speech at scale. Voicebox is a non-autoregressive flow-matching model trained to infill speech, given audio context and text. Voicebox can be used for mono or cross-lingual zero-shot text-to-speech synthesis, noise removal, content editing, style conversion, and diverse sample generation.", "questions": "1. It could be more natural to use diffusion models since the masking strategy is similar to adding noise, so what is your consideration in using flow-matching models?\n2. Citations [42] and [43] are the same", "limitations": "/", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "It is interesting that Voicbox performs in-context learning via the masking strategy. Besides, the authors train the model in extensive data and present the SOTA results in several downstream tasks.", "weaknesses": "1. One of the weaknesses should be novelty. It sounds like the combination of masked-based A3T and flow-based speech synthesis model. Besides, duration prediction models and classifier-free guidance are not new.\n2. Evaluation. Firstly, why present a portion of MOS in Table 2? VITS-LJ and VITS-VCTK are compared as the baseline in the LibriTTS test-other set, but why not train in the LibriTTS dataset instead? The differences in data amount could encounter unfair comparison. \n3. Unclear presentation. Voicebox still needs to provide a clear illustration. Do you need tags for different task inferences? A3T finds that it is challenging to generate speech given full-mask samples, and what do you find is the most suitable proportion of masked and unmasked regions?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful feedback. We address common questions in Author Rebuttal above and other questions here. We will incorporate all feedback in the final version.\n\n**1. Do you need tags for different task inferences?**\n\n**No.** Voicebox performs different tasks by preparing input differently, which we illustrated in Figure 1. We have added refined illustrations in the supplementary PDF for the rebuttal (Figure 1-3).\n\n**2. Why present a portion of MOS in Table 2?**\n\n**VALL-E is not publicly available** and there are only 8 samples from their demo page, which are not sufficient for MOS studies. We did not present A3T MOS because **the performances of A3T on WER and SIM-o are very bad**, and the MOS scores are expected to be very bad after we listen to a number of samples.\n\n**3. A3T finds that it is challenging to generate speech given full-mask samples, and what do you find is the most suitable proportion of masked and unmasked regions?**\n\nA3T finds it challenging because it assumes the mapping between the input (text and audio context) and the output (target audio) is deterministic, as discussed in paragraph 3 Section 2 and the response to common comment 1 above. In particular, it struggles more as the duration to infill becomes longer (the distribution of possible speech becomes broader and less described by the mean targeted by regression). \n\nIn contrast, Voicebox addresses the issue by modeling with a CNF model, which models the full distribution over possible speech, and not just the mean like for a regression model. Hence, Voicebox is capable of infilling audio of any length.\n\nDuring training, we mask the entire audio with 30% probability, and with 70% probability masking a contiguous chunk that is [70%, 100%] the length of the entire audio. During inference, it depends on the task and the input. For zero-shot TTS, the masked length is the length of predicted duration of the target transcript. For denoising, we explore infilling 50% (Table 3) and 30%/70% (Table B4 in Appendix). For diverse speech sampling (Section 5.3), it is 100% masked.\n\n**4. Citations [42] and [43] are the same**\n\nWe will fix that. Thank you.\n"}, {"review_id": "h3PdXmBsH0", "reviewer": "Reviewer_K4NY", "summary": "This paper proposes VoiceBox, a speech infilling model based on a flow-matching generative model. VoiceBox is trained to fill in masked speech based on unmasked speech and given text, and it can perform various tasks depending on how the mask is applied to the speech during inference. VoiceBox allows for speech editing by masking the speech corresponding to the text portion that needs to be edited, and it can also perform zero-shot TTS by infilling the speech for the desired text. By masking the entire speech during generation, the model can generate voices from various speakers in a speaker-unconditional manner for a given sentence. This paper demonstrates significantly improved performance compared to existing models in various tasks and even shows that a generative model for speech can aid in performance improvement in speech recognition tasks through diverse speech generation.", "questions": "* Despite VALL-E already demonstrating impressive zero-shot TTS performance, VoiceBox appears to have excessively higher SIM-o and SIM-r scores. It is difficult to determine that VoiceBox's zero-shot TTS samples are significantly better than VALL-E when listening to only the VoiceBox samples in the demo. For a one-to-one comparison with VALL-E, would it be possible to provide VoiceBox samples generated with the same prompt as VALL-E's demo samples?\n\n* In order to calculate SIM-r as mentioned in the paper, were both the reference speech prompt and all generated samples encoded and decoded using Encodec? If calculated differently, please provide an explanation. Additionally, when measuring SIM-o or SIM-r with the reference speech, was the similarity measured with the speech prompt only or with the entire reference speech?\n\n* ", "limitations": "This paper provides additional experiments on a classifier designed to detect potential misuse in order to mitigate such risks.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "* This paper demonstrates the versatility of VoiceBox across various tasks while achieving impressive performance.\n* This paper demonstrates that utilizing synthesized speech generated by VoiceBox can improve the generalizability of ASR models, thereby showing its ability to effectively model the distribution of general speech.\n* In the zero-shot TTS task, VoiceBox particularly outperforms the previous state-of-the-art model, VALL-E, by a significant margin.\n* The use of a flow-matching generative model enables fast sampling.\n* The extensive experimental results provide strong support for the advantages of the model.", "weaknesses": "* For zero-shot TTS and speech editing, both the duration model and audio model require prompting with the transcript of the reference speech as well as the duration per phoneme. This necessitates the use of MFA (Montreal Forced Aligner) during inference.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful feedback. We address common questions in Author Rebuttal above and other questions here. We will incorporate all feedback in the final version.\n\n**1. In order to calculate SIM-r as mentioned in the paper, were both the reference speech prompt and all generated samples encoded and decoded using Encodec?  If calculated differently, please provide an explanation.**\n\n**No.** SIM-r aims to measure the similarity with respect to the audio feature target a generative model predicts. For Voicebox, we compute the mel spectrogram of the ground truth audio and decode the mel spectrogram into a waveform using HiFi-GAN to create the reference speech. We decode Voicebox output mel spectrogram into a waveform using HiFi-GAN to create the generated speech.\n\nSIM-r serves as the upper bound for the audio model given the selected audio feature (Encodec code for VALL-E / mel spectrogram for Voicebox). We noted in Section 4 that this number is not comparable when two models use different audio features or vocoders. Hence, we focus on SIM-orig and argue that SIM-orig should be used for comparison across papers.\n\n**2. when measuring SIM-o or SIM-r with the reference speech, was the similarity measured with the speech prompt only or with the entire reference speech?**\n\nWhen measuring SIM-{r,o}, the reference is the speech prompt. We confirmed this is the same protocol VALL-E used through personal communication with the authors.\n\n**3. Would it be possible to provide VoiceBox samples generated with the same prompt as VALL-E's demo samples?**\n\nThe rebuttal period does not allow uploading audio samples, and we would also need explicit consent from a speaker to put samples resembling that speaker online. We will include them in the final demo if we could obtain such consents. We also note that all the samples presented in the supplementary material use audio prompts from internally recruited volunteers, who recorded audio from their own devices. These audio would likely be more out-of-domain.\n\n**4. For zero-shot TTS and speech editing, both the duration model and audio model require prompting with the transcript of the reference speech as well as the duration per phoneme. This necessitates the use of MFA (Montreal Forced Aligner) during inference.**\n\nWe will add discussion on this in the main paper if space permits or in the appendix. We also note that for speech editing forced aligner is always needed during inference to identify the location of the source word(s) to be deleted/replaced, regardless of whether a forced aligner is used during training. In that regard the inference requirement of Voicebox is the same as prior work.\n\nTo further resolve the limitation, there are two potential solutions to address the limitation in the future work. **First, we could use a mix of phonetic and self-supervised learning (SSL) units (e.g., HuBERT units) as the content representation similar to [1].** Specifically, the frame-level phone units corresponding to audio context can be replaced with SSL units, such that transcript of the context is not needed and duration of the SSL units can be easily derived since SSL units are originally at the frame level. **Second, we could explore using a separate encoder for audio context**, which duration model and audio model attend to through cross attention.\n\n[1] Fong, Jason, et al. \"Speech Audio Corrector: using speech from non-target speakers for one-off correction of mispronunciations in grapheme-input text-to-speech.\" Interspeech’22\n"}, {"review_id": "0EVvNtXnDu", "reviewer": "Reviewer_pReF", "summary": "This paper proposed Voicebox, a non-autoregressive flow-matching model designed to infill speech by leveraging given audio context and text. Notably, Voicebox capitalizes on a substantial amount of data, consisting of 50,000 hours of speech, which contributes to its impressive performance across various speech generation tasks. The model demonstrates notable capabilities in generating coherent and high-quality speech outputs.", "questions": "It is unclear from the paper what motivated the authors to choose the flow-matching model as the audio model. ", "limitations": "Authors are encouraged to add some discussions about the potential negative social impact.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper exhibits a comprehensive and well-executed evaluation, yielding impressive experimental results.\n2. The authors have effectively presented their work with clear and accessible writing, ensuring ease of understanding for readers.\n3. The proposed framework showcases a high level of flexibility, enabling its application to various speech generation tasks and settings.", "weaknesses": "1. The demos provided in the supplementary materials are not as satisfying as described in the paper. For example, issues can be found in the perceived speaker similarity in the zero-tts task and the quality of the articulation position of the mask. It would be beneficial if the authors engage in further discussions regarding these phenomena.\n2. Given that speech encompasses various components (such as prosody, content, timbre, and noise), it would be better to have a more comprehensive discussion on how Voicebox specifically handles these different aspects. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful feedback. We address common questions in Author Rebuttal above and other questions here. We will incorporate all feedback in the final version.\n\n**1. The demos provided in the supplementary materials are not as satisfying as described in the paper. For example, issues can be found in the perceived speaker similarity in the zero-tts task and the quality of the articulation position of the mask. It would be beneficial if the authors engage in further discussions regarding these phenomena.**\n\nAll the samples presented in the supplementary material use audio prompts from internally recruited volunteers, who recorded samples from their own devices and provided consents for the processing and sharing samples online. Moreover, many of these speakers are non-native and with accents. These audio would be more out-of-domain relative to the training data, which may contribute to the perceptually lower similarity. VALL-E has observed similar results when tested on VCTK which is out-of-domain relative to their training data (Librivox). We will include audio samples for comparison with VALL-E if we can obtain consents for the prompts they used.\n\nIn terms of the quality of the articulation position of the mask, we are unsure which samples have the issue reviewers mention. We would be more than happy to discuss if the reviewer could provide more details!\n\n**2. Given that speech encompasses various components (such as prosody, content, timbre, and noise), it would be better to have a more comprehensive discussion on how Voicebox specifically handles these different aspects.**\n\nVoicebox decouples speech into textual content and audio style, where audio style encompasses everything other than textual content. To generate an audio sample, the textual content is specified by the phone transcript, and the audio style is specified through the surrounding audio (audio context).\n\nVoicebox does not differentiate different aspects of audio style (voice, noise, emotion, etc) and does not need such labels either. Through learning to infill from large quantities of data, Voicebox learns that these attributes tend to be consistent within an utterance and it can infer the audio style of the target given the context.\n\nWe can take the first “Transient Noise Removal” sample on the demo page as an example (transcript starts with “in zero weather in mid-winter…”). “Voicebox Input” audio shows the audio context input to the model. From it we hear a feminine voice speaking calmly, and there is static noise that is particularly noticeable when the speaker speaks. We can find the same audio style (low static noise, voice, slow pace, calm emotion) from the generated segments (“to a great depth below the surface when in driving over the”) presented in “Voicebox Output” audio.\n\n**3. Authors are encouraged to add some discussions about the potential negative social impact.**\n\nWe have included some discussion of potential negative social impact and mitigation in Section 6. We expanded the discussion in the thread above for “Ethics Review” and will incorporate those in the final version.\n"}, {"review_id": "pTCYUc4pFv", "reviewer": "Reviewer_Peh7", "summary": "The paper proposes voicebox, a text-guided generative model for speech at scale. By abstracting many speech tasks into speech infilling tasks, voicebox is able to conduct zero-shot text to-speech synthesis, noise removal, content editing, style conversion, and diverse sample generation in monolingual or crosslingual scenario. Experiments show that equipped with the advanced technology such as CNF and classifier-free guidance, voicebox show better performance than baseline methods on many tasks,", "questions": "1. As Table 5 shows, VB is able to generate speech with different styles. Can this feature be used in Table 6? To be more specific, if VB synthesize more than one sample per text from the Librispeech training set, can the corresponding WER be further reduced?\n\n2. If there is any insight for using Continuous Normalizing Flows to modeling speech infilling rather than other generative models such as diffusion models?", "limitations": "The authors have discussed the limitations in conclusion.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The paper abstracts many different speech tasks into speech infilling task.\n2. The paper scales up the model to 50K hours of speech and multilingual scenario, where the model is able to perform in-context learning for speech style.\n3. The proposed method is well-studied on many tasks including zero-shot TTS, speech enhancement, speech editing and style generation. The experiments on ASR also show the potential value of proposed method.", "weaknesses": "About the experiments in Section 5.2, it is unfair to compare VB with speech enhancement model since VB can have access to the text of noised speech, which significantly help the model to generate speech with lower WER. Also, the text of noised speech is usually not accessible in the conventional formulation of speech enhancement.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful feedback. We address common questions in Author Rebuttal above and other questions here. We will incorporate all feedback in the final version.\n\n**1. As Table 5 shows, VB is able to generate speech with different styles. Can this feature be used in Table 6? To be more specific, if VB synthesize more than one sample per text from the Librispeech training set, can the corresponding WER be further reduced?**\n\n**Yes.** We have conducted such experiments before using an earlier version of the model (trained on 1K hour, using a different forced aligner and word-position-independent phones, using regression duration model). We generated three copies of the Librispeech training set, trained 1 ASR model on each copy, and trained another model combining three copies. The WER of the first three ASR models are 5.34%/5.38%/5.54% on test-clean and 15.40%/15.79%/15.83% on test-other. The WER of the ASR is 4.43% and 12.37% when trained on the combination of the three copies. The results demonstrate the potential of using Voicebox for data augmentation and we plan to explore more in future work.\n\n**2. About the experiments in Section 5.2, it is unfair to compare VB with speech enhancement model since VB can have access to the text of noised speech, which significantly help the model to generate speech with lower WER. Also, the text of noised speech is usually not accessible in the conventional formulation of speech enhancement.**\n\nWe have noted in text that “It should be noted that A3T and Voicebox utilize transcript and location of the noise while Demucs does not.” We will revise this text to emphasize this difference more\n\nWe agree that this is an unconventional setup for speech enhancement where transcript is usually unavailable. That said, we believe it is still a worthy application as transcripts can still be available in some scenarios. For example, it can be used when one records a scripted speech or when a robust audio-visual speech recognition model is available, which can transcribe accurately even at a very low SNR.\n"}, {"review_id": "FCinqalWLx", "reviewer": "Reviewer_MQeA", "summary": "This work presents a text-conditional speech synthesis model with an optimal transport path and conditional flow matching. The model is trained with a large-scale cross-lingual dataset for zero-shot style transfer and content editing.", "questions": "1. The authors utilized a 100 Hz frame rate to extract the high-resolution Mel-spectrogram. For a fair comparison, each model is also trained with the same time resolution for Mel-spectrogram. For example, VITS and YourTTS should be trained with the linear spectrogram extracted at a 100 Hz frame rate for high-quality. However, the details are not described for baseline models.\n\n2. Are there any scenarios for automatic noise removal or editing? In this work, the location of audio for noise removal or editing should be segmented by the user. [5] presented the fully automated speech correction scenario with detection, correction, and generation. It would be nice incorporating ASR with this work for an automatic speech editing system. \n\n3. I think replacing the HiFi-GAN with BigVGAN could improve the audio quality. \n\n4. For the AR model, the dropout in the pre-net of the decoder may improve the diversity of speech. Also, VITS and YourTTS can increase the sample diversity by controlling the temperature T. The details you used should be described.", "limitations": "They stated the limitations and potential negative societal impact on Section Conclusion.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 1, "strengths": "The work adopts a conditional normalizing flow with an optimal transport path for speech synthesis. The proposed model can synthesize and edit the speech with different styles.", "weaknesses": "Although this work successfully executes the zero-shot TTS and speech editing, I have doubts on the originality of this work and I think that the authors should have conducted more comparisons with the recently proposed TTS models (not YourTTS, not Vall-E), and speech editing or speech correcting paper. \n\nI have some comments on this work.\n\n1. The concept to train the model by infilling speech given audio and text has been already presented in many works. First, SpecAugment [1] presented a time masking for context learning and Conformer [2] successfully adopted it for the ASR task. Self-supervised speech representation also models such as wave2vec 2.0 successfully learned the context with masking. Moreover, there are many similar models which can edit the speech conditional text information in speech editing [3] [4], correcting domains [5] [6], and also in the TTS domain [7] [8]. The authors only adopt the conditional normalizing flow with an optimal transport path for text-to-speech.  \n\n[1] Park, Daniel S., et al. \"Specaugment: A simple data augmentation method for automatic speech recognition.\" Interspeech, 2019.\n\n[2] Gulati, Anmol, et al. \"Conformer: Convolution-augmented transformer for speech recognition.\" Interspeech, 2020.\n\n[3] Tae, Jaesung, Hyeongju Kim, and Taesu Kim. \"EdiTTS: Score-based editing for controllable text-to-speech.\" Interspeech, 2022.\n\n[4] Wang, Tao, et al. \"Campnet: Context-aware mask prediction for end-to-end text-based speech editing.\" IEEE/ACM Transactions on Audio, Speech, and Language Processing 30 (2022): 2241-2254.\n\n[5] Tan, Daxin, et al. \"CorrectSpeech: A Fully Automated System for Speech Correction and Accent Reduction.\" ISCSLP, 2022.\n\n[6] Fong, Jason, et al. \"Speech Audio Corrector: using speech from non-target speakers for one-off correction of mispronunciations in grapheme-input text-to-speech.\" Proc. Interspeech 2022 (2022): 1213-1217.\n\n[7] Ao, Junyi, et al. \"Speecht5: Unified-modal encoder-decoder pre-training for spoken language processing.\" ACL, 2022.\n\n[8] Wang, Tao, et al. \"Non-Autoregressive End-to-End TTS with Coarse-to-Fine Decoding.\" INTERSPEECH. 2020.\n\n2. Specifically, the diffusion-based model (EdiTTS [3] and Guided-TTS [9]) can edit the speech even without training. The author should have compared the conditional normalizing flow with a diffusion-based model. However, the authors only referred that OT path leads to faster training, faster generation, and better performance compared to diffusion paths. I wonder if the conditional normalizing flow with OT path is better than diffusion-based models.       \n\n[9] Kim, Heeseung, Sungwon Kim, and Sungroh Yoon. \"Guided-tts: A diffusion model for text-to-speech via classifier guidance.\" ICML, 2022.\n\n3. The audio quality is not good on the demo pages.\n\n4. For a fair comparison, all models should be trained with the same dataset.\n\n5. YourTTS is not a good text-to-speech model for zero-shot text-to-speech models. The audio quality of YourTTS is bad. I recommend training the VITS with the reference encoder you used and the same configuration. In addition, transferring the voice style from a reference audio, not speaker ID, is utilized in many works. For a fair comparison, the same transferring method is used for each model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful feedback. We address common questions in Author Rebuttal above and other questions here. We will incorporate all feedback in the final version.\n\n**1. The concept to train the model by infilling speech given audio and text has been already presented in many works. SpecAugment [1] presented a time masking for context learning and Conformer [2] successfully adopted it for the ASR task. Self-supervised speech representation also models such as wave2vec 2.0 successfully learned the context with masking.**\n\nWe would like to emphasize that this work focuses on building a **generalist speech generation model**. In contrast, **SpecAugment, Conformer, and wav2vec 2.0 are ASR/representation learning models that do not infill speech and cannot generate speech. They should not be considered related work.** [1] and [2] use masking for regularization. wav2vec 2.0 is similar to CPC which masks to create different views for contrastive learning.\n\n**2. there are many similar models which can edit the speech conditional text information in speech editing [3] [4], correcting domains [5] [6], and also in the TTS domain [7] [8]. The diffusion-based model (EdiTTS [3] and Guided-TTS [9]) can edit the speech even without training**\n\n**CampNet [4] is very similar to A3T, which we have already compared with.** Both assume a deterministic input/output mapping, preventing speech infilling to generalize to longer spans. The key differences are [4] does not require alignment during training, and uses a two-stage NAR decoder to refine spectrogram.\n\n**[3-9] are all very different from Voicebox and cannot be compared in the same setup.** [3] and [6] require an audio sample containing the new text to be swapped in for editing. To edit “I’m happy” into “I’m heavy”, [3] requires an audio sample of the same speaker saying “heavy”, while [6] requires a sample of “heavy” that can be from a different speaker. [5] presents an automatic way to align source text with target text and reuse [4] for editing, which is complementary but orthogonal to Voicebox.\n\nSpeechT5 [7] is an unsupervised pre-training framework, which requires separate fine-tuning to perform each task, and cannot edit speech or perform in-context style transfer. [8] is a regression-based NAR TTS model evaluated on a 20-hour single speaker dataset and does not edit speech. It will suffer similar issues as A3T when trained on diverse data. [9] did not show that it is capable of editing..\n\n**3. each model trained with the same time resolution for Mel-spectrogram. For example, VITS and YourTTS should be trained with the linear spectrogram extracted at a 100 Hz frame rate for high-quality.**\n\n**What the reviewer suggests is similar to Glow-TTS+ HiFi-GAN. We believe VITS serves as a stronger baseline for flow-based TTS,** given VITS has shown that it is better (Table 1 and 3 in VITS). The frame rate of Glow-TTS is 22K / 256 = 86Hz which is close to the 100Hz we used. Moreover, higher time resolution does not necessarily mean better quality, it depends on if a model has enough model capacity.\n\n**4. For the AR model, the dropout in the pre-net of the decoder may improve the diversity of speech. Also, VITS and YourTTS can increase the sample diversity by controlling the temperature T.**\n\n**Pre-net dropout, which serves for regularization, can lead to stochastic behavior but the output diversity is limited.** Moreover, AR models like Tactoron still make an overly strong conditional independence assumption where each feature dimension is conditionally independent for a given time step. \n\nFor VITS trained on diverse speech without a speaker encoder, one can draw samples from the low-dimensional Gaussian prior to generate diverse samples. We have presented results of this in the common comment 3 above.\n\nWe follow the recommended temperature setup for YourTTS for sampling. It can be seen in Table 5 and 6 that the gap to Voicebox is huge (FSD: 277.9 vs 159.8/test-o WER: 54.6% vs 8.3%). YourTTS also conditions generation on speaker embedding inferred from a reference audio, and hence it cannot perform diverse sampling without conditioning on any audio like Voicebox.\n\n**5. The audio quality is not good on the demo pages.**\n\nAll the samples presented in the supplementary material use audio prompts from recruited volunteers, who recorded samples from their own devices and provided consents for sharing. Hence it can be heard that some prompts also contain noise and are not high quality. The quality of the produced audio samples should be compared against the quality of the audio prompt (“Voicebox Input”, “Original Speech”, and “Prompt”). This is because the Voicebox transfers audio style from the prompt, including not only voice, but also audio quality (noise, reverberation, etc). We would be more than happy to discuss if the reviewer could point out any specific samples with noticeable worse quality compared to their input audio prompt.\n\n**6. Are there any scenarios for automatic noise removal or editing?**\n\nThis paper did not explore automatic methods for determining noisy segments. One can consider using reference-free quality estimation methods like torchaudio-squim [1] or WADA-SNR [2] to determine which segments are noisy and should be re-generated. For editing, CorrectSpeech is applicable to not only CampNet but also A3T and Voicebox. \n\n[1] Kumar et al. \"Torchaudio-Squim: Reference-Less Speech Quality and Intelligibility Measures in Torchaudio.\" ICASSP’23\n\n[2] Kim and Richard. \"Robust signal-to-noise ratio estimation based on waveform amplitude distribution analysis.\" Interspeech’08\n\n**7. replacing the HiFi-GAN with BigVGAN could improve the audio quality.**\n\nWe thank the reviewer for their suggestion and agree that BigVGAN would likely lead to better audio quality. We will explore in the future work, and we also want to note that our main contribution is orthogonal to the choice of the vocoder.\n"}]

Author rebuttal:
We thank the reviewers for their thoughtful feedback. We are glad that they found Voicebox is highly versatile and can perform many different tasks (**FPC6, K4NY, pReF, Peh7**) by abstracting them into speech infilling (**Peh7**). We are encouraged that they found Voicebox is well-evaluated (**K4NY, pReF, Peh7**), scales effectively to 50K hours and multilingual setups (**Peh7, F9C6, MQeA**), shows SoTA performance (**F9C6, K4NY, pReF, Peh7**), enables faster inference through flow-matching with OT  (**K4NY**), and has the potential as a data generator for training other models  (**K4NY, Peh7**). We are also pleased that reviewer **pReF** found the paper easy to follow. We address common questions here and individual ones in separate threads. We will incorporate all feedback in the final version.

**Comment 1: Novelty (MQeA, F9C6)**
> **MQeA**: “The concept to train the model by infilling speech given audio and text has been already presented in many works.” **F9C6**: “It sounds like the combination of masked-based A3T and flow-based speech synthesis model”

**We highlight that no prior speech infilling models has attempted to or is capable of generalizing to as many tasks in the context of generative modeling as Voicebox does.** The goal of this paper is to build a generalist speech generative model that can solve many tasks without fine-tuning, just like LLMs solving many NLP tasks. Below we discuss the key novelties:

1. We show that speech infilling subsumes many tasks which have not been presented before. While speech infilling models exist, they struggle to generate long or diverse samples, because they assume deterministic input/output mapping and formulate it as a regression task.
2. We show that NAR flow-matching overcomes deterministic mapping and speeds up training and inference. See Table B3 (our reduced setup with 150K steps achieves better WER/spk similarity than Vall-E). See Fig. 2 for inference time comparison.
3, Voicebox enables unprecedented scaling and a single multilingual generative model. We added experiments in the rebuttal PDF Table 1 to show the benefit.

**Comment 2: Why flow-matching w/ OT path (F9C6, pReF, Peh7, MQeA)**
> **F9C6**: “use diffusion models since the masking strategy is similar to adding noise.” **pReF**: “what motivated the authors to choose the flow-matching model.” **Peh7**: “insight for using CNF to model speech infilling rather than other generative models such as diffusion models?” **MQeA**: “compare the conditional normalizing flow with a diffusion-based model.”

**To generalize speech infilling, any powerful non-autoregressive generative models, including diffusion models, should work. We chose flow-matching (FM) with optimal transport (OT) path because [1] showed that FM w/ OT > FM w/ diffusion > score-matching (SM) w/ diffusion (the typical diffusion model) on training speed and inference compute-quality trade-off.** See the comparison in Table 1 and Fig 4-7 in [1].

FM and SM models are in fact similar. Both transform a noise distribution to the data distribution, and learn to predict the gradient (score and flow, respectively) given a time step and a noisy sample at that time step. Diffusion and OT are simply two different paths (with the same initial and final marginal distribution). A path dictates how a sample transforms between initial and final distribution. As shown in Figure 3, Lipman et al. (2023), OT corresponds to a “simpler path” with constant speed and direction compared to diffusion, which the authors argue is easier to learn (faster training) and integration can be estimated more accurately with fewer steps (faster inference), leading to better empirical results.

[1] Lipman et al. "Flow matching for generative modeling." ICLR’23

**Comment 3: Baselines (F9C6, MQeA)**
> **F9C6**: “why not train [VITS] in the LibriTTS dataset?” **MQeA**: “YourTTS is not a good TTS model for zero-shot TTS models. The audio quality of YourTTS is bad. I recommend training the VITS with the reference encoder you used and the same configuration.” 

**YourTTS is exactly a VITS model trained on LibriTTS (and a few other datasets) with a reference encoder.** We have included such comparisons in Table 5 which reviewer F9C6 asked for. In addition, the Voicebox with a reduced setup (12 layers, trained on 1K hours) in Table B3 in the appendix still shows superior performance than VITS/YourTTS. Moreover, we trained a VITS on LibriSpeech without a reference encoder. It leads to much worse results (16.64% WER and 311.75 FSD for Table 5)

**Unfortunately, we cannot train VITS with same reference encoder because Voicebox doesn't have an explicit encoder**. Voicebox is a decoder only model taking text, masked audio, noisy audio) as input to predict flow. Also, given YourTTS is exactly VITS with a reference encoder trained on data similar to ours, we are not sure why the suggested experiment would lead to better results than YourTTS.

> **MQeA**: “all models should be trained with the same dataset.” **MQeA**: “transferring the voice style from a reference audio, not speaker ID, is utilized in many works. [...] the same transferring method is used for each model.” **MQeA**: “more comparisons with the recently proposed TTS models (not YourTTS, not Vall-E)”

Given that this work targets generalist speech generative model with unprecedented scaling, we believe **YourTTS is an appropriate baseline** because it is recent (ICML’22), open-sourced, achieves SoTA prior to VALL-E on zero-shot TTS, and is trained on in-the-wild data which enables better generalization than those trained on VCTK/LJ.

Furthermore, **we compared with most related models trained with the same dataset and using the same transferring method**, which are VALL-E and A3T-style models (Section B.3 in the appendix). We could better address the comments if reviewer could kindly clarify **why these are not appropriate baselines**, and **what zero-shot TTS models we should compare with and why.**

Meta-review: This paper proposes a non-autoregressive flow-matching model for generating speech given audio context and text. The model was trained using in-context learning with a large-scale dataset of over 50,000 hours of speech. It also provides a comprehensive comparison with the state-of-the-art VALL-E and other models. 

The paper was thoroughly discussed and ultimately received acceptance from all reviewers, with reviewer MQeA particularly changing their score from 2 to 5, leading to an “Accept” decision.
Decision: Accept (poster)