Title: Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale
Venue: NeurIPS 2023
Accepted: yes
Keywords: speech generation

flow-matching

diffusion

in-context learning

text-to-speech
Abstract: Large-scale generative models such as GPT and DALL-E have revolutionized the research community. These models not only generate high fidelity outputs, but are also generalists which can solve tasks not explicitly taught. In contrast, speech generative models are still primitive in terms of scale and task generalization. In this paper, we present Voicebox, the most versatile text-guided generative model for speech at scale. Voicebox is a non-autoregressive flow-matching model trained to infill speech, given audio context and text, trained on over 50K hours of speech that are not filtered or enhanced. Similar to GPT, Voicebox can perform many different tasks through in-context learning, but is more flexible as it can also condition on future context. Voicebox can be used for mono or cross-lingual zero-shot text-to-speech synthesis, noise removal, content editing, style conversion, and diverse sample generation. In particular, Voicebox outperforms the state-of-the-art zero-shot TTS model VALL-E on both intelligibility (5.9\% vs 1.9\% word error rates) and audio similarity (0.580 vs 0.681) while being up to 20 times faster. Audio samples can be found in \url{https://voicebox.metademolab.com}.

Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale

Abstract

Large-scale generative models such as GPT and DALL-E have revolutionized the research community. These models not only generate high fidelity outputs, but are also generalists which can solve tasks not explicitly taught. In contrast, speech generative models are still primitive in terms of scale and task generalization. In this paper, we present V oicebox, the most versatile text-guided generative model for speech at scale. Voicebox is a non-autoregressive flow-matching model trained to infill speech, given audio context and text, trained on over 50K hours of speech that are not filtered or enhanced. Similar to GPT, Voicebox can perform many different tasks through in-context learning, but is more flexible as it can also condition on future context. Voicebox can be used for mono or cross-lingual zero-shot textto-speech synthesis, noise removal, content editing, style conversion, and diverse sample generation. In particular, Voicebox outperforms the state-of-the-art zeroshot TTS model VALL-E on both intelligibility (5.9% vs 1.9% word error rates) and audio similarity (0.580 vs 0.681) while being up to 20 times faster. Audio samples can be found in https://voicebox.metademolab.com .

1 Introduction

Recent advances in large-scale generative models [6, 42, 50] have led to a major paradigm shift towards building general-purpose models, which can perform many new tasks not explicitly trained on. These generative models learn to predict the missing data given the context. Post training, we can directly input a question, optionally with a few contextual question-answer examples, instead of fine-tuning with labeled data. While the training objective appears simple, it subsumes many tasks as one can convert them into some form of context. For the model to perform well at every task, it implies that the estimation of p ( missing data | context ) needs to be accurate for every context. Hence, scale and diversity are the most crucial factors for building general-purpose models [20, 1].

Despite the success of large-scale generative models in other areas, most speech models are still trained on datasets at the scale of tens to hundreds of hours [51, 31, 32, 46, 24, 58, 7]. Previous works consider highly curated datasets such as VCTK [64], which contains only clean audio recorded in studio from about 100 speakers with little speaking style and text variation. Such models struggle to synthesize speech with rich variation in emotion, voice, background noise, acoustic condition, and have not been tested on the abilities to generalize to tasks not explicitly trained on.

This paper presents Voicebox, the most versatile text-conditioned speech generative model at scale. Voicebox is trained on a text-guided speech infilling task, where the goal is to generate masked speech

Raw

data

Model

Input

Model

Output

Text-Only Sampling

'DON'T MOVE AROUND…'

Editing

'OUTSIDE OF THESE…'

noise

Concat

('HIS … GOURMET', 'FROM … ALONE')

Ref

Target

Ref

'FROM … ALONE'

Style

'HIS … GOURMET'

Content

'OUTSIDE OF THESE…'

'DON'T MOVE AROUND…'

Sample #1

Sample #2

Sample #3

given its surrounding audio and text transcript. This can be considered as a guided in-context learning problem, where audio style is inferred from the audio context and textual content is specified through transcript. V oicebox does not require any audio style labels (e.g., speaker, emotion, and noise), which differentiates V oicebox from the majority of prior work where such labels are used extensively. Prior work uses labels to make the mapping between input (text and audio style) and output (speech) more deterministic to reduce underfitting [60, 46]. We show that Voicebox's text-guided speech infilling approach is much more scalable in terms of data while subsuming many common generative tasks.

In terms of modeling, Voicebox is a non-autoregressive (NAR) continuous normalizing flow (CNF) model [10]. Similar to diffusion models [19], CNFs model the transformation from a simple distribution to a complex data distribution, parameterized by a neural network. We train Voicebox with flow-matching [38], a recently proposed method that enables efficient and scalable training of CNFs via a simple vector field regression loss. In contrast to auto-regressive models, Voicebox can consume context not only in the past but also in the future. Moreover, the number of flow steps can be controlled at inference time to flexibly trade off quality and runtime efficiency.

Voicebox is trained on 60K hours of English audiobooks and 50K hours of multilingual audiobooks in 6 languages for the mono and multilingual setups. Voicebox achieves state-of-the-art performance on mono-lingual/cross-lingual zero-shot TTS, speech denoising, speech editing, diverse speech sampling and an application to data creation for speech recognition. To tackle the lack of comparability due to the use of subjective metrics, this paper presents a series of metrics using public models to facilitate reproducible comparison and model development for speech generation studies.

2 Related Work

Generative speech models: Most speech generative models are task-specific and trained on different datasets. One common type of task is audio style conversion , which aims to convert only a specific attribute while keeping other attributes the same. Voice conversion [27, 39], emotion conversion [53, 34], speech enhancement [63, 11, 55] belong to this category. Many of these models are supervised and trained on pairs of data that only differ in one attribute, for example, emotion [34]. It is hard to obtain such data. Moreover, some attributes, such as speaking style, are hard to annotate. Hence, these models are often trained on small datasets.

Controllable text-to-speech synthesis (TTS) is another common task, which aims to synthesize speech in a target audio style given text. While some styles like voice can be specified through labels [32] or pre-trained embeddings like YourTTS [7] and Jia et al. [25]; others like prosody are hard to annotate or embed. Previous studies [62] tried to control them by learning a residual embedding. However, these models encode style in a low-dimensional space and impose an overly simple distribution of speech given text and residual embedding [51, 56]. They cannot generate realistic noisy speech given a low dimensional vector, and performance degrades when conditioned on noisy references [21].

Infilling can be considered as another type of task. It aims to predict speech given context [36, 4] and optionally text guidance [3, 5, 61]. Instead of learning an explicit embedding to control style, infilling models predict speech coherent to the context. In other words, these models perform incontext learning similar to Large Language Models (LLM). While this is a step toward building large

scale generalist models using little explicit supervision, most prior work using text guidance still assumes a deterministic mapping from text and context to target [3, 5], which is only realistic for very short segments. Voicebox is a text-guided infilling model, but it leverages the CNF model that can parameterize any distribution. Hence, Voicebox can infill speech of any length and can be trained on in-the-wild datasets with rich variation, and provide a general solution that subsumes many tasks in a text-guided fashion.

Large scale in-context learning models: With the advancement in neural codec for speech [22, 12, 67], many recent studies explore token-based language modeling for speech generation. The GSLM-family [36, 28, 41] are textless language models built upon HuBERT units [22] for speech continuation without using text. HuBERT units encode mostly content, and the generated speech does not preserve the voice of the prompt. To tackle this, AudioLM [4] considers a cascaded approach which first generates HuBERT-like tokens and then predicts SoundStream [67] tokens, a reconstruction based codec that preserves style. These models are not conditioned on text and are evaluated on spoken language modeling tasks.

VALL-E [61] is most related to Voicebox. It is a text conditioned LM trained on Encodec [12] tokens (similar to SoundStream). Encodec encodes each frame with 8 ordered codebooks at 75Hz using a residual quantization layer. V ALL-E has two modules. The first is an auto-regressive (AR) model that predicts the first code of each frame given text and the audio prompt. The second is an NAR model that predicts the remaining seven codebooks sequentially.

VALL-E demonstrates state-of-the-art (SOTA) zero-shot TTS performance through in-context learning, where speech of the desired style is used as prompt. The model considers the prompt as part of the whole utterance such that it generates the rest of the utterance containing the target text in the same audio style. Voicebox has several design advantages compared to this. 1) Voicebox can use context both in the past and future, which is useful for editing where only a segment in the middle needs to be generated. 2) Voicebox can generate speech much faster than VALL-E because flow-matching can produce high quality samples with less than 10 NAR steps, while VALL-E requires 1 AR and 7 NAR steps. 3) Voicebox decouples duration and audio modeling, enabling finer grained alignment control. 4) Voicebox is compatible with any continuous features including Encodec embeddings.

3 Method

3.1 Background: Flow Matching with an optimal transport path

Let R d be the data space with data points x ∈ R d drawn from some unknown distribution q ( x ) . Continuous Normalizing Flows (CNFs) [10] are a family of generative models that learn the transformation from a simple prior distribution p 0 (e.g., normal distribution) to the data distribution p 1 ≈ q . CNFs parameterize a time-dependent vector field v t : [0 , 1] × R d → R d that is used to construct a flow : ϕ t : [0 , 1] × R d → R d that pushes points from the prior towards the target distribution. The relationship is defined via the ordinary differential equation (ODE) as: dϕ t ( x ) /dt = v t ( ϕ t ( x )) and ϕ 0 ( x ) = x . For a flow ϕ t , the probability path (time-dependent probability density function) p : [0 , 1] × R d → R > 0 can be derived via the change of variables formula: p t ( x ) = p 0 ( ϕ -1 t ( x )) det [ ∂ϕ -1 t ( x ) /∂x ] . To sample from p t ( x ) , we first draw x 0 from p 0 and then solve the initial value problem (IVP) for ϕ t ( x 0 ) given dϕ t ( x ) /dt = v t ( ϕ t ( x )) and ϕ 0 ( x ) = x 0 . We use x t and ϕ t ( x 0 ) interchangeably.

Let p t be a probability path and u t be the corresponding vector field that generates p t . The vector field v t ( x ; θ ) parameterized by a neural network θ can be trained with the Flow Matching objective: L FM ( θ ) = E t,p t ( x ) || u t ( x ) -v t ( x ; θ ) || 2 , where t ∼ U [0 , 1] and x ∼ p t ( x ) . While the objective appears simple, in practice we do not have the prior knowledge of p t or v t , and cannot directly compute the loss or its gradient estimator.

Let x 1 be a random variable distributed according to data distribution q . Lipman et al. [38] first notes that a probability path p t ( x ) can be constructed via a mixture of simpler conditional paths p t ( x | x 1 ) whose vector field u t ( x | x 1 ) can be easily computed. To construct p t ( x ) , a conditional path is defined such that 1) p 0 ( x | x 1 ) = p 0 ( x ) and 2) p 1 ( x | x 1 ) = N ( x | x 1 , σ 2 I ) , a Gaussian distribution centered at x 1 with a sufficiently small σ (typically 10 -5 ). The marginal path is computed as ∫ p t ( x | x 1 ) q ( x 1 ) dx 1 , which closely approximates q ( x 1 ) at t = 1 . With that, [38] presents the Conditional Flow Matching (CFM) objective, L CFM ( θ ) = E t,q ( x 1 ) ,p t ( x | x 1 ) || u t ( x | x 1 ) -v t ( x ; θ ) || 2 .

It is proven that FM and CFM have identical gradients w.r.t. θ . More importantly, one can easily draw samples from p t ( x | x 1 ) and compute u t ( x | x 1 ) to derive an unbiased gradient estimator.

The next question is how to choose a conditional flow. A flow defines trajectories , which dictates how each point moves between p 0 and p 1 . Intuitively, a simpler trajectory (e.g., a straight line) can be learned faster and the IVP can be solved more accurately and efficiently. Lipman et al. [38] presents a conditional flow called optimal transport (OT) path , which has the form of p t ( x | x 1 ) = N ( x | tx 1 , (1 -(1 -σ min ) t ) 2 I ) and u t ( x | x 1 ) = ( x 1 -(1 -σ min ) x ) / (1 -(1 -σ min ) t ) . The flow is arguably simple because points move with a constant speed and direction. We adopt it for Voicebox.

Lipman et al. [38] also presents another flow that recovers the path of diffusion models [57], which is more complex than the OT path. We will present ablation studies comparing different paths (OT vs diffusion) and different objectives (CFM vs score-matching). Results show the superiority in performance and efficiency of CFM with OT path.

3.2 Problem formulation

Given a dataset of transcribed speech ( x, y ) where x and y denote an audio sample and its transcript, respectively, the goal is to build a single model that can perform many text-guided speech generation tasks through in-context learning. We propose to train such a generative model on the text-guided speech infilling task , which predicts a segment of speech given its surrounding audio and the complete text transcript. Let m be a binary temporal mask which is of the same length as x , and x mis = m ⊙ x and x ctx = (1 -m ) ⊙ x be the complementary masked versions of x . The generative model learns p ( x mis | y, x ctx ) . In other words, y and x ctx are the context and x mis is the missing data.

'How are you doing'

Phonemize &

Forced alignment

Random span

masking

Compute sample at

flow step t for OT path

Sampled

flow step

Sampled noise

from the prior

F

Sinusoidal

position embedder

Phone

embedder

H

F

F

N

Linear

Projection

D

N

1

Transformer

F

N

Frame-level

phone emb

Noisy

speech

Masked

speech

Predicted

flow (gradient)

Voicebox Audio Model

N

(a) Voicebox audio model. The lower half illustrates how inputs are created during training.

Voicebox Audio Model Inference

ODE step 1

ODE step 2

ODE step N

…

conditionals:

Voicebox audio model:

(b) Inference is equivalent to solving an ODE with an initial condition x 0 sampled from the prior, a derivative dx t d t specified by the audio model, and conditional inputs ( z, x ctx ) .

3.3 Model and Training

Motivated by the need that some applications require fine-grained alignment control between speech and text, we decouple Voicebox into two components: an audio model and a duration model. Let x = ( x 1 , x 2 , · · · , x N ) be an audio sample of N frames, y = ( y 1 , y 2 , · · · , y M ) be a text sequence of M phones, and l = ( l 1 , l 2 , · · · , l M ) be the per-phone duration where l j denotes how many audio frames y j correspond to and ∑ M j =1 l j = N . We further define z = rep ( y, l ) = ( z 1 , z 2 , · · · , z N ) to be the frame-level phone transcript, which repeats each y j by l j times such that z i denotes the phone label of the audio frame x i . For a pair of ( x, y ) , l and z can be estimated through forced alignment using a speech recognition model. The estimation of q ( x mis | y, x ctx ) is then broken down into the audio model q ( x mis | z, x ctx ) and the duration model q ( l mis | y, l ctx ) , where l mis and l ctx denote l masked by m ′ and 1 -m ′ , and m ′ is downsampled from m based on l , detailed in Appendix A.2.

Audio Model: Given a context z and x ctx of length N , the distribution of x mis is highly stochastic especially when x mis has a large temporal span. Hence, we parameterize it with a CNF and train

it using the flow matching objective with the optimal transport path. Audio x is represented as an 80 -dimensional log Mel spectrogram ( x i ∈ R 80 ) extracted at a 100Hz frame rate. The audio context x i ctx = 0 where m i = 1 and x i ctx = x i where m i = 0 . For simpler conditioning, we model the conditional distribution q ( x | z, x ctx ) of all frames x instead of only masked frames x mis . A neural network is used to parameterize the conditional vector field v t ( x t , x ctx , z ; θ ) that additionally takes x ctx and z as input. Note that x t is a sample at flow step t and x = x 1 .

Given as input x ctx ∈ R N × F , x t ∈ R N × F , phone sequence z ∈ [ K ] N with K denoting the number of phone classes, and a time step t ∈ [0 , 1] , we employ a Transformer model to parameterize the vector field v t . A lookup table, denoted as L ∈ R K × H , is used to embed the phone sequence z , resulting in the embedded sequence z emb ∈ R N × H where z i emb = L ( z i ) for i ∈ 1 , . . . , N . Subsequently, the three sequences ( x t , x ctx , and z emb ) are concatenated frame-by-frame and projected by employing matrix W p ∈ R (2 F + H ) × D , thereby obtaining the sequence H c ∈ R N × D where D represents the embedding dimension of the Transformer model.

To embed the flow step, a sinusoidal positional encoding is applied to map t ∈ [0 , 1] to h t ∈ R D . The sequence ˜ H c ∈ R ( N +1) × D , which serves as the input to the Transformer model, is derived by concatenating H c with the vector h t along the time dimension. Given the Transformer output v t ( x t , x mis , z ; θ ) ∈ R N × F , which is the sub-sequence corresponding to H c , the loss is computed as:

by reparameterization. During training, given an audio sample x and a prior sample x 0 , we have x t = (1 -(1 -σ min ) t ) x 0 + tx and u t ( x t | x ) = x -(1 -σ min ) x 0 . This function computes the loss on all frames, including those that are not masked and would not be required during inference. To divert the model's focus to masked frames, we present a masked version of L audio-CFM:

where the loss is only computed on masked frames. Appendix B.3 shows it leads to better results

Duration model: We consider two solutions. The first one closely follows the audio model. It models q ( l | y, l ctx ) via a conditional vector field which swaps ( x, x ctx , z ) with ( l, l ctx , y ) and accordingly for the flow, where l, l ctx ∈ R M × 1 and y ∈ [ K ] M . The masked version of the CFM loss is used for training. On the other hand, previous studies have shown that regression duration models can produce reasonable speech [51, 37]. Hence we consider a second solution that regresses the masked duration l mis given the context duration l ctx and phonetic transcript y . The same Transformer model is used, except that there are only two input sequences instead of three, and the time embedding is not used. The model is trained with an L 1 regression loss on masked phones:

where g denotes the regression-based duration model. This is similar to the duration model used in FastSpeech2 [51], but with additional duration context l ctx as input.

3.4 Inference

To sample from the the learned audio distribution p 1 ( x | z, x ctx ) , a noise x 0 is first sampled from p 0 , and then an ODE solver is used to evaluate ϕ 1 ( x 0 ) given dϕ t ( x ) /dt = v t ( ϕ t ( x ) , x ctx , z ; θ ) and the initial condition ϕ 0 ( x 0 ) = x 0 . Intuitively, the ODE solver computes ϕ 1 ( x 0 ) by evaluating v t at multiple t to approximate the integration from t = 0 to t = 1 given the initial condition ϕ 0 ( x 0 ) = x 0 . The number of function evaluation (NFE) is defined as how many times dϕ t ( x 0 ) /dt is evaluated. A higher NFE often leads to a more accurate solution of ϕ 1 ( x 0 ) at the cost of longer run time. This provides great flexibility for users to decide the trade-off between speed and accuracy. Moreover, we find that empirically Voicebox can already generate very high quality speech with less than 10 NFEs, making it significantly faster compared to auto-regressive models.

3.5 Classifier-Free Guidance

Classifier guidance (CG) [14] is a technique used to trade off mode coverage and sample fidelity for diffusion models post training. It modifies the score estimate of a diffusion model to include the gradient of the log likelihood of an auxiliary classifier. Ho and Salimans [18] notes that CG approximates sampling from p ( x | c ) p ( c | x ) α where c is the conditioner, and this can be simulated

without a classifier by mixing the score estimate of a conditional model and an unconditional model. The unconditional model can be jointly trained by dropping the conditioner c with some probability, and the same model provides score estimates for both p ( x ) and p ( x | c ) .

We extend the idea of classifier free guidance (CFG) to flow-matching models. The conditioner c is equivalent to ( z, x ctx ) for audio models and ( y, l ctx ) for duration models, which is dropped with p uncond during training. During inference, the modified vector field ˜ v t for the audio model becomes ˜ v t ( w,x mis , z ; θ ) = (1 + α ) · v t ( w,x ctx , z ; θ ) -α · v t ( w ; θ ) , where α is the strength of the guidance, and v t ( w ; θ ) is obtained by dropping x ctx and z . We use α and α dur for the CFG strengths for the audio and the duration model, selected based on validation. Note that the computation is doubled for the same NFE when using CFG, because the model forward is called twice to compute ˜ v t .

3.6 Applications

We demonstrate that Voicebox exhibits in-context learning abilities similar to LLMs by presenting a few examples of how to create context to perform tasks Voicebox was not explicitly trained on. Fig. A1 shows a detailed diagram of how inputs are formatted for each task.

Zero-shot TTS & alignment-preserved style transfer: Given a target text ˆ y and a transcribed reference audio ( x, y ) , zero-shot TTS aims to synthesize speech resembling the possibly unseen audio style of the reference. Voicebox performs the task by treating the reference audio and the target speech as one utterance where the target speech is masked. Let l and z be phone duration and frame-level transcript of ( x, y ) . The target duration ˆ l is sampled given the duration context l and concatenated phone sequence cat ( y, ˆ y ) . The target speech ˆ x is then sampled given the context x and concatenated frame-level phones cat ( z, rep (ˆ y, ˆ l )) .

Voicebox can also convert the audio style for speech ¯ x while preserving its alignment ¯ z . This is useful for editing audio that is synchronized with other modalities such as video. Similar to zero-shot TTS, Voicebox can simply perform the task by sampling target speech ˆ x given the context x and concatenated frame-level phones cat ( z, ¯ z )

Transient noise removal & content editing: When recording speech, one might misspeak a few words or the recording my be interrupted by unexpected background noise. In these scenarios it is desired to just edit the problematic segment instead re-recording the speech. Voicebox can perform transient noise removal through re-generating the noise corrupted segment given the original frame-level transcript and the surrounding clean audio.

For content editing, Voicebox first samples duration for the new phones given the edited phone transcript and the duration of existing phones to create the edited frame-level phone transcript. Given the new frame-level phone transcript and the audio for existing frames, Voicebox then samples the audio for frames corresponding to the new phones.

Diverse speech sampling & alignment-preserved style shuffling: Voicebox can generate diverse speech samples by infilling the whole utterance. We first use the duration model to sample ˆ l given the phone transcript ˆ y . We then use the audio model to sample ˆ x given ˆ z = rep (ˆ y, ˆ l ) . Similar to style transfer, V oicebox can also shuffle the audio style while keeping the alignment by sampling ˆ x conditioning on the frame-level transcript ¯ z of the target speech clip ¯ x .

4 Metrics

The common goal of audio-conditioned tasks is to produce realistic speech that is coherent with the context and has the correct textual content. For tasks not conditioned on audio context, it is desired to generate diverse and realistic samples with distribution similar to training data. Prior studies often adopt subjective metrics like mean opinion scores (MOS) [52] which are not comparable across papers, or quantitative metrics like mel cepstral distortion [35] that assume the output is deterministic given input, which is often not realistic [54]. In this paper, we advocate the following reproducible model-based perceptual metrics.

Correctness and intelligibility: We measure it by the word error rate ( WER ) of the synthesized speech's transcription with respect to the input text, which has been adopted in prior work [62]. Public automatic speech recognition (ASR) models are used for comparability. For English-only setups, we

follow [61] and use HuBERT-L [22] pre-trained on 60K hours of Librilight [26] and fine-tuned on 960 hours of Librispeech [43]. For multilingual setups we use the Whisper large-v2 model [49].

Coherence: This is measured by the similarity between the embedding of generated speech and that of the audio context, where different embedding models would reflect coherence of different attributes. V ALL-E proposed to use WavLM-TDCNN speaker embedding model, which maps an audio clip to a fixed dimensional vector, to measure voice similarity. We consider the same model to compare with VALL-E. In particular, VALL-E reports similarity with respect to resynthesized audio context by its vocoder (Encodec-decoder), which we call SIM-resyn (SIM-r) . SIM-resyn is not comparable across models using different vocoders. Hence, we advocate for computing similarity against the original audio context, which we call SIM-orig (SIM-o) .

Diversity and quality: Fréchet Inception Score (FID) [17] is widely adopted for image generation evaluations, which captures the similarity between generated and real images at the distribution level in some feature space. A shorter distance implies the distributions are more similar and generally reflects both higher sample quality and diversity. We adapt the metric for speech by using selfsupervised wav2vec 2.0 feature [2] and refer to it as Fréchet Speech Distance ( FSD ). We verify its effectiveness in Appendix C.1 along with alternative features.

As supplementary metrics, we include quality MOS ( QMOS ) for subjective audio quality evaluation, and similarity MOS ( SMOS ) for subjective audio similarity evaluation given pairs of prompt and system-generated audio clips. Both of which are in the scale of 1 to 5 with 5 being the best. The MOS instructions and standalone metrics for duration models can be found in Appendix C.

5 Experiment

Data: We train the English-only model on 60K hours ASR-transcribed English audiobooks and the multilingual model on 50K hours of multilingual audiobooks from six languages: English (En), French (Fr), German (De), Spanish (Es), Polish (Pl) and Portuguese (Pt). The two models are abbreviated as VB-En and VB-Multi. The Montreal Forced Aligner (MFA) [40] is used to phonemize and force align the transcript based on the MFA phone set. Word position postfixes are added. Audio is represented as a 80-dimensional log Mel spectrogram and a HiFi-GAN vocoder trained on the same 60K hours English speech is used to generate waveform. More details about phone representation, data transformation, and vocoder can be found in Appendix A1-A3.

Model: Transformer [59] with convolutional positional embedding [2] and symmetric bi-directional ALiBi self-attention bias [48] are used for both the audio and the duration model. ALiBi bias for the flow step x t is set to 0. More details in Appendix Appendix A.7. The audio model has 24 layers, 16 attention heads, 1024/4096 embedding/feed-forward network (FFN) dimension, 330M parameters. The duration model has 8 heads, 512/2048 embedding/FFN dimensions, with 8/10 layers for English/multilingual setup (28M/34M parameters in total).

Training: VB-En/VB-Multi audio models are trained for 500K/750K updates with an effective batch size of 240K frames. For training efficiency, audio length is capped at 1,600 frames and chunked randomly if length exceeds. Duration models are trained for 600K updates with an effective batch size of 60K frames. The Adam [33] optimizer is used with a peak learning rate of 1e-4, linearly warmed up for 5K steps and decays over the rest of training. The audio/duration sequence is masked with p drop = 0 . 3 / 0 . 2 , and otherwise a segment of r % sequence length is masked, where r ∼ U [70 , 100] / U [10 , 100] . p uncond is set to 0 . 2 for audio/duration models.

Inference: The torchdiffeq [9] package is used, which implements both fixed and adaptive step ODE solvers. By default, the midpoint solver is used with a step size of 0.0625 (NFE=32). The regression duration model is used by default. Silence at both ends are trimmed to 0.1 second max.

Baselines: We consider three baselines: 1) VALL-E [61], SOTA for English zero-shot TTS trained on Librilight. 2) YourTTS [7], SOTA multilingual zero-shot TTS model trained on VCTK, LibriTTS, TTS-Portugese [8], and M-AILABS French. It is a flow-based model adapted from VITS [32] using a pre-trained speaker embedder. 3) A3T [3], SOTA for NAR speech editing and infilling trained with a regression loss on VCTK. We also consider Demucs [11], a SOTA speech enhancement model trained with regression and adversarial losses for denoising experiments.

5.1 Monolingual and cross-lingual zero-shot TTS

Table 2 presents the zero-shot TTS results of the English model VB-En. Following [61], the test set is constructed by selecting 4 to 10 second long samples from Librispeech test-clean. We consider cross-sentence prompting where a 3 second clip from another sample of the same speaker is used as audio context, and continuation prompting where the first 3 seconds of each utterance is used. Voicebox outperforms all baselines on all metrics in both cases. In particular, V oicebox transfers style much more effectively (+0.101/+0.108 SIM-r on cross-sentence/continuation) than VALL-E, and the gap is even bigger when compared against raw audio (+0.141 SIM-o on continuation). MOS studies also confirm the quality and similarity of Voicebox are subjectively better than YourTTS.

Table 4 presents cross-lingual zero-shot TTS results, where the audio context and the target text are in different languages. Note that VB-Multi is not trained on any sample with multiple languages in an utterance spoken by the same speaker. The test set is constructed using filtered MLS test split described in Appendix A.4. For each target text, we sample one 3-second long audio context from each language, which creates 36 language transfer directions in total. Voicebox yields better performance than YourTTS everywhere. Specifically, on En/Fr/Pt which YourTTS supports, Voicebox obtains 3.1%/5.9%/8.1% lower WERs and 0.136/0.141/0.160 higher similarity averaged across audio context in six languages. Addition studies on prompt lengths are presented in Appendix B.2

5.2 Transient noise removal

We construct a noisy test set by mixing the filtered Librispeech test-clean from Section 5.1 with non-speech noise such that it overlaps with 50% of the duration at a -10dB signal-to-noise ratio. Additional conditions can be found in Appendix B.4. Table 3 presents the results comparing Voicebox with A3T and Demucs. It should be noted that A3T and Voicebox utilize transcript and location of the noise while Demucs does not. Compared to the baselines, Voicebox generates samples best on all metrics. A3T is better than Demucs on intelligibilty and quality, but the infilled speech is not coherent because it is only trained on VCTK.

5.3 Diverse speech sampling and application to ASR data generation

Table 5 compares the ability to generate diverse samples for Librispeech test-other text. We consider English Voicebox (VB-En) with regression (regr) or flow-matching (FM) duration models. VITS-VCTK additionally conditions on a speaker ID, which we randomly sample for each sentence. YourTTS conditions on text and a reference audio, which we draw from the LS train splits. Qualitatively, A3T generates the same robotic voice and VITS-LJ generates high quality but from a single voice, hence both yield high FSD (bad quality or diversity) but VITS-LJ has a low WER. VITS-VCTK improves the voice diversity and FSD and YourTTS further advances it as it is trained on more speakers. Voicebox models (with different duration samplers) outperform the baseline on FSD by large margins, showing Voicebox's ability to produce realistic and diverse samples whose distribution is close to the training data. Among them, the FM duration model creates more varying speaking styles compared to the regression one which ASR may struggle more to recognize.

We next train an ASR model using only synthetic speech and evaluate it on real speech , which has not been successful before because synthetic data were not realistic and representative enough. Table 6 compares real and synthetic data from Voicebox and three baseline models. Each TTS model generates one sample per text from the Librispeech training set, resulting in 281K utterances per system. For real data, we consider train-960 and train-clean-100. Details about the ASR model and training are in Appendix A.5. The results are highly correlated with the FSD scores for synthetic data. In particular, the ASR model trained on Voicebox data with FM duration model reduces WERs by over 85% compared to baselines, and only lags behind real data by 0.4% and 1.7% absolute.

5.4 Inference efficiency versus performance

We examine the trade-off between the metrics of interest (WER, SIM, FSD) for different settings of guidance strength ( α ) and NFE specified by the user. Fig. 3a shows the Voicebox inference time to generate an audio sample of 10 seconds (including vocoding and predicting duration) as NFE varies and compares that to VALL-E. 3 For NFE=2 without CFG, Voicebox takes about 0 . 31 seconds, about 20 times faster than V ALL-E. At NFE=64, Voicebox is only 4% slower than VALL-E.

Next, we study the cross-sentence setup of Section 5.1 to analyze the impact on WER and SIM-r. We find that for all settings V oicebox has better WER than V ALL-E. WER remains stable with mean of 2 . 0 and variance of 0 . 005 . WER plot can be found in Appendix B.5. As shown in Fig. 3b, in the case of SIM-r, lower classifier guidance strength values ( α = 0 or 0 . 3 ) produce higher speaker similarity when operating in a lower NFE regime ( ≤ 4). However, starting from NFE=8, a higher classifier guidance strength improves speaker similarity. Finally, in Fig. 3c we examine FSD by generating samples for Librispeech test-other text. We find that lower classifier guidance strength produces lower FSD scores and more diverse samples. Increasing the NFE for each setting improves FSD.

5.5 Ablation on generative modeling approaches

We compare three generative modeling approaches: the proposed flow-matching with the OT path (FM w/ OT), flow-matching with the variance preserving (VP) diffusion path (FM w/ diff), and score-

Voicebox

=0

=0.3

=0.7

VALL-E

2 1

2 2

2 3

2 4

2 5

2 6

NFE

0

2

4

6

Inference time (10s audio)

20.4X

(a) NFE vs inference time w/o CFG

2 1

2 2

2 3

2 4

2 5

NFE

0.575

0.600

0.625

0.650

0.675

SIM-r

2 1

2 2

2 3

2 4

2 5

NFE

160

180

200

220

FSD

(c) NFE vs FSD

matching with the VP diffusion path (SM w/ diff). A reduced setup described in B.3 is adopted, with a lowered learning rate (1e-4) and the loss in Eq. (1) to ensure convergence for all three objectives.

We vary the number of training and inference steps, and evaluate models on the zero-shot TTS task (Section 5.1). Results in Table 7 shows that FM w/ OT trains significantly faster than the other two objectives, achieving the best performance with 100K training steps, and even outperforms SM w/ diff using only 50K updates. Results in Table 8 shows superior inference efficiency of FM w/ OT, which can produce good results with just 8 NFEs, while FM w/ diff requires at least 8 NFEs and SM w/ diff requires over 32 NFEs. Complete results are in Table B5

6 Conclusion

This paper presents Voicebox, the most versatile generative model for speech. By learning to solve a text-guided speech infilling task on large scale multilingual datasets with a power model and training objective Voicebox demonstrates impressive task generalization capabilities. Voicebox achieves state-of-the-art performance on mono and cross-lingual zero-shot TTS, speech inpainting, and diverse speech sampling, and can generate speech 20 times faster than the best autoregressive models.

With high fidelity speech generation models like Voicebox, it brings the potential of misuse and unintended harm. To mitigate the risk, we also detail in Appendix B.1 that a highly effective classifier can be built to distinguish between authentic and synthetic speech. Voicebox is now trained only on read speech from audiobooks in six languages, and cannot transfer one attributes (e.g., emotion) from a reference while transferring another attribute (e.g., voice) from another reference. Due to space limit, we expand our discussion of limitation and broader impact in Appendix D. For future work, we would continue scaling the model and the data to include more languages and diverse types of speech such as conversations, and explore disentangled prompting for different attributes.

Acknowledgment

The authors would like to thank Kristin Lauter and Joelle Pineau for supporting the project, thank Ricky Chen, Yaron Lipman, Alexandre Defossez, Gabriel Synnaeve for the technical discussion, thank Jade Copet and Gabriel Synnaeve for the compute support, thank Eleonora Presani and Jackie Pan for discussing the responsible AI studies, thank William Ngan, Somya Jain, Lydia Baillergeau, Dana Beaty, Chantal Mora, Daniel Duncan, Gopika Jhala, Steph Miles, Josh Terry, Valeryia Aranovich, Ashton Evans, Aly Gill, Andrea Mileskiewicz, Emily Richards, and Aaron Vasquez for developing the visual assets and the website, thank Alyssa Newcomb and Oliver Libaw for developing the posts, thank Peter Gray, Natalie Hereth, Shauna Kelleher, Ashley Gabriel, Seine Kim, Ana Paula Kirschner Moffarej and Aiman Farooq for coordinating the launch, thank Harrison Rudolph, Mallika Malhotra, Carolyn Krol, Lauren Cohen and Mo Metanat for the reviews, and thank Alexandra Gualdino, Ana Paula Kirschner Mofarrej, Benjamin Muller, Chloe Rolland, Daniella Kalfa, Darcie Da Silva, Gabriel Synnaeve, Hunter Goldman, Juan Pino, Karen Ulrich, Kris Sekula, Manuel Ribeiro, Marina Zannoli, Mary Williamson, Rashel Moritz, Stephanie Castillo, Tu Anh Nguyen, Vlad Sobal, Volker Seeker, and anonymous volunteers for sharing speech samples for the demo.

References

A. Aghajanyan, L. Yu, A. Conneau, W.-N. Hsu, K. Hambardzumyan, S. Zhang, S. Roller, N. Goyal, O. Levy, and L. Zettlemoyer. Scaling laws for generative mixed-modal language models. ArXiv , abs/2301.03728, 2023.

A. Baevski, Y. Zhou, A. Mohamed, and M. Auli. wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in neural information processing systems , 2020.

H. Bai, R. Zheng, J. Chen, X. Li, M. Ma, and L. Huang. A3T: Alignment-aware acoustic and text pretraining for speech synthesis and editing. In International Conference on Machine Learning , 2022.

Z. Borsos, R. Marinier, D. Vincent, E. Kharitonov, O. Pietquin, M. Sharifi, O. Teboul, D. Grangier, M. Tagliasacchi, and N. Zeghidour. AudioLM: a language modeling approach to audio generation. ArXiv , abs/2209.03143, 2022.

Z. Borsos, M. Sharifi, and M. Tagliasacchi. SpeechPainter: Text-conditioned speech inpainting. In Interspeech , 2022.

T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. J. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei. Language models are few-shot learners. ArXiv , abs/2005.14165, 2020.

E. Casanova, J. Weber, C. D. Shulby, A. C. Júnior, E. Gölge, and M. A. Ponti. YourTTS: Towards zero-shot multi-speaker tts and zero-shot voice conversion for everyone. In International Conference on Machine Learning , 2021.

E. Casanova, A. C. Junior, C. Shulby, F. S. d. Oliveira, J. P. Teixeira, M. A. Ponti, and S. Aluísio. Tts-portuguese corpus: a corpus for speech synthesis in brazilian portuguese. Language Resources and Evaluation , 56(3):1043-1055, 2022.

R. T. Q. Chen. torchdiffeq, 2018. URL https://github.com/rtqichen/torchdiffeq .

R. T. Q. Chen, Y. Rubanova, J. Bettencourt, and D. K. Duvenaud. Neural ordinary differential equations. In Neural Information Processing Systems , 2018.

A. Défossez, G. Synnaeve, and Y. Adi. Real time speech enhancement in the waveform domain. ArXiv , abs/2006.12847, 2020.

A. Défossez, J. Copet, G. Synnaeve, and Y. Adi. High fidelity neural audio compression. ArXiv , abs/2210.13438, 2022.

B. Desplanques, J. Thienpondt, and K. Demuynck. ECAPA-TDNN: Emphasized Channel Attention, propagation and aggregation in TDNN based speaker verification. In Interspeech , 2020.

P. Dhariwal and A. Nichol. Diffusion models beat GANs on image synthesis. Advances in Neural Information Processing Systems , 2021.

J. J. Godfrey, E. C. Holliman, and J. McDaniel. Switchboard: Telephone speech corpus for research and development. In Acoustics, Speech, and Signal Processing, IEEE International Conference on , volume 1, pages 517-520. IEEE Computer Society, 1992.

A. Gulati, J. Qin, C.-C. Chiu, N. Parmar, Y. Zhang, J. Yu, W. Han, S. Wang, Z. Zhang, Y. Wu, et al. Conformer: Convolution-augmented transformer for speech recognition. arXiv preprint arXiv:2005.08100 , 2020.

M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter. GANs trained by a two time-scale update rule converge to a local Nash equilibrium. Advances in neural information processing systems , 2017.

J. Ho and T. Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598 , 2022.

J. Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems , 2020.

J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. de Las Casas, L. A. Hendricks, J. Welbl, A. Clark, T. Hennigan, E. Noland, K. Millican, G. van den Driessche, B. Damoc, A. Guy, S. Osindero, K. Simonyan, E. Elsen, J. W. Rae, O. Vinyals, and L. Sifre. Training compute-optimal large language models. ArXiv , abs/2203.15556, 2022.

W.-N. Hsu, Y. Zhang, R. J. Weiss, H. Zen, Y. Wu, Y. Wang, Y. Cao, Y. Jia, Z. Chen, J. Shen, et al. Hierarchical generative modeling for controllable speech synthesis. In International Conference on Learning Representations , 2019.

W.-N. Hsu, B. Bolte, Y.-H. H. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM Transactions on Audio, Speech, and Language Processing , 29:3451-3460, 2021.

W.-N. Hsu, T. Remez, B. Shi, J. Donley, and Y. Adi. Revise: Self-supervised speech resynthesis with visual input for universal and generalized speech enhancement. arXiv preprint arXiv:2212.11377 , 2022.

R. Huang, M. W. Y. Lam, J. Wang, D. Su, D. Yu, Y. Ren, and Z. Zhao. FastDiff: A fast conditional diffusion model for high-quality speech synthesis. In International Joint Conference on Artificial Intelligence , 2022.

Y. Jia, Y. Zhang, R. Weiss, Q. Wang, J. Shen, F. Ren, P. Nguyen, R. Pang, I. Lopez Moreno, Y. Wu, et al. Transfer learning from speaker verification to multispeaker text-to-speech synthesis. Advances in neural information processing systems , 2018.

J. Kahn, M. Rivière, W. Zheng, E. Kharitonov, Q. Xu, P.-E. Mazar'e, J. Karadayi, V. Liptchinsky, R. Collobert, C. Fuegen, T. Likhomanenko, G. Synnaeve, A. Joulin, A. rahman Mohamed, and E. Dupoux. Libri-Light: A benchmark for asr with limited or no supervision. International Conference on Acoustics, Speech and Signal Processing , 2019.

H. Kameoka, T. Kaneko, K. Tanaka, and N. Hojo. StarGAN-VC: non-parallel many-to-many voice conversion using star generative adversarial networks. IEEE Spoken Language Technology Workshop , 2018.

E. Kharitonov, A. Lee, A. Polyak, Y. Adi, J. Copet, K. Lakhotia, T. Nguyen, M. Rivière, A. rahman Mohamed, E. Dupoux, and W.-N. Hsu. Text-free prosody-aware generative spoken language modeling. In Annual Meeting of the Association for Computational Linguistics , 2021.

E. Kharitonov, D. Vincent, Z. Borsos, R. Marinier, S. Girgin, O. Pietquin, M. Sharifi, M. Tagliasacchi, and N. Zeghidour. Speak, read and prompt: High-fidelity text-to-speech with minimal supervision, 2023.

K. Kilgour, M. Zuluaga, D. Roblek, and M. Sharifi. Fréchet audio distance: A reference-free metric for evaluating music enhancement algorithms. In Interspeech , 2019.

J. Kim, S. Kim, J. Kong, and S. Yoon. Glow-TTS: A generative flow for text-to-speech via monotonic alignment search. Advances in Neural Information Processing Systems , 2020.

J. Kim, J. Kong, and J. Son. Conditional variational autoencoder with adversarial learning for end-to-end text-to-speech. In International Conference on Machine Learning , 2021.

D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. CoRR , abs/1412.6980, 2014.

F. Kreuk, A. Polyak, J. Copet, E. Kharitonov, T.-A. Nguyen, M. Rivière, W.-N. Hsu, A. Mohamed, E. Dupoux, and Y. Adi. Textless speech emotion conversion using decomposed and discrete representations. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing , 2022.

R. Kubichek. Mel-cepstral distance measure for objective speech quality assessment. In Proceedings of IEEE pacific rim conference on communications computers and signal processing , volume 1, pages 125-128. IEEE, 1993.

K. Lakhotia, E. Kharitonov, W.-N. Hsu, Y. Adi, A. Polyak, B. Bolte, T. Nguyen, J. Copet, A. Baevski, A. B. Mohamed, and E. Dupoux. On generative spoken language modeling from raw audio. Transactions of the Association for Computational Linguistics , 9:1336-1354, 2021.

A. Ła´ ncucki. Fastpitch: Parallel text-to-speech with pitch prediction. In International Conference on Acoustics, Speech and Signal Processing , 2021.

Y. Lipman, R. T. Q. Chen, H. Ben-Hamu, M. Nickel, and M. Le. Flow matching for generative modeling. In International Conference on Learning Representations , 2023.

J. Lorenzo-Trueba, J. Yamagishi, T. Toda, D. Saito, F. Villavicencio, T. H. Kinnunen, and Z. Ling. The voice conversion challenge 2018: Promoting development of parallel and nonparallel methods. ArXiv , abs/1804.04262, 2018.

M. McAuliffe, M. Socolof, S. Mihuc, M. Wagner, and M. Sonderegger. Montreal forced aligner: Trainable text-speech alignment using kaldi. In Interspeech , 2017.

T. Nguyen, E. Kharitonov, J. Copet, Y. Adi, W.-N. Hsu, A. M. Elkahky, P. Tomasello, R. Algayres, B. Sagot, A. Mohamed, and E. Dupoux. Generative spoken dialogue language modeling. Transactions of the Association for Computational Linguistics , 11:250-266, 2022.

A. Nichol, P. Dhariwal, A. Ramesh, P. Shyam, P. Mishkin, B. McGrew, I. Sutskever, and M. Chen. GLIDE: Towards photorealistic image generation and editing with text-guided diffusion models. In International Conference on Machine Learning , 2021.

V. Panayotov, G. Chen, D. Povey, and S. Khudanpur. Librispeech: An asr corpus based on public domain audio books. International Conference on Acoustics, Speech and Signal Processing , 2015.

D. S. Park, W. Chan, Y. Zhang, C.-C. Chiu, B. Zoph, E. D. Cubuk, and Q. V. Le. SpecAugment: A simple data augmentation method for automatic speech recognition. In Interspeech , 2019.

A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems , 2019.

V. Popov, I. Vovk, V. Gogoryan, T. Sadekova, and M. Kudinov. Grad-TTS: A diffusion probabilistic model for text-to-speech. In International Conference on Machine Learning , 2021.

D. Povey, A. Ghoshal, G. Boulianne, L. Burget, O. Glembek, N. Goel, M. Hannemann, P. Motlicek, Y. Qian, P. Schwarz, et al. The kaldi speech recognition toolkit. In IEEE 2011 workshop on automatic speech recognition and understanding , number CONF. IEEE Signal Processing Society, 2011.

O. Press, N. A. Smith, and M. Lewis. Train short, test long: Attention with linear biases enables input length extrapolation. ArXiv , abs/2108.12409, 2021.

A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever. Robust speech recognition via large-scale weak supervision. ArXiv , abs/2212.04356, 2022.

A. Ramesh, M. Pavlov, G. Goh, S. Gray, C. Voss, A. Radford, M. Chen, and I. Sutskever. Zero-shot text-to-image generation. ArXiv , abs/2102.12092, 2021.

Y. Ren, C. Hu, X. Tan, T. Qin, S. Zhao, Z. Zhao, and T.-Y. Liu. Fastspeech 2: Fast and highquality end-to-end text to speech. In International Conference on Learning Representations , 2021.

F. Ribeiro, D. Florêncio, C. Zhang, and M. Seltzer. CrowdMOS: An approach for crowdsourcing mean opinion score studies. In International Conference on Acoustics, Speech and Signal Processing , 2011.

C. Robinson, N. Obin, and A. Roebel. Sequence-to-sequence modelling of F0 for speech emotion conversion. In International Conference on Acoustics, Speech and Signal Processing , 2019.

C. Saharia, W. Chan, H. Chang, C. Lee, J. Ho, T. Salimans, D. Fleet, and M. Norouzi. Palette: Image-to-image diffusion models. In ACM SIGGRAPH 2022 Conference Proceedings , 2022.

J. Serrà, S. Pascual, J. Pons, R. O. Araz, and D. Scaini. Universal speech enhancement with score-based diffusion. ArXiv , abs/2206.03065, 2022.

J. Shen, R. Pang, R. J. Weiss, M. Schuster, N. Jaitly, Z. Yang, Z. Chen, Y. Zhang, Y. Wang, R. J. Skerry-Ryan, R. A. Saurous, Y. Agiomyrgiannakis, and Y. Wu. Natural TTS synthesis by conditioning wavenet on mel spectrogram predictions. International Conference on Acoustics, Speech and Signal Processing , 2017.

Y. Song and S. Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems , 32, 2019.

X. Tan, J. Chen, H. Liu, J. Cong, C. Zhang, Y. Liu, X. Wang, Y. Leng, Y. Yi, L. He, F. K. Soong, T. Qin, S. Zhao, and T.-Y. Liu. NaturalSpeech: End-to-end text to speech synthesis with human-level quality. ArXiv , abs/2205.04421, 2022.

A. Vaswani, N. M. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. ArXiv , abs/1706.03762, 2017.

C. Wang, W.-N. Hsu, Y. Adi, A. Polyak, A. Lee, P.-J. Chen, J. Gu, and J. M. Pino. fairseq s 2 : A scalable and integrable speech synthesis toolkit. In Conference on Empirical Methods in Natural Language Processing , 2021.

C. Wang, S. Chen, Y. Wu, Z.-H. Zhang, L. Zhou, S. Liu, Z. Chen, Y. Liu, H. Wang, J. Li, L. He, S. Zhao, and F. Wei. Neural codec language models are zero-shot text to speech synthesizers. ArXiv , abs/2301.02111, 2023.

Y. Wang, D. Stanton, Y. Zhang, R. J. Skerry-Ryan, E. Battenberg, J. Shor, Y. Xiao, F. Ren, Y. Jia, and R. A. Saurous. Style tokens: Unsupervised style modeling, control and transfer in end-to-end speech synthesis. In International Conference on Machine Learning , 2018.

Y. Xu, J. Du, L.-R. Dai, and C.-H. Lee. A regression approach to speech enhancement based on deep neural networks. IEEE/ACM Transactions on Audio, Speech, and Language Processing , 23(1):7-19, 2014.

J. Yamagishi, C. Veaux, and K. MacDonald. Cstr vctk corpus: English multi-speaker corpus for cstr voice cloning toolkit (version 0.92). 2019.

R. Yamamoto, E. Song, and J.-M. Kim. Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogram. In International Conference on Acoustics, Speech and Signal Processing , 2020.

N. Yu, V. Skripniuk, S. Abdelnabi, and M. Fritz. Artificial fingerprinting for generative models: Rooting deepfake attribution in training data. In Proceedings of the IEEE/CVF International conference on computer vision , pages 14448-14457, 2021.

N. Zeghidour, A. Luebs, A. Omran, J. Skoglund, and M. Tagliasacchi. Soundstream: An end-to-end neural audio codec. IEEE/ACM Transactions on Audio, Speech, and Language Processing , 30:495-507, 2022.

A Additional Details of Experiment Setup

A.1 Vocoder

We adapt the HiFi-GAN V1 configuration to generate 16kHz audio from 80 dimensional log Mel spectral features sampled at 100Hz. To compute the log Mel spectrogram, we use a 1024-point short time Fourier transform with a 640-sample (40ms) analysis window, 160-sample (10ms) shift, and the Hann windowing function to compute the amplitude spectrogram, and then apply an 80 dimension Mel filter with a cutoff frequency at 8kHz. The original HiFi-GAN V1 has four transposed convolution blocks for upsampling. The upsampling factors are [8 , 8 , 2 , 2] and the corresponding kernel sizes are [16 , 16 , 4 , 4] . Here we only need a total upsampling factor of 160 instead of 256, and we adjust the upsampling factors to [5 , 4 , 4 , 2] and kernel sizes to [11 , 8 , 8 , 4] accordingly. The other parameters are identical to the HiFi-GAN V1 configuration. Total number of parameters is 13M. We train the adapted HiFi-GAN on the 60K hours of English audiobook data for 1.5M updates on 8 GPUs, which takes 7.5 days.

A.2 Phone representation

Ghost silence The frame-level phonetic transcript used for training is obtained through forcealigning speech and phonetic transcript. In particular, a forced aligner may align some frames to a special phone ' SIL ' for non-speech frames (silence or noise). For most forced aligners, only frames between words and frames at the beginning and at the end of an utterance can be aligned to SIL .

During inference, we are only given the text transcript, which does not tell us where we should insert silence to. Hence, it is desired to have the duration model not only predict the duration for each phone ( SIL included), but also predict the existence of SIL at eligible locations (between words and at the two ends of the utterance). To tackle it, we introduce ghost silence to our phonetic transcript, which are silences in between words with duration of zero frames.

To give an example, suppose the transcript contains three words: ' Hey what's up ' with pronunciation ' {Hey:[A,B], what's:[C], up:[D,E,F]} ', and the frame-level phonetic transcript z obtained through forced alignment is z = ( SIL A B B SIL C D D D E E F SIL SIL ) . The phonetic transcripts becomes y = ( SIL A B SIL C SIL D E F SIL ) , where the ghost silence is highlighted in green. The corresponding duration would be l = (1 , 1 , 2 , 1 , 1 , 0 , 3 , 2 , 1 , 2) . A ghost silence is inserted between what's and up during training, and the duration model should predict the duration of it as zero to indicate that there should not be a pause between the two words.

Word-position-dependent phone The possible absence of silence between words in the framelevel phone transcript can make it hard for the audio model to identify word boundaries. To help the audio model identify the word boundary which is important when reading a sentence, we introduce word-position-dependent phones which are commonly used in Hidden Markov Model based acoustic models for speech recognition [47]. This adds a postfix to each phone in the transcript to denote where it is in the corresponding word. There are four postfixes: _B for beginning, _E for end, _I for intermediate, and _S for singleton. The above example becomes ' {Hey:[A_B,B_E], what's:[C_S], up:[D_B,E_I,F_E]} ' with frame-level phonetic transcript z = ( SIL A_B B_E B_E SIL C_S D_B D_B D_B E_I E_I F_E SIL SIL ) .

Phone-level mask In terms of masking, given duration l , the relationship of phone-level mask m ′ and frame-level mask m can be written as m = rep ( m ′ , l ) . For the applications where a duration model is involved (zero-shot TTS, content editing, diverse speech sampling), the frame-level mask m is extended such that no phone is partially masked. In other words, all the frames corresponding to a phone is either entirely masked or entirely unmasked. During training, we mask a contiguous chunk of audio, infilling of which is a more challenging task compared to infilling multiple smaller segments. All frames that are aligned to a phone are either entirely masked or unmasked. Note that masking all frames for a phone is not a necessity but was chosen due to ease of implementation.

A.3 Data transformation

The Mel spectrogram is normalized with the global mean (-5.8843) and standard deviation (2.2615) to stabilize training. The statistics are estimated on 30k random training samples from the 1K hours of

English audio. Input and output duration are dequantized ( x ∼ U [ x -0 . 5 , x +0 . 5] ) and transformed with log(1 + x ) following [51]. Prediction of duration is quantized and clipped such that the minimal duration is greater than or equal to zero.

A.4 Cross-lingual zero-shot TTS test data filtering

We create a test set for each language by selecting samples from the MLS test split which have Whisper transcription WER lower than 20% (or 30% for Polish and Portugueses test splits which contains less than 1K samples), because we found MLS test set contains many examples with incomplete transcriptions missing a large portion of the utterance. In addition, a small amount of utterances were excluded due to MFA alignment failure. Table A1 lists the number of samples remained for each language.

A.5 Setup for training ASR models with synthetic speech

To train an ASR model in Section 5.3, we extract 80-dimensional log Mel features with a 25ms window and a 10ms frame shift, and then apply global mean-variance normalization. The ASR model is an RNN-T with a Conformer-based encoder [16]. The conformer applies time scale reduction to the input features with stride 6, embeds them into 512-dimensional vectors, passes these vectors through a 20-layer conformer which has 8 attention heads and 2048-dimensional fully-connected layers. The conformer output is further mapped to 1024 dimensions through a linear layer followed by layer normalization before being passed to the joiner. The predictor of the network first embeds wordpiece units into 512 dimensional embeddings, applies layer normalization, a 512-dimensional LSTM, a dropout layer and a linear layer that maps the LSTM output to 1024 dimensions. The joiner adds the encoder and predictor outputs, applies tanh non-linearity and uses a linear layer that maps the 512-dimensional joiner input into wordpiece units. There are 4096 wordpiece units estimated from the LibriSpeech 960hr training text.

We apply SpecAugment [44] in all ASR runs. The models are trained using PyTorch [45] with Adam [33] optimizer for 120 epochs unless otherwise noted. The learning rate follows a tri-stage schedule with a maximum of 0.001. We applied gradient clipping at 10 and a weight decay parameter of 0.1. For the 960hr setting, we used a variable batch size capped at 1K utterances or 30K frames, whichever is smaller. This corresponds to about 45K update steps for 120 epochs. For the 100hr setting, we set the maximum learning rate to 0.0001 and used smaller batch size (capped at 200 utterances or 5K frames). In this case, 120 epochs corresponded to about 120K updates. For decoding, we used n-best decoding with a beam-size of 15, and evaluated the WER on the 1-best path.

A.6 How to format input for different task

Fig. A1 shows how input are formatted to perform diverse speech sampling, content editing, and zero-shot TTS using Voicebox.

A.7 Bi-directional ALiBi Bias

We use a symmetric variant bi-directional variant of ALiBi bias where any query Q i and key K j with | i -j | = N use the same representations 4 . Furthermore, for any query Q i the bias corresponding

[SIL HH AH L OW SIL W ER L D SIL]

[0   0  0  0 0  0   0 0  0 0 0  ]

Duration

model

[8   3  2  2 1  0   4 5  1 1 10 ]

Audio

model

[SIL*8 HH*3 … D*1 SIL*10]

upsample

'Hello world'

phonemize

[SIL HH AH L OW SIL T AA M SIL]

[8   3  2  2 1  7   0 0  0 10 ]

Duration

model

[8   3  2  2 1  7   1 5  2 10 ]

Audio

model

upsample

'Hello world' → 'Hello tom'

Forced-align &

Replace

(audio of 'Hello world')

'world' segment removed,

insert 8 frames of mask

(predicted duration of 'tom')

[SIL HH … L D SIL HH EY SIL F OW K S SIL]

[8   3  … 1 1 10  0  0  0   0 0  0 0 0  ]

Duration

model

Audio

model

[SIL*8 … K*3 S*3 SIL*4]

upsample

'Hello world'

Forced-align

(audio of 'Hello world')

mask whose length is

the sum of predicted

duration for 'Hey folks'

'Hey folks'

phonemize

[8   3  … 1 1 10  2  4  0   1 5  1 3 4  ]

(audio of 'Hello world')

Diverse sampling

Editing / infilling

Style transfer

一一一

一

to the flow step x t is set to 0. Similarly the bias from the flow-step x t to any other query 0. In our experiments, we find ALiBi Bias to improve convergence and extrapolation to longer sequences.

B Additional Experiments

B.1 Detecting generated speech

We recognize the potential risks of a model capable of generating speech in the style of arbitrary users. In an effort to diminish these risks we show that a binary classification model is able to consistently distinguish between real world speech and that which is generated from our model.

Inspired by [29], we train a convolutional binary classification model to distinguish between real and generated speech. The model consists of 6 blocks with hidden dimension sizes: [64, 128, 256, 256, 512, 512]. Each block contains a (3 x 1) convolution along the time axis, a (1 x 3) convolution along the frequency axis, followed by a ReLU activation and batch normalization. After each block that

increases the hidden dimension size we also apply max pooling with a stride of 2 across both the time and frequency dimensions. Finally, global max pooling is applied and a linear layer projects to a single value that is fed into a binary cross entropy loss. At inference time we create a sliding window with hop length equal to 250ms and run each chunk of audio through the classifier and average the outputs.

The model is tested on the dev-clean split of Librispeech. We then take a 100 hour subset of the 60K hour-English data and set aside 2,703 random utterances (to match the size of dev-clean) which is used as a validation split. The remaining utterances from the 100 hours subset are used as the ground truth utterances for training. For each split we synthesize audio, conditioned on each utterance of the split by masking out frames in the spectrogram corresponding to 90%, 50%, and 30% of the phonemes of the utterance. All samples are generated using classifier-free guidance with w = 0 . 7 , midpoint ODE solver (step size 0.0625 / NFE=64), and the regression duration model.

We consider two detection tasks. The first one is to distinguish between original audio and Voiceboxgenerated audio. The second one is to distinguish resynthesized audio and Voicebox-generated audio. The resynthesized audio is created by extracting the Mel Spectrogram from original audio and then vocoding it with the HiFi-GAN vocoder.

Table B2 presents the results for each setting. The model can trivially distinguish original audio from Voicebox-generated audio. This results from the fact that a model can also trivially distinguish original audio from resynthesized audio, most likely by recognizing artifacts produced by the vocoder. The task of differentiating V oicebox-generated audio from resynthesized audio is much harder. When 90% of the audio is masked, the model is able to reliably classify the audio as Voicebox-generated. In lower masking regimes this decreases a bit, but this is likely due to a naive inference method of averaging the outputs of all sliding windows. Since the majority of windows are non-synthetic, this leads to mis-classifications.

B.2 How context length affects monolingual and cross-lingual zero-shot TTS

Monolingual: For in-context zero-shot TTS in Section 5.1, we used 3 . 0 seconds of prompt audio. Here we examine how WER / SIM-r vary with different amounts of prompt audio using duration from regression duration model for the target text. If the desired prompt is longer than the available audio, the shorter audio is used as the prompt. Results are shown in Figure B2. As expected, WER mildly decreases and SIM-r grows quickly flattens with longer audio prompts. Comparing against VALL-E, Voicebox is more efficient at leveraging an audio prompt, achieving the same speaker similarity as VALL-E with roughly two thirds the input audio.

Cross-lingual: Here we examine the effect of increasing the prompt length for the case of crosslingual zero-shot TTS. As described in 5.1, this setting has a total 36 language transfer directions for each pair of source and target language. For each target text in a given transfer setting, we examine how WER / SIM-o 5 vary as the prompt length increases. Similarly, the regression duration model is used for the target text. Fig. B3 and Fig. B4 plot the SIM-o (speaker similarity) and WER trends respectively. When concatenating the prompt to the target for MLS, we find that the samples are quite a bit longer than what the model was trained on (16s max length), because MLS test set samples are in average 15 seconds long. To alleviate this out of domain issue and focus the study

1

2

3

4

5

6

7

8

Seconds

2

3

4

5

6

WER

(a) WER

1

2

3

4

5

6

7

8

Seconds

0.2

0.3

0.4

0.5

0.6

0.7

SIM-r

english

french

german

polish

portuguese

spanish

2

4

6

8

10

Prompt (s)

0.3

0.4

0.5

0.6

SIM-o

(a) English

2

4

6

8

10

Prompt (s)

0.3

0.4

0.5

0.6

0.7

SIM-o

(b) French

2

4

6

8

10

Prompt (s)

0.3

0.4

0.5

0.6

0.7

SIM-o

(c) Spanish

2

4

6

8

10

Prompt (s)

0.3

0.4

0.5

0.6

0.7

SIM-o

(d) Portuguese

2

4

6

8

10

Prompt (s)

0.3

0.4

0.5

0.6

0.7

SIM-o

(e) German

2

4

6

8

10

Prompt (s)

0.3

0.4

0.5

0.6

0.7

SIM-o

(f) Polish

on varying the prompt length, we truncate the target sequences to 4 seconds (at word boundaries). We notice that WERs are higher compared to Table 4, likely because the ASR model struggles with incomplete sentences. Each subplot contains the trend for one of the target languages from all six source languages.

The speaker similarity consistently improves as the prompt length is increased, similar to the monolingual setting. In contrast, we find that WER increases as we increase the prompt length for most directions. The WER increases much more for En → non-En directions. We hypothesize that this is due to training data imbalance across languages, where English accounts for over 90% of the multilingual training data. Hence, when transferring from English, the model is more likely to assume that the whole sentence is in English as the prompt length increases and produce incorrect

english

french

german

polish

portuguese

spanish

2

4

6

8

10

Prompt (s)

6.5

7.0

7.5

8.0

8.5

WER

(a) English

2

4

6

8

10

Prompt (s)

10

20

30

40

WER

(b) French

2

4

6

8

10

Prompt (s)

10

20

30

40

WER

(c) Spanish

2

4

6

8

10

Prompt (s)

10

20

30

40

WER

(d) Portuguese

2

4

6

8

10

Prompt (s)

8

10

12

14

16

18

WER

(e) German

2

4

6

8

10

Prompt (s)

20

40

60

80

WER

(f) Polish

pronunciation for the non-English target. Note that during the training phase, the model was only exposed to audio samples and phonemes originating from a single language.

B.3 Comparing audio model training objectives

While A3T is considered the regression-based speech infilling baseline, it is trained on a smaller dataset and uses a smaller model compared to Voicebox. Here we present a controlled study comparing the flow-matching and regression objectives, as well as the effectiveness of masked loss.

We consider a reduced setup for this ablation to save the compute. All models were trained on an English audiobook dataset with 1K hours of speech using a smaller model configuration (12 layers, 1024-dimensional Transformer embedding, 2048-dimensional feed-forward layer, 8 attention heads) for 150k steps with an effective batch size of 120k frames. These models are evaluated on the cross-sentence zero-shot TTS setup (Section 5.1) and diverse speech sampling (Section 5.3).

Results in Table B3 show that while regression audio models produce comparable WER, the audio similarity and diversity are significantly worse. Subjective listening also reveals that the audio quality and audio similarity are much worse. On the other hand, masked loss improves audio similarity and diversity while having little impact on intelligibility.

B.4 Effectiveness on data scaling

We create four subsets of the 60K hour English data (0.1%, 1%, 10%, 100% in duration). In particular, the x % subset would contain roughly x % of the speakers from the original set. We train one model on each subset with a reduced setup described in Appendix B.3 and evaluate them on zero-shot TTS (cross-sentence) and diverse sampling. Results show that scaling data constantly improves the zero-shot TTS performance (WER and SIM-r) as well as WER on diverse sampling. For FSD it shows regression when scaling from 6K hour to 60K hour, but this could result from the the reference distribution is computed from the 1K hour English audiobook data that has less diverse samples.

B.5 Complete results on comparing generative modeling approaches

Table B5 presents the full results of Section 5.5 on all combinations of training steps, inference steps with results on both monolingual zero-shot TTS (Section 5.1) and diverse speech sampling (Section 5.3) for the ablation study presented in Section 5.5. In all settings Flow Matching with OT paths performs strictly better than both of the other approaches.

B.6 Transient noise removal in more conditions

We expand the experiments in Section 5.2 by comparing the models on two noise levels (low noise: 10dB and high noise: -10dB), three overlapping ratios (30%, 50%, 70%), and also two types of noise (speech noise and non-speech noise).

Results are presented in Table B6. Voicebox consistently produces the most intelligible audio at all conditions (indicating the percentage of speech to infill). In terms of audio similarity, V oicebox is constantly better in the high noise condition with gains ranging from 0.265 to 0.324 compared to Demucs, and is on par with Demucs in low noise condition.

B.7 Additional results on inference efficiency versus performance

As explained in Section 5.4, for the cross-sentence setup of Section 5.1, we find that WER remains stable with mean of 2 . 0 and variance of 0 . 005 . This can be also be observed from Fig. B5a.

Voicebox

=0

=0.3

=0.7

VALL-E

2 1

2 2

2 3

2 4

2 5

NFE

2

3

4

5

6

WER

2 1

2 2

2 3

2 4

2 5

NFE

2.9

3.0

3.1

WER

(a) NFE vs WER (Zero-shot TTS)

(b) NFE vs WER (Divers speech sampling)

Figure B5: Trade-off between NFE and WER for different classifier-free guidance strengths (a) presents the WER for cross-sentence zero-shot TTS (Section 5.1) and (b) presents the WER for diverse speech sampling (Section 5.3).

In Fig. B5b, we show the WER for the samples generated on Librispeech test-other text. We find that for α = 0 , WER increases slightly from 2 . 8 to 3 . 1 as NFE goes from 2 to 32 . For a larger classifier-free guidance strength, WER remains more stable. Subjective listening and FSD reveal that 1) a lower NFE leads to less natural samples with lower diversity, and 2) a higher guidance weight leads to lower diversity. In addition, we observe that ASR can perform well with unnatural samples that contains artifacts, but degrades when the samples are more diverse and expressive (e.g., with whispering voice or with strong reverberation). As a result, we see that the combination of low guidance weight and high NFE leads to a higher WER due to the higher diversity.

B.8 Choice of audio model output features

The performance of our model is upper bounded by how well the chosen acoustic features can be reconstructed to waveform. The reconstruction performance is determined jointly by the encoding

process, as in how much information is lost when encoding waveform into the features, and the decoding process, as in how well the vocoder can translate the encoded information into waveform.

To motivate the choice of the acoustic feature and the vocoder, we compare four combinations: the first one is Mel spectrogram + HiFi-GAN which is what this paper adopts. The second is Mel spectrogram + Parallel WaveGAN [65] that is used by A3T [3]. The third one is Encodec postquantization dense feature + Encodec decoder, which is analogous to VALL-E's setup. The last one is also Encodec but with pre-quantization dense feature, which we include to study how much information is lost during quantization.

We also note that Mel spectrogram features are 80 dimensional encoded at 100Hz, which is 8K dimensions per second, while Encodec features are 128 dimensional encoded at 75Hz, which is 9.6K dimensions per second, higher than the Mel spectrogram features.

Table B7 presents the results evaluated on the Librispeech dev-clean and dev-other splits. All three models have the same WER resynthesizing dev-clean split, but ParallelWaveGAN degrades the most on dev-other. Interestingly Encodec even produces audio of lower WER than the ground truth.

In terms of audio similarity, besides the default audio feature extractor WavLM-TDCNN, we also include results of similarity computed with another speaker encoder ECAPA [13]. Parallel WaveGAN is consistently the worst. However, it is unclear whether HiFi-GAN or Encodec performs better. Encodec prevails with the WavLM-TDCNN embedder and HiFi-GAN wins using ECAPA. It may require subjective MOS test to conclude which one reconstructs the audio better, and we leave exploration of modeling Encodec dense features for future study.

C Additional Details and Studies on Metrics

C.1 Measuring speech diversity and quality with FSD

Diversity We first validate if FSD reflects the diversity for a set of speech samples and study its sensitivity to sample size. To achieve that, we design controlled experiments to compute FSD on sets of samples with varying diversity and sample sizes. Specifically, we create two partitions from 1K hours of English speech, where each partition has the same set of speakers and the same number of utterances for each speaker. The first partition is considered the reference set.

To test the sensitivity to sample size, we use the second partition to create subsets by sampling r % of utterances from each speaker in that partition. This sampling method is denoted as 'utt'. We computed that on average, each speaker contributed approximately 2 . 33 sessions, with each session containing around 52 . 45 utterances. Therefore, the subsets created using the sampling method are expected to have similar audio style distributions to the reference set and the FSD is expected to stay low regardless of the subset size. We consider r ∈ { 1 , 5 , 10 , 25 , 50 , 100 } .

To test the correlation with diversity, we again use the second partition to create subsets by sampling r % of speakers and including all the utterances in the partition from those speakers. This sampling method is denoted as 'spk' where a smaller r leads to a subset with fewer speakers and hence lower diversity. Therefore the FSD is expected to increase as r decreases. The same set of values for r is considered. For the same r , the 'utt' subset should always have a lower FSD than the 'spk' subset.

We compare three different features for computing the FSD score. The first is the supervised WavLM-TDCNN feature used for computing audio similarity (SIM-r and SIM-o). The second is the self-supervised wav2vec 2.0 BASE [2] feature reduced to 128 dimensions using principle component analysis (PCA). The last one is the supervised audio event classification model feature that is used to compute FAD [30] for non-speech audio generation.

Figure C6 first compares using different layers of wav2vec 2.0 features. All of them yield similar desirable results where 'utt' stays low and 'spk' increases drastically when the sample size reduces and speaker diversity decreases. We then decide to use the middle layer (layer 6) as the default feature for FSD computation.

10 0

10 1

10 2

0

50

100

150

200

250

layer-2

10 0

10 1

10 2

0

50

100

150

200

250

layer-4

10 0

10 1

10 2

0

50

100

150

200

10 0

10 1

10 2

ratio(%)

0

50

100

150

200

250

300

layer-8

utt

spk

10 0

10 1

10 2

ratio(%)

0

50

100

150

200

layer-10

10 0

10 1

10 2

ratio(%)

0

25

50

75

100

125

150

layer-12

Figure C7 further compares wav2vec 2.0-layer 6 with the two other features. WavLM-TDCNN and wav2vec 2.0-layer 6 present similar trends and both have low variance. Both of them are suitable for measuring diversity, and we decide to use wav2vec 2.0 features as it is self-supervised and would be able to capture more holistic information of speech such as prosody and emotion.

In contrast, FAD score [30] is not appropriate for measuring speech diversity. The score does not increase much between r = 25% and r = 1% for 'spk' sampling method, showing that the score does not reflect the decreasing speaker diversity. On the other hand, 'utt' sampling method observes huge FAD score increase when reducing the sample size from r = 25% to r = 1% where the diversity does not change much as the number of speakers remains the same. Moreover, at r = 1% both sampling methods result in similar FAD score while the two subsets exhibit very different levels of diversity. We hypothesize that this is because FAD score is computed based on features extracted from an audio even classifier trained on AudioSet, which learns to distinguish between events like lawn mower, car engine, and human speech, but does not learn to capture the variation within speech, such as different voices.

10 0

10 1

10 2

ratio(%)

0

1

2

3

WavLM-TDCNN

utt

spk

10 0

10 1

10 2

ratio(%)

0

50

100

150

200

Wav2vec 2.0-layer 6

10 0

10 1

10 2

ratio(%)

0

100

200

300

400

FAD

Quality In addition to measuring diversity, Fréchet distance is a commonly used metric for assessing quality in image generation [19]. To show its applicability for speech generation, we evaluate the FSD score of speech utterances with varying levels of quality. The reference set samples are 1K hours of English training data, and the hypothesis set is the Librispeech test-clean split with noise added. We added Gaussian noise at different SNRs, ranging from 0 to 50 dB. Lower SNR values correspond to lower quality. We use the default speech feature extractor (i.e., wav2vec 2.0, layer-6) throughout the experiments.

Our results, summarized in Figure C8, show that a subset with a lower SNR has a higher FSD score. Therefore, a lower FSD score indicates higher acoustic quality for the set of test samples when diversity is fixed.

0

10

20

30

40

50

SNR (dB)

0

500

1000

1500

2000

2500

3000

3500

C.2 Standalone metrics for duration models

As mentioned in the main text, we can utilize end-to-end metrics of WER, SIM, and FSD to evaluate duration models, but also consider metrics specifically for duration.

First, we consider two metrics aimed at the quality of duration predictions, here denoted ˆ l ( l ctx , y ) . For a regression model, we use ˆ l ( l ctx , y ) = g ( l ctx , y ; θ ) . For a flow matching model, we set ˆ l as the mean over 20 samples, ensuring a fairer comparison.

Duration correctness (MS-MAE) Our first metric, multi-sample mean-absolute error (MS-MAE), is the masked absolute error per-utterance divided by the average number of masked phonemes per-utterance

Speaking rate correlation (MS-Corr) Our next metric, multi-sample correlation (MS-Corr), computes the average masked predicted duration and unmasked duration context per utterance, and computes their correlation across utterances. Comparing MS-Corr with the same correlation computed from the ground truth, we observe to what extent predicted durations capture appropriate correlations with the context.

Duration diversity and quality (FDD) Additionally, we evaluate the quality and diversity of duration samples at the distribution level, similar to our audio evaluation of diversity and quality via FSD. We produce one sample per utterance from a duration model and collect all sampled phoneme durations, possibly many per-utterance, into an empirical distribution. We compare means and variances of this sampled distribution versus the means and variances of the training distribution, labeled µ , s , and µ ′ , s ′ respectively. We define the Fréchet duration distance (FDD) as the Fréchet distance between the distributions √

treated as though they were Gaussians. FDD depends on the sampled durations accurately reflecting the training distribution of real durations. As for FSD, this metric is specific to unconditional text-to-speech generation.

C.3 Duration model evaluation with standalone metrics

We evaluate three duration model variants. The first and second utilizes flow matching and regression, trained using masked conditional flow matching and regression respectively as described in Section 3.3. The third is a regression model that ignores duration context l ctx and only uses phonetic transcript y , referred to as unconditional regression below. This is the duration model used in FastSpeech2 [51], A3T [3] and many other non-autoregressive speech synthesis models.

We evaluate our three duration model variants on the Librispeech test-other split on two tasks. The first is unconditional TTS where we generate all durations from given phonemes (i.e. l ctx is entirely masked). The second task is infilling the second half of each utterance's durations, where l ctx are durations from the unmasked half of the utterance. This second infilling task distinguishes between the two regression model variants, since the unconditional regression ignores l ctx , and hence predicts identical durations for the tasks. Duration metrics are computed for TTS and infilling in Table C8 and C9. The prefix Phn or Sil indicates the associated metric was either computed across all nonsilence or all silence phonemes. Start and end silences were not trimmed for these duration metric evaluations.

Starting with prediction quality metrics (MS-MAE and MS-Corr), the duration-conditional regression performs slightly better on MS-MAE overall than the other models. Larger differences are seen on Phn-MS-Corr where the unconditional regression has a correlation substantively below the other models (Phn-MS-Corr of ground truth is 0 . 47 ), indicating conditioning on duration context l ctx is beneficial. Flow-matching shows the largest distinction versus regression on the distributional comparison captured by FDD. The regression models have generally larger FDD because they underestimate the standard deviation in phoneme and silence durations, and hence produce samples with less duration diversity and more regular duration lengths.

C.4 Duration model evaluation with end-to-end metrics

We now present end-to-end metrics for our three duration variants for zero-shot TTS cross-sentence and continuation, as well as diverse speech generation, corresponding to Sections 5.1 and 5.3. Zero-shot TTS cross-sentence and continuation results are shown in Table C11 and diverse speech generation results in Table C10. These results are not comparable with the main text as they utilize the flow-matching model described in Appendix B.3, denoted as VB-En-1K.

Overall, FSD and SIM are similar across duration variants. On the other hand, WER is sensitive to the choice of duration model, where the duration-conditional regression achieves a substantially lower WER. Subjective listening from the duration-conditional regression and flow-matching confirms that the regression model is producing more regular patterns of speech, that may be easier for ASR to recognize, while sacrificing some duration diversity.

C.5 MOSinstructions

Table C12 shows the instruction presented to the raters for quality mean opinion score study. Table C13 shows the instruction presented to the raters for similarity mean opinion score study.

D Limitation and Broader Impact

Limitation Voicebox models presented in this paper are trained on read speech from audiobooks in up to six written languages. Hence, the current models may not transfer well to conversational speech [15], which is more casual and contains more non-verbal sounds such as laughing and backchanneling (e.g., um-hmm). We plan to tackle the problem by scaling the training data to incorporate more diverse speech.

On the other hand, Voicebox depends on a phonemizer and a forced aligner to produce frame-level phonetic transcript. In addition, many existing phonemizers [40] are word-based, which does not take neighboring words of the target into account when predicting the pronunciation. Such phonemizers cannot accurately predict phonetic transcript given text because pronunciation is context-dependent in many languages (e.g., liaisons in French). In the future, we will explore more end-to-end methods where a model would be able to take raw text with punctuation as input [7], and eliminate the need of phonemizers and forced aligners to improve the performance and increase the language coverage.

Introduction

Your task is to evaluate the subjective quality and intelligibility of the speech from short (2-8 second) audio files. Each HIT can be completed in roughly around 120 seconds.

Task Instructions

In this task you will hear samples of speech recordings. The purpose of this test is to evaluate the quality and intelligibility of each file in terms of its overall sound quality and the amount of mumbling and unclear phrases in the recording.

Please keep in mind that speech samples can be distorted and noisy, however these are only specific examples.

Please use a headset for listening and adjust your volume level to your comfort during this training, and do not change later during the experiment.

You should give a score according to the following scale, known as the MOS (mean opinion score) scales:

Score (Quality and Intelligibility of the speech)

5 (Excellent)

4 (Good)

3 (Fair)

2 (Poor)

1 (Bad)

Last but not least, while V oicebox yields impressive results on transferring audio style (voice, speaking style, emotion, and acoustic condition), the model does not allow independent control of each attribute. In other words, one cannot ask the model to generate speech that resembles voice of one sample while resembling the emotion of another sample. We leave disentangled control of attributes through prompting or text description for future work.

Broader impact A high-quality and versatile generalist speech generation model like Voicebox can enable many applications that improve the quality of our life. For example, zero-shot TTS could bring the voice back to people who suffer from diseases or underwent surgeries such as laryngectomy the causes inability to speak. Zero-shot TTS can also be combined with visual speech recognition systems [23] to avoid the need of typing. When paired with speech translation models, cross-lingual zero-shot TTS enables everyone to speak any language in their own voice. Content editing and speech denoising can be productivity tools for users to create content more effortlessly. Diverse speech sampling, as shown in the paper, can significantly reduces the cost of creating data for training speech-input models.

While Voicebox can bring many positive social impacts, it also carries the potential of misuse and unintended harm. To mitigate the risk, we have presented a highly effective classifier showing that the

Table C13: Similarity mean opinion score (SMOS) instruction.

Task Name

Rate the similarity of the synthesized speech samples to a given prompt.

Task Instructions

Your task is to evaluate the similarity of the synthesized speech samples to the given speech prompt. You should focus on the similarity of the speaker, speaking style, acoustic conditions, background noise, etc. You should rank the recordings on the scale between 1-5, where 5 is the best quality and 1 is the worst.

In other words, please rank the recordings according to their acoustic similarity to the given prompt, meaning as if they were recorded in the same place by the same speaker speaking in similar styles. This task typically requires approximately 120 seconds to complete.

Please use a headset for listening and adjust your volume level to your comfort during this training, and do not change later during the experiment.

model can accurately distinguish between real and synthetic speech. For future work, we also plan to investigate proactive methods for training the generative model such that the synthetic speech can be more easily detected, such as embedding artificial fingerprints [66] that can be trivially detected without hurting the speech quality.

To prevent Voicebox from learning biases, we also need to carefully select its training data. First, if Voicebox is only trained on a smaller number of speakers from a specific group with similar accents, it will not be able to generate diverse speech representing the accents around the globe, and downstream models trained on Voicebox generated speech would perform worse on groups with underrepresented accents. For zero-shot style transfer, the performance would also degrade for underrepresented accents. To mitigate this, we have leveraged in-the-wild speech that includes a wide variety of accents, and will continue investing in collecting diverse speech to avoid such biases.

Second, if Voicebox is trained on data where samples from one ethnic group always have lower audio quality (e.g., more noise) while the other ethnic group always has higher audio quality samples, the model would also learn undesired association. To mitigate this, we want the distribution of audio quality (and other audio attributes) and ethnic group to be less correlated, which is usually the case when we have larger scale data collected from in-the-wild sources. We can further tackle this by leveraging data augmentation to decorrelate the distribution, such as adding noise and enhancing speech to widen the audio quality distribution.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work presents a text-conditional speech synthesis model with an optimal transport path and conditional flow matching. The model is trained with a large-scale cross-lingual dataset for zero-shot style transfer and content editing.\n\nSTRENGTHS:\nThe work adopts a conditional normalizing flow with an optimal transport path for speech synthesis. The proposed model can synthesize and edit the speech with different styles.\n\nWEAKNESSES:\nAlthough this work successfully executes the zero-shot TTS and speech editing, I have doubts on the originality of this work and I think that the authors should have conducted more comparisons with the recently proposed TTS models (not YourTTS, not Vall-E), and speech editing or speech correcting paper. \n\nI have some comments on this work.\n\n1. The concept to train the model by infilling speech given audio and text has been already presented in many works. First, SpecAugment [1] presented a time masking for context learning and Conformer [2] successfully adopted it for the ASR task. Self-supervised speech representation also models such as wave2vec 2.0 successfully learned the context with masking. Moreover, there are many similar models which can edit the speech conditional text information in speech editing [3] [4], correcting domains [5] [6], and also in the TTS domain [7] [8]. The authors only adopt the conditional normalizing flow with an optimal transport path for text-to-speech.  \n\n[1] Park, Daniel S., et al. \"Specaugment: A simple data augmentation method for automatic speech recognition.\" Interspeech, 2019.\n\n[2] Gulati, Anmol, et al. \"Conformer: Convolution-augmented transformer for speech recognition.\" Interspeech, 2020.\n\n[3] Tae, Jaesung, Hyeongju Kim, and Taesu Kim. \"EdiTTS: Score-based editing for controllable text-to-speech.\" Interspeech, 2022.\n\n[4] Wang, Tao, et al. \"Campnet: Context-aware mask prediction for end-to-end text-based speech editing.\" IEEE/ACM Transactions on Audio, Speech, and Language Processing 30 (2022): 2241-2254.\n\n[5] Tan, Daxin, et al. \"CorrectSpeech: A Fully Automated System for Speech Correction and Accent Reduction.\" ISCSLP, 2022.\n\n[6] Fong, Jason, et al. \"Speech Audio Corrector: using speech from non-target speakers for one-off correction of mispronunciations in grapheme-input text-to-speech.\" Proc. Interspeech 2022 (2022): 1213-1217.\n\n[7] Ao, Junyi, et al. \"Speecht5: Unified-modal encoder-decoder pre-training for spoken language processing.\" ACL, 2022.\n\n[8] Wang, Tao, et al. \"Non-Autoregressive End-to-End TTS with Coarse-to-Fine Decoding.\" INTERSPEECH. 2020.\n\n2. Specifically, the diffusion-based model (EdiTTS [3] and Guided-TTS [9]) can edit the speech even without training. The author should have compared the conditional normalizing flow with a diffusion-based model. However, the authors only referred that OT path leads to faster training, faster generation, and better performance compared to diffusion paths. I wonder if the conditional normalizing flow with OT path is better than diffusion-based models.       \n\n[9] Kim, Heeseung, Sungwon Kim, and Sungroh Yoon. \"Guided-tts: A diffusion model for text-to-speech via classifier guidance.\" ICML, 2022.\n\n3. The audio quality is not good on the demo pages.\n\n4. For a fair comparison, all models should be trained with the same dataset.\n\n5. YourTTS is not a good text-to-speech model for zero-shot text-to-speech models. The audio quality of YourTTS is bad. I recommend training the VITS with the reference encoder you used and the same configuration. In addition, transferring the voice style from a reference audio, not speaker ID, is utilized in many works. For a fair comparison, the same transferring method is used for each model.\n\nQUESTIONS:\n1. The authors utilized a 100 Hz frame rate to extract the high-resolution Mel-spectrogram. For a fair comparison, each model is also trained with the same time resolution for Mel-spectrogram. For example, VITS and YourTTS should be trained with the linear spectrogram extracted at a 100 Hz frame rate for high-quality. However, the details are not described for baseline models.\n\n2. Are there any scenarios for automatic noise removal or editing? In this work, the location of audio for noise removal or editing should be segmented by the user. [5] presented the fully automated speech correction scenario with detection, correction, and generation. It would be nice incorporating ASR with this work for an automatic speech editing system. \n\n3. I think replacing the HiFi-GAN with BigVGAN could improve the audio quality. \n\n4. For the AR model, the dropout in the pre-net of the decoder may improve the diversity of speech. Also, VITS and YourTTS can increase the sample diversity by controlling the temperature T. The details you used should be described.\n\nLIMITATIONS:\nThey stated the limitations and potential negative societal impact on Section Conclusion.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-29"}

comments: SUMMARY:
The paper proposes voicebox, a text-guided generative model for speech at scale. By abstracting many speech tasks into speech infilling tasks, voicebox is able to conduct zero-shot text to-speech synthesis, noise removal, content editing, style conversion, and diverse sample generation in monolingual or crosslingual scenario. Experiments show that equipped with the advanced technology such as CNF and classifier-free guidance, voicebox show better performance than baseline methods on many tasks,

STRENGTHS:
1. The paper abstracts many different speech tasks into speech infilling task.
2. The paper scales up the model to 50K hours of speech and multilingual scenario, where the model is able to perform in-context learning for speech style.
3. The proposed method is well-studied on many tasks including zero-shot TTS, speech enhancement, speech editing and style generation. The experiments on ASR also show the potential value of proposed method.

WEAKNESSES:
About the experiments in Section 5.2, it is unfair to compare VB with speech enhancement model since VB can have access to the text of noised speech, which significantly help the model to generate speech with lower WER. Also, the text of noised speech is usually not accessible in the conventional formulation of speech enhancement.

QUESTIONS:
1. As Table 5 shows, VB is able to generate speech with different styles. Can this feature be used in Table 6? To be more specific, if VB synthesize more than one sample per text from the Librispeech training set, can the corresponding WER be further reduced?

2. If there is any insight for using Continuous Normalizing Flows to modeling speech infilling rather than other generative models such as diffusion models?

LIMITATIONS:
The authors have discussed the limitations in conclusion.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-03

comments: SUMMARY:
This paper proposed Voicebox, a non-autoregressive flow-matching model designed to infill speech by leveraging given audio context and text. Notably, Voicebox capitalizes on a substantial amount of data, consisting of 50,000 hours of speech, which contributes to its impressive performance across various speech generation tasks. The model demonstrates notable capabilities in generating coherent and high-quality speech outputs.

STRENGTHS:
1. The paper exhibits a comprehensive and well-executed evaluation, yielding impressive experimental results.
2. The authors have effectively presented their work with clear and accessible writing, ensuring ease of understanding for readers.
3. The proposed framework showcases a high level of flexibility, enabling its application to various speech generation tasks and settings.

WEAKNESSES:
1. The demos provided in the supplementary materials are not as satisfying as described in the paper. For example, issues can be found in the perceived speaker similarity in the zero-tts task and the quality of the articulation position of the mask. It would be beneficial if the authors engage in further discussions regarding these phenomena.
2. Given that speech encompasses various components (such as prosody, content, timbre, and noise), it would be better to have a more comprehensive discussion on how Voicebox specifically handles these different aspects.

QUESTIONS:
It is unclear from the paper what motivated the authors to choose the flow-matching model as the audio model.

LIMITATIONS:
Authors are encouraged to add some discussions about the potential negative social impact.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes VoiceBox, a speech infilling model based on a flow-matching generative model. VoiceBox is trained to fill in masked speech based on unmasked speech and given text, and it can perform various tasks depending on how the mask is applied to the speech during inference. VoiceBox allows for speech editing by masking the speech corresponding to the text portion that needs to be edited, and it can also perform zero-shot TTS by infilling the speech for the desired text. By masking the entire speech during generation, the model can generate voices from various speakers in a speaker-unconditional manner for a given sentence. This paper demonstrates significantly improved performance compared to existing models in various tasks and even shows that a generative model for speech can aid in performance improvement in speech recognition tasks through diverse speech generation.\n\nSTRENGTHS:\n* This paper demonstrates the versatility of VoiceBox across various tasks while achieving impressive performance.\n* This paper demonstrates that utilizing synthesized speech generated by VoiceBox can improve the generalizability of ASR models, thereby showing its ability to effectively model the distribution of general speech.\n* In the zero-shot TTS task, VoiceBox particularly outperforms the previous state-of-the-art model, VALL-E, by a significant margin.\n* The use of a flow-matching generative model enables fast sampling.\n* The extensive experimental results provide strong support for the advantages of the model.\n\nWEAKNESSES:\n* For zero-shot TTS and speech editing, both the duration model and audio model require prompting with the transcript of the reference speech as well as the duration per phoneme. This necessitates the use of MFA (Montreal Forced Aligner) during inference.\n\nQUESTIONS:\n* Despite VALL-E already demonstrating impressive zero-shot TTS performance, VoiceBox appears to have excessively higher SIM-o and SIM-r scores. It is difficult to determine that VoiceBox's zero-shot TTS samples are significantly better than VALL-E when listening to only the VoiceBox samples in the demo. For a one-to-one comparison with VALL-E, would it be possible to provide VoiceBox samples generated with the same prompt as VALL-E's demo samples?\n\n* In order to calculate SIM-r as mentioned in the paper, were both the reference speech prompt and all generated samples encoded and decoded using Encodec? If calculated differently, please provide an explanation. Additionally, when measuring SIM-o or SIM-r with the reference speech, was the similarity measured with the speech prompt only or with the entire reference speech?\n\n*\n\nLIMITATIONS:\nThis paper provides additional experiments on a classifier designed to detect potential misuse in order to mitigate such risks.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}

comments: SUMMARY:
The authors present Voicebox, the text-guided generative model for speech at scale. Voicebox is a non-autoregressive flow-matching model trained to infill speech, given audio context and text. Voicebox can be used for mono or cross-lingual zero-shot text-to-speech synthesis, noise removal, content editing, style conversion, and diverse sample generation.

STRENGTHS:
It is interesting that Voicbox performs in-context learning via the masking strategy. Besides, the authors train the model in extensive data and present the SOTA results in several downstream tasks.

WEAKNESSES:
1. One of the weaknesses should be novelty. It sounds like the combination of masked-based A3T and flow-based speech synthesis model. Besides, duration prediction models and classifier-free guidance are not new.
2. Evaluation. Firstly, why present a portion of MOS in Table 2? VITS-LJ and VITS-VCTK are compared as the baseline in the LibriTTS test-other set, but why not train in the LibriTTS dataset instead? The differences in data amount could encounter unfair comparison. 
3. Unclear presentation. Voicebox still needs to provide a clear illustration. Do you need tags for different task inferences? A3T finds that it is challenging to generate speech given full-mask samples, and what do you find is the most suitable proportion of masked and unmasked regions?

QUESTIONS:
1. It could be more natural to use diffusion models since the masking strategy is similar to adding noise, so what is your consideration in using flow-matching models?
2. Citations [42] and [43] are the same

LIMITATIONS:
/
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-11