SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Matthew Le* Apoorv Vyas* Bowen Shi* Brian Karrer* Leda Sari Rashel Moritz
Mary Williamson Vimal Manohar Yossi Adi† Jay Mahadeokar Wei-Ning Hsu*
Large-scale generative models such as GPT and DALL-E have revolutionized the research community. These models not only generate high fidelity outputs, but are also generalists which can solve tasks not explicitly taught. In contrast, speech generative models are still primitive in terms of scale and task generalization. In this paper, we present Voicebox, the most versatile text-guided generative model for speech at scale. Voicebox is a non-autoregressive flow-matching model trained to infill speech, given audio context and text, trained on over 50K hours of speech that are not filtered or enhanced. Similar to GPT, Voicebox can perform many different tasks through in-context learning, but is more flexible as it can also condition on future context. Voicebox can be used for mono or cross-lingual zero-shot text-to-speech synthesis, noise removal, content editing, style conversion, and diverse sample generation. In particular, Voicebox outperforms the state-of-the-art zero-shot TTS model VALL-E on both intelligibility (5.9% vs 1.9% word error rates) and audio similarity (0.580 vs 0.681) while being up to 20 times faster. Audio samples can be found in https://voicebox.metademolab.com.
Recent advances in large-scale generative models [6, 42, 50] have led to a major paradigm shift towards building general-purpose models, which can perform many new tasks not explicitly trained on. These generative models learn to predict the missing data given the context. Post training, we can directly input a question, optionally with a few contextual question-answer examples, instead of fine-tuning with labeled data. While the training objective appears simple, it subsumes many tasks as one can convert them into some form of context. For the model to perform well at every task, it implies that the estimation of  $p$  (missing data | context) needs to be accurate for every context. Hence, scale and diversity are the most crucial factors for building general-purpose models [20, 1].
Despite the success of large-scale generative models in other areas, most speech models are still trained on datasets at the scale of tens to hundreds of hours [51, 31, 32, 46, 24, 58, 7]. Previous works consider highly curated datasets such as VCTK [64], which contains only clean audio recorded in studio from about 100 speakers with little speaking style and text variation. Such models struggle to synthesize speech with rich variation in emotion, voice, background noise, acoustic condition, and have not been tested on the abilities to generalize to tasks not explicitly trained on.
This paper presents Voicebox, the most versatile text-conditioned speech generative model at scale. Voicebox is trained on a text-guided speech infilling task, where the goal is to generate masked speech
*Equal contribution. Corresponding authors: {mattle, wnhsu}@meta.com
$^{\dagger}$ FAIR & Hebrew University of Jerusalem.
Figure 1: Task generalization via in-context learning.
Table 1: Comparing Voicebox with baselines on task capabilities. *Through infilling, A3T and Voicebox can remove transient noise but not stationary background noise.
Model	ZS TTS	Denoise	Edit	Sampling
YourTTS	✓	X	X	✓
VALL-E	✓	X	X	✓
A3T	✓	*	✓	X
Demucs	X	✓	X	X
Voicebox	✓	*	✓	✓
given its surrounding audio and text transcript. This can be considered as a guided in-context learning problem, where audio style is inferred from the audio context and textual content is specified through transcript. Voicebox does not require any audio style labels (e.g., speaker, emotion, and noise), which differentiates Voicebox from the majority of prior work where such labels are used extensively. Prior work uses labels to make the mapping between input (text and audio style) and output (speech) more deterministic to reduce underfitting [60, 46]. We show that Voicebox's text-guided speech infilling approach is much more scalable in terms of data while subsuming many common generative tasks.
In terms of modeling, Voicebox is a non-autoregressive (NAR) continuous normalizing flow (CNF) model [10]. Similar to diffusion models [19], CNFs model the transformation from a simple distribution to a complex data distribution, parameterized by a neural network. We train Voicebox with flow-matching [38], a recently proposed method that enables efficient and scalable training of CNFs via a simple vector field regression loss. In contrast to auto-regressive models, Voicebox can consume context not only in the past but also in the future. Moreover, the number of flow steps can be controlled at inference time to flexibly trade off quality and runtime efficiency.
Voicebox is trained on 60K hours of English audiobooks and 50K hours of multilingual audiobooks in 6 languages for the mono and multilingual setups. Voicebox achieves state-of-the-art performance on mono-lingual/cross-lingual zero-shot TTS, speech denoising, speech editing, diverse speech sampling and an application to data creation for speech recognition. To tackle the lack of comparability due to the use of subjective metrics, this paper presents a series of metrics using public models to facilitate reproducible comparison and model development for speech generation studies.
Generative speech models: Most speech generative models are task-specific and trained on different datasets. One common type of task is audio style conversion, which aims to convert only a specific attribute while keeping other attributes the same. Voice conversion [27, 39], emotion conversion [53, 34], speech enhancement [63, 11, 55] belong to this category. Many of these models are supervised and trained on pairs of data that only differ in one attribute, for example, emotion [34]. It is hard to obtain such data. Moreover, some attributes, such as speaking style, are hard to annotate. Hence, these models are often trained on small datasets.
Controllable text-to-speech synthesis (TTS) is another common task, which aims to synthesize speech in a target audio style given text. While some styles like voice can be specified through labels [32] or pre-trained embeddings like YourTTS [7] and Jia et al. [25]; others like prosody are hard to annotate or embed. Previous studies [62] tried to control them by learning a residual embedding. However, these models encode style in a low-dimensional space and impose an overly simple distribution of speech given text and residual embedding [51, 56]. They cannot generate realistic noisy speech given a low dimensional vector, and performance degrades when conditioned on noisy references [21].
Infilling can be considered as another type of task. It aims to predict speech given context [36, 4] and optionally text guidance [3, 5, 61]. Instead of learning an explicit embedding to control style, infilling models predict speech coherent to the context. In other words, these models perform in-context learning similar to Large Language Models (LLM). While this is a step toward building large
scale generalist models using little explicit supervision, most prior work using text guidance still assumes a deterministic mapping from text and context to target [3, 5], which is only realistic for very short segments. Voicebox is a text-guided infilling model, but it leverages the CNF model that can parameterize any distribution. Hence, Voicebox can infill speech of any length and can be trained on in-the-wild datasets with rich variation, and provide a general solution that subsumes many tasks in a text-guided fashion.
Large scale in-context learning models: With the advancement in neural codec for speech [22, 12, 67], many recent studies explore token-based language modeling for speech generation. The GSLM-family [36, 28, 41] are textless language models built upon HuBERT units [22] for speech continuation without using text. HuBERT units encode mostly content, and the generated speech does not preserve the voice of the prompt. To tackle this, AudioLM [4] considers a cascaded approach which first generates HuBERT-like tokens and then predicts SoundStream [67] tokens, a reconstruction based codec that preserves style. These models are not conditioned on text and are evaluated on spoken language modeling tasks.
VALL-E [61] is most related to Voicebox. It is a text conditioned LM trained on Encodec [12] tokens (similar to SoundStream). Encodec encodes each frame with 8 ordered codebooks at  $75\mathrm{Hz}$  using a residual quantization layer. VALL-E has two modules. The first is an auto-regressive (AR) model that predicts the first code of each frame given text and the audio prompt. The second is an NAR model that predicts the remaining seven codebooks sequentially.
VALL-E demonstrates state-of-the-art (SOTA) zero-shot TTS performance through in-context learning, where speech of the desired style is used as prompt. The model considers the prompt as part of the whole utterance such that it generates the rest of the utterance containing the target text in the same audio style. Voicebox has several design advantages compared to this. 1) Voicebox can use context both in the past and future, which is useful for editing where only a segment in the middle needs to be generated. 2) Voicebox can generate speech much faster than VALL-E because flow-matching can produce high quality samples with less than 10 NAR steps, while VALL-E requires 1 AR and 7 NAR steps. 3) Voicebox decouples duration and audio modeling, enabling finer grained alignment control. 4) Voicebox is compatible with any continuous features including Encode embeddings.
3.1 Background: Flow Matching with an optimal transport path
Let  $\mathbb{R}^d$  be the data space with data points  $x\in \mathbb{R}^d$  drawn from some unknown distribution  $q(x)$ . Continuous Normalizing Flows (CNFs) [10] are a family of generative models that learn the transformation from a simple prior distribution  $p_0$  (e.g., normal distribution) to the data distribution  $p_1\approx q$ . CNFs parameterize a time-dependent vector field  $v_{t}:[0,1]\times \mathbb{R}^{d}\to \mathbb{R}^{d}$  that is used to construct a flow:  $\phi_t:[0,1]\times \mathbb{R}^d\to \mathbb{R}^d$  that pushes points from the prior towards the target distribution. The relationship is defined via the ordinary differential equation (ODE) as:  $d\phi_t(x) / dt = v_t(\phi_t(x))$  and  $\phi_0(x) = x$ . For a flow  $\phi_t$ , the probability path (time-dependent probability density function)  $p:[0,1]\times \mathbb{R}^d\to \mathbb{R}_{>0}$  can be derived via the change of variables formula:  $p_t(x) = p_0(\phi_t^{-1}(x))\operatorname *{det}\left[\partial \phi_t^{-1}(x) / \partial x\right]$ . To sample from  $p_t(x)$ , we first draw  $x_0$  from  $p_0$  and then solve the initial value problem (IVP) for  $\phi_t(x_0)$  given  $d\phi_t(x) / dt = v_t(\phi_t(x))$  and  $\phi_0(x) = x_0$ . We use  $x_{t}$  and  $\phi_t(x_0)$  interchangeably.
Let  $p_t$  be a probability path and  $u_t$  be the corresponding vector field that generates  $p_t$ . The vector field  $v_t(x; \theta)$  parameterized by a neural network  $\theta$  can be trained with the Flow Matching objective:  $\mathcal{L}_{FM}(\theta) = \mathbb{E}_{t, p_t(x)} ||u_t(x) - v_t(x; \theta)||^2$ , where  $t \sim \mathcal{U}[0,1]$  and  $x \sim p_t(x)$ . While the objective appears simple, in practice we do not have the prior knowledge of  $p_t$  or  $v_t$ , and cannot directly compute the loss or its gradient estimator.
Let  $x_{1}$  be a random variable distributed according to data distribution  $q$ . Lipman et al. [38] first notes that a probability path  $p_t(x)$  can be constructed via a mixture of simpler conditional paths  $p_t(x \mid x_1)$  whose vector field  $u_t(x \mid x_1)$  can be easily computed. To construct  $p_t(x)$ , a conditional path is defined such that 1)  $p_0(x \mid x_1) = p_0(x)$  and 2)  $p_1(x \mid x_1) = \mathcal{N}(x \mid x_1, \sigma^2 I)$ , a Gaussian distribution centered at  $x_{1}$  with a sufficiently small  $\sigma$  (typically  $10^{-5}$ ). The marginal path is computed as  $\int p_t(x \mid x_1) q(x_1) dx_1$ , which closely approximates  $q(x_{1})$  at  $t = 1$ . With that, [38] presents the Conditional Flow Matching (CFM) objective,  $\mathcal{L}_{CFM}(\theta) = \mathbb{E}_{t,q(x_1),p_t(x|x_1)}||u_t(x \mid x_1) - v_t(x;\theta)||^2$ .
It is proven that FM and CFM have identical gradients w.r.t.  $\theta$ . More importantly, one can easily draw samples from  $p_t(x \mid x_1)$  and compute  $u_t(x \mid x_1)$  to derive an unbiased gradient estimator.
The next question is how to choose a conditional flow. A flow defines trajectories, which dictates how each point moves between  $p_0$  and  $p_1$ . Intuitively, a simpler trajectory (e.g., a straight line) can be learned faster and the IVP can be solved more accurately and efficiently. Lipman et al. [38] presents a conditional flow called optimal transport (OT) path, which has the form of  $p_t(x \mid x_1) = \mathcal{N}(x \mid tx_1, (1 - (1 - \sigma_{\min})t)^2I)$  and  $u_t(x \mid x_1) = (x_1 - (1 - \sigma_{\min})x) / (1 - (1 - \sigma_{\min})t)$ . The flow is arguably simple because points move with a constant speed and direction. We adopt it for Voicebox.
Lipman et al. [38] also presents another flow that recovers the path of diffusion models [57], which is more complex than the OT path. We will present ablation studies comparing different paths (OT vs diffusion) and different objectives (CFM vs score-matching). Results show the superiority in performance and efficiency of CFM with OT path.
Given a dataset of transcribed speech  $(x,y)$  where  $x$  and  $y$  denote an audio sample and its transcript, respectively, the goal is to build a single model that can perform many text-guided speech generation tasks through in-context learning. We propose to train such a generative model on the text-guided speech infilling task, which predicts a segment of speech given its surrounding audio and the complete text transcript. Let  $m$  be a binary temporal mask which is of the same length as  $x$ , and  $x_{mis} = m \odot x$  and  $x_{ctx} = (1 - m) \odot x$  be the complementary masked versions of  $x$ . The generative model learns  $p(x_{mis} \mid y, x_{ctx})$ . In other words,  $y$  and  $x_{ctx}$  are the context and  $x_{mis}$  is the missing data.
(a) Voicebox audio model. The lower half illustrates how inputs are created during training.
(b) Inference is equivalent to solving an ODE with an initial condition  $x_0$  sampled from the prior, a derivative  $\frac{dx_t}{dt}$  specified by the audio model, and conditional inputs  $(z, x_{ctx})$ .
Figure 2: Illustration of Voicebox training and inference.
Motivated by the need that some applications require fine-grained alignment control between speech and text, we decouple Voicebox into two components: an audio model and a duration model. Let  $x = (x^{1},x^{2},\dots ,x^{N})$  be an audio sample of  $N$  frames,  $y = (y^{1},y^{2},\dots ,y^{M})$  be a text sequence of  $M$  phones, and  $l = (l^1,l^2,\dots ,l^M)$  be the per-phone duration where  $l^j$  denotes how many audio frames  $y^{j}$  correspond to and  $\sum_{j = 1}^{M}l^{j} = N$ . We further define  $z = \mathsf{rep}(y,l) = (z^1,z^2,\dots ,z^N)$  to be the frame-level phone transcript, which repeats each  $y^{j}$  by  $l^j$  times such that  $z^i$  denotes the phone label of the audio frame  $x^{i}$ . For a pair of  $(x,y)$ ,  $l$  and  $z$  can be estimated through forced alignment using a speech recognition model. The estimation of  $q(x_{mis}\mid y,x_{ctx})$  is then broken down into the audio model  $q(x_{mis}\mid z,x_{ctx})$  and the duration model  $q(l_{mis}\mid y,l_{ctx})$ , where  $l_{mis}$  and  $l_{ctx}$  denote  $l$  masked by  $m'$  and  $1 - m'$ , and  $m'$  is downsampled from  $m$  based on  $l$ , detailed in Appendix A.2.
Audio Model: Given a context  $z$  and  $x_{ctx}$  of length  $N$ , the distribution of  $x_{mis}$  is highly stochastic especially when  $x_{mis}$  has a large temporal span. Hence, we parameterize it with a CNF and train
it using the flow matching objective with the optimal transport path. Audio  $x$  is represented as an 80-dimensional log Mel spectrogram  $(x^{i} \in \mathbb{R}^{80})$  extracted at a  $100\mathrm{Hz}$  frame rate. The audio context  $x_{ctx}^{i} = 0$  where  $m^{i} = 1$  and  $x_{ctx}^{i} = x^{i}$  where  $m^{i} = 0$ . For simpler conditioning, we model the conditional distribution  $q(x \mid z, x_{ctx})$  of all frames  $x$  instead of only masked frames  $x_{mis}$ . A neural network is used to parameterize the conditional vector field  $v_{t}(x_{t}, x_{ctx}, z; \theta)$  that additionally takes  $x_{ctx}$  and  $z$  as input. Note that  $x_{t}$  is a sample at flow step  $t$  and  $x = x_{1}$ .
Given as input  $x_{ctx} \in \mathbb{R}^{N \times F}$ ,  $x_{t} \in \mathbb{R}^{N \times F}$ , phone sequence  $z \in [K]^{N}$  with  $K$  denoting the number of phone classes, and a time step  $t \in [0,1]$ , we employ a Transformer model to parameterize the vector field  $v_{t}$ . A lookup table, denoted as  $L \in \mathbb{R}^{K \times H}$ , is used to embed the phone sequence  $z$ , resulting in the embedded sequence  $z_{emb} \in \mathbb{R}^{N \times H}$  where  $z_{emb}^{i} = L(z^{i})$  for  $i \in 1, \ldots, N$ . Subsequently, the three sequences  $(x_{t}, x_{ctx},$  and  $z_{emb}$ ) are concatenated frame-by-frame and projected by employing matrix  $W_{p} \in \mathbb{R}^{(2F + H) \times D}$ , thereby obtaining the sequence  $H_{c} \in \mathbb{R}^{N \times D}$  where  $D$  represents the embedding dimension of the Transformer model.
To embed the flow step, a sinusoidal positional encoding is applied to map  $t \in [0,1]$  to  $h_t \in \mathbb{R}^D$ . The sequence  $\tilde{H}_c \in \mathbb{R}^{(N + 1)\times D}$ , which serves as the input to the Transformer model, is derived by concatenating  $H_{c}$  with the vector  $h_t$  along the time dimension. Given the Transformer output  $v_{t}(x_{t},x_{mis},z;\theta) \in \mathbb{R}^{N\times F}$ , which is the sub-sequence corresponding to  $H_{c}$ , the loss is computed as:
$$
\mathcal {L} _ {\text {a u d i o - C F M}} (\theta) = \mathbb {E} _ {t, m, q (x, z), p _ {0} (x _ {0})} | | u _ {t} (x _ {t} \mid x) - v _ {t} (x _ {t}, x _ {c t x}, z; \theta) | | ^ {2}, \tag {1}
$$
by reparameterization. During training, given an audio sample  $x$  and a prior sample  $x_0$ , we have  $x_{t} = (1 - (1 - \sigma_{\min})t)x_{0} + tx$  and  $u_{t}(x_{t}\mid x) = x - (1 - \sigma_{\min})x_{0}$ . This function computes the loss on all frames, including those that are not masked and would not be required during inference. To divert the model's focus to masked frames, we present a masked version of  $\mathcal{L}_{\mathrm{audio-CFM}}$ :
$$
\mathcal {L} _ {\text {a u d i o - C F M - m}} (\theta) = \mathbb {E} _ {t, m, q (x, z), p _ {0} (x _ {0})} \| m \odot \left(u _ {t} \left(x _ {t} \mid x\right) - v _ {t} \left(x _ {t}, x _ {c t x}, z; \theta\right)\right) \| ^ {2}, \tag {2}
$$
where the loss is only computed on masked frames. Appendix B.3 shows it leads to better results
Duration model: We consider two solutions. The first one closely follows the audio model. It models  $q(l \mid y, l_{ctx})$  via a conditional vector field which swaps  $(x, x_{ctx}, z)$  with  $(l, l_{ctx}, y)$  and accordingly for the flow, where  $l, l_{ctx} \in \mathbb{R}^{M \times 1}$  and  $y \in [K]^M$ . The masked version of the CFM loss is used for training. On the other hand, previous studies have shown that regression duration models can produce reasonable speech [51, 37]. Hence we consider a second solution that regresses the masked duration  $l_{mis}$  given the context duration  $l_{ctx}$  and phonetic transcript  $y$ . The same Transformer model is used, except that there are only two input sequences instead of three, and the time embedding is not used. The model is trained with an  $L_1$  regression loss on masked phones:
$$
\mathcal {L} _ {\text {d u r -} \operatorname {g r e g - m}} (\theta) = \mathbb {E} _ {m, q (l, y)} | | m ^ {\prime} \odot \left(l _ {\text {m i s}} - g \left(l _ {\text {c t x}}, y; \theta\right)\right) | | _ {1}, \tag {3}
$$
where  $g$  denotes the regression-based duration model. This is similar to the duration model used in FastSpeech2 [51], but with additional duration context  $l_{ctx}$  as input.
To sample from the the learned audio distribution  $p_1(x \mid z, x_{ctx})$ , a noise  $x_0$  is first sampled from  $p_0$ , and then an ODE solver is used to evaluate  $\phi_1(x_0)$  given  $d\phi_t(x) / dt = v_t(\phi_t(x), x_{ctx}, z; \theta)$  and the initial condition  $\phi_0(x_0) = x_0$ . Intuitively, the ODE solver computes  $\phi_1(x_0)$  by evaluating  $v_t$  at multiple  $t$  to approximate the integration from  $t = 0$  to  $t = 1$  given the initial condition  $\phi_0(x_0) = x_0$ . The number of function evaluation (NFE) is defined as how many times  $d\phi_t(x_0) / dt$  is evaluated. A higher NFE often leads to a more accurate solution of  $\phi_1(x_0)$  at the cost of longer run time. This provides great flexibility for users to decide the trade-off between speed and accuracy. Moreover, we find that empirically Voicebox can already generate very high quality speech with less than 10 NFEs, making it significantly faster compared to auto-regressive models.
3.5 Classifier-Free Guidance
Classifier guidance (CG) [14] is a technique used to trade off mode coverage and sample fidelity for diffusion models post training. It modifies the score estimate of a diffusion model to include the gradient of the log likelihood of an auxiliary classifier. Ho and Salimans [18] notes that CG approximates sampling from  $p(x \mid c)p(c \mid x)^{\alpha}$  where  $c$  is the conditioner, and this can be simulated
without a classifier by mixing the score estimate of a conditional model and an unconditional model. The unconditional model can be jointly trained by dropping the conditioner  $c$  with some probability, and the same model provides score estimates for both  $p(x)$  and  $p(x \mid c)$ .
We extend the idea of classifier free guidance (CFG) to flow-matching models. The conditioner  $c$  is equivalent to  $(z, x_{ctx})$  for audio models and  $(y, l_{ctx})$  for duration models, which is dropped with  $p_{\text{uncond}}$  during training. During inference, the modified vector field  $\tilde{v}_t$  for the audio model becomes  $\tilde{v}_t(w, x_{mis}, z; \theta) = (1 + \alpha) \cdot v_t(w, x_{ctx}, z; \theta) - \alpha \cdot v_t(w; \theta)$ , where  $\alpha$  is the strength of the guidance, and  $v_t(w; \theta)$  is obtained by dropping  $x_{ctx}$  and  $z$ . We use  $\alpha$  and  $\alpha_{dur}$  for the CFG strengths for the audio and the duration model, selected based on validation. Note that the computation is doubled for the same NFE when using CFG, because the model forward is called twice to compute  $\tilde{v}_t$ .
3.6 Applications
We demonstrate that Voicebox exhibits in-context learning abilities similar to LLMs by presenting a few examples of how to create context to perform tasks. Voicebox was not explicitly trained on. Fig. A1 shows a detailed diagram of how inputs are formatted for each task.
Zero-shot TTS & alignment-preserved style transfer: Given a target text  $\hat{y}$  and a transcribed reference audio  $(x,y)$ , zero-shot TTS aims to synthesize speech resembling the possibly unseen audio style of the reference. Voicebox performs the task by treating the reference audio and the target speech as one utterance where the target speech is masked. Let  $l$  and  $z$  be phone duration and frame-level transcript of  $(x,y)$ . The target duration  $\hat{l}$  is sampled given the duration context  $l$  and concatenated phone sequence  $\operatorname{cat}(y,\hat{y})$ . The target speech  $\hat{x}$  is then sampled given the context  $x$  and concatenated frame-level phones  $\operatorname{cat}(z,\operatorname{rep}(\hat{y},\hat{l}))$ .
Voicebox can also convert the audio style for speech  $\bar{x}$  while preserving its alignment  $\bar{z}$ . This is useful for editing audio that is synchronized with other modalities such as video. Similar to zero-shot TTS, Voicebox can simply perform the task by sampling target speech  $\hat{x}$  given the context  $x$  and concatenated frame-level phones  $\operatorname{cat}(z, \bar{z})$