Audiobox: Unified Audio Generation with Natural Language Prompts
Authors: name: Apoorv Vyas, name: Bowen Shi, name: Matthew Le, name: Andros Tjandra, name: Yi-Chiao Wu, name: Baishan Guo, name: Jiemin Zhang, name: Xinyue Zhang, name: Robert Adkins, name: William Ngan, name: Jeff Wang, name: Ivan Cruz, name: Bapi Akula, name: Akinniyi Akinyemi, name: Brian Ellis, name: Rashel Moritz, name: Yael Yungster, name: Alice Rakotoarison, name: Liang Tan, name: Chris Summers, name: Carleigh Wood, name: Joshua Lane, name: Mary Williamson, name: Wei-Ning Hsu

Audio is an essential part of our life, but creating it often requires
expertise and is time-consuming. Research communities have made great progress
over the past year advancing the performance of large scale audio generative
models for a single modality (speech, sound, or music) through adopting more
powerful generative models and scaling data. However, these models lack
controllability in several aspects: speech generation models cannot synthesize
novel styles based on text description and are limited on domain coverage such
as outdoor environments; sound generation models only provide coarse-grained
control based on descriptions like "a person speaking" and would only generate
mumbling human voices. This paper presents Audiobox, a unified model based on
flow-matching that is capable of generating various audio modalities. We design
description-based and example-based prompting to enhance controllability and
unify speech and sound generation paradigms. We allow transcript, vocal, and
other audio styles to be controlled independently when generating speech. To
improve model generalization with limited labels, we adapt a self-supervised
infilling objective to pre-train on large quantities of unlabeled audio.
Audiobox sets new benchmarks on speech and sound generation (0.745 similarity
on Librispeech for zero-shot TTS; 0.77 FAD on AudioCaps for text-to-sound) and
unlocks new methods for generating audio with novel vocal and acoustic styles.
We further integrate Bespoke Solvers, which speeds up generation by over 25
times compared to the default ODE solver for flow-matching, without loss of
performance on several tasks. Our demo is available at
https://audiobox.metademolab.com/