AudioX: Diffusion Transformer for Anything-to-Audio Generation
Authors: name: Zeyue Tian, name: Yizhu Jin, name: Zhaoyang Liu, name: Ruibin Yuan, name: Xu Tan, name: Qifeng Chen, name: Wei Xue, name: Yike Guo

Audio and music generation have emerged as crucial tasks in many
applications, yet existing approaches face significant limitations: they
operate in isolation without unified capabilities across modalities, suffer
from scarce high-quality, multi-modal training data, and struggle to
effectively integrate diverse inputs. In this work, we propose AudioX, a
unified Diffusion Transformer model for Anything-to-Audio and Music Generation.
Unlike previous domain-specific models, AudioX can generate both general audio
and music with high quality, while offering flexible natural language control
and seamless processing of various modalities including text, video, image,
music, and audio. Its key innovation is a multi-modal masked training strategy
that masks inputs across modalities and forces the model to learn from masked
inputs, yielding robust and unified cross-modal representations. To address
data scarcity, we curate two comprehensive datasets: vggsound-caps with 190K
audio captions based on the VGGSound dataset, and V2M-caps with 6 million music
captions derived from the V2M dataset. Extensive experiments demonstrate that
AudioX not only matches or outperforms state-of-the-art specialized models, but
also offers remarkable versatility in handling diverse input modalities and
generation tasks within a unified architecture. The code and datasets will be
available at https://zeyuet.github.io/AudioX/