Towards Diverse and Efficient Audio Captioning via Diffusion Models
Authors: name: Manjie Xu; name: Chenxing Li; name: Xinyi Tu; name: Yong Ren; name: Ruibo Fu; name: Wei Liang; name: Dong Yu
Published: 2024-09-14 10:23:35

Abstract: We introduce Diffusion-based Audio Captioning (DAC), a non-autoregressive
diffusion model tailored for diverse and efficient audio captioning. Although
existing captioning models relying on language backbones have achieved
remarkable success in various captioning tasks, their insufficient performance
in terms of generation speed and diversity impede progress in audio
understanding and multimedia applications. Our diffusion-based framework offers
unique advantages stemming from its inherent stochasticity and holistic context
modeling in captioning. Through rigorous evaluation, we demonstrate that DAC
not only achieves SOTA performance levels compared to existing benchmarks in
the caption quality, but also significantly outperforms them in terms of
generation speed and diversity. The success of DAC illustrates that text
generation can also be seamlessly integrated with audio and visual generation
tasks using a diffusion backbone, paving the way for a unified, audio-related
generative model across different modalities.