ZeroSep: Separate Anything in Audio with Zero Training
Authors: name: Chao Huang, name: Yuesheng Ma, name: Junxuan Huang, name: Susan Liang, name: Yunlong Tang, name: Jing Bi, name: Wenqiang Liu, name: Nima Mesgarani, name: Chenliang Xu

Audio source separation is fundamental for machines to understand complex
acoustic environments and underpins numerous audio applications. Current
supervised deep learning approaches, while powerful, are limited by the need
for extensive, task-specific labeled data and struggle to generalize to the
immense variability and open-set nature of real-world acoustic scenes. Inspired
by the success of generative foundation models, we investigate whether
pre-trained text-guided audio diffusion models can overcome these limitations.
We make a surprising discovery: zero-shot source separation can be achieved
purely through a pre-trained text-guided audio diffusion model under the right
configuration. Our method, named ZeroSep, works by inverting the mixed audio
into the diffusion model's latent space and then using text conditioning to
guide the denoising process to recover individual sources. Without any
task-specific training or fine-tuning, ZeroSep repurposes the generative
diffusion model for a discriminative separation task and inherently supports
open-set scenarios through its rich textual priors. ZeroSep is compatible with
a variety of pre-trained text-guided audio diffusion backbones and delivers
strong separation performance on multiple separation benchmarks, surpassing
even supervised methods.