ZeroSep: Separate Anything in Audio with Zero Training
Authors: Chao Huang
Yuesheng Ma
Junxuan Huang
Susan Liang
Yunlong Tang
Jing Bi
Wenqiang Liu
Nima Mesgarani
Chenliang Xu
Venue: NeurIPS 2025 poster ()
Keywords: Audio Separation
Zero-shot
Open-set
Training-free
Primary area: applications

Abstract: Audio source separation is fundamental for machines to understand complex acoustic environments and underpins numerous audio applications. Current supervised deep learning approaches, while powerful, are limited by the need for extensive, task-specific labeled data and struggle to generalize to the immense variability and open-set nature of real-world acoustic scenes. Inspired by the success of generative foundation models, we investigate whether pre-trained text-guided audio diffusion models can overcome these limitations. We make a surprising discovery: zero-shot source separation can be achieved purely through a pre-trained text-guided audio diffusion model under the right configuration. Our method, named ZeroSep, works by inverting the mixed audio into the diffusion model's latent space and then using text conditioning to guide the denoising process to recover individual sources.  Without any task-specific training or fine-tuning, ZeroSep repurposes the generative diffusion model for a discriminative separation task and inherently supports open-set scenarios through its rich textual priors. ZeroSep is compatible with a variety of pre-trained text-guided audio diffusion backbones and delivers strong separation performance on multiple separation benchmarks, surpassing even supervised methods.

Separate Anything in Audio with Zero Training
Chao Huang', Yuesheng , Junxuan Huang', Susan Liang', Yunlong Tang', Jing Bi', Wenqiang Liu?, Nima Mesgarani', Chenliang Xu' University of Rochester, "Columbia University, Teencent America
Abstract
Audio source separation is fundamental for machines to understand complex acous- tic environments and underpins numerous audio applications. Current supervised deep learning approaches, while powerful, are limited by the need for extensive, task-specific labeled data and struggle to generalize the immense variability and open-set nature real-world acoustic scenes. Inspired by the success of generative foundation models, investigate whether pre-trained text-guided audio diffusion models can overcome these limitations. We make a surprising discovery: zero-shot source separation can be achieved purely through pre-trained texl-guided audio diffusion model under the right configuration. Our method, named ZeroSep, works by inverting the mixed audio into the diffusion model's latent space and then using text conditioning to guide the denoising process to recover individual sources. With- out any task- specifp training or fine-tuning. ZeroSep repurposes the generative diffusion model for discriminative separation task and inherently supports open- set scenarios through its rich textual priors. ZeroSep compatible with variety of pre-trained text-guided audio diffusion backbones and delivers strong separa- tion performance on multiple separation benchmarks, surpassing even supervised methods. Our project page is here: https://wikichan gi thub. io/ ZeroSep/
Introduction
At the heart of acoustic scene perception lies the fundamental task of source separation, which aims to isolate individual sound sources from complex audio mixture. Accurate source separation is crucial for a wide range of applications, including media production, surveillance systems, automatic speech recognition in noisy environments, and analysis of complex soundscapes.
The dominant approach audio separation in recent years has relied heavily on supervised learning: deep neural networks are trained on large datasets of paired mixtures and clean sources [Luo and Mesgarani] 2019 Subakan al. 202T]. While these methods have achieved impressive performance on specific. well-represented source types and datasets, they often fall short when faced with the open- variability of real-wer world acoustic scenes. Consequently, training a foundation-lewe separation model becomes exceptionally challenging due to the need for vast amounts of labeled data, the difficulty of defining training objectives and mixing strategies, and the design of effective conditioning mechanisms.
Recent efforts, such LASS-nee DesS-netLiia Liu et al. 2022a AudioSep Liu al. 2023a and FlowSep Yin et al. 2024), have explored leveraging natural language queries for more flexible separation. Despite these advances, they still contend with the same core challenges: vast data requirements, complex task-specific training regimes, and limited generalization to unseen acoustic scenes. Inspired by the transformative success of large language models in unifying diverse NLP tasks under generative framework [Brown et al. 2020]. we pose central question: Can generative foundation model similarly emerge for audio tasks? In this work, we explore this question by investigating the capabilities of pre-trained text-guided audio diffusion models.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

We discover that text- text-guided audio diffusion model can, out of the box, separate a mixture into its sources no training or fine-tuning, relying solely on latent inversion and conditioned denoising: (i) Given a mixed audio signal, we can find corresponding point in the diffusion model's latent space through an inversion process. This latent representation captures the composite information from all the sound sources present in the mixture. (ii) Subsequently, by guiding the generative denoising process from this latent state using text prompts corresponding to individual sources in the mixture, the model can be steered to reconstruct each source in isolation. Surprisingly, even though this is generative process, the separated sources are highly faithful to the original sources, especially with classifier-fre free guidance < 1, which prevents hallucination. This effectively repurposes the generative model for a discriminative task, offering a fundamentally different approach to separation.
Based on the above observations, we introduce ZeroSep, a zero-training troo-trannin framework for audio source separation that repurposes pretrained text-guided diffusion models. By casting separation as two- step generative inference, latent inversion followed by text-conditioned denoising, ZeroSep offers three key advantages:
Open-set Separation: As the core of ZeroSep is pre-trained text-guided audio diffusion model, which has learned to generate realistic audios from diverse, open-domain descriptions and mixing styles, ZeroSep naturally handles open-set queries and is able to separate from diverse mixtures. Model-agnostic Versatility: The inversion plus denoising pipeline is generic to diffusion architectures, allowing ZeroSep to leverage different pre-trained audio diffusion backbones. Interestingly. we observe a trend that the better the audio diffusion model can generate, the better it can separate, which could suggest continuous improvement whenever there is a more advanced audio generation model available.
Training-Free Efficacy: Without any fine-tuning task-specific data, ZeroSep matches or exceeds the performance of existing training- based generative separators, overturning the assumption that high-quality separation requires dedicated training.
In summary, our contributions to the community includes
1. We introduce ZeroSep, a training- audio suuree-epparation framework that repurposes pre-trained text-guided diffusion models, representing a fundamental shift away from super- vised separation paradigms.
2. We demonstrate that pure generative inference lateence--eeett inversion followed by text- conditioned denoising -eiiids ing training-based generative methods.
yields state-of-the-aa separation performance, outperforming exist-
3. We establish ZeroSep 's versatility and open-set capability: it seamlessly handles diverse mixtures and textual queries and can be applied to many pre-trained audio diffusion back- bones, improving separation quality as the underlying model's generative fidelity increases.
2 Related Works
Diffusion probabilistic models have rapidly emerged as a leading paradigm for generating high-quality and diverse audio content. Early works like DiffWave (Kong al. 2021] and WaveGrad [Chen et al.] [021a] demonstrated the potential of applying denoising diffusion to synthesize raw audio waveforms, achieving high-fidelity unconditional audio generation. Building on this foundation, diffusion models were successfully extended to conditional audio generation tasks. In text-to-speech (TTS), models such as Diff-TTS [Jeong et al. 202T and Grad-TTS [Popov et al. 2021] showed that diffusion processes could generate high-fidelity mel-spectrograms conditioned on text input. Researchers also focused on improving the efficiency and controllability of diffusion sampling: for instance, Guided-Tie Kim et al. (2022) introduced classifier guidance for TTS, and PriorGrad [Lee al. 2022) addressed sampling speed in vocoders through data-dependent priors. Diffusion models have also been applied to other audio synthesis tasks, including singing voice synthesis with DiffSinger [Liu et al., (022b) and waveform super-resolution with NU- (Lee] and Han 2021]. More recently, the focus has shifted towards latent-space diffusion models and text-conditioned generation of general audio. AudioLDMM [Liu et al., 2023b) pioneered combining diffusion with CLAP embeddings to enable text-conditioned generation of diverse sounds and music. AudioLDM2 [Liu et al. 2024) and Tango [Ghosal et al.) 2023) further advanced this direction, providing enhanced control and quality. These text-condittoned latent diffusion models [Evans et al.
Audio Diffusion Models.

---

2025a capable of generating complex audio scenes from natural language. form the technological foundation for our training-free separation method ZeroSep.
Audio Separation. The problem of source separation has long been tackled by both classic signal-processing techniques and, more recently, deep learning. Traditional methods such NMF- MFCC [Stöter et (021] decompose mixtures under assumptions about timbral or spectral structure. While training-free free, they often fail on complex heavily overlapping sources that lack clear dis- tinguishing features. Deep learning revolutionized the field by learning representations directly from data. Deep Clustering [Hershey et al., 2016) trains embeddings for clustering source-specific time Lime-frequency bins, and Permutation-Invaatt Invariant Training (PIT) Yu et al. 2017 resolves the label- permutation problem during training. Conv- TasNet (Luo and Mesgarani 2019] further advanced performance with end-to end-to-end waveform separation, frequently surpassing traditional masking ap- proaches. Diffusion-barn baased approaches such as SepDiff [Chen et al., 2023 and DiffSep (Scheibler et al. 2023 reformulate separations as generative problem. However, these models remain "blind" to user intent: once trained, they separate every detectable component rather than targeting a spe- cific source. To introduce controllability, recent works condition separation on auxiliary modalities. Video-guided methods [Huang et al. 2024a] use visual cues, while language-basee frameworks, such LASS-Net [Liu al. 2022a AudioSep (Liu et al. 2023a and FlowSep [Yin et al. [024)) leverage text prompts to guide mask estimation. Although more flexible, they still require large supervised corpora of synthetic mixtures, inheriting closed- world biases. Zero-shot diffusion editors like AUDIT [Wang et al.) 2023], DITTO [Novack et al.) [2024), and AudioEdit [Manor and Michaeli 2024] fine-tune or invert latent trajectories delete components, but focus editing rather than explicit separation.
In contrast, ZeroSep repurposes pre-trained text-guided audio diffusion model as universal, training-free (Postolache et al. 2024] prior for open-set separation. By (i) inverting an audio mixture into the model's is latent space and (ii) re-denoising under user-provided text prompts with unit classifier- free guidance, ZeroSep generates one isolated waveform per prompt, achieving comprehensive, zero-shot source separation without fine-tuning.
3 Method
In this section, we first review the foundational knowledge of text-guided diffusion models and diffusion inversion techniques, which form the basis of our method. Next, we discuss the separation task setup with generative diffusion models. Lastly, we introduce ZeroSep, zero-shot separation adaptation of existing text-guided audio diffusion models.
3.1 Preliminary: Text-Guided Audio Diffusion and Inversion
Text-guided audio diffusion models typically operate in a learned latent space: An initial audio signal is first encoded into latent representation, denoted as Zo; the forward diffusion process progressively adds Gaussian noise to this latent vector, transforming it into a pure noise vector Ør. A neural network, parameterized by set of 0, learns to predict and remove the noise added at each step t, effectively reversing the diffusion process and generating mel-spectrograms which are then converted to waveforms using vocoder" In text-guided models, this denoising process is driven by text condition c, derived from a text encoder, ensuring the generated audio aligns with the text prompt.
DDIM Inversion. To enable manipulation of existing audio content, inversion techniques are used to map real audio sample back into the noisy latent space. common approach DDIM inversion, which leverages the deterministic nature of DDIM sampling [Song et al. 2020). The standard DDIM sampling process iteratively denoises a noisy latent x; to produce a less noisy version x,-1
X
X₁ +
(V 1 ā
-1 co cxt, V
1
'This pipeline underlies many audio diffusion models, such as the AudioLDM family [Liu et al 2023b and Tango Ghosal et al. 2023), but does not apply to architectures like Stable Audio Open Evans et al.
2024
2025a
3

---

where {ār defines the noise schedule and ca(-, c, is the model's noise prediction conditioned on c. DDIM inversion reverses this process, estimating the noisy latent X from x,:
X
X
-
(V
ac+1
-11 Eo [xe c,t).
(2)
so that iterating from Xy recovers an estimate of the pure noise xy. Cumulative errors can, however, cause deviations from the true noise trajectory.
DDPM Inversion. In contrast to DDIM inversion, DDPM inversion Hluberman-SSiegelle et al. 2024 leverages the probabilistic forward diffusion to obtain an exact noise path. Given clean latent Xo. one constructs an auxiliary sequence of noisy latents
Vā X + vI - āi Eτ, & & t= I....,T,
(3)
and then extracts the per-step noise vectors
2, X - 11-1M((X)) (x(x))
1= T,....,,
0, where f4 (x+) and or follow the DDPM simply re-injects X7 and } via
Ho et al. 2020 reverse-step definitions. Reconstruction
pr ) + ot Zt:
(S)
exactly recovering By scaling or replacing {z,) (e.g. using text embeddings at select timesteps), DDPM inversion offers a probabilistic framework for precise, text-guided edits.
From a high-level perspective, both DDIM and DDPM inversion can be viewed as a single mapping, which we denote by Xy Finy (Xo, c). This operator Fine encapsulates the step-wise recovery of the noise trajectory corresponding
given clean latent. Whether implemented via the deterministic updates of DDIM the probabilistic steps of DDPM, Finy produces the pure noise Xy that, when re-injected into the standard diffusion sampler, exactly reconstructs the original sample Xy-
3.2
Task Setup
In real-world scenarios, audio stream be mixture of N individual sound sources: where each source s(*) can be of various categories. To work in the diffusion latent space, we first convert a to a mel-spectrogram and encode with a Variational Autoencoder ( (VAE), yielding latent features C where C' is the number of channels, F the numbers of frequency bins, and T the number of time frames. Let mix denote the VAE encoding of the mixture and (( the encoding of source i. Our goal is to find a separation mapping f (xxmix, c(f) - x(9), where c(i) conditioning signal (e.g., text description) that specifies which source extract. x(() is then fed to the VAE decoder and Vocoder to convert latent features back to waveform level to obtain g(f)
3.3 From Generation to Separation: The ZeroSep Principle
The core of ZeroSep lies in repurposing a pre-trained text-guided audio diffusion model, originally designed for generating audio from text, to perform the task of audio source separation.
Let Cinv be the prompt used during the inversion process (mapping the mixed audio mix to noisy latent X,), and Cze be the text prompt used during the subsequent reverse denoising process (reconstructing clean source from the noisy latent).
Diffusion models typically employ classifier-free guidance during denoising, where the noise predic- tion at step / is combination of an unconditional prediction and conditional prediction guided by Crev
cx Ø,t) + (Eo(Xi, Crev, t) cx t)
(6)
Here, 60 (x,, Ø, t) is the unconditional noise prediction, cx C t) is the prediction guided by Grvv and w is the classifier-fir free guidance weight controlling the influence of the text condition Crev-
While this formulation typically used to amplify the presence the desired content during generation, we discover that specific choices of Cinv, Crev. and w, enable effective source separation. This shifts the model's 's function from synthesizing new audio to dissecting existing mixtures. Here the key principles for transforming the generative process into separation tool:

---

Cjov
Blind: null prompt 0
Cinv
prompt
T timest
Nolsy Latent X
VLM/ALM prompt
(a) Inversion
x
mii
Crev
Separation
c
x
Crev
T times
(c) Effect of Prompts
LDM
Estimate Noise é
ruf 0.6)
XT
si (eal Srevi ca(s, 0.4)
Separation
Generation
LDM
Update Xe
Separated sound Xg
Value
(b) Separation
(d) Effect of guidance
Figure 1: The overview of ZeroSep, which includes (a) an inversion process to obtain a latent representation for the mixture, and (b) separation denoising process effectively extract the target source with text conditions. We show the choice of inversion prompt Cinv and reverse prompt Crev in (c), and demonstrate the valid separation region defined by w in (d).
The Reverse Prompt Crev: To isolate a specifie source the reverse denoising prompt Cze must explicitly describe that target source:
Crev
separating source
This directs the denoising process to reconstruct the audio components associated with the target source described by c( Using any other prompt would result in guided generation, not separation.
The Inversion Prompt Ciav: The inversion prompt Ciav influences how the mixed audio is mapped to the noisy latent space. We found flexibility here, with effective choices including a null prompt Ø or prompts describing the other sources present the mixture (c(3) for j / i). While describing other sources can potentially refine the latent representation by emphasizing non-target components, requires prior knowledge of the mixture's s contents, yet can be achieved with user query or by prompting Vision- Language Models Audio Language Models (as shown in Fig. [T)).. A simpler and often effective approach is to use a null prompt (Cms = Ø) as the default. This inverts the mixed signal based the model's 's general audio understanding without imposing specific content constraints during the inversion phase. The effect of Cinv and Crev can be found in Tab. 9
The Crucial Role of Guidance Weight wt key discovery that achieving separation hinges on setting the classifier-free guidance weight w appropriately, specifically w - 1. This is counter- intuitive to typical generative usage where high w values (e.g., w = 3.5 for AudioLDM2 [Liu et al.) 2024 amplify the conditional signal for strong generation. In our context, when using Cinv Ø:
Setting 0 removes the conditional influence, effectively leading to reconstruction of the
original mixed audio.
Setting w removes the unconditional noise estimation from the combined prediction in Eq. 6 leaving only the component aligned with the target source described by Czev- This effectively isolates the target souree during denoising.
Settinn w 1, in standard generation, overly amplifies the conditional signal, leading to the synthesis of new content rather than the separation of existing audio components.
We empirically find that = yields the best separation results (as shown in Fig. B(a)). This finding reveals that controlling the balance between conditional and unconditional predictions via is critical for steering the diffusion process from generation towards faithful separation.
By carefully selecting Cine, Crev, and setting (in practice, we set 1), we effectively repurpose the pre-trained audio diffusion model's generative capabilities to perform high-quality source separation without requiring any task-specific training.
5

---

3.4 Why Do T2A Diffusion Models Exhibit Zero-Shot Separation Ability?
The emergent source separation capability in Text-to-Audio (T2A) diffusion models is rooted in two core pillars: the latent disentanglement achieved during polyphonic training. and the repurposing of the score function via Classifier-Free Guidance (CFG) for an inverse problem.
Latent Disentanglement via Data Compositionality, T2A diffusion models are trained to estimate the probability density of multi-label, polyphonic audio (e.g., AudioSet (Gemmeke et al.) 2017]. with an average of 2.7 concurrent labels per clip). To accurately generate a complex mixture, the model must implicitly learn the compositional struciure of the acoustic environment. The model is forced to encode disentangled latent factors corresponding to individual sources (s(1), ..... s(N) such that their superposition correctly predicts the mixture x This compositional training establishes an inherent, semantically-alignee structure in the latent space, where the representation of the mixture x xii is a combination of the latent representations of its individual sources. This prerequisite knowledge makes the inverse task of decomposition feasible.
The Score Function View and Guided Filtering. The separation task is realized by repurposing the score-based generative process. The T2A model estimates the conditional score function via the noise prediction Eg, which typically combined using Classifier- The procedure for separation involves two steps: first, the mixture audio Xmix is mapped into the noisy latent space via DDIM Inversion. Second, the key to separation is setting the guidance weight w 1. This condition simplifies the guidance to (x(x,, c) co (x,, c), effectively removing the unconditional score component cx 2) (which models the mixture's 's density) from the denoising process. By solely following the conditional score estimate, the model's reverse process is steered to follow the gradient direction leading to the data distribution consistent only with the target source defined by the text prompt c. This mechanism thus transforms the generative model into a zero-shot guided filtering tool for source extraction.
Guidance (illustrated in Eq. (6))
Experiments
4.1
Experimental Settings
Baselines. To evaluate our training- free diffusion- separation method, we compare against two categories of existing approaches: (i) Training-based methods. These methods rely on large-scale supervised training and leverage text queries for targeted separation. We include: LASS-Ner [Liu et al., 2022a which conditions mask estimator text queries; AudioSep [Liu et al. 2023a scaled-up version of LASS-Net trained on massive multimodal data for zero-shot capabilities across diverse sources; and FlowSep (Yin et al. 2024]. which enhances query-based separation using rectified continuous normalizing flows. We note that AUDIT [Wang et al. 2023] also uses audio diffusion models for cntructtionguidded audio editing (including source manipulation) but is not included as a direct baseline due to the lack of public code and data release for comparison. (ii) Training-free methods. These methods perform separation without requiring task task-specific training data. We compare against: NMF-MFCC [Stöter et al. 2021]. a classical non-negative matrix factorization approach operating MFCC features for blind source separation; and AudioEditor [Manor and Michaeli 2024), which achieves unsupervised separation by discovering principal components within the denoising process of a pre-trained diffusion model. In summary, training- based baselines require extensive annotated data for training, whereas other training-free baselines employ different underlying principles from our generative diffusion-based approach.
Datasets. We evaluate the open-set separation capabilities of our training-free method on two bench- mark multimodal datasets with paired audio and text labels: The Audio-Visuaa Visual Event (AVE) (Tian] et al. 2018) dataset contains 4,143 video clips, each 10 seconds long, covering 28 distinct sound categories (e.g., church bell, fryinn. AVE is valuable for evaluating separation in complex, real-world scenarios due to the presence of background noise, off-screen sounds, and varying event durations. The MUSIC dataset [Zhao et al., 2018) consists of clean solo performances from 11 musical instruments, thereby offering controlled environment to assess the separation of individual, isolated sources with minimal interference. To facilitate comparison with prior research and ensure reproducibility, we use the official separation data splits for both AVE and MUSIC as provided by the
DAVIS repository (Huang et al. 2024a

---

Table 1: Main audio separation results comparing ZeroSep with training- based and training- free baselines on the AVE (Tian et al. 2018 and MUSIC (Zhao et al. 2018 datasets. Metrics are reported on the test sets. f indicates higher is better, + indicates lower better. The best results are bold. More metrics can be found in Appendix [ Method
MUSIC
AVE
FAD 4
LPAPS
C-A
C-T
FAD
LPAPS 4
C-A1
C-T
Require Separation Training
LASS-Net
1.039
5.602
0.204
0.014
0.626
6.062
0.232
0.011
AudioSep
0.725
5.209
0.450
0.204
0.446
5.733
0.457
0.167
FlowSep
0.402
5.578
0.564
0.245
0.258
4.719
0.493
0.082
Separation Training Free
NMF-MFCC.
1.286
5.618
0.239
-0.055
1.246
5.851
0.174
0.211
AudioEdit
0.568
4.869
0.453
0.196
0.372
4.959
0.341
0.074
ZeroSep (Ours)
0.377
4.669
0.615
0.271
0.269
4.537
0.442
-0.001
Separation Prompt: "dog bark"
Mixture
GT
ZeroSep (Ours)
LASS-Net
AudioEdit
AudioSep
FlowSep
Figure 2: Qualitative visualization of audio separation results. The figure shows the input mixture (containing speech and dog barking) and the separated "dog barking" source produced by different baselines and ZeroSep. ZeroSep, guided by the text prompt "dog bark", successfully isolates the target sound, demonstrating its effectiveness compared to baseline methods. More separation results can be found in the appendix.
Evaluation Metries. Traditional separa-
Method
SDR
SIR
SAR
tion metrics metrics-Signal-tor Sognallto-Distoreee Ratio (SDR),
Original VAE+ + Vcoder
0.31
0.31
149.90
Signal-to-Interfereen Ra tio (SIR), and Signal- to-Artifact Ratio (SAR) [Raffel et al. 2014]- quantify sample-level differences between a sep- arated output and the ground truth s. These metrics assume the output lies the same wave- form manifold as s, an assumption violated by generative models that may produce perceptu- ally accurate but sample-we wise divergent signals. Tab. 2 demonstrates how a AAcodo reconstruc- tion of the mixture yields misleadingly poor SDR/SIR/SAR scores. Such an issue previously studied in [aayaram and Thickstun) 2020].
23.07
00.79
19.09
Table 2: Breakdown of SDR/SIR/SAR (with re-
spect to the individual target source), for a gen- erative reconstruction of the mixture versus the original mixture.
To capture perceptual and semantic fidelity of generative separation, we adopt metrics in embedding spaces: Frechét Audio Distance (FAD) [Kilgour al. 2018): measures the distance between embedding distributions of separated and ground-truth audio. Learned Perceptual Audio Patch

---

Table 3: Evaluation of AudioLDM [Lu al. 2024]. AudioLDM2 Liu et al., 2024 and
Tango [Ghosal et al. 2023] on the MUSIC and AVE benchmarks. We compare two U-Net sizes for each AudioL.MM variant-S (181 M) L. (739 M) and AudioL.DM2-S (350 M)
(750
M), and Tango's 866 M-parameter U-Net. Results are reported for both DDIM and DDPM inversion methods, and for AudioL we include full vs. music-only training data.
Model
Size
Data
MUSIC
AVE
FADL
LPAPS!
C-At
C-T1
FADL
LPAPS.
C-At
C-T1
DDIM Inversion
S
Full
0.460
4.690
0.562
0.284
0.275
4.821
0.484
0.114
Audiol.DM
L
Full
0.470
4.625
0.577
0.260
0.253
4.742
0.490
0.102
Full
0.421
4.630
0.375
0.261
0.251
4.560
0.477
0.039
AMd
Music
0.439
4.620
0.584
0.259
0.325
4.666
0.424
0.106
L
Full
0.377
4.669
0.615
0.271
0.269
4.537
0.442
-0.001
Tango
L
Full
0.606
4.511
0.544
0.204
0.724
4.451
0.437
0.077
DDPM Inversion
Full
0.417
4.580
0.605
0.300
0.239
4.681
0.504
0.133
AudioLDM
L
Full
0.388
4.536
0.626
0.283
0.266
4.629
0.496
0.108
S
Full
0.390
4.586
0.595
0.238
0.272
4.546
0.488
0.041
Audiol.DMr
Music
0.384
4.596
0.609
0.259
0.238
4.628
0.467
0.126
L
Full
0.397
4.581
0.598
0.239
0.267
4.523
0.445
-0.088
Tango L
Full 0.539
4.474
0.581
0.189
0.723
4.471
0.451
0.032
Similarity (LPAPS) [Manor and Michaeli, 2024]: evaluates perceptual audio similarity in learned embedding space. CLAP-A and CLAP-T [Yin et al. 2024]: CLAP-A is the cosine similarity between audio embeddings of the separation output and the ground-truth source; CLAP-T is the cosine similarity between audio embeddings and the text embedding of the target class. These feature- based metrics better reflect perceptual quality and semantic alignment, addressing the shortcomings of waveform-level reference metrics for generative audio separation. We further assess whether the model truly separates the original source content merely generates audio with similar semantic qualities that may not remain consistent with the ground-truth mixture (Appendix A).
4.2 Main Comparison
Tab. presents the core results of our evaluation, comparing the performance our training- frre method, ZeroSep, against representative training-based and other training-free free baselines the AVE and MUSIC datasets. Remarkably, ZeroSep demonstrates performance that surpasses the leading supervised methods, effectively challenging the necessity of large-scale supervised training for state-of-the-an audio separation. On the MUSIC dataset, ZeroSep outperforms the strongest training- graseing-based based baseline FlowSep across all metrics. On the more complex and open-domain AVE dataset, ZeroSep achieves performance comparable to FlowSep.
The training- based baselines show clear improvement trend with increasing model size and data: LASS-Net is surpassed by AudioSep, which in turn is surpassed by FlowSep. underscoring the benefits of extensive supervised training data and better models. The other training-free methods evaluated, NMF-MFCC and AudioEditor, yield substantially lower performance than the top supervised methods, highlighting the difficulty of achieving high-quality separation without leveraging separation training, which ZeroSep has addressed.
Beyond quantitative scores, the qualitative visualization in Fig. 2 provides further evidence of ZeroSep's effectiveness, illustrating the successful separation target sound (e.g.. "dog barking") from complex mixture containing other sources like human speech. These results collectively indicate that pre-trained text-guided diffusion models possess powerful inherent capabilities that can be effectively harnessed for audio separation without the need for task-specific training.

---

199/5
C.12 Saule
AVE cutaset
MUSIC dutasd
410
0.00
120
-
1075
430
0.0
0.5
20
1.0 23 2.1
1.0
Vakke
Figure 3: (a) Impact of guidance weight w: increasing w from 0 to improves separation metrics (LPAPS and CLAP-A), whereas w degrades performance below the mixture baseline (w 0), underscoring the critical role of w. (b)-(c) Positive correlation between separation quality (normalized all scores from Tab. 3 and generative capability (normalized FAD scores on AudioCap [Liu et al.) 2023b). [Liu et al.) (0244) ) across AudioLDM variants, indicating that stronger generation can potentially lead better separation.
4.3 Ablation Studies
In this section, analyze the influence of various components on ZeroSep's separation performance to identify factors contributing to its effectiveness. We investigate aspects including the choice and capacity of the base generative model, the impact of its training data domain, inversion strategies, guidance weight effects, and prompt selection. More analysis can be found in the appendix.
How Does the Base Generative Model Affect Separation? Since ZeroSep is built upon a pre- trained diffusion model, understanding how this base model affects separation is crucial. First, we compare separation performance using different base model architectures, including models from the AudioLDM [Liu et al. [2023b] AudioLDM2 [Liu et al. (024). and Tango [Ghosal et al. 2023 families, as shown in Tab. 3 The results indicate that various base models can yield separation performance comparable to the best training-basse -baeed baseline, FlowSep, demonstrating versatility in base model selection. Specifically, models from the AudioLDM and AudioLDM2 families generally outperform Tango in this separation task.
Second, analyze the effect of model capacity by comparing different sizes within the AudioLDM and AudioLDME families (e.g., AudioLDMMS vs. AudioLDM--L. AudioLDMM2-S vs. AMdioLDM2-LL. As shown in Tab. B increasing model size consistently leads to improved separation performance. This suggests positive correlation between the generative power of the base model and its effectiveness for separation. We further visualize this trend by plotting the correlation between generative performance (measured by FAD) and separation metrics in Fig. 3 3b) and (c), which confirms that stronger generative models tend to yield better separation results.
Third, we investigate the impact of the base model's training data domain. Tab. 3 includes results for model trained specifically MUSIC data compared to the same model trained a broader data corpora. We observe that the MUSIC-data-traine better than the full-training model for certain metrics (e.g., LPAPS
model achieves performance on the MUSIC
dataset that is similar to and CLAP-A). This finding suggests that the target separation domain is narrow, a base generative model trained domain-specifee data can be sufficient even advantageous, potentially increasing the separation flexibility.
Inversion Strategy. As shown in Tab. both DDIM and DDPM inversion methods enable competi- tive separation performance relative to the baselines. Analyzing their behavior across different base model capacities and training data domains, observe that DDPM inversion tends to yield more stable metrics, exhibiting less fluctuation with respect to changes In contrast, DDIM inversion shows larger variations under these different conditions. This analysis indicates that ZeroSep's ' effectiveness is not strictly tied to a single inversion technique, offering flexibility in implementation.
model size and training data.
Effect of w. As detailed in Sec. 3.3 setting 3 = 0 effectively reduces the process to an unconditional reconstruction, while higher values increase the adherence to the text prompt Crev- We analyze the impact of w on separation performance by evaluating values in the set {0, 0.5, 1, 1.5, 2}. Fig. 3 a) presents the results for LAPAS and CLAP-A metrics. It can be observed that as w increases from

---

Table 4: Effect of Ciuy and Crev on separation metrics. Triangles indicate change relative to the baseline, with V denoting improvement and V denoting degradation.
MUSIC
AVE
Ciav
FAD 4
LPAPS 4
C-A
C-T
FAD 4
LPAPS
C-A
C-T
0.377
4.669
0.615
0.271
0.269
4.537
0.442
0..01
8
random
0.577
4.900
0.363
0.125
0.325
4.749
0.289
0.019
0,200
0.231
0.252
0.146
0.056
0.212
0.153
0.020
c(3)
e
0.454
4.547
0.581
0.254
0.321
4.599
0.496
0.055
0.077
0.122
0.034
0.017
0.052
0.062
0.054
0.056
O to 1, both separation metrics generally improve, indicating that conditioning on the target source prompt effectively guides the separation. However, beyond = 1, performance deteriorates sharply, suggesting that excessively strong guidance can lead to suboptimal reconstructions or introduce artifacts. Based on this analysis, we empirically set w = 1 for our main experiments to achieve the best balance between adherence to the target prompt and reconstruction quality.
Effect of Crev and Cluv. Tab. summarizes the separation performance under different prompt configurations. First, replacing the prompt Crev specifying the target sound source with random, prompt results in drastic performance drop across all metrics. This highlights the essential role of accurate text conditioning towards separating target source. For Cive. we explore replacing the null prompt with a prompt for different source present the mixture but not the target. This substitution leads to a slight degradation in performance, which demonstrates that while Ciny provides some contextual information, the method is less sensitive to its precise content.
5 Conclusion
This paper demonstrates a new paradigm for audio source separation, moving away from reliance on extensive supervised training. In particular, we introduce ZeroSep, a novel training-free approach that leverages the power of pre-trained text-guided audio diffusion models. Our evaluation reveals that ZeroSep achieves performance on par with or exceeds leading supervised separation baselines across benchmark datasets, and our analysis further illuminates the factors critical for successful transformation from generation to separation. The effectiveness of ZeroSep showcases a new application for the growing family of audio diffusion models and offers a compelling alternative direction for developing open-set audio source separation models.
Limitations. While we have demonstrated the efficacy of ZeroSep on popular audio diffusion models (e.g., AudioLDM families and Tango), how it works on larger models and alternative architectures remains untested. In addition, our reliance on latent inversion can introduce approximation errors that may impair separation fidelity. Due to computational constraints, we did not include these experiments in this work. We will explore how scale evaluations to diverse, high-capacity diffusion models and develop more accurate inversion techniques in future work.
Beyond Basic Separation. ZeroSep's inherent mechanism unlocks diverse application scenarios beyond simple text-to-sound separation. First, text prompts can be automatically generated. Audio event detection MMssaros al. 2021] audio language models [Gong al. 2023 Ghosh al. 2024) can derive labels or frre-form descriptions from mixtures, enabling automated separation. Second, ZeroSep facilitates cross- modal applications; for instance, leveraging audio-visual localiza- tion [Huang et al.) 20233 Chen et al. [021b] and vision-language models (Liu et al.) [023c] users could separate sounds
video by visually describing sounding objects. Third, recognizing the importance of spatial audio understanding and rendering [Gao and Grauman 2019 Liang et al. 2023a b] Huang al., 2024b) for human-lewe level acoustic perception, ZeroSep can directly extended to spatial audio separation using diffusion models that support multi multi-channel input, such as Stable Audio Open [Evans et al., 2025b]. Finally, our method naturally enables a continuous transition from mixture reconstruction to sound highlighting (Gandikota et al. 2024a] Huang et al. [024c] 2025a b) by varying the value of classifier-free guidance, allowing scaling of target sound elements from full presence to complete separation (Gandikota et al. 2024b).
10

---

References
Yi Luo and Nima Mesgarani. Conv-tasnet: Surpassing ideal time-frequency magnitude masking for speech separation. IEEE/ACM Trans. Audio Speech Language Process. 27(8):125--126, 2019.
doi: 10.11099TASLLP:019229..
Cem Subakan, Mirco Ravanelli, Samuele Cornell, Mirko Bronzi, and Jianyuan Zhong. Attention is all you need in speech separation. In Proc. IEEE International Conference on Acousties, Speech and Signal Processing (ICASSP), 2021.
Xubo Liu, Haohe Liu, Qiuqiang Kong, Xinhao Mei, Jinzheng Zhao, Qiushi Huang, Mark D. Plumbley, and Wenwu Wang. Separate what you describe: Language-queried audio source separation. In Proc. Interspeech, pages 1801-1805, 2022a.
Xubo Liu, Qiuqiang Kong, Yan Zhao, Haohe Liu, Yi Yuan, Yuzhuo Liu, Rui Xia, Yuxuan Wang, Mark D. Plumbley, and Wenwu Wang. Separate anything you describe. arXiv preprint arXiv:2308.00 05037, 2023a.
Han Yin, Jisheng Bai, Yang Xiao, Hui Wang, Siqi Zheng, Yafeng Chen, Rohan Kumar Das, Chong Deng, and Jianfeng Chen. Flowsep: Language-queried sound separation with rectified flow, arXiv preprint urXiv:2409.076/1 2024.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis. In International Conference on Learning Representations (ICLR), 2021.
Nanxin Chen, Yu Zhang, Heiga Zen, Ron Weiss, Mohammad Norouzi, and William Chan. Wave- grad: Estimating gradients for waveform generation. In International Conference on Learning Representations (ICLR), 2021a.
Myeonghun Jeong, Hyeongju Kim, Sung Jun Cheon, Byoung Jin Choi, and Nam Soo Kim. Diff-tts: A denoising diffusion model for text-to-speech. In Proc. Interspeech 2021, pages 3605-3609, 2021. doi: 10.21457/mterrpecchh222111-
Vadim Popov, Ivan Vovk, Vladimir Gogoryan, Tasnima Sadekova, and Mikhail Kudinov. Grad-tts: A diffusion probabilistic model for text-to-speech. In International Conference on Machine Learning, pages 8599-8608. PMLR, 2021.
Heeseung Kim, Sungwon Kim, and Sungroh Yoon. Guided-tts: A diffusion model for text-to-speech via classifier guidance. In Proceedings of the 39th International Conference on Machine Learning (ICML), 2022.
Sang-gil Lec, Heeseung Kim, Chachun Shin, Xu Tan, Chang Liu, Qi Meng, Tao Qin, Wei Chen, Sungroh Yoon, and Tie-Yan Liu. Priorgrad: Improving conditional denoising diffiusion models with depe-depn adaptive prior. In International Conference on Learning Representations (ICLR), 2022.
Jinglin Liu, Chengxi Li, Yi Ren, Feiyang Chen, and Zhou Zhao. Diffsinger: Singing voice synthesis via shallow diffusion mechanism. In Proceedings of the AAAI Conference on Artificial Intelligence, 2022b.
Junhyeok Lee and Seungu Han. Nu-wave: A diffusion probabilistic model for neural audio upsam- pling. In Proc. Interspeech 2021, pages 1634-1638, 2021. doi: 10.21457/nnerspecch.22.. 2022ers2pppchh.22-66666
Haohe Liu, Zehua Chen, Yi Yuan, Xinhao Mei, Xubo Liu, Danilo Mandic, Wenwu Wang, and Mark D. Plumbley. Audioldm: Text-to-audio generation with latent diffusion models. In Proceedings of the 40th International Conference Machine Learning (ICML), pages 1440--21474, PMLR, 2023b.
11

---

Haohe Liu, Yi Yuan, Xubo Liu, Xinhao Mei, Qiugiang Kong, Qiao Tian, Yuping Wang, Wenwu Wang, Yuxuan Wang, and Mark D. Plumbley. Audioldm Learning holistic audio generation with self-supervised pretraining. IEEF/ACM Transactions on Audio, Speech, and Language Processing. 32:2871-2883, 2024. doi: 10.1199/AASS.P22243399077
weepanway Ghosal, Navonil Majumder, Ambuj Mehrish, and Soujanya Poria. Text-to-audio gener- ation using instruction tuned Ilm and latent diffusion model. arXiv preprint arXiv:2304. 13731, 2023.
Zach Evans, Julian D Parker, CJ Carr, Zack Zukowski, Josiah Taylor, and Jordi Pons. Stable audio open. In ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5, IEEE, 2025a.
Fabian-Rober Stöter, Zahra Hafida Benslimane, et al. Separation by timbre via non negative matrix factorization with mfce clustering, 2021. Implementation the nussl Python library.
John R. Hershey, Zhuo Chen, Jonathan Le Roux, and Shinji Watanabe. Deep clustering: Discrimi- native embeddings for segmentation and separation. In Proc. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2016.
Dong Yu, Morten Kolbæk, Zheng-Hure Tan, and Jesper Jensen. Permutation invariant training of deep models for speaker-independent multi-talker speech separation. In Proc. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 241-245, 2017. doi: 10.1 1..11099CASSPP.21..7799922
Bo Chen, Chao Wu, and Wenbin Zhao. Sepdiff: Speech separation based denoising diffusion model. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5. IEEE, 2023.
Robin Scheibler, Youna Ji, Soo- Whan Chung, Jaeuk Byun, Soyeon Choe, and Min-Seok Choi. Diffusion-based generative speech source separation. In ICASSP 2023-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5. IEEE, 2023.
Chao Huang, Susan Liang, Yapeng Tian, Anurag Kumar, and Chenliang Xu. High-quality visually- guided sound separation from diverse categories. In Proceedings of the Asian Conference on Computer Vision (ACCV), pages 35-49, December 2024a.
Yuancheng Wang. Zeqian Ju, Xu Tan, Lei He, Zhizheng Wu, Jiang Bian, and Sheng Zhao Audit: Audio editing by following instructions with latent diffusion models. In Advances in Neural Information Processing Systems (NeurIPS), 2023.
Zachary Novack, Julian McAuley, Taylor Berg-Kirkkae Kirkpatrick, and Nicholas J Bryan. Ditto: Diffusion inference- -imee t-optimization for music generation. arXiv preprint arXiv:2401.12119, 2024.
Hila Manor and Tomer Michaeli. Zero-shot unsupervised and text-based audio editing using DDPM inversion. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors, Proceedings of the 41 International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 34603- 34629. PMLR, 21-27 Jul 2024. URL htttps://proceddii .ww.iinngmmrrr...mmmmmmm... prw.s/ html
httppp/wwss.seeeeeeeeeeeeeeeeeee he////235////// v235/manor224 manor24.
Emilian Postolache, Giorgio Mariani, Luca Cosmo, Emmanouil Benetos, and Emanuele Rodolà. Generalized multi-source inference for text conditioned music diffusion models. In ICASSP 2024-2024 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pages 6980 6984. IEEE, 2024.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02500 2020.
Inbar Huberman-Spiegelgee Vladimir Kulikov, and Tomer Michaeli. An edit friendly ddpm noise space: Inversion and manipulations. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages -2478,, 2024.
12

---

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840 6851, 2020.
Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter. Audio set: An ontology and human-labeled dataset for audio events. In 2017 IEFE international conference acoustics, speech and signal processing (ICASSP), pages 776-780. IEEE, 2017.
Yapeng Tian, Jing Shi, Bochen Li, Zhiyao Duan, and Chenliang Xu. Audio-visual event localization in unconstrained videos. In Proceedings of the European Conference on Computer Vision (ECCV), pages 247-263, 2018.
Hang Zhao, Chuang Gan, Andrew Rouditchenko, Carl Vondrick, Josh McDermott, and Antonio Torralba. The sound of pixels. In Proceedings of the European conference computer vision (ECCV), pages 570-586, 2018.
Colin Raffel, Brian McFee, Eric Humphrey, Justin Salamon, Oriol Nicto, Dawen Liang, Daniel PW Ellis, and C Colin Raffel. Mir_ Mir_eval:: transparent implementation of common mir metrics. In ISMIR, pages 367-372, 2014.
Vivek Jayaram and John Thickstun. Source separation with deep generative priors. In International Conference Machine Learning, pages 4724 4735- PMLR, 2020.
Kevin Kilgour, Mauricio Zuluaga, Dominik Roblek, and Matthew Sharifi. Fri, 'echet audio distance: A metric for evaluating music enhancement algorithms. arXiv preprint arXiv:/8/2.08846. 2018.
Annamaria Mesaros, Toni Heittola, Tuomas Virtanen, and Mark D Plumbley. Sound event detection: A tutorial. IEEE Signal Processing Magazine, 38(5):67-83, 2021.
Yuan Gong, Hongyin Luo, Alexander H Liu, Leonid Karlinsky, and James Glass. Listen, think, and understand. arXiv preprint arXiv:2305. 200. 2023.
Sreyan Ghosh, Sonal Kumar, Ashish Seth, Chandra Kiran Reddy Evuru, Utkarsh Tyagi, S Sakshi, Oriol Nieto, Ramani Duraiswami, and Dinesh Manocha. Gama: A large audio-language model with advanced audio understanding and complex reasoning abilities, arXiv preprint 1rXiv:2406.11768. 2024.
Chao Huang, Yapeng Tian, Anurag Kumar, and Chenliang Xu. Egocentric audio-visual object localization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 229 10-22921, 2023.
Honglie Chen, Weidi Xie, Triantafyllos Afouras, Arsha Nagrani, Andrea Vedaldi, and Andrew Zisserman. Localizing visual sounds the hard way. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16876. 2021b
Haotian Liu, Chumyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 10:34892-34911 2023c.
Ruohan Gao and Kristen Grauman. 2.5 d visual sound. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 324-333, 2019.
Susan Liang, Chao Huang, Yapeng Tian, Anurag Kumar, and Chenliang Xu. Av-nerf: Learning neural fields for real-world audio-visual scene synthesis. Advances in Neural Information Processing Systems,3:374450 36:37472-37444 2023a.
Susan Liang, Chao Huang, Yapeng Tian, Anurag Kumar, and Chenliang Xu. Neural acoustic context field: Rendering realistic room impulse response with neural fields. arXiv preprint arXiv:2309. 15977, 2023b.
Chao Huang, Dejan Marković, Chenliang Xu, and Alexander Richard. Modeling and driving human body soundfields through acoustic primitives. In European Conference on Computer Vision, pages 1-17. Springer, 2024b.
13

---

Zach Evans, Julian D Parker, CJ Carr, Zack Zukowski, Josiah Taylor, and Jordi Pons. Stable audio open. In ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5. IEEE, 2025b.
Rohit Gandikota, Joanna Materzyúska, Tingrui Zhou, Antonio Torralba, and David Bau. Concept sliders: Lora adaptors for precise control in diffusion models. In European Conference on Computer Vision, pages 1 172-188 Springer, 2024a.
Chao Huang, Susan Liang, Yunlong Tang, Yapeng Tian, Anurag Kumar, and Chenliang Xu. Scaling concept with text-guided diffusion models. arXiv preprint arXiv:2410.24151, 2024c.
Chao Huang, Ruohan Gao, JMF Tsang, Jan Kurcius, Cagdas Bilen, Chenliang Xu, Anurag Kumar, and Sanjeel Parekh. Learning highlight audio by watching movies. arXiv preprint 12114 2025a.
Chao Huang, Susan Liang. Yunlong Tang, Li Ma, Yapeng Tian, and Chenliang Xu. Fresca: Unveiling the scaling space diffusion models. arXiv preprint wrXiv:2504.02/54 2025b.
Rohit Gandikota, Joanna Materzyúskka Tingrui Zhou, Antonio Torralba, and David Bau. Erasing concepts from diffusion models. In Proceedings of the 2024 IEEE European Conference Computer Vision, 2024b. arXiv preprint arXiv:231..12092.
Alexandre Défossez, Jade Copet, Gabriel Synnaeve, and Yossi Adi. High fidelity neural audio compression. arXiv preprini urXiv:2210.13438, 2022.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint 1rX1v:2207.12598. 2022.
Seyedmorteza Sadat, Otmar Hilliges, and Romann M Weber. Eliminating oversaturation and artifacts of high guidance scales in diffusion models. In The Thirteenth International Conference on Learning Representations, 2024.
14

---

Table 5: Additional evaluation on the AVE [Tian al. 2018 and MUSIC [Zhao al. 2018 datasets with MSE and Encodec-basee metrics. + indicates lower is better. The best results are bold.
Method
MUSIC
AVE
MSE 4
(Emb.) 4
1.2 (Emb.) 4
MSE.J. +
(Emb.) &
1.2 (Emb.) !
Require Separation Training
LASS-Net
32.860
0.6205
0.7338
12.612
0.5984
0.6737
AudioSep
10.857
0,4232
0,4046
9.383
0.4949
0,5095
FlowSep
15.134
0.5525
0.5869
7.785
0.6706
0.8402
Separation Training Free
NMF-MFCC
36.761
0.4230
0.4180
59.594
0.5766
0.6906
AudioEdit
29.357
0.8059
1.3088
22.253
0.9588
1.7090
ZeroSep (Ours)
15.470
0.5540
0,5912
11,076
0.7959
1.1258
A
More Metrics
To further assess separation quality, we report additional metrics, including L1/L2 loss on En- codec [Défossez et al. 2022] embeddings and MSE mel-spectrograms, which capture temporal fidelity more effectively. y. Results are shown in Tab. 5 On these newly introduced metrics, ZeroSep achieves performance comparable the supervised FlowSep model and surpasses LASS-Net on the MUSIC dataset, providing stronger evidence of genuine source separation rather than prompt- consistent hallucination.
An interesting observation arises with NMF-MFCC. Unlike other methods, which typically fail by leaving residual interference, NMF-MFCC often produces silence overly smoothed outputs. This distinct failure mode can artificially benefit the Encodec embedding distances: the smoothed, low-detail outputs align more closely with Encodec's coarse embedding space, despite discarding much the fine geaigrai content. As a result, NMF-MFCC appears competitive on L1/L2 embedding metrics, yet its poor performance perceptual and semantic measures (e.g., FAD, LPAPS, and mel-spectrogram MSE) highlights its limitations.
B Effect of w and Scheduled Guidance
Large guidance weights w amplify the conditional score. Since both the unconditional and conditional are approximations, excessive scaling can push samples into oowppobbbiility regions off the data manifold, improving prompt adherence and sample diversity but risking artifacts phenomenon broadly reported in image/audio diffusion models (Ho and Salimans 2022 Sadat et al.) 2024). Consequently, large w favors generative hallucination rather than accurate separation. Empirically, we find 1 strikes good balance between suppressing interfering sounds and avoiding spurious generations.
Static vs. Scheduled Guidance. Intuitively, early diffusion steps shape the global structure, while later steps refine details. We therefore compared constant and scheduled w on the DMd-o..MM--Large backbone (DDIM, 50 steps). Results on MUSIC are shown Tab.
Table 6: Comparison of static vs. scheduled guidance (w) on MUSIC.
schedule
FAD 4
LPAPS
C-A f
C-T
constant
0.377
4.669
0.615
0.271
lincar 0-+1
0.471
4.526
0,520
0.202
linear 1 1-0
0.332
4.667
0.618
0.281
sine 01+1
0.431
4.523
0.550
0.227
Observations. (i) Low-to-high schedules (0~>1, sine) degrade separation, as early under-
conditioning causes loss of clean target structure, (ii) High-to-low schedules (1-1) improve over constant w 1, consistent with reports that guidance is most useful in the carly-to-mid noise range
15

---

but less helpful at the end. This supports our main claim that = is an effective default while also motivating dynamic scheduling as a promising future direction.
C Runtime and Scalability for Multiple Sources
Let S denote the number of inference steps and Ta the per-step time. ZeroSep requires a single inversion of the mixture (shared for all branches) plus one guided denoising per target source:
TeroSop 2 STa +
+N = (N 1)STa.
inversion
N
targets
Discriminative models (eAs AudioSep) require only a single forward pass per target (N Ta). FlowSep, rectified-Ffe flow model, requires multi-step denoising (N S'Ta). Tab. plexity and measured runtimes.
T summarizes com-
Table 7: Runtime complexity and average cost (separating source) on A GPU.
Method
Complexity
Runtime (s)
LASS-Net
Ta
0.05
AudioSep
N Ta
0.04
FlowSep
N S
1.07
ZeroSep (AudioLDM-S)
(N + 1)S 0.98
While ZeroSep is slower than discriminative models, it matches FlowSep. Crucially, runtime scales down with advanced samplers: DPM-Solver++ attains high quality in 15-20 steps, and consistency models in 2-8 or even step, which would directly reduce ZeroSep 's runtime.
D Polyphonic Mixtures
ZeroSep naturally extends to multi-source separation. 3-source MUSIC mixtures (e.g., vio- lin-flute-trumpet),, results degrade moderately compared to 2-uuue
Table 8: ZeroSep's performance on and 3-source mixtures from the MUSIC dataset.
Sources
FAD |
LPAPS
C-A
C-T
2
0.377
4.669
0.615
0.271
0.508
4.248
0.501
0.246
For especially challenging cases where sources share the same event class, attribute-qualified prompts (c.g., "soft violin arpeggio vs. "long sustained violin note") ") may help.
E Prompt Sensitivity and Hierarchical/Open-See Behavior
We tested prompt robustness by perturbing MUSIC class names with hypernyms (e.g., violin + string instrument) and underspecified descriptors (e.g., trumpet - bright instrument). Results are summarized in Tab. [
Table 9: Prompt sensitivity analysis on MUSIC.
Prompt type
FAD 1
LPAPS
C-A f
C-T
canonical
0.377
4.669
0.615
0.271
hypernym
0.388
4.692
0.583
0.280
underspecified
0.336
4.714
0.577
0.248
16

---

Observations. Hypernyms preserve family-level correctness, slightly degrading aceuracy but improving consistency (C-T). Underspecified descriptors sometimes improve distributional metrics (FAD) while reducing class accuracy, indicating plausible but less precise extractions. These findings suggest hierarchical robustness: ZeroSep generalizes reasonably from coarse categories (e.g., "string instrument") to specific instances (e.g., violin), though we avoid over-claiming open-set separation.
Table 10: Effect of prompt descriptiveness on MUSIC.
Prompt type FAD +
LPAPS +
C-A 1
C-T
Class label
0.377
4.669
0.615
0.271
Descriptive
0,484
4.541
0.512
0.196
AudioSep
0.725
5.209
0.450
0.204
F Prompt Length and Descriptiveness
We conducted a focused study on the MUSIC dataset to test robustness to descriptive prompts (shown in Tab. I0P. Each of the 11 instrument class labels was expanded into a free-form caption roughly 10x longer than the label. For example, "saxophone" was expanded to "warm, breathy reed tone with slight rasp and expressive bends". and "accordion" " to "reedy, wheezy sustained chords with gentle tremolo and slow swells". ". This setting stresses ZeroSep with more detailed but potentially noisy prompts.
Observations. ZeroSep shows moderate degradation under long. descriptive prompts but still outperforms the strong supervised baseline AudioSep across most metries. We attribute this drop to mismatch between our prompt expansions and the captions used to train text-to-audio diffusion models, which are typically short, coarse descriptions (e.g., man speaking while a baby crying" "). This highlights a promising direction: aligning prompt style with training data or finetuning more descriptive captions to better exploit natural- natural-language expressiveness.
GT stem
OURS stom
Figure Failure case analysis of ZeroSep. Mixture: Man speech (stem 1) Shofar (stem 2).
G Failure Case Analysis
While generally effective, ZeroSep can sometimes fail fully isolate the target source. This typically occurs when an interfering source possesses significant energy that the model cannot eliminate in single iteration. An illustrative example of such failure is presented in Fig. We postulate that, given the inherent progressive operation of diffusion models, the removal of interfering sources also proceeds incrementally. Consequently, this performance limitation may be tied the number of inference steps utilized. Potential avenues for improvement include increasing the inference steps or iteratively applying the separation process.
H
More Separation Results
These figures present mel-spectrograms that visualize the audio separation performance on two-source mixtures. For each figure, the rows are ordered from top to bottom as follows: the first source's Ground Truth, followed by its separation results from LASS-Ne FlowSep, AudioEdit, AudioSep.
17

---

and Ours. This sequence is then repeated for the second source: Ground Truth 2. LASS-Net 2. FlowSep 2, AudioEdit 2, AudioSep 2. and Ours 2. You might notice some white or empty areas on the right side of the mel-spectrograms; these are simply due to the varying lengths of the audio samples.
LAGS mem.
LASS dm
Di
SEP stom
SSP San
27aie
LSSS 9mn
EDIT tte 2
EMT can
Figure 5: Mixture: Cello (stem 1) + Erhu
Figure 6: Mixture: Acoustic Guitar (stem 1)
(Stem 2)
Tuba (Stem 2)
18

---

MUSC- CeCC---iro------ VCCC-rondo--eeercooooo Fel--
M.SII -colo- 5% spO uite O
GT etem
ST Stve
smm
iti
EBIT scan
OU1S
DJRS
HLW
FUPWsten
S/P
OUBS spm
OunS
Figure Mixture: Accordion (stem 1)
Figure 8: Mixture: Cello (stem 1) + Erhu (Stem 2)
Flute (Stem 2)
19

---

N:SC-1 11w.- COA CAT.--GEEEN SGU
MUSIC www.phomm Sw..R.SOO. (ump Coan
GT etemn
ST stve
simn
ItiT 1
BBIT scan
OU1S
DJRS kem
stee
HLOW
FUPWsten
pDIt
OUBS spa
ounSaten1
Figure Mixture: Tuba (stem 1) + Cello
Figure 10: Mixture: Xlyophone (stem 1) +
(Stem 2)
Trumpet (Stem 2)
20

---

We HiSoo Rango
C2 Jlan
GT siam
FLO
SDIT Ston
EXI
0J65 1em
OU15
ctmm
FDW stem
HLW Stan
0 men
CUFS CHN
Figure 11: Mixture: Truck (stem 1) + Banjo
Figure 12: Mixture: Chainsaw (stem 1) +
(Stem 2)
Accordion (Stem 2)
21

---

Soot EOMES homr www.ee--ppoooo.. NO00O Hore
TeTTENAAA cpnadligg clcci fund slrontt
22 Jtan
LASS Stm
26-
FLOM
SDIT Simn
EMT cw
52P sian
OJRS Mem
DURS Mem
FDW stem
TooWatem2
S/P Ston.2
0.85 mtem
0J81 stem
24
1000
Figure 13: Mixture: Train Horn (stem 1)
Figure 14: Mixture: Male Speech (stem 1) + Airplane (Stem 2)
Bark (Stem 2)
22

---

1CP00 202 Trurce (irple RCC
5.004 940---77-- 2002 sirch
27 Jtan
GT slam
FLN
SDIT Simn
EXT den
DJRS Mem
CU15
etem
FLW
HLON: Stan
DJRS mten
0415 cven
Figure 15: Mixture: Truck (stem 1) Ukulele
Figure 16: Mixture: Bark (stem 1) + Toilet
(Stem 2)
Flush (Stem 2)
23

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The claims are provided in the introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Limitations will be provided in the appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]

---

Justification:
No theory results are provided.573
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: All details discussed in the method part.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
25

---

Answer: [No]
Justification:
The code will be released upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https guwdes/ddeeeeee Cooeeeoodeesssseeeeee //nips.com /ttc://dlllliiiiiee////eeeaaaass ssiniiiiccy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: Details are provided the setup.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: We fix the random seed and verify the experiments large amount of data to minimize the randomness.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
26

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Details are provided the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question:
Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics htttpp///barrrrs.cc//aaa///hccct
Answer: [Yes]
Justification: Confirmed.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss it in the introduction.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
27

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: No such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We discuss it in the appendix.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
28

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
No assets released.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: No human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: No human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: Not involve LLMs.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy fattps://neurip:::: /ttpaarrpp..cco/ooofeeeeeee////s for what should or should not be described.
30


Reviews:
[{"review_id": "lIkOhrckw3", "reviewer": "Reviewer_rKJV", "summary": "Authors propose the first latent inversion approach to sound-source separation, and by virtue of a completely training-free approach. Particularly, ZeroSep repurposes pretrained generative diffusion models for discriminative separation tasks without any task-specific training, representing a fundamental departure from past supervised/finetuning approaches.", "questions": "Some minimal exploration on optimal text conditioning for different types of acoustic scenes, and how sensitive the method is to prompt phrasing, would help improve the quality of this paper.\n\nCan authors comment on the hierarchical boundaries of events? Specifically, what is the permissible deviation of text prompts from the training taxonomy while still ensuring effective separation performance? It would be interesting to see the extent of open-set separation, say, distinguishing unseen leaf node events that belong to already seen parent events. Open-set separation is considered a particularly strong claim.", "limitations": "yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths\nZeroSep surpasses existing training-based generative methods on multiple benchmarks - all metrics on MUSIC dataset, and comparable to flowsep on AVE -while being training-free.\nMore importantly, the overall approach is model-agnostic and is bound to improve with better T2A models in the future.\nOverall, paper is well-written with relevant references and easy to read-understand.\n\nWeakeness\nWhile respecting the page limit, the authors fail to touch on some theoretical analysis explaining why T2A diffusion models possess inherent separation capabilities; Is this an emergent property of the generative objective? -Some analysis regarding how much percentage of T2A model’s training data included polyphonic sounds would be helpful.\n\nSimilarly, \\omega = 1 removes unconditional noise estimation deserves more rigorous theoretical grounding; model follow only the conditional score? Is it the case that higher \\omega pushes off the manifold? Do you think a dynamic guidance schedule adds further to improving performance? One quick suggestion would be to test a simple linear or cosine ramp that peaks at 1 around the middle 40-60% of steps.\n\nCan the authors provide detailed runtime comparisons + scalability analysis for separating multiple sources simultaneously compared to single-pass (non-diffusion) methods?\n\nThe paper mentions that latent inversion can introduce approximation errors that may impair separation fidelity, but it does not thoroughly investigate how these errors accumulate or propose mitigation strategies. MEDIC argues that imprecise approximation of the original audio using DDIM inversion can allow authors to comment on the same, or argue why ZeroSep is not impacted by the assumption of perfect reversibility of ODE, often not met during text-based conditioning? \n\nPlease provide analysis on complex polyphonic mixtures involving more than 2 sources/instances of the same event class.", "was_revised": false, "final_justification": "", "rebuttal": "**We thank the reviewer for acknowledging the novelty and importance of our paper, and also for providing constructive comments to help improve the paper.** Below, we address each point.\n\n### Q1. Why do T2A diffusion models exhibit zero‑shot separation ability?\n\n**Data perspective.** Text‑to‑audio (T2A) diffusion models are usually trained on **multi‑label, polyphonic** audio corpora, so the model must learn *both* the acoustics of individual sources and how they combine. Thus, the model learns *composable* latent factors that align with text semantics (e.g., *dog bark*, *rain*) and their superposition (e.g., *a dog bark and raindrops*). This polyphonic nature exists in many T2A pretraining datasets. For example, AudioSet -- the predominant pretraining resource -- has **2.7 labels per 10‑s clip on average**[1], indicating widespread mixtures, which encourages the model to encode disentangled source structure. \n\n[1] Audio set: An ontology and human-labeled dataset for audio events\n\n**Score-function view.** Classifier‑free guidance (CFG) combines an unconditional score $\\nabla_{x_t}\\log p(x_t)$ with a conditional increment that steers toward $\\nabla_{x_t}\\log p(c\\mid x_t)$:\n\n$$\n\\hat\\epsilon_\\theta(x_t,c)=\\epsilon_\\theta(x_t,\\varnothing)\n+ \\omega\\big(\\epsilon_\\theta(x_t,c)-\\epsilon_\\theta(x_t,\\varnothing)\\big),\n$$\n\nwhich is a standard derivation in CFG. Setting $\\omega=1$ removes the unconditional term and retains only the conditional direction, effectively **filtering the inverted latent toward the target source** specified by text. Therefore, the zero-shot separation capability is an emerging property of T2A diffusion models trained with mixture audio. We will add this discussion to the revised manuscript.\n\n### Q2. Effect of $\\omega$ and scheduled guidance\n\nLarge $\\omega$ amplifies the conditional score. Because both scores are learned approximations, excessive scaling can move samples toward low‑probability regions (“off the data manifold”), improving prompt adherence and diversity of generated samples but risking artifacts -- an effect documented broadly in image/audio diffusion [2, 3].Therefore, it will opt for generation instead of accurate separation with large $\\omega$. This potential artifact, along with large $\\omega$, does not contribute to accurate separation, and empirically we show $\\omega=1$ achieves a good trade-off between removing the other sounds and introducing generated content/artifact. \n\n**Static vs. scheduled guidance.** Thank you for the great suggestion! Intuitively, the early stage of diffusion models builds the global structure, while the later denoising steps act as a refinement stage. We tested constant and scheduled $\\omega$ on **AudioLDM2‑Large** backbone (DDIM, 50 steps). Results on the MUSIC dataset:\n\n| ω schedule |     FAD ↓ |   LPAPS ↓ |     C‑A ↑ |     C‑T ↑ |\n| ---------- | --------: | --------: | --------: | --------: |\n| constant 1 |     0.377 |     4.669 |     0.615 |     0.271 |\n| linear 0→1 |     0.471 | **4.526** |     0.520 |     0.202 |\n| linear 1→0 | **0.332** |     4.667 | **0.618** | **0.281** |\n| sine 0→1 |     0.431 |     4.523 |     0.550 |     0.227 |\n\n\n**Observations.** (i) **Low-to-high**, Starting at 0 (0→1, sine) harms separation (global structure lost of clean target audio). (ii) **High‑to‑low (1→0)** improves over constant 1.0, aligning with findings that guidance is most useful in the **early to middle** noise range and less beneficial at the end. This further enhances our claim that $\\omega=1$ is beneficial for accurate separation, while dynamic scheduling based on this finding can also be promising in future exploration, which we will add discussion in the revision.\n\n[2]: CLASSIFIER-FREE DIFFUSION GUIDANCE. NeurIPS 2021 Workshop\n\n[3]: ELIMINATING OVERSATURATION AND ARTIFACTS OF HIGH GUIDANCE SCALES IN DIFFUSION MODELS. ICLR 2025.\n\n### Q3. Runtime and scalability for multiple sources\n\nLet $S$ be the number of network inference steps and $T_d$ the time per step. ZeroSep performs **one inversion** of the mixture (shared for all separation branches) and then runs one guided denoising per target source. Thus complexity is\n\n$$\nT_{\\text{ZeroSep}} \\approx \\underbrace{S\\,T_d}_{\\text{inversion}}+\\underbrace{N\\,S\\,T_d}_{\\text{N targets}}=(N+1)S\\,T_d.\n$$\n\nFor single‑pass discriminative models like LASS-Net and AudioSep, the runtime is ${N\\,T_d}$ where only 1 step inference is required. For FlowSep, which is also based on a rectified flow model and requires multi-step inference, its runtime is $N\\,S\\,T_d$ for multi‑target separation. \nIn summary, we list the table below with the runtime complexity for each method and the actual time cost on an A100 GPU.\n\n| Method                            | Complexity  | Avg. Runtime (Separate one source on A100)                         |\n| --------------------------------- | ----------- | ---------------------------- |\n| LASS‑Net                          | ${N\\,T_d}$        | 0.05 s                   |\n| Audio-Sep                         | ${N\\,T_d}$        | 0.04 s                   |\n| FlowSep                           | ${N\\,S\\,T_d}$      | 1.07 s                       |\n| ZeroSep (AudioLDM-S)              | ${N\\,S\\,T_d}$  | 0.98 s                       |\n\n\nNote that $S$ and $T_d$ for different methods are different. In our implementation, ZeroSep’s runtime is larger than LASS-Net and Audio-Sep while remaining similar as FlowSep. In practice, **$S$** can be reduced substantially with advanced samplers or distillation: **DPM‑Solver++** attains high quality in **15–20 steps**, and **consistency models** reduce to **2–8 or even 1 step**, which directly scales our runtime down.\n\n\n\n\n### Q4. Latent inversion approximation and mitigation\n\nDDIM inversion is fast but introduces **stepwise drift**. As the reviewer mentioned, MEDIC found DDIM inversion works for very small edits, yet it accumulates reconstruction error, breaks musical structure, and offers no fine‑grained control. MEDIC tackles all three pain points with a plug‑and‑play framework that needs no extra training. Therefore, MEDIC pointed out a direction to mitigate the latent inversion error, which will also apply to our method. Apart from MEDIC, there are more correction methods that exist in vision. ZeroSep is drop‑in compatible with each:\n\n* **Null‑text (embedding) optimization**: optimizes the unconditional embedding along the trajectory to match the source mixture, compensating for score error.\n* **PnP‑style inversions**: explicitly estimate and inject residual noise to correct drift.\n* **DDPM‑style stochastic inversion**: By recovering per‑step noise latents, it better matches the forward process and reduces accumulation compared with purely deterministic DDIM inversion.\n\nWe currently use a DDPM‑based inversion variant precisely to **mitigate drift**; we will expand the discussion on inversion methods and possible mitigation.\n\n### Q5. Polyphonic mixtures\n\nZeroSep supports **multi‑source separation** naturally. We additionally include results on **3-source** mixtures built from MUSIC dataset (e.g., violin+flute+trumpet), which shows moderate degradation as the number of mixing sources grows. \n\n| # sources | FAD ↓ | LPAPS ↓ | C‑A ↑ | C‑T ↑ |\n| :-------: | ----: | ------: | ----: | ----: |\n|     2     | 0.377 |   4.669 | 0.615 | 0.271 |\n|     3     | 0.508 |   4.248 | 0.501 | 0.246 |\n\nFor the challenging cases the reviewer mentions, where sources from the same event class are mixed, we argue that adding **attribute qualifiers** in text (e.g., “*soft violin arpeggio*” vs. “*long sustained violin note*”) could help. We will include the new 3-source separation results and add the discussion on challenging instance-level separation in the revised manuscript.\n\n## Q6. Prompt sensitivity & hierarchical/open‑set behavior\n\nTo test the prompt sensitivity and hierarchical prompting performance, given the class labels from the MUSIC dataset, we probe 11 instrument class names with two perturbations per class:\n\n* hypernyms:** *violin → string instrument*, *flute → woodwind*.\n* underspecified descriptors:** *trumpet → bright instrument*, *cello → low tone*.\n\n**Results (vs. canonical prompts).**\n\n| Prompt type    |              FAD ↓ |        LPAPS ↓ |          C‑A ↑ |              C‑T ↑ |\n| -------------- | -----------------: | -------------: | -------------: | -----------------: |\n| canonical      |              0.377 |          4.669 |          0.615 |              0.271 |\n| hypernym       |     0.388 (+0.011) | 4.692 (+0.023) | 0.583 (−0.032) | **0.280 (+0.009)** |\n| underspecified | **0.336 (−0.041)** | 4.714 (+0.045) | 0.577 (−0.038) |     0.248 (−0.023) |\n\n\n**Observation.** Hypernyms keep the correct **family**, yielding small degradations; the slight **C‑T rise** suggests the classifier judges family‑level matches as more consistent, while **C‑A** (accuracy to ground‑truth class) drops. Underspecified descriptors sometimes improve **distributional proximity (FAD)** yet hurt **class accuracy**, indicating plausible but less precise extractions. We will add this table and clarify metric roles.\n\n**Open‑set note.** These results indicate **good generalization** from parent nodes (e.g., *string instrument*) to leaves (e.g., a violin articulation), but we avoid over‑claiming “strong open‑set separation.” We will frame it as **hierarchical robustness** and include success/failure cases in the revised manuscript."}, {"review_id": "QSttWLFcfm", "reviewer": "Reviewer_2kNp", "summary": "### **Summary**\n\nThis paper introduces a novel audio source separation framework named ZeroSep, whose core idea is to leverage a pre-trained text-guided audio diffusion model to achieve the separation of arbitrary audio sources without any additional training or fine-tuning. The main mechanism of this method consists of two steps: first, through a diffusion inversion process, the mixed audio to be separated is mapped into the latent noise space of the diffusion model; then, a text prompt describing the target source is used to guide the denoising process, thereby reconstructing a single target source from the noise.", "questions": "### **Questions**\n\n1.  Are the datasets used for evaluation too few or lacking in generality? Most research on general-purpose separation uses AudioSet for training and evaluation, including the selected baselines LASSNet, AudioSep, and Flowsep. How does ZeroSep perform on this dataset? Does it still outperform traditional methods? Furthermore, the original papers for all the baselines did not use the AVE dataset. Is the choice of this dataset somewhat niche or less conventional?\n\n2.  Additionally, since prompts are used, is it possible to use descriptive captions for separation instead of just simple class labels? Does the length or complexity of the prompt content affect the framework's performance?\n\n3.  Regarding the separation demo shown in the paper (Figure 2), is ZeroSep's performance really significantly superior to the other methods in this specific example? Could a more compelling example have been chosen? Also, it appears that the spectrogram for the LASSNet result is displayed upside down.", "limitations": "yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "### **Strengths**\n\n1.  It directly applies a pre-trained generative diffusion model to a discriminative separation task without any training, which is a departure from the current mainstream supervised separation paradigm that relies on large amounts of paired data.\n2.  It explores the inconsistent role of the classifier-free guidance weight in the separation task compared to its role in generative tasks.\n3.  ZeroSep's performance on multiple benchmark datasets is comparable to or even surpasses supervised separation models, providing a new paradigm for open-set audio separation.\n\n### **Weaknesses**\n\n1.  Diffusion models typically have high computational costs during inference, as they require a multi-step iterative denoising process. The ZeroSep framework needs to perform a complete 'inversion + denoising' process for *each* target source in the mixture. This means that separating a mixture containing N sources requires approximately N times the computation of generating a single audio clip. The paper does not mention or compare the inference time differences between ZeroSep and the baseline methods at all.\n\n2.  The experiments show that using a null prompt (cinv = 'none') performs better than using a prompt that describes other (non-target) sources in the mixture. This seems somewhat counter-intuitive, as providing prior information about the background sounds should theoretically help the model perform a better inversion.\n\n3.  In the ablation study for `crev`, the conclusion does not fully align with the experimental results. On the AVE dataset, the CLAP-T metric improved when using a random prompt, so the claim that performance dropped \"across all metrics\" is inaccurate.\n\n4.  The ablation study for `cinv` does not seem to lead to a clear conclusion; it feels like it was just performed for the sake of it. Theoretically, changing `cinv` to the label of a non-target audio in the mixture should improve performance, but the actual results do not support this, and this discrepancy was not further analyzed.", "was_revised": true, "final_justification": "My final score is 4 (weakly accepted), and my concerns have been addressed.", "rebuttal": "**We thank the reviewer for highlighting ZeroSep’s novelty and for the constructive questions.** We clarify and address each point below.\n\n### Q1. Inference cost\n\nLet $S$ be the number of network inference steps and $T_d$ the time per step. ZeroSep performs **one inversion** of the mixture (shared for all separation branches) and then runs one guided denoising per target source. Thus complexity is\n\n$$\nT_{\\text{ZeroSep}} \\approx \\underbrace{S\\,T_d}_{\\text{inversion}}+\\underbrace{N\\,S\\,T_d}_{\\text{N targets}}=(N+1)S\\,T_d.\n$$\n\nFor single‑pass discriminative models like LASS-Net and AudioSep, the runtime is ${N\\,T_d}$ which only 1 step inference is required. For FlowSep, which also based on a rectified flow model and requires multi-step inference, its runtime is $N\\,S\\,T_d$ for multi‑target separation. \nIn summary, we list the table below with the runtime complexiity for each method and the actual time cost on a A100 GPU.\n\n| Method                            | Complexity  | Avg. Runtime (Separate one source on A100)                         |\n| --------------------------------- | ----------- | ---------------------------- |\n| LASS‑Net                          | ${N\\,T_d}$        | 0.05 s                   |\n| Audio-Sep                         | ${N\\,T_d}$        | 0.04 s                   |\n| FlowSep                           | ${N\\,S\\,T_d}$      | 1.07 s                       |\n| ZeroSep (AudioLDM-S)              | ${N\\,S\\,T_d}$  | 0.98 s                       |\n\n\nNote that $S$ and $T_d$ for different methods is different. In our implementation, ZeroSep’s runtime is larger than LASS-Net and Audio-Sep while keeping similar as FlowSep. In practice, **$S$** can be reduced substantially with advanced samplers or distillation: **DPM‑Solver++** attains high quality in **15–20 steps**, and **consistency models** reduce to **2–8 or even 1 step**, which directly scales our runtime down.\n\n### Q2/Q4. Why a null inversion prompt outperforms non‑target prompts\n\nA non‑target prompt biases the latent inversion toward irrelevant sources, constricting the representation available for the true target during guided denoising. A null prompt leverages the diffusion prior’s rich, unbiased mixture representation, after which the target‑specific prompt can act more effectively. Table 4 shows a consistent – albeit modest – drop when conditioning on non‑targets. We will make this intuition explicit.\n\n### Q3.  $c_{\\text{rev}}$ ablation wording\nThank you for point out this issue. Indeed, the CLAP‑T score on AVE improves with a random prompt. We will replace “across all metrics” with “across all metrics except CLAP‑T on AVE”\n\n\n### Q5. Dataset choice\nWe clarify that our dataset choice is reasonable: \n* First, the two datasets, MUSIC and AVE, are the de‑facto multimodal (including text query) source separation benchmarks [1, 2, 3, 4]. Therefore, they are suitable for assessing the performance of unsupervised or text-guided separation methods as shown in the Tab 1. \n\n[1] The Sound of Pixels\n[2] CLIPSep: Learning Text-queried Sound Separation with Noisy Unlabeled Videos\n[3] iQuery: Instruments As Queries for Audio-Visual Sound Separation\n[4] Language-Guided Audio-Visual Source Separation via Trimodal Consistency\n\n* Second, not evaluating on AudioSet can test open‑set generalization since all supervised baselines except FlowSep were trained on AudioSet yet are evaluated out‑of‑domain here. ZeroSep’s gains therefore reflect true open domain robustness.\n\n### Q6 │ Prompt length / descriptiveness\n\nWe ran a focused prompt‑descriptiveness study on the MUSIC dataset. Specifically, We expanded each of the 11 MUSIC instrument labels into a free‑form descriptive caption (~10x length). For example: \"saxophone\" to \"Warm, breathy reed tone with slight rasp and expressive bends,\"accordion\" to \"Reedy, wheezy sustained chords with gentle tremolo and slow swells.\" This create a challenging setting to test ZeroSep where the prompt is descriptive but potentially noisy.\n\n| Prompt type | FAD ↓     | LPAPS ↓   | CLAP‑A ↑  | CLAP‑T ↑  |\n| ----------- | --------- | --------- | --------- | --------- |\n| Class label | **0.377** | 4.669     | **0.615** | **0.271** |\n| Descriptive | 0.484     | **4.541** | 0.512     | 0.196     |\n| AudioSep    | 0.725     | 5.209     | 0.450     | 0.204     |\n\nWe observe a moderate drop of ZeroSep yet still beat the strong supervised baseline AudioSep. We agrue that the degradtation likely stems from a mismatch between our expansion and the caption used to train Text-to-Audio diffusion model, which is often a simple audio description like \"a man speak while the baby is crying\". \n\n### Q7. Qualitative demo (Fig. 2)\nThank you for suggesting improvement on the choice of qualitative example. We have included additional qulitative examples in the supplement, and we will correct LASSNet’s spectrogram orientation in the revised manuscript plus adding more qualitative examples in the main paper."}, {"review_id": "KaReC0lTbZ", "reviewer": "Reviewer_KHgX", "summary": "The paper introduces ZeroSep, a novel zero-training framework for audio source separation that repurposes pre-trained text-guided audio diffusion models. Unlike traditional supervised methods, ZeroSep leverages latent inversion and text-conditioned denoising to separate audio sources without task-specific training. The key insight is that a pre-trained diffusion model, originally designed for generation, can be steered via text prompts to disentangle mixed audio into individual sources. Experiments on AVE and MUSIC datasets show that ZeroSep outperforms supervised baselines, demonstrating its effectiveness in open-set scenarios. The framework offers three main advantages: open-set separation capability, model-agnostic versatility, and training-free efficacy.", "questions": "See weaknesses.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 2, "strengths_and_weaknesses": "## Strengths\n1. Clear Mechanistic Analysis: The paper thoroughly investigates the impact of key components (inversion strategies, guidance weight, prompt selection), providing insights into how generative models can be adapted for separation. The correlation between generative fidelity and separation performance is particularly noteworthy.\n2. Open-Set Capability: By leveraging text-guided priors, ZeroSep naturally handles unseen source types, a critical advantage for real-world acoustic scenes. The framework’s compatibility with different diffusion backbones (e.g., AudioLDM, Tango) showcases its versatility.\n3. Conceptual Innovation: ZeroSep represents a paradigm shift by repurposing generative diffusion models for discriminative separation tasks, eliminating the need for supervised training. This approach addresses the data scarcity and generalization challenges of traditional methods.\n\n## Weaknesses\n1. Latent Inversion Limitations: The inversion process may introduce approximation errors, as acknowledged by the authors. While both DDIM and DDPM inversion are tested, the impact of these errors on separation fidelity—especially for complex mixtures—requires deeper analysis.\n2. Prompt Engineering Dependence: Effective separation relies on precise text prompts. The paper demonstrates performance with ideal prompts but does not address how suboptimal or ambiguous prompts affect results, nor does it explore automated prompt generation in depth.\n3. Lack of novelty: The author emphasizes that the model is used for audio separation, so the baseline systems compared in the experimental and method sections are very limited. In fact, separation is a subtask of audio editing, and authors should compare more editing models to demonstrate the innovation and effectiveness of their methods. Because there are many very similar methods in the field of audio editing[1][2][3][4], the author should **analyze and emphasize differentiation** between the proposed method and these methods, especially for [1][2].\n\n[1] Meng C, He Y, Song Y, et al. Sdedit: Guided image synthesis and editing with stochastic differential equations[J]. arXiv preprint arXiv:2108.01073, 2021.\n\n[2] Jia Y, Chen Y, Zhao J, et al. AudioEditor: A Training-Free Diffusion-Based Audio Editing Framework[J]. arXiv preprint arXiv:2409.12466, 2024.\n\n[3] Liu H, Wang J, Li X, et al. MEDIC: Zero-shot Music Editing with Disentangled Inversion Control[J]. arXiv preprint arXiv:2407.13220, 2024.\n\n[4] Zhang Y, Ikemiya Y, Xia G, et al. MusicMagus: zero-shot text-to-music editing via diffusion models[C]//Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence. 2024: 7805-7813.", "was_revised": true, "final_justification": "The author proposed an audio separation method that does not require training and has advantages in audio separation tasks. The drawback is that although the separation task is not the same as the extraction task in audio edit, it is still quite similar in itself.", "rebuttal": "**We thank the reviewer for highlighting our mechanistic analysis, open‑set capability, and the “paradigm‑shift” use of text‑guided diffusion for separation.** We clarify and address each question below.\n\n### Q1. Latent inversion approximation and mitigation\n\nDDIM inversion is fast but introduces **stepwise drift**. As the reviewer mentioned, MEDIC found DDIM inversion works for very small edits, yet it accumulates reconstruction error, breaks musical structure, and offers no fine‑grained control. MEDIC tackles all three pain points with a plug‑and‑play framework that needs no extra training. Therefore, MEDIC pointed out a direction to mitigate the latent inversion error, which will also apply to our method. Apart from MEDIC, there are more correction methods that exist in vision. ZeroSep is drop‑in compatible with each:\n\n* **Null‑text (embedding) optimization**: optimizes the unconditional embedding along the trajectory to match the source mixture, compensating for score error.\n* **PnP‑style inversions**: explicitly estimate and inject residual noise to correct drift.\n* **DDPM‑style stochastic inversion**: by recovering per‑step noise latents, it better matches the forward process and reduces accumulation compared with purely deterministic DDIM inversion.\n\nWe currently use a DDPM‑based inversion variant precisely to **mitigate drift**; we will expand the discussion on inversion methods and possible mitigation.\n\n### Q2.  Prompt sensitivity & hierarchical/open‑set behavior\n\nWe probe 11 instruments with two perturbations per class:\n\n* **Ambiguous (hypernym / family):** *violin → string instrument*, *flute → woodwind*.\n* **Suboptimal (underspecified / distractor adj.):** *trumpet → bright instrument*, *cello → low tone*.\n\n**Results (vs. canonical prompts).**\n\n| prompts    |              FAD ↓ | LPAPS ↓ |              C‑A ↑ |              C‑T ↑ |\n| ---------- | -----------------: | ------: | -----------------: | -----------------: |\n| canonical  |              0.377 |   4.669 |              0.615 |              0.271 |\n| ambiguous  | 0.388 (**+0.011**) |   4.692 | 0.583 (**−0.032**) | 0.280 (**+0.009**) |\n| suboptimal | **0.336 (−0.041)** |   4.714 |     0.577 (−0.038) |     0.248 (−0.023) |\n\n**Interpretation.** Hypernyms keep the correct **family**, yielding small degradations; the slight **C‑T rise** suggests the classifier judges family‑level matches as more consistent, while **C‑A** (accuracy to ground‑truth class) drops. Suboptimal descriptors sometimes improve **distributional proximity (FAD)** yet hurt **class accuracy**, indicating plausible but less precise extractions. We will add this table and clarify metric roles.\n\n### Q3. Novelty vs. editing‑oriented methods \\[1‑4]**\n\n| Method             | Task                 | Per‑clip optimization  | Separation analysis                  |\n| ------------------ | -------------------- |  -------------- | ---------------------------------- |\n| SDEdit \\[1]        | **Image** editing    | ✗              | ✗                                 |\n| AudioEditor \\[2]   | Audio **editing**    | **✓ (> 1 min)** | ✗                        |\n| MEDIC \\[3]         | Music editing        | ✗              | ✗                                  |\n| MusicMagus \\[4]    | Music editing        | ✗              | ✗                                 |\n| **ZeroSep (Ours)** | **Audio separation** | ✗              | **✓**  |\n\nKey differences:\n\n1. **Task** – We target *source separation*, not waveform replacement. Separation demands preserving mixture consistency and yielding *N* disjoint sources, which editing pipelines cannot guarantee (see Table 1 AudioEdit row)\n2. **Training‑free *and* optimization‑free** – Unlike AudioEditor’s per‑prompt null‑text fitting, ZeroSep is runtime‑constant.\n3. **Separation analysis** – Our paper dissects the role of inversion, guidance weight, and prompt specificity—analysis particularly related to separation task, yielding different findings compared to \\[1‑4].\n\nNonetheless, we will add \\[1‑4] to Related Work to include a in-depth discussion."}, {"review_id": "7Fdcsz5uII", "reviewer": "Reviewer_V2hC", "summary": "In this paper the authors propose a training-free way of doing audio source separation via a diffusion model, without requiring the user to re-train or fine-tune the diffusion model for the particular separation task (like LASS-Net / AudioSep). \n\nThe method first inverts the observed variable (the mixture $x^{mix}$) to the latent domain via the forward process (additionally given a conditional text embedding $c_{inv}$ describing the mixture). Then, given the resulting noisy latent $x_T$ one does the reverse process with classifier free guidance, setting the $c_{rev}$ prompt to describe a source in the mix. \n\nThe authors show that the best results are obtained with classifier free guidance weighting of 1. The authors also ablate the choice of the diffusion model, the inversion strategy (DDIM inversion vs DDPM inversion), the effect of the classifier free guidance weight, and the type of text embedding $c_{inv}, c_{rev}$ to use during inversion and conditional inference, respectively.", "questions": "- Line 125: Bold subscripts $\\textbf{0}$ an $\\textbf{T}$ while following text uses normal font.\n- Line 127: \"parameterized by θ\" should be \"parameterized by a set of weights \\theta\".\n- Line 128: \"reversing the diffusion\" should be \"reversing the diffusion process\".\n- Line 128: If one assume using AudioLDM2, one first has to decode the final latents to obtain the mel-spectrograms. Also, there are many models (e.g., Stable Audio Open) which do not operate over mel-spectrograms.\n- Line 155: Here again the latent space doesn't have nothing to do with mel-spectraograms. Specify this is your particular setting. \n- Line 163: Audio source separation is not an inherently discriminative task. It can be called discriminative if one uses discriminative models to solve the task. The presented method and many other works effectively engage in generative source separation. See this recent survey for a detailed listing of works in generative source separation: https://arxiv.org/abs/2506.08457v1 (references [247]–[253]).", "limitations": "yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n\n- The authors apply successfully DDIM and DDPM inversion for solving source separation via diffusion without requiring to have a pre-trained audio model for this specific task. Solving this task zero-shot via a powerful pre-trained text-conditioned diffusion model feels a natural direction, instead of having to retrain (e.g., LASS-Net / AudioSep), not taking advantage of an available prior model. \n\n**Weaknesses**\n\n**Missing citations**\n\n- First and foremost, there are many important citations that the paper is missing. For example:\n  - It does not cite modern latent diffusion models such as the state-of-the-art Stable Audio Open (Evans et al. 2025) which greatly improves over AudioLDM2.\n  - It does not cite prominent previous baselines for performing source separation with diffusion models like SepDiff (Chen et al. 2023) and DiffSep (Scheibler et al. 2023). Those were the two first papers solving source separation via diffusion models, so citing them in a paper about source separation via diffusion models should be the minimum. To be even more correct, the first model to use score-based methods for source separation was (Jayaram et al. 2020), appearing even before the release of the DDPM paper dated 19 June 2020 (see https://arxiv.org/abs/2006.11239v1), and was based on the first formulation of Yang Song of denoising score-matching employing noise-conditioned score networks (NCSNs) (Song et al. 2019); more about the paper of Jayaram et al. in the following).\n  - The authors do not cite the highly relevant DITTO baseline (Novack et al. 2024) that also does inference-only editing without modifying the base diffusion model.\n\n**Novelty overclaim**\n\n- It is not true that ZeroSep is the *\"first training-free audio source separation framework repurposing pre-trained text-guided audio diffusion models\"* as the paper claims (verbatim). Generalized Multi-Source Diffusion Inference (GMSDI) (Postolache et al. 2024) sets out to acomplish the same task (to be precise more tasks, also including accompaniment generation) with a pre-trained text-guided audio diffusion model without any custom training or fine-tuning for the task. That method differs from the present paper since it uses posterior sampling with a Dirac-based likelihood function instead of a inversion mechanism, but it does accomplish literally the same task with the same assumptions on not pre-training / fine-tuning the base diffusion model for the separation task.\n- Also regarding the technical contribution:\n   1. The methodology is very similar to the text version of *AudioEditor*, which the authors compare to, albeit in the unsupervised modality. The authors of AudioEditor presented a text-based inversion method for audio that largely overlaps with the proposed methodology, albeit being applied to editing instead of source separation, where it is applied in this paper.\n   2. Relating the GMSDI paper, a very similar principle is presented for source extraction in (Eq. 14) (*GMSDI extractor*). While doing posterior sampling instead of inversion, the user only has to provide a description of the mixture and a description of the source to be extracted. While the authors compare with training free methods, those are not text-guided like GMSDI. The authors should compare with such text-based posterior sampling, keeping the underlying generative model fixed, in order to show that their methodology improves over it.\n  3. Understanding that CFG weighting is crucial is not something surprising, since it is well known that setting $w= 0$ removes conditioning and by setting it too high results degrade (for example see Fig. 8 in (Karras et al. 2024) and Table 1 in (Postolache et al. 2024)). \n\n**Concerning metrics**\n\nFirst thing, the inadequacy of standard SS metrics like SDR illustrated in Table 3, was previously studied in (Jayaram et al. 2020), see Figure 4 (I believe this merits a citation). They also proposed to use FID as a distribution-based metric (they were doing image source separation in that paper). Regarding FAD, the authors do not detail which embedding space is used for computing FAD (we have vggish, Encodec, CLAP, etc. and they work in different ways (Gui et al. 2024)).\n\nI do not think that the FAD metric (which measure semantic similarity on sets), together with the CLAP / LPAPS metrics\n(which encode all temporal content in a global compressed vector) are sufficient to quantify if we are really separating the content or we are only generating something with the same semantic qualities which, however could not be consistent with the ground truth source in the mixture. For example, doing an L1 / L2 loss on Encodec embeddings could give something more robust An even better metric is the  MSE on mel-spectograms like in (Karchkhadze et al. 2025). They use the AudioLDM baseline (like the authors) for doing supervised source separation. They show improvement over a baseline in source separation, showcasing that such a metric is useful for measuring  source separation even when generating via a latent diffusion model. \n\nThus, Table 1, should be improved,  because the authors only match the semantic content at the moment, which is clearly enforced by the textual prompt during generation.\n\n**Summary**\n\nWhile the direction is positive, there are problems concerning (i) missing important citations (ii) novelty overclaim and (most importantly) (iii) the authors do not compute any metric that matches the outcomes with the ground-truths at a temporal level (they do only semantic level). As such, I am inclined to reject the paper at the moment; authors can improve their paper solving these three issues.\n\n**References**\n\n- Evans, Zach et al. “Stable Audio Open.” ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2025.\n- B. Chen, C. Wu, and W. Zhao, “SEPDIFF: Speech separation based on denoising diffusion model,” in ICASSP, Jun. 2023, pp. 1–5.\n- R. Scheibler, Y. Ji, S.-W. Chung, J. Byun, S. Choe, and M.-S. Choi, “Diffusion-based generative speech source separation,” in ICASSP, Jun. 2023, pp. 1–5.\n- Jayaram, Vivek, and John Thickstun. \"Source separation with deep generative priors.\" International Conference on Machine Learning. PMLR, 2020.\n- Song, Yang, and Stefano Ermon. \"Generative modeling by estimating gradients of the data distribution.\" Advances in neural information processing systems 32 (2019).\n- Novack, Zachary, et al. \"DITTO: Diffusion Inference-Time T-Optimization for Music Generation.\" Forty-first International Conference on Machine Learning. 2024.\n- Hila Manor and Tomer Michaeli. Zero-shot unsupervised and text-based audio editing using DDPM inversion. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors, Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 34603–34629. PMLR, 21–27 Jul 2024.\n- Postolache, Emilian, et al. \"Generalized multi-source inference for text conditioned music diffusion models.\" ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2024.\n- Karras, Tero, et al. \"Guiding a diffusion model with a bad version of itself.\" Advances in Neural Information Processing Systems 37 (2024): 52996-53021.\n- Gui, Azalea, et al. \"Adapting frechet audio distance for generative music evaluation.\" ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2024.", "was_revised": true, "final_justification": "I decided to raise the score given that authors have addressed my concerns (such modifications should appear in the camera ready version), namely adjusting the novelty claim, adding missing references, stating in what way the method differs from AudioEditor and integrating L1 / L2 embedding metrics in the quantitative evaluation.", "rebuttal": "We sincerely thank you for the thorough, constructive review. Below we (i) add the missing citations, (ii) clarify novelty and positioning, (iii) extend the evaluation with time‑aligned metrics, and (iv) correct wording/formatting issues.; we will incorporate all changes in the revised manuscript.\n\n---\n\n### Q1 Missing Citations\nWe thank the reviewer for providing the paper list to help us make a more comprehensive bibliography. We have added all requested references as followed:\n\n| Ref.                                                | Where added                                           |\n| --------------------------------------------------- | ----------------------------------------------------- |\n| Stable Audio Open (Evans ’25)                       | p3 L91‑92 (modern T2A backbones)                      |\n| SepDiff / DiffSep (Chen & Scheibler ’23)            | p3 L103‑106 (diffusion‑based separation)              |\n| Jayaram et al. ’20                                  | p3 L106 (earliest score‑based separation)             |\n| DITTO (Novack ’24)                                  | p3 L110‑112 (inference‑only editing)                  |\n| GMSDI (Postolache ’24)                              | p3 L108‑110  (zero‑shot text‑guided separation) |\n\n\nThese changes ensure the paper accurately cites prior works.\n\n---\n\n### Q2. Novelty Clarification & Positioning\n\nWe agree that our original phrasing overstated novelty.  We now state:\n\n> “ZeroSep is *a* training‑free audio source‑separation framework that repurposes pre‑trained text‑guided diffusion models…”\n\n**Difference  to GMSDI.**\nGMSDI (Postolache et al. 2024) performs zero‑shot separation, but via **posterior sampling with a Dirac likelihood**. ZeroSep instead performs **latent‑space inversion plus text‑conditioned denoising**. While GMSDI can indeed perform zero-shot separation given appropriate textual descriptions of the input during inference, it requires train a text-conditioned diffusion model on mixtures only, which is not as versatile as our method where any off-the-shelf text-guided diffusion model can work. Also, GMSDI's effectiveness heavily depends on the quality and relevance of the textual embeddings and the accuracy of the provided text descriptions. And the open-set ability is constrained due to the mixture dataset it train on. Moreover, as **no open-source implementation or pretrained checkpoints** for GMSDI are currently available, we were unable to include comparative results from this method in our evaluation during the rebuttal window. However, we will include a detailed discussion with GMSDI in the revised manuscript. We summarize the difference in the table below:\n\n| Aspect                 | GMSDI                                       | **ZeroSep (ours)**                                         |\n| ---------------------- | ------------------------------------------- | ---------------------------------------------------------- |\n| Mechanism              | Posterior *sampling* with Dirac likelihood  | *Latent inversion* followed by conditional denoising       |\n| Base model requirement | Text‑conditioned *mixture*‑only training    | Any pretrained T2A model (AudioLDM1, AudioLDM2, …) |\n| Conditioning interface | Needs *two* text prompts (mixture & target) | One prompt per target; no mixture text needed              |\n| Open‑set ability       | Bound by training mixture corpus            | Inherits full vocabulary of the base T2A model             |\n\n\n**Difference to AudioEditor.**\nAudioEditor targets *editing*; ZeroSep targets *separation*.  Beyond task difference, ZeroSep introduces **separation-based analysis** that focus on at what situations a T2A diffusion model can be repurposed to a separation model. These design principles are critical for accurate multi‑source separation; AudioEditor does not address them.\n\n**Insight on CFG Weighting.**\nWe agree with the reviewer that generation-oriented CFG trade‑offs is well taken, where lower CFG reduce conditioning strength and higher CFG may degrade result quality. However we sweep CFG $\\in$ [0, 2] and find CFG=1 yields good trade-off between separation and degradation (under-separation or over-generation, see Fig. 3(a)). This specific CFG choice is not well stuided as most papers focus on scenarios where CFG > 1. For example, Fig. 8 in (Karras et al. 2024) studies CFG=4 and Table 1 in (Postolache et al. 2024) stuides CFG=(3,7.5,15,24) We clarify this distinction and will add more discussion in the revised manuscript.\n\n---\n\n### Q3 Metric Suite Expansion\n\n**Citation of Jayaram et al. 2020**\nThank the reviewer for providing the reference to better motivate our finding. We will cite Jayaram et al. 2020 in the Table 3.\n\n**Implementation of FAD**\nIn our paper, we use the embedding space of CLAP model to compute FAD.\n\n**Improve Table 1 with More Metrics:**\nWe thank the reviewer for the great suggestion! We agree that additional metrics like L1/L2 loss on Encodec embedding and MSE on mel-spectrogram can capture temporal fidelity better. Below we show the improved table:\n\n| Method | MUSIC FAD ↓ | MUSIC LPAPS ↓ | MUSIC C-A ↑ | MUSIC C-T ↑ | MUSIC MSE ↓ | MUSIC Encodec L1 (Emb.) ↓ | MUSIC Encodec L2 (Emb.) ↓ | AVE FAD ↓ | AVE LPAPS ↓ | AVE C-A ↑ | AVE C-T ↑ | AVE MSE ↓ | AVE Encodec L1 (Emb.) ↓ | AVE Encodec L2 (Emb.) ↓ |\n|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|\n| **Require Separation Training** | | | | | | | | | | | | | | |\n| LASS-Net | 1.039 | 5.602 | 0.204 | 0.014 | 32.860 | 0.6205 | 0.7338 | 0.626 | 6.062 | 0.232 | 0.011 | 12.612 | 0.5984 | 0.6737 |\n| AudioSep | 0.725 | 5.209 | 0.450 | 0.204 | **10.857** | 0.4232 | **0.4046** | 0.446 | 5.733 | 0.457 | 0.167 | 9.383 | **0.4949** | **0.5095** |\n| FlowSep | 0.402 | 5.578 | 0.564 | 0.245 | 15.134 | 0.5525 | 0.5869 | **0.258** | 4.719 | **0.493** | 0.082 | **7.785** | 0.6706 | 0.8402 |\n| **Separation Training Free** | | | | | | | | | | | | | | |\n| NMF-MFCC | 1.286 | 5.618 | 0.239 | -0.055 | 36.761 | **0.4230** | 0.4180 | 1.246 | 5.851 | 0.174 | **0.211** | 59.594 | 0.5766 | 0.6906 |\n| AudioEdit | 0.568 | 4.869 | 0.453 | 0.196 | 29.357 | 0.8059 | 1.3088 | 0.372 | 4.959 | 0.341 | 0.074 | 22.253 | 0.9588 | 1.7090 |\n| ZeroSep (Ours) | **0.377** | **4.669** | **0.615** | **0.271** | 15.470 | 0.5540 | 0.5912 | 0.269 | **4.537** | 0.442 | -0.001 | 11.076 | 0.7959 | 1.1258 |\n\nGenerally, on the newly added metrics, ZeroSep matches the supervised method FlowSep and outperform LASS-net on the MUSIC dataset, confirming genuine separation rather than prompt‑consistent hallucination.\n\n---\n\n### Q5 Minor Corrections & Wording\nWe will incorporate the following edits verbatim:\n\n| Line | Fix                                                                                                                                                                                |\n| ---- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |\n|  125 | Bold subscripts → normal font                                                                                                                                                      |\n|  127 | “parameterized by a set of weights \\$\\theta\\$”                                                                                                                                     |\n|  128 | “reversing the diffusion **process**”                                                                                                                                              |\n|  128 | Clarify decoding step for AudioLDM2; note that Stable Audio Open operates in waveform space                                                                                        |\n|  155 | Specify it as our particular setting                                                                                                                                 |\n|  163 | Rephrase: “While separation has often been framed as a discriminative task, we follow the recent line of *generative* source separation methods.” |"}]

Author rebuttal:

Meta-review: Summary:

The paper addresses the issue of open set source separation by leveraging a pre-trained text-to-audio diffusion model. The method uses text conditioning to guide the denoising process in the diffusion latent space (latent inversion approach to sound/source separation). The method is also model-agnostic, with the potential to improve with better generative models.

Strengths:

- Promising new paradigm for open set audio separation.
- Good results when compared with existing training-based generative methods. 
- Explores the role of classifier-free guidance (CFG) in the context of sound/source separation, which is not the standard understanding of CFG. Also interesting is the combination of inversion strategies with guidance weight and prompt selection.
- Well-written paper and relevant references (after reviewers' suggestions).

Weaknesses:

- Lack of intuition/motivation on why text-to-audio generative models should possess some sort of source separation capabilities. Of course the results provide part of the justification, but one would like to understand further...
- Computational cost (this is extensively addressed by the authors in the rebuttal).
- Possible prompt engineering dependence.

Reasons to accept/reject:

I think the novelty and the new paradigm for open set audio separation is already a very good reason to accept the paper. The authors also provide convincing results.

Discussion and rebuttal:

The authors mostly addressed all reviewers' comments and concerns. They also provided some partial justification on the lack of intuition/motivation in the rebuttal, which they promise to add to the main paper. In general, they provided very detailed and informed answers (and tables/results), which I hope they can add to the main paper, perhaps removing some other less important parts.
Decision: Accept (poster)