Multi-Source Diffusion Models for Simultaneous Music Generation and Separation
Authors: Giorgio Mariani
Irene Tallini
Emilian Postolache
Michele Mancusi
Luca Cosmo
Emanuele Rodolà
Venue: ICLR 2024 oral (2024)
Keywords: source separation
probabilistic diffusion models
music generation
Primary area: generative models
TL;DR: In this work, we define a diffusion-based generative model which is the first to be capable of both music generation and source separation. We also introduce the partial generation task, where we generate a subset of the sources given the others.

Abstract: In this work, we define a diffusion-based generative model capable of both music generation and source separation by learning the score of the joint probability density of sources sharing a context. Alongside the classic total inference tasks (i.e., generating a mixture, separating the sources), we also introduce and experiment on the partial generation task of source imputation, where we generate a subset of the sources given the others (e.g., play a piano track that goes well with the drums). Additionally, we introduce a novel inference method for the separation task based on Dirac likelihood functions. We train our model on Slakh2100, a standard dataset for musical source separation, provide qualitative results in the generation settings, and showcase competitive quantitative results in the source separation setting. Our method is the first example of a single model that can handle both generation and separation tasks, thus representing a step toward general audio models.

Published as a conference paper at ICLR 2024
MULTI-SOURCE DIFFUSION MODELS FOR SIMULTANE- OUS MUSIC GENERATION AND SEPARATION
Giorgio Mariani'
Irene Tallini"
Sapienza University of Rome
Sapienza University of Rome
mariani @di. unirummaa romal. it
tallini@di uni romal 1 it
Emilian Postolache*
Michele Mancusi
Sapienza University Rome
Sapienza University of Rome
postolache@di: uni romal.
mancusi @dii unn nomal.
Luca Cosmo!
Emanuele Rodolà
Ca' Foscari University of Venice luca. cos.o@unive..i.....
Sapienza University of Rome rodol Fodola@di. uni romal it
ABSTRACT
In this work, we define a diffusion- based generative model capable both music synthesis and source separation by learning the score of the joint probability density of sources sharing context. Alongside the classic total inference tasks (i.e., generating mixture, separating the sources), we also introduce and experiment on the partial generation task of source imputation, where we generate a subset of the sources given the others (e.g., play piano track that goes well with the drums). Additionally, we introduce a novel inference method for the separation task based Dirac likelihood functions. We train model on Slakh2100, standard dataset for musical source separation, provide qualitative results in the generation settings, and showcase competitive quantitative results in the source separation setting. Our method the first example of a single model that can handle both generation and separation tasks, thus representing step toward general audio models.
INTRODUCTION
Generative models have recently gained much attention thanks to their successful application in
many fields, such as NLP [OpenAl 2023 Touvron et al.) [2033] Santilli et al. [023), image synthesis
(Ramesh et al. 2022, Rombach et al.) 2022) protein design (Shin et al. 2021 Weiss al. 2023
Minello et al. 2024). Audio no exception to this trend (Agostinelli et al. 2023 Liu al. 2023
A peculiarity of the audio domain is that audio sample can be seen the sum of multiple individual sources (x1,..., the audio domain (e.g., speech), sources present in musical mixtures (stems) share a context given their strong interdependence. For example, the bass line of song follows the drum's rhythm and
Xx x. resulting in mixture y => X,, Unlike in other sub-fields of
"Equal contribution. Listing order is random. G.M. wrote most of the code, performed most objective experiments, and contributed the development of the Dirae separator. I.T. proposed and developed the idea Dirac separator and partly formalized its proof, contributed to the code and the objective experiments, especially concerning the Dirac separator, performed the subjective listening tests, and wrote substantial parts of the paper. E.P. proposed and developed the ideas of the source-joint Bayesian separator and the sub-FAD metric, partly formalized the proof of the Dirac separator, contributed the code and the objective experiments, especially concerning source imputation,
wrote substantial parts of paper. M.M. proposed the idea of using the source-joint model for music (and accompaniment) generation, proposed using the correction steps, and contributed to the objective experiments. Shared last authorship.

---

Published as a conference paper at ICLR 2024
dx(t)
-o (1) Vx(1) lonloog p(g)()()))
X(T
x(t)
X(O)
x
xx(4)
xx(0)
Xa(0) HIjiiuu
xx(4)
X.(T)
X2(()
xx(0) |
xx(0) [[()))
x(T
x(1)
x(O)
dx(f) (t) Vxto log p(((()
Figure 1: Proposed method. We leverage forward Gaussian process (right-to-left) to learn the score over contextual sets (the large boxes) of instrumental sources (the waveforms) across different time steps f. During inference, the process reversed (left-to-right), letting us perform the tasks of total generation, partial generation, or source separation (Figure
harmonizes with the melody of the guitar. Mathematically, this fact can be expressed by saying that the joint distribution of the sources p(X1,..... X,) does not factorize into the product of individual source distributions distribution over the mixtures p(y) since the latter can be obtained through the sum. The converse is more difficult mathematically, being inverse problem.
(xx))>=....... Knowing the joint p(X1,.... ......... .Ny implies knowing the
Nevertheless, humans have developed the ability to process multiple sound sources simultaneously in terms of synthesis (i.e., musical composition generation) and analysis (i.e., source separation). More specifically, composers can invent multiple sources {xI..... XN) that sum to consistent mixture y and, extract information about the individual sources (xi,..,,Xy XN] from mixture y.
This ability to compose and decompose sound is crucial for a generative music model. A model designed assist in music composition should be capable isolating individual sources within mixture and allow for independent operation on each source. Such a capability would give the composer maximum control over what to modify and retain in composition. Therefore, we argue that compositional (waveform) music generation is highly connected to music source separation.
To our knowledge, no model in deep learning literature can perform both tasks simultaneously. Models designed for the generation task directly learn the distribution p(y) over mixtures, collapsing information needed for the separation task. this case, we have accurate mixture modeling but no information about the individual sources. It is worth noting that approaches that model the distribution mixtures conditioning on textual data (Schneider et al. 2023 Agostinelli et al. 2023) face the same limitations. Conversely, models for source separation (Défossez et al. 2019) either target p(x1,.... XN y), conditioning on the mixture, or learn a single model (x,,) for each source distribution (e.g., G, in weakly-supervised manner) and condition the mixture during inference (Jayaram & Thickstun| 2020 Postolache et al. (202a)). In both cases, generating mixtures is impossible. In the first case, the model inputs mixture, which hinders the possibility of unconditional modeling, not having direct access to p(x, p(.,,.... x (or equivalently to p(y)). ). In the second case, while we can accurately model each source independently, all essential information about their interdependence is lost, preventing the possibility of generating coherent mixtures.
Contribution. Our contribution is three-fold. (i) First, we bridge the gap between source separation and music generation by learning p(x1.... XN ). the joint (prior) distribution of contextual sources the same song). For this purpose, we use the denoising score-matchine
(i.e.. those belonging framework to train Multi-Source Diffusion Model (MSDM). We can perform both source separation and music generation during inference by training this single model. Specifically, generation is achieved by sampling from the prior, while separation is carried out by conditioning the prior
2

---

Published as a conference paper at ICLR 2024
the mixture and then sampling from the resulting posterior distribution. (ii) This new formulation opens the doors to novel tasks in the generative domain, such as source imputation, where we create accompaniments by generating a subset of the sources given the others (e.g.. play a piano track that goes well with the drums). (iii) Lastly, to obtain competitive results on source separation with respect to state-of-the-art regressor models (Manilow et al., 2022) on the Slakh2100 (Manilow al. 2019) dataset, we propose new procedure for computing the posterior score based on Dirac delta functions, exploiting the functional relationship between the sources and the mixture.
RELATED WORK
2.1 GENERATIVE MODELS FOR AUDIO
Deep generative models for audio learn, directly or implicitly, the distribution of mixtures, represented in our notation by (y), possibly conditioning on additional data such as text. Various general-pun ppurpsse generative models, such as autoregressive models, GANs (Donahue et al. 2019), and diffusion models, have been adapted for use in the audio field.
Autoregressive models well-established in audio modeling (van den Oord et al. 2016). Jukebox (Dhariwal et al., 2020) proposed to model musical tracks with Scalable Transformers Vaswani cet |l. on hierarchical discrete representations obtained through VQ-VAEs (van den Oord et al. 2017). J. Furthermore, using a lyrics conditioner, this method generated tracks with vocals following the text. However, while Jukebox could model longer sequences in latent space, the audio output suffered from quantization artifacts. Newer latent autoregressive models (Borsos et al. 2023 Kreuk al. 2023) can handle extended contexts, more coherent generations and, by incorporating residual quantization (Zeghidour et al., (02T). output more naturally sounding samples. State-of-the-as latent autoregressive models for music, such as MusicLM (Agostinelli al., 2023), can guide generation by conditioning on textual embeddings obtained via large-scale contrastive pre-training (Manco et al. 2022; Huang et al., 2022). MusicLM can also input a melody and condition on text for style transfer. A concurrent work, SingSong (Donahue et al. (023), introduces docal-to-mixture accompaniment generation. Our accompaniment generation procedure differs from the latter since we perform generation at the stem level in composable way, while the former outputs single mixture.
DiffWave (Kong et al., 202T) and WaveGrad (Chen et al., 202T) were the first diffusion (score) based generative models in audio, tackling speech synthesis. Many subsequent models followed these
preliminary works, mainly conditioned to solve particular tasks such as speech enhancement (Lu
et al. 2021 Serra al. 2022, Sawata al.) 2023 Saito et al. 2023). audio upsampling (Lee &
Han 2021] Yu et al.] 2023). MIDI-to-wavefone
(Mittal et al. 2021 Hawthorne et al.) 2022). or
spectrogram-to-MID generation (Cheuk al. (002)). The first work in source-specific generation
with diffusion models is CRASH (Rouard & Hadjeres 2021). (Yang et al.) 2023] Pascual et al.
2023 Liu et al., 2023) proposed text-conditioned diffusion models to generate general sounds, not focusing on restricted classes such as speech or music. Closer to our work, diffusion models targeting
the musical domain are Riffusion (Forsgren & Martiros (022) and Moûsai (Schneider et al. (023). Riffusion fine-tunes Stable Diffusion (Rombach et al. 2022). large pre-trained text-conditioned vision diffusion model, over STFT magnitude spectrograms. Moûsai performs generation in a latent domain, resulting in context lengths that surpass the minute. Our score network follows the design of the U-Net proposed in Moûsai, albeit using the waveform data representation.
2.2 AUDIO SOURCE SEPARATION
Existing audio source separation models can be broadly classified into deterministic and generative.
Deterministic source separators are parametric models that input the mixtures and systematically
extract one or all sources. These models are typically trained with regression loss (Gusó al.
2022] on the estimated signal represented as waveform (Llus et al. [2019] Luo & Mesgarani 2019
Défossez al. 2019), STFT (Takahashi al., 2018 Choi et al.] (021). or both (Défossez) 2021)
On the other hand, generative source separation models based on (independent) Bayesian inference
learn prior model for each source, thus targeting the distributions (px. (x,)))==.....)) The mixture
is observed only during inference, where likelihood function connects it to its constituent sources.
The literature has explored different priors, such as GANs (Subakan & Smaragdis 2018: Kong et al.)
3

---

Published as a conference paper at ICLR 2024
x(7)
x(()
x(0]
6.07
(6
8: (0)
x(T)
x(()
x(0)
k,7%
8,00
8.07
8:10
0,10
8,10
8,00
Total generation
8.10)
x(7)
xx()
x:07
4,07
Au10
A,(0]
y
y
y
X:10
X,
Source separation
Partial generation
Figure 2: Inference tasks with MSDM. Oblique lines represent the presence of noise in the signal, decreasing from left to right, with the highest noise level at time I'' when we start the sampling procedure. Top-left: We generate all stems in mixture, obtaining total generation. Bottom-leftt We perform partial generation (source imputation) by fixing the sources x; (Bass) and X3 (Piano) and generating the other (Drums) and (0) (Guitar). We denote with x1 (t) and xs(t), the noisy stems obtained from x; and x3 via the perturbation kernel in Eq. (T). Right: We perform souree separation by conditioning the prior with a mixture following Algorithm!] !
sources *y (0)
2019
Narayanaswamy et al. 2020), normalizing flows (Jayaram & Thickstun 2020 Zhu al.
2022
and autoregressive models (Jayaram & Thickstun 2021 Postolache et al., 2023a
The separation method closer to ours is NCSN-BASIS (Jayaram & Thickstun 2020). This method was proposed for image source separation, using Langevin Dynamics to separate the mixtures with an NCSN score-based model. It employs a Gaussian likelihood function during inference, which, demonstrate experimentally. is sub-optimal compared our novel Dirac-based likelihood function. The main difference between our method and other generative source separation methods (including NCSN-BASIS) is the modeling of the full joint distribution. As such, source separation and synthesize mixtures or subsets of stems with a single model.
perform
Contextual information between sources is explicitly modeled in (Manilow et al.) 2022) and (Posto- lache et al. 2023b). The first work models the relationship between sources by training an orderless NADE estimator, which predicts subset of the sources while conditioning on the input mixture and the remaining sources. The subsequent study achieves universal source separation (Kavalerov al. 2019 Wisdom al., (2220) through adversarial training, utilizing a context-bart discriminator to model the relationship between sources. Both methods are deterministic and conditioned on the mixtures architecturally. The same architectural limitation present in diffusion- -asedd (Scheibler
et al. 2023 Lutati al.) (023) diffusion-inspired inspired (Plaja-Roglans et al. (0022) conditional ap- proaches. Our method sets itself apart as it proposes a model not constrained architecturally by a mixture conditioner, so we can also perform unconditional generation.
BACKGROUND
The foundation of our model lies in estimating the joint distribution of the sources p p(Xi,... .., xN). Our approach is generative because we model an unconditional distribution (the prior). The different tasks are then solved at inference time, exploiting the prior.
We employ a diffusion- (Sohl-Dickstein et al. 2015) Ho et al. 2020 generative model trained via denoising score-matching (Song Ermon 2019) learn the prior. Specifically, we present our formalism by utilizing the notation and assumptions established in (Karras et al.) (022). The central idea of score-matching (Hyvärinen] 2005 Kingma & LeCun 2010 Vincent 2011 is to
Ho
approximate the "score" function of the target distribution p(x). namely Vx log p(x), rather than the distribution itself. To effectively approximate the score in sparse data regions, denoising diffusion methods introduce controlled noise to the data and learn to remove it. Formally, the data distribution perturbed with a Gaussian perturbation kernel:
p(x(t) x()) ((x())=x(0,,<((1)
(1)
4

---

Published as a conference paper at ICLR 2024
where the parameter (t) regulates the degree of noise added to the data. Following the authors in (Karras et al., 2022). we consider an optimal schedule given by ott) = t. With that choice of (t)), the forward evolution of data point x(t) in time is described by probability flow ODE (Song et al. 2021]
dx(() - (1) V x(1) log px))()(()) dt.
For 1 7 >> 0, data point x (T) is approximately distributed according a Gaussian distribution (x)(t);00 0, (T))),, from which sampling is straightforward. Eq. (2) can be inverted in time,
resulting in the following backward ODE that describes the denoising process:
dx(() - VL)V x(1) logp(x(()) dc.
Sampling can be performed integrating Eq. (3) with standard ODE solver, starting from initial
(noisy) sample drawn from (x((t);00 0, o (T))). The score function, is approximated by neural network 5 (()), minimizing the following score-matching loss:
(1-77 (x00)) (0000000000000000000000000000)00 (| (x(t), - Vx(() log (x(t) xx(0))]
By expanding ((((() x()) with Eq. ① the score-matching loss simplifies to:
Bioe Bo ((,o))) x(0)]l
where we define S
METHOD
4.1
MULTI- AUDIO DIFFUSION MODELS
In our setup, we have N distinct source waveforms (xi...., xN) with x E RD for each The sources coherently sum to a mixture y = =E Xx. We also use the aggregated form (xI...., xN) E R-xxD_ In this setting, multiple tasks can be performed: one may generate consistent mixture or separate the individual sources x from given mixture We refer to the first task as generation and the second as source separation. A subset of sources can also be fixed the generation task, and the others can be generated consistently. We call this task partial generation or source imputation. Our key contribution is the ability to perform all these tasks simultaneously by training a single multi-soured diffusion model (MSDM), capturing the prior p(x1,..., xx). The approximates the noisy score function:
model, illustrated in Figure !
V xt) log p(x(t)) V (xxx log p(x: (t)..... xx(()).
with neural network:
5 (x((1, (()) R x R - -
(4)
where x()) = (t), (t), (t) denotes the sources perturbed with the Gaussian kernel in Eq. We describe the three tasks (illustrated in Figure 2 using the prior distribution:
(
Total Generation.
This task requires generating plausible mixture y. It can be achieved by sampling the sources {x1, {x,, from the prior distribution and summing them obtain the mixture y.
Partial Generation. Given a subset sources, this task requires generating a plausible accompaniment. We define the subset of fixed sources Xx and generate the remaining sources X7 by sampling from the conditional distribution p(x7 xg).
Source Separation. Given a mixture y. this task requires isolating the individual sources that compose it. It can be achieved by sampling from the posterior distribution p(x y).
4.2 INFERENCE
The three tasks of our method are solved during inference by discretizing the backward Eq. (3) Although different tasks require distinct score functions, they all originate directly from the prior score function in Eq. 4 We analyze each of these score functions in detail. For more details on the discretization method, refer to Section
5

---

Published as a conference paper at ICLR 2024
Algorithm 'MSDM Dirac' sampler for source separation.
Require: number discretization steps for the ODE, R number of corrector steps, (o.... noisc
schedule, Schum
1: Initialize & M/0, I 2: a f- min(S
v2 1) 3: for irrie / to do
4: for R to do
- d. (a+1) + <~ (0,1)
6:
x V0² 0.f
zt -IN Σ Xn.
for to do
10:
5n < Sn (z,
SN (2, 6)
11:
end for
12:
g+ gl.... -N-11
X1NN-1 1 + XI:N 11 (0, 8)g
14:
& RINN-1,
Σ Dー knl
15:
ifr > Othen
16:
6 E~N(0,1
17:
x & + oo 0 1
18: end if
19: end for
20:
end for
21:
return &
4.2.1
TOTAL GENERATION
The total generation task is performed by sampling from Eq. 3 using the score function in Eq. (4)
The mixture is then obtained by summing over all the generated sources.
4.2.2 PARTIAL GENERATION
In the partial generation task, we fix subset of source indices I {I...., .N} and the relative sources == {x,, J The goal is to generate the remaining sources X = {x, we consistently,
where 7 (,,..., N} I. To do so, estimate the gradient of the conditional distribution:
Vxy(1) log p(x>(t)) xz())).
(5)
This falls into the setting of imputation or, as is more widely known in the image domain, inpainting. We approach imputation using the method in (Song 202I). The gradient in Eq. (5) approximated as follows:
Vxx(1) log
where スy is sample from the forward process: & (t) ((xx(t); xy (x)), (t) The square bracket operator denotes concatenation. Approximating the score function, we write:
(xz(t) log (()
x (t) ~ (()) (t)),
where St denotes the entries of the score network corresponding to the sources indexed by T.
4.2.3 SOURCE SEPARATION
We view source separation as a specific instance of conditional generation, where we condition the generation process the given mixture y = ((). This requires computing the score function of the posterior distribution:
(6) Standard methods for implementing conditional generation for diffusion models involve directly estimating the posterior score Eq. (6) at training time (i.e., Classifier Free Guidance (Ho &
Vx(1) log p(x(t) (())).
Salimans 2021) estimating the likelihood function p(y(0) x(()) and using the Bayes formula

---

Published as a conference paper at ICLR 2024
to derive the posterior. The second approach typically involves training separate model, often classifier, for the likelihood score (i.c., Classifier Guidance (Dhariwal & Nichol 2021)).
In diffusion--aasee bbasdd generative source separation, learning a likelihood model is typically unnecessary because the relationship between x(t) and y(t) is represented by a simple function, namely the sum. A natural approach is to model the likelihood function based on such functional dependency. This is the approach taken by (Jayaram & Thickstun 2020), where they use Gaussian likelihood function:
N
(() x(1) x(0)) -v((()
Σ
with the standard deviation given by hyperparameter (t). The authors argue that aligning the r((t) value be proportionate (tt) optimizes the outcomes of their NCSN-BASI separator.
We present novel approximation of the posterior score function in Eq. (6) by modeling p(y(t)
|x())
Dirac delta function centered in Σ X, (t):
p(y(t) x(t)) I y00==
(8)
The complete derivation be found in Appendix A and present only the final formulation, which we call 'MSDM Dirac", The method constrains a source, without loss of generality xN. by setting (1) = (0) E xn. (1) and estimates:
N-1
V x.. logp(x(() (()) ~ 5% (f)......N_(())))))
Σ
X. (t))),
Σ xx (t))
where 1 S m S - and Sq S denote the entries of the score network corresponding to the m-th and N-th sources. Our approach models the limiting case wherein y(t) - in the Gaussian likelihood function. This represents scenario where the functional dependence between x(t) and yt) becomes increasingly tight, thereby sharpening the conditioning on the given mixture during the generation process. The pseudo-code for the 'MSDM Dirac' source separation sampler, using the Euler ODE integrator of (Karras et al., 2022), is provided in Algorithm [] The Euler ODE discretization logic uses the Schum mechanism of (Karras et al.) 2022) and optional correction steps (Song et al., 2021) (see Section C.3 for more details).
The separation procedure be additionally employed in the weakly-supervised source separation scenario, typically encountered in generative source separation (Jayaram & Thickstun 2020 /hu et al.) 2022 Postolache et al. 2023a). This scenario pertains to cases where know that specific audio data belongs to a particular instrument class while not having access to sets of sources sharing a context. To adapt to this scenario, we assume independence between sources p(x1,.... .x) I Pr (x,) and train separate model for each source class. We call the resulting model 'Independent Source Diffusion Model with Dirac Likelihood" ('ISDM Dirac'). We derive its formula together with formulas for the Gaussian versions 'MSDM Gaussian' and 'ISDM Gaussian' in Appendix B
EXPERIMENTAL RESULTS
We experiment on Slakh2100 Manilow al. 2019). a standard dataset for music source separation. We chose Slakh2100 because it has significantly larger quantity of data (145h) than other multi- source waveform datasets like MUSDB18-HO RRaii et al.) 2019) (10h). The amount of data plays a decisive role in determining the quality of a generative model, making Slakh2100 a preferable choice. Nevertheless, in Appendix [ we conduct study of data efficiency on MUSDB 18-H. Details on datasets, architecture, training, and sampling are provided in Appendix C
5.1 MUSIC GENERATION
The performance of MSDM on the generative tasks is tested through subjective and objective evaluation.
7

---

Published as a conference paper at ICLR 2024
Table Comparison between total generation capabilities of MSDM (Slakh2100) and equivalent architecture trained on Slakh2100 mixtures. Both subjective (quality and coherence, higher is better) and objective (FAD, lower is better) evaluations are shown. The quality and coherence columns refer to the average scores of the listening tests, with respective variances.
Model
FAD +
Quality 1
Coherence f
MSDM
6.55
6.51 + 2.19
6.35 +: 2.36
Mixture Model
6.67
6.15 2.47
5.67 2.60
Table Quantitative and qualitative results for the partial generation fask on Slakh2100. We use both subjective (quality and density, higher is better) and objective (sub-FAD, lower is better) evaluation metrics. The sub-FAD metric is reported for all combinations of generated sources (B; Bass, D: Drums, G: Guitar, P: Piano). The quality and density columns refer to the average scores of the listening tests, with respective variances.
Slakh2100
B
G
BD
BG
BP
DG
DP
GP
BDG
BDP
BGP
DGP
Quality
Density
MSDM
0.45
1.09
0,11
0.76
2.09
1,00
2.32
1.45
1.82
1.65
293
3.30
4.90
3.10
6.3
2.7
6.1 2.6
Subjective evaluation is done through listening tests, whose form format is reported in Appendix [] Concisely, we produce two forms, one for total generation and one for partial generation. In the first, subjects are asked to rate, from to 10, the quality and instrument coherence (i.e., how the instruments sound plausible together) of 30 generated chunks, of which 15 are generated by MSDM and 15 by model trained on mixtures (using the same diffusion architecture MSDM). In the second one, knowing the fixed instruments, subjects are asked to rate, from I to 10, the quality and the density of the generated accompaniment. Namely, 'quality" tests how the full chunk sounds plausible with respect to the ground truth data, and 'density' tests how much the generated instruments are present in the chunk. We also provide examples of music and accompaniment generation]]
For the objective evaluation of the generative tasks, we generalize the FAD protocol in Donahue et al. (2023)
total and partial generation tasks with ground truth mixtures chunks and I a set indexing conditioning sources (0 for total generation), we build dataset Dgen whose elements are the sum between conditioning sources (indexed by Σ) and the respective generated sources. We define the sub-FAD as FFFDFFFDDDD FAD((Draat Dgm)). We use VGGish embeddings (Hershey al. 2017 for computing the metric.
than source. Given Dreal dataset of
Results for total and partial generations are reported in Tables T and respectively, both for subjective and objective evaluations. Results in Table show a minimal difference between the model trained on mixtures and MSDM. This suggests that, given the same dataset and architecture, the generative power of MSDM is the same as the model trained on mixtures while being able to perform separation and partial generation. Table shows via the subjective results that the task of partial generation can be performed with non-trivial quality. Our method being the first able to generate any combination of partial sources, does not have competitor baseline for the objective metrics. We thus report the sub-FAD results of our method as baseline metrics for future research.
5.2
SOURCE SEPARATION
In order evaluate source separation, we use the scale-invariant SDR improvement (SI-SDR;) metric (Roux et al. 2019). The SI-SDR between a ground-truie truth source x,, and an estimate x,, is defined as:
||ax,, 1|2 +e
SI-SDR x,.) = 10 ||xx, x, 1|2 +e
where x._ te and 10-8 The improvement with respect the mixture baseline is defined
x *4
as SI-SDR, = SI-SDR(X,, xx.) - (X-SDR(xn.y))
On Slakh, compare supervised MSDM and weakly-supervised MSDM with the 'Demues' Défossez et al. 2019) and 'Demucs + Gibbs (512 steps)' regressor haselines from MMnnlow et al.
8

---

Published as a conference paper at ICLR 2024
Table 3: Quantitative results for source separation the Slakh2100 test set. We the SI-SDR; evaluation metric (dB higher better). present both supervised ('MSDM Dirac'. 'MSDM Gaussian") and weakly-supervised ('ISDM Dirac". 'ISDM Gaussian') ") separators and specify if correction step is used. "All" reports the average over the four stems.
Model
Bass
Drums
Guitar
Piano
All
Demucs Défossez et al. 2019 Manilow et al.
2022
15.77
19.44
15.30
13.92
16.11
Demucs Gibbs (512 steps)
Manilow et
2022)
17.16
19.61
17.82
16.32
17.73
Dirac Likelihood
ISDM
18.44
20.19
13.34
13.25
16.30
ISDM (correction)
19.36
20.90
14.70
14.13
17.27
MSDM
16.21
17.47
12.71
13.29
14.92
MSDM (correction)
17.12
18.68
15.38
14.73
16.48
Gaussian Likelihood (Jayaram & Thickstun
2020)
ISDM
13.48
18.09
11.93
11.17
13.67
ISDM (correction)
14.27
19.10
12.74
12.20
14.58
MSDM
12.53
16.82
12.98
12.90
MSDM (correction)
13.93
17.92
14.19
12.11
14.54
2022). the state-of-the-art for supervised music source separation on Slakh2100, aligning with the evaluation procedure of (Manilow et al. 2022). We evaluate over the test set of Slakh2100, using chunks of 4 seconds in length (with an overlap of two seconds) and filtering out silent chunks and chunks consisting of only one source, given the poor performance of SI-SDR, on such segments. We report results comparing our Dirae score posterior with the Gaussian score posterior (Jayaram & Thickstun 2020), using the best parameters of the ablations in Appendix and 150 inference steps.
Results are reported in Table 3 and show that: (i) The Dirac likelihood improves overall results, even outperforming the state of the art when applied to ISDM Bass and Drums (ii) adding correction step is beneficial (ii) MSDM with Dirac likelihood and one step of correction gives results comparable with the state of the art and superior to standard Demucs overall. We stress again that, while the baselines can perform the separation task alone, MSDM can also perform generative tasks.
CONCLUSIONS
We have presented a general method, based on denoising score-matching for source separation, mixture generation, and accompaniment generation in the musical domain. Our approach utilizes single neural network trained once, with tasks differentiated during inference. Moreover, we have defined sampling method for source separation. We quantitatively tested the model source separation, obtaining results comparable to state-of-the-art regressor models. We qualitatively and quantitatively tested the model on total and partial generation. Our model's ability handle both total and partial generation and source separation positions as significant step toward the development of general audio models. This flexibility paves the way for more advanced music composition tools, where users can easily control and manipulate individual sources within mixture.
6.1 LIMITATIONS AND FUTURE WORK
The amount of available contextual data constrains the performance of our model. To address this, pre-separating mixtures and training the separations, as demonstrated in (Donahue al.) (023). may prove beneficial. Additionally, it would be intriguing to explore the possibility of extending method situations where the sub-signals different function. Finally, future work could adapt the model to jointly model MIDI information (for example, extracted from sources (Lin al. 202 for further control.
not related by addition but rather by known but

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENTS
The authors were partially supported by the ERC grant no. 802554 (SPECGEO), PRIN 2020 project no. 2020TA3K9N (LEGO.AI), PRIN 2022 project no. 2022AL45R2 (EYE-FLAI, CUP H53D2230035000001), and PNRR MUR project no. PE0000013-FAIR,
REFERENCES
Andrea Agostinelli, Timo Denk, Zalán Borsos, Jesse Engel, Mauro Verzetti, Antoine Caillon, Qingqing Huang, Aren Jansen, Adam Roberts, Marco Tagliasacchi, et Musiclm: Generating music from text. arXiv preprint arXiv:2301.1133 2023.
Zalán Borsos, Raphaël Marinier, Damien Vincent, Eugene Kharitonov, Olivier Pietquin, Matt Sharifi, Dominik Roblek, Olivier Teboul, David Grangier, Marco Tagliasacchi, et al. Audiolm: language modeling approach to audio generation. IEEE/ACM Transactions Audio, Speech, and Language Processing, 2023.
Nanxin Chen, Yu Zhang, Heiga Zen, Ron Weiss, Mohammad Norouzi, and William Chan. Wave- grad: Estimating gradients for waveform generation. In International Conference on Learning Representations, 2021. URL /ttttttss////tteeeet//
net http:/oommmmmdnsMMi.--- a080
Kin Wai Cheuk, Ryosuke Sawata, Toshimitsu Uesaka, Naoki Murata, Naoya Takahashi, Shusuke Takahashi, Dorien Herremans, and Yuki Mitsufuji. Diffroll: Diffusion- based generative music transcription with unsupervised pretraining capability. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2023.
Woosung Choi, Minseok Kim, Jaehwa Chung, and Soonyoung Jung. Lasaft: Latent source attentive frequency transformation for conditioned source separation. In ICASSP 2021-2021 IEEE Inter- national Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 171-175. IEEE, 2021.
Alexandre Défossez. Hybrid spectrogram and waveform source separation. In Proceedings of the ISMIR 2021 Workshop on Music Source Separation, 2021.
Alexandre Défossez, Nicolas Usunier, Léon Bottou, and Francis Bach. Music source separation in the waveform domain. arXiv preprint arXiv:1911.13254 2019.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Ad- vances in Neural Information Processing Systems, volume 34, pp. 8780-8794. Curran Asso- ciates, Inc., 2021. URL http:ttttt/toooceeeiiiiiii mewreeedddinnnnnnnnneeeeeeeeeeee neurips. http:rasppppppppppeeeeeeeee cc//ppaperr/222//E1lll
Prafulla Dhariwal, Heewoo Jun, Christine Payne, Jong Wook Kim, Alec Radford, and Ilya Sutskever. Jukebox: A generative model for music. arXiv preprins arXiv:2005.00341 2020.
Chris Donahue, Julian McAuley, and Miller Puckette. Adversarial audio synthesis. In International Conference on Learning Representations, 2019.
Chris Donahue, Antoine Caillon, Adam Roberts, Ethan Manilow, Philippe Esling, Andrea Agostinelli, Mauro Verzetti, Ian Simon, Olivier Pietquin, Neil Zeghidour, et al. Singsong: Generating musical accompanimentss from singing. arXiv preprint arXiv:2301. 12662, 2023. URL (tRl::ttp:: htttttts///xxiii
lorg/bbs/230 2301. 12662
Seth Forsgren and Hayk* Martiros. Riffusion Stable diffusion for real-time music generation, 2022. URL htttttss//xiffusiooo com/ about
Enric Gusó, Jordi Pons, Santiago Pascual, and Joan Serrà. On loss functions and evaluation metrics for music source separation. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 306-310. IEEE, 2022.
10

---

Published as a conference paper at ICLR 2024
Curtis Hawthorne, Ian Simon, Adam Roberts, Neil Zeghidour, Josh Gardner, Ethan Manilow, and Jesse Engel. Multi-instrumenn music synthesis with spectrogram diffusion. In International Society for Music Information Retrieval Conference, 2022.
Shawn Hershey, Sourish Chaudhuri, Daniel PW Ellis, Jort F Gemmeke, Aren Jansen, R Channing Moore, Manoj Plakal, Devin Platt, Rif A Saurous, Bryan Seybold, et al. Can architectures for large-scale audio classification. In 2017 ieee international conference on acoustics, speech and signal processing (icassp), pp. 131-135. IEEE, 2017.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NeurlPS 202/ Workshop on Deep Generative Models and Downstream Applications, 2021. URL htttps://openreeet
net/ / wwr.uorriiiioe........ (rrrum/idddqBAAAXEEED
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Proceedings of the 34th International Conference on Neural Information Processing Systems, pp. 6840-6851, 2020.
Qingqing Huang, Aren Jansen, Joonseok Lee, Ravi Ganti, Judith Yue Li, and Daniel W. Ellis. Mulan: A joint embedding of music audio and natural language. In International Society for Music Information Retrieval Conference, 2022.
Aapo Hyvärinen. Estimation of non nor-normali statistical models by score matching. Journal of Machine Learning Research, 6(24):695-709) 709, 2005. URL CRRLhttp:/// http://mmm..rr/pppps/////////
hyvarinen05a. html
Vivek Jayaram and John Thickstun. Source separation with deep generative priors. In Proceedings of the 37th International Conference on Machine Learning, pp. 4724-4735, 2020.
Vivek Jayaram and John Thickstun. Parallel and flexible sampling from autoregressive models via langevin dynamics. In Proc. ICMI,, pp. 4807-4818 4818. PMIR, 2021.
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. In Advances in Neural Information Processing Systems, 2022. URL
htttps://oppenreett eew net/ tavee/foorune forum d= wFw.d==FFFFFoooccc
Yitzhak Katznelson. An Introduction to Harmonic Analysis. Cambridge Mathematical Library. Cambridge University Press, 3 edition, 2004. doi: 10. 10.1007/CB098 011880999811111653997 139165372.
Ilya Kavalerov, Scott Wisdom, Hakan Erdogan, Brian Patton, Kevin Wilson, Jonathan Le Roux, and John R Hershey. Universal sound separation. 2019 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA), pp. 175-179. IEEE, 2019.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun ). 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. httpt
7/ 1/arxiv. org/ abs/ 1111/1111111 6980
Durk P Kingma and Yann LeCun. Regularized estimation of image statistics by score matching. In Lafferty, C. Williams, J. Shawe-Taylor, Zemel, and A. Culotta (eds.), Advances in Neural Information Processing Systems, volume 23, Curran Associates, Inc., 2010.
Qiuqiang Kong, Yong Xu, Wenwu Wang, Philip J. B. Jackson, and Mark D. Plumbley, Single-channel signal separation and deconvolution with generative adversarial networks. In Proc. IJCAI, pp. 2747-2753. AAAI Press, 2019. ISBN 9780999241 141.
Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis. In International Conference on Learning Representations, 2021. URL httttttt///ppezeeets/s
net/ feet:/ooum?idda... xFK8Ymz5J
Felix Kreuk, Gabriel Synnaeve, Adam Polyak, Uriel Singer, Alexandre Défossez, Jade Copet, Devi Parikh, Yaniv Taigman, and Yossi Adi. Audiogen: Textually guided audio generation. In The Eleventh International Conference on Learning Representations, 2023. URL https
11

---

Published as a conference paper at ICLR 2024
Junhyeok Lee and Seungu Han. NU-Wave: A Diffusion Probabilistic Model for Neural Audio Upsampling. In Proc. Interspeech 2021, pp. 1634-1633, 1638, 2021. doi: 1022149//mccrspeechh2221111
Liwei Lin, Qiuqiang Kong, Junyan Jiang, and Gus Xia. A unified model for zero-shot music source separation, transcription and synthesis. In Proceedings of 22st International Conference on Music Information Retrieval, ISMIR, 2021.
Haohe Liu, Zehua Chen, Yi Yuan, Xinhao Mei, Xubo Liu, Danilo Mandic, Wenwu Wang, and Mark D Plumbley. Audioldm: Text-to-audio generation with latent diffusion models. arXiv preprint arXiv:2301. 2023.
Francesc Lluís, Jordi Pons, and Xavier Serra. End-to-end music source separation: Is it possible in the waveform domain? In INTERSPEECH, pp. 4619-4623, 2019. URL httptt:t/oo.....o
10. I0.21437/ Interspeeetii.e www.cch...1111111 2019-1177
Yen-Ju Lu, Yu Tsao, and Shinji Watanabe. A study on speech enhancement based on diffusion probabilistic model. In 2021 Asia-Pacific Signal and Information Processing Association Anmual Summit and Conference (APSIPA ASC), pp. 659-666. IEEE, 2021.
Yi Luo and Nima Mesgarani. Conv-tasnet: Surpassing ideal frequency magnitude masking for speech separation. IEEE/ACM transactions audio, speech, and language processing, 27(8): 1256-1266, 2019.
Shahar I Lutati, Eliya Nachmani, and Lior Wolf. Separate and diffuse: Using a pretrained diffusion model for improving source separation. arXiv preprint arXiv:2301.10752, 2023.
llaria Manco, Emmanouil Benetos, Elio Quinton, and György Fazekas. Learning music audio repre- sentations via weak language supervision. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing ( (ICASSP), pp. 456-460. IEEE, 2022.
Ethan Manilow, Gordon Wichern, Prem Seetharaman, and Jonathan Le Roux. Cutting music source separation some Slakh: A dataset to study the impact of training data quality and quantity. In Proc. IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA). IEEE, 2019.
Ethan Manilow, Curtis Hawthorne, Cheng-Zhi Anna Huang, Bryan Pardo, and Jesse Engel. Improving source separation by explicitly modeling dependencies between sources. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 291-295. IEEE, 2022.
Giorgia Minello, Alessandro Bicciato, Luca Rossi, Andrea Torsello, and Luca Cosmo. Graph generation via spectral diffusion. arXiv preprint arXiv:2402.18974 2024.
Gautam Mittal, Jesse Engel, Curtis Hawthomee, and Ian Simon. Symbolic music generation with diffusion models. In Proceedings of the 22nd International Society for Music Information Re- trieval Conference, 2021. URL https::t.... http///rchiieeeeeeee iwtiiiiieeiisiimm.i.. ismir www.ir.net/..eeee.mmeeemmme..eee net memeee/ismiittiim r2011 200058.paF
http://mmmmm2221ppppppep /
Vivek Narayanaswamy, Jayaraman Thiagarajan, Rushil Anirudh, and Andreas Spanias. Unsuper- vised audio source separation using generative priors. Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH, 2020-0ctober2267-2266.. 2020. doi: (0.21437/nterrpeechh2202
OpenAl. Gpt-4 technical report, 2023.
Santiago Pascual, Gautam Bhattacharya, Chunghsin Yeh, Jordi Pons, and Joan Serrà. Full-band general audio synthesis with score-based diffusion. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023.
Genís Plaja-Roglans, Miron Marius, and Xavier Serra. diffusion-inspired training strategy for singing voice extraction in the waveform domain. In Proc. of the 23rd Int. Society for Music Information Retrieval, 2022.
12

---

Published as a conference paper at ICLR 2024
Emilian Postolache, Giorgio Mariani, Michele Mancusi, Andrea Santilli, Luca Cosmo, and Emanuele Rodolà. Latent autoregrressive source separation. Proceedings of the AAAI Conference on Artificial Intelligence, 37(8) Jun. 2023a. doi:
v3718.26131 URL https:///os
httaa.orrr/ddee..pp/AAAAAA.//.. http://AAA//artiiaca/aa 2 26131
Emilian Postolache, Jordi Pons, Santiago Pascual, and Joan Serrà. Adversarial permutation invariant training for universal sound separation. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023b.
Zafar Rafii, Antoine Liutkus, Fabian- Stöter, Stylianos Ioannis Mimilakis, and Rachel Bittner. Musdb 8-hq an uncompressed version of musdb18, August 2019, URL httpt:://oo...ooo org/ 10. 028552811/2 enooo.. 3338373
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.061155 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pp. 10684-10695, 2022.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. Medical Image Computing and Computer-Assisted Intervention-MICC 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part IIl 18, pp. 234-241 Springer, 2015.
Simon Rouard and Gaëtan Hadjeres. CRASH: raw audio score-based generative modeling for controllable high-resolution drum sound synthesis. In Proceedings of the 22nd International Sociity for Music Information Retrieval Conference, ISMIR 2021, pp. 579-585 2021.
Jonathan Lee Roux, Scott Wisdom, Hakan Erdogan, and John R. Hershey. Sdr half-baked or well done? In ICASSP 2019 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 626-630, 2019, doi: 10.1 10.1109/CAASSP22099..00 86838555
Koichi Saito, Naoki Murata, Toshimitsu Uesaka, Chieh-Hsin Lai, Yuhta Takida, Takao Fukui, and Yuki Mitsufuji. Unsupervised vocal dereverberation with diffusion-based generative models. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023.
Andrea Santilli, Silvio Severino, Emilian Postolache, Valentino Maiorca, Michele Mancusi, Riccardo Marin, and Emanuele Rodola. Accelerating transformer inference translation via parallel decoding. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings the 6l Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 12336-12355, Toronto, Canada, July 2023, Association for Computational Linguistics. doi: 10. 10.118633/1120223.o acI-long.688) URL htttpe///acllattoooooooo.r//////
689
Ryosuke Sawata, Naoki Murata, Yuhta Takida, Toshimitsu Uesaka, Takashi Shibuya, Shusuke Takahashi, and Yuki Mitsufuji. Diffiner: A Versatile Diffusion-based Generative Refiner for Speech Enhancement. In Proc. INTERSPEECH 2023, pp. 3824 3828, 2023. doi: 10.21437/
mnterspeech.2023-1544
Robin Scheibler, Youna Ji, Soo-Whan Chung, Jaeuk Byun, Soyeon Choe, and Min-Seok Choi. Diffusion-based generative speech source separation. In ICASSP 2023-2023 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023.
Flavio Schneider, Zhijing Jin, and Bernhard Schölkopf. Moûsai: Text-to-music generation with long-context latent diffusion. arXiv preprint arXiv:2301.11757 2023. BtLLhttps://aam
orc/abs/2301. 11757
Joan Serrà, Santiago Pascual, Jordi Pons, Oguz Araz, and Davide Scaini. Universal speech enhancement with acore-based diffusion. arXiv preprint arXiv:2206. 03366, 2022.
13

---

Published as a conference paper at ICLR 2024
Jung-Eun Shin, Adam J Riesselman, Aaron W Kollasch, Conor McMahon, Elana Simon, Chris Sander, Aashish Manglik, Andrew Kruse, and Debora S Marks. Protein design and variant prediction using autoregressive generative models. Nature communications, 12(1):2403, 2021. URL
wgs. neurips. cc www.cc//pppppeeeeeeeeee 9/001//hsh/
(e1153119666699999 a768cc3477769 html
Jascha Sohl-Dickstein, Eric A. Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In Francis R. Bach and David M. Blei (eds.). Proceedings ICML 2015, Lille, France, 6-11 July 2015, volume 37 of JMLR Workshop and Conference Proceedings, pp. 2256-2265. Jorg. 2015. URL http::/prrocedd mlr.
htmll
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' A'Alché-Bul Buc, Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 11895-113 1907, 2019.
Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differenttial equations. In International Conference on Learning Representationss 2021. URL
Px-1x21112RRHS
Cem Subakan and Paris Smaragdis. Generative adversarial source separation. In Proc. ICASSP, pp. 26-30. IEFE, 2018
Naoya Takahashi, Nabarun Goswami, and Yuki Mitsufuji. Mmdenselstm: An efficient combination of convolutional and recurrent neural networks for audio source separation. In Proc. IWAENC, pp. 106-110, 2018. doi: 10.1 119//AAEE..20...222222
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprini arXiv:2302. 13971, 2023.
Aaron van den Oord, Oriol Vinyals, al. Neural discrete representation learning, Advances in neural information processing systems, 30, 2017.
Aäron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. WaveNet: Generative Model for Raw Audio. In Proc. 9th ISCA Workshop on Speech Synthesis Workshop (SSW 9), pp. 125, 2016.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E. ukasz. Kaiser, and Illia Polosukhin. Attention all you need. U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Gamett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Asso- ciates, Inc., 2017. URL httpp:::/ _et. mew.rippsccc/pppet///////
Guyon,
pdf]
Pascal Vincent. A connection between score matching and denoising autoencoders. Neural Computa- tion, 2367)11611-1674, 2011.
Tomer Weiss, Eduardo Mayo Yanes, Sabyasachi Chakraborty, Luca Cosmo, Alex M. Bronstein, and Renana Gershomi-Poranne, Guided diffusion for inverse molecular design. Nature Computational Science, 888)) Oct 2023. ISSN 2662-8457. doi: 10. 011888444888002--0055550 htttps://doi.org/11 1tt///001100000 1038/ 001444588888000005----2
URL
Scott Wisdom, Efthymios Tzinis, Hakan Erdogan, Ron Weiss, Kevin Wilson, and John Hershey. Unsupervised sound separation using mixture invariant training. Advances in Neural Information Processing Systems, 33:3846-3857 2020.
Dongchao Yang. Jianwei Yu, Helin Wang, Wen Wang, Chao Weng, Yuexian Zou, and Dong Yu. Diffsound: Discrete diffusion model for text-to-sound generation. JEEE/ACM Transactions on Audio, Speech, and Language Processing, 2023.
14

---

Published as a conference paper at ICLR 2024
Chin Yun Yu, Sung-Lin Yeh, György Fazekas, and Hao Tang. Conditioning and sampling in variational diffusion models for speech super-resolution. In ICASSP 2023-2023 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023.
Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. Sound- stream: An end-to-end neural audio codec. IEEE/ACM Transactions on Audio, Speech, and Language Processing,
2021.
Ge Zhu, Jordan Darefsky, Fei Jiang, Anton Selitskiy, and Zhiyao Duan. Music source separation with generative flow. IEEE Signal Processing Letters, 29:2288- 2022. doi: 10.1109/-.P.2022. 3219355.
A DERIVATION OF MSDM DIRAC POSTERIOR SCORE FOR SOURCE SEPARATION
We prove the main result of Section 4.2.3] We condition the generative model over the mixture (0) y. As such, compute the posterior:
y(t) y(x y(0))dy(t) - / p(x(() ((), ((0))((())))) (t))
p(x(t) (0))
(t))
The first equality is given by marginalizing over (t) and the second by the chain rule. Following Eq. (50) in Song et al. (2021). we can eliminate the dependency on y()) from the first term, obtaining the approximation:
p(x(t) (())) se [ p(x(t) (y()) (()))dy(t).
y(t)
We compute p(y(t) (())), using the chain rule after marginalizing over x(0) and x(t):
(t) y(0)) = [
x(0))x(() p(y((), (())))x( (0)))xx)))xx((()
-f x(())(())) p(y(t) x(t), (())),x(0),
x(0). (0))(((())) x(0))dx(0)dx(().
By the Markov property of the forward diffusion process. y(t is conditionally independent from x0) given x(t) and we drop again the conditioning on (0) from the first two terms, following Eq. (50) in (Sng et (20TT). As such, we have:
((y()) (()) ~
p(x(0) ((0))p(x(( x(0))p(>() (())(dx(()xx(().
(10)
x(1).((1)
We model the likelihood function (L) x())) with the Dirac delta function in Eq.
8 The
posterior p(x(0) (0) is obtained via Bayes theorem substituting the likelihood:
p(x (()) = p(x(0))1 y(0) ( (..(0)
pix(0))
D.' x. (x) -0)
ply())
(()
otherwise
We substitute it in Eq. 10 together with Eq. I and Eq.
8
obtaining:
p(x(0)) {
((x()):x(0), VO= dx(())x())
(11)
XX EN x(0)=y(0) (0) x(())
We sum over the first N sources in the inner integral, setting x. (f) y(t) -EN' x,,
N-1
/
XI.N- ((x))-1(0).10)--
X
(12)
N-1
N-1 xx(():
N (0),
XL-N (1t)
==1
n=1
=N((y(t; Σ x, No))
(13)
15

---

Published as a conference paper at ICLR 2024
The second equality is obtained by factorizing the Gaussian, which has diagonal covariance matrix, while the last equality obtained by iterative application of the convolution theorem Katznelson (2004).
We substitute Eq. (13) in Eq. (TT), obtaining:
p(x(0)) (yy(t);
N Σ Xn No)
ply(t) y()) ~
X(O)) (0) ((0) p(y(0))
n=l
p(x(0)) dx(()
= -N)
-N, (0. P(y(0)
(14)
At this point, we apply Bayes theorem in Eq. substituting the Dirac likelihood:
p(x(t) y(0)) S p(x(4)p(x(1) x())
p(y(t) y(0))dy(1)
(15)
p(y(t))
S p(x(t)) y(t) y(t) y(1)- x.,(
p(y(t) ((0))dy())
(16)
p(y(t))
p(x(t)) E (t) ly(0))
N
(17)
p P (t)
n=1
Estimating Eq. (7), however, requires knowledge of the mixture density p p(E x, (t)), which we do not acknowledge. As such, we approximate Eq. (16) with Monte Carlo, using the mean of p(y(t) y())), namely (0) (see Eq. (14), obtaining:
p(x(t) v(0) ~ p(x(()) y(O) EN x..(
p(x(e))
if E (t)
Ply
(18)
((y(0))
otherwise
Similar to how we constrained the integral in Eq. (12 we parameterize the posterior, without loss of generality, using the first N - sources (() = (x1 (f),.... (1)). The last source is constrained
setting (t) (0) Σn x,, (t) and the parameterization is defined as:
= :
=
Σ xt). n=1
(19)
Plugging Eq.
19
Eq.
18)
we obtain the parameterized posterior:
p(F(((t)))
p(F(((()) (()) ~
(20)
p(y(0))
At this point, we compute the gradient of the logarithm Eq. 20 with respect to x(():
Vx()) log logg((((x(t)) y()) x Vi( log p(F(x(t)))
p(y(0))
Vx( log ((F(((())) - Vx()) log logp(y(())
Vx(e) log p( ((x(t)))
(21)
Using the chain-rule for differentiation on Eq. 21 we have: V Vog log (((((((((()) (()) V F((()) log (xx)(((
(22)
where (x((((t) E ((NxD))((N-1)XD is the Jacobian of F' computed in x(t), equal to:
I
0
0
0
I
0
Jx(((t))
:
0
0
I
-I
-I
-I
16

---

Published as a conference paper at ICLR 2024
The gradient with respect to source (t) with 1 I S N - in Eq. 22 is thus equal to:
V Xm (1) log ((((()) (()) ~ V F(X()))
V F(((0) log (F((()))
where we index the components of the m-th and N-th sources in V F(((()) logp(((((())) Finally, replace the gradients with the score networks:
N-1
xm. (t) (ogp(((((())x(() = (x.
Σ
X (t))))
((s).)...... - Σ xm, (t)( ot)).
(23)
n=1
where Sh and Sy are the entries of the score network corresponding to the m-th and N-th sources.
B DERIVATION OF GAUSSIAN AND WEAKLY-SUPERVISHD POSTERIOR SCORES FOR SOURCE SEPARATION
In this Section we derive the formulas for 'MSDM Gaussian', 'ISDM Dirac' and 'ISDM Gaussian', We first adapt the Gaussian posterior introduced in Jayaram & Thickstun (2020) to continuous-time score-based diffusion models Karras al. 2022). We plug the Gaussian likelihood function (Eq.
into Eq. (15). obtaining:
p(x(t))(((()) EN (t), p(y(() (y()))y(() p(y(1))
p(x(1) (())
(24)
(())
Following Jayaram & Thickstun 2020 (t) is not re-sampled during inference and is always set to
y0). As such, we perform Monte Carlo in Eq. (24) with y0)), the mean of p(y(t) y(0) (see Eq. obtaining:
p(x(t) (()) (yx()))N((()) ΣN (1), ((y(0))
(25)
At this point, we compute the gradient of the logarithm Eq. 25 with respect to xm (t):
V xm (t) logr(x(() (0)) V (mm()) (1) log ((x(1))((((0))
En X (1,,
p(y(0))
V (1) logp(x(()) V xm (0) log.N Σ x, (1),
Vxm((1) logp(x(()) 2t²(t) (x.(()))l(()))
x (1)
N
Vx, logp(x(()) (t) (y(0) Σ
(t)
(26)
We obtain the 'MSDM Gaussian' posterior score by replacing the contextual prior with the score network:
V V (t) logp(x(t) (y()) (xx)))).....x(((().)))))))))) 72(t) (v(0)
Σ x (t) (27)
The weakly-supervised posterior scores are obtained by approximating:
(tx) ... (t) II
P. (x, (t))
n=1
17

---

Published as a conference paper at ICLR 2024
Model
Inference Time (s)
# of parameters
Demucs
0.111
40M
Demucs + Gibbs (512 steps)
0.111
512 56.832 36.3552
40M
Demucs + Gibbs (256 steps)
0.111 x 256 28.416=18..10
40M
ISDM (correction)
4.640.345 X 4 = 18.4
405M x4
MSDM (correction)
0.64
405M
Table 4: Inference times for single 12s long separation and number of parameters for each model in T] We report Demues Gibbs (256 steps) since the minimum number of steps that makes the SI-SDR: over all instruments (17.59 dB) greater than ISDM. While ISDM and MSDM are not time-competitive Demucs, they are more time-efficient than Demucs + Gibbs (256 and 512 steps).
where Ps, are estimated with independent score functions S. In the contextual samplers in Eq. 25 (MMSDM Dirac') and Eq. (27) ("MSDM Gaussian") ), 5% ((x1 (f),.... (t)),, (f()....x((t))))
(t)) refers to a slice
of the full score network on components of the n-th source. In the weakly- suukly-uupevvieed cases, S
is an individual function. To obtain the 'ISDM Dirac' posterior score, we factorize the prior in Eq. (2)), then use the chain rule differentiation, as in Appendix A to obtain:
N-1
Xm (1) log ((Fp(((((())) l (0)) V xx.(( log Pm (Xm (f) + (x)((e) logpN(y(())
Σ X (1) R=1
-
N-1
2hh (xm (1) (x Σ x (1))(())
n=1
We obtain the 'ISDM Gaussian' posterior score by factorizing the joint prior in Eq. 26
V Vx,.()) (t) logp(x(()) (x)))= Sth (xm(())
72(t) (y(0) Σ (t((t))
C EXPERIMENTAL SETUP
C.I DATASET
We perform experiments mainly Slakh2100 (Manilww al. 2019). a standard dataset for music source separation. Slakh2100 is collection of multi-track waveform music data synthesized from MIDI files using virtual instruments of professional quality. The dataset comprises 2100 tracks, with a distribution of 1500 tracks for training. 375 for validation, and 225 for testing. Each track is accompanied by its stems, which belong to 31 instrumental classes. For fair comparison, only used the four most abundant classes as in (Manilow et al. (022). namely Bass, Drums, Guitar, and Piano; these instruments are present in the majority the songs: 94.7% (Bass), 99.3% (Drums), 100.0% (Guitar), and 99.3% (Piano).
In Appendix E we experiment on MUSDBI8-HE Rafi et al. (2019). a benchmark dataset for the music source separation task. It contains 150 tracks, with 100 allocated for training and 50 for testing, amounting to roughly 10 hours of professional-gras audio. Each piece within the dataset separated into the stems: Bass, Drums, Vocals, and Other, with the latter encompassing any elements not included the categories above.
C.2 ARCHITECTURE AND TRAINING
The implementation of the score network is based on a time domain (non-latent) unconditional version of Moûsai (Schneider et al. 2023).
We used the publicly available repository audio-dee -il--difffiiooo pytorch/v0.. 0. 432 The score network U-Net Ronneberger et al. 2015) comprised of encoder, bottleneck, and de-
httpp:://gittub.ccmm/
d www.iion--yyoorcc//rrreeeeeeeee
18

---

Published as a conference paper at ICLR 2024
Table 5: Hyperparameter search for source separation. We use 'MSDM Dirac' (top-left), 'ISDM Dirac' (bottom-left), 'MSDM Gaussian' (top-right) and 'ISDM Gaussian' (bottom-right) posteriors. We report the SI-SDR, values in dB (higher is better) averaged over all instruments (Bass, Drums, Piano, Guitar).
Dirac Likelihood
Gaussian Likelihood
Constrained Source
7(f)
Schura
Bass
Drums
Guitar
Piano
(1)
0.5e(()
(175
la(()
1.25a(1)
L.5(())
(x(1)
4.41
5.05
3.28
2.87
-41.54
6.37
6.05
5.67
5.729
5.13
4.33
MSDM
7.90
8.18
7.03
7.05
-47.24
6.79
6.51
6.15
6.19
5.66
4,45
20
14.29
12.99
12.19
11.69
-47.17
11.07
10.51
9.43
10.19
9.18
7.58
40
14.28
13.02
5.51
4.78
-47.17
-36.92
12.48
11.25
11.87
10.80
9.03
5.05
3.69
-2,50
6.93
-45.46
7.12
6.50
5.78
5.02
4.49
3.69
ISDM
9.23
8.57
7.28
9.20
-47.54
7.57
7.20
6.32
5.35
4.82
3.83
15.35
15.08
13.20
15.36
-46.86
12.89
12.21
10.87
9.32
8.32
6.47
17.26
15.77
15.30
14.98
-46.86
-35.97
14.09
12.82
10.85
10.02
8.26
16.21
15.57
15.51
14.20
-46.80
-46.85
14.06
12.57
11.83
10.81
9.24
coder with skip connections between the encoder and the decoder. The encoder has six layers comprising two convolutional ResNet blocks, followed by multi-head attention in the final three layers. The signal sequence is downsampled in each layer by factor of 4. The number of chan- nels in the encoder layers is [256, 512, 1024, 1024, 1024, 1024]. The bottleneck consists a ResNet block, followed by self-attention, and another ResNet block (all with 1024 channel lay- ers). The decoder follows reverse symmetric structure with respect to the encoder. We employ audio-di f -wtoooooootccccheooooooooooo train the score network with four stacked mono channels for MSDM (i.e., one for each stem) and one mono channel for each model in ISDM, using a context length of 12 seconds. All our models were trained until convergence on an NVIDIA RTX A6000 GPU with 24 GB of VRAM. We trained all our models using Adam Kingma & Ba 2015). with learning rate of 10-4 31 0.9, 82 = 0.99 and batch size of 16.
ne for training. We downsample data 22kHz and
We report inference times and number of parameters of the various models Table
C.3 THE SAMPLER
We use a first-order ODE integrator based on the Euler method and introduce stochasticity following
(Karras et al. 2022). The amount of stochasticity is controlled by the parameter Setum- As shown in Appendix Daand explained in detail in (Karras et al.) 2022), stochasticity significantly improves sample quality. We implemented a correction mechanism (Song et al., 2021 Jayaram & Thickstun 2020) iterating for R steps after each prediction step adding additional noise and re-optimizing with the score network fixed at d,. As per Karras et al. (2022), we adopt non-linear schedule for time discretization that gives more importance lower noise levels. It defined as:
t; = o, = Omax +
I- min
Tmax)²
where 0 S i < I, with 7 the number of discretization steps. We set amii 10-4, 0max 1, 7.
D HYPERPARAMETER SEARCH FOR SOURCE SEPARATION
We conduct a hyperparameter search over Schum to evaluate the importance of stochasticity in source separation over fixed subset of 100 chunks of the Slakh2100 test set, each spanning 12 seconds (selected randomly). To provide a fair comparison between the Dirac ('MSDM Dirac", 'ISDM Dirac') and Gaussian ('MSDM Gaussian', 'ISDM Gaussian') ) posterior scores, we execute search over their specific hyperparameters, namely the constrained source for the Dirac separators and the
https:://gt // thub. htwp:///eeeeettoooooooooeeeeeoee
79229912
19

---

Published as a conference paper at ICLR 2024
Table 6: Comparison of results of MSDM and Demucs v2 (Défossez et al. 2019) We report the SI-SDRi values (higher better). The network same as one trained Slakh2100, except that sampling rate 44kHz and is trained on 6s long chunks.
Tested MUSDB
Finetuned on MUSDB
Trained MUSDB
Model
Bass
Drums
All
Bass
Drums
All
Bass
Drums
Other
Vocals
Demucs v2
13.28
11.53
8.59
16.80
12.55
MSDM
-0.83
-0.94
-0.88
3.46
5.03
4.25
4.87
3.28
1.97
6.83
4.24
(t) coefficient for the Gaussian separators. Results are illustrated in Table We observe that: (i) stochasticity proves beneficial for all separators, given that the highest values of SI-SDR are achieved with Sctum = 20 and Sehurn = 40, (ii) using the Dirac likelihood we obtain higher values of SI-SDR, with respect the Gaussian likelihood, both with the MSDM and ISDM separators, and (iii) the ISDM separators perform better than the contextual MSDM separators (at the expense of not being able to perform total and partial generation).
E DATA EFFICIENCY STUDY: RESULTS ON MUSDB
We report in Table the results of MSDM and Demucs v2 (Défossez et al.) 2019) on the MUSDB18. HQ test set Rafii al. (2019). We try three different strategies, first check the out-of-distribution ability the model trained on Slakh2 100 by testing directly on MUSDB I8-HQ. Then, we try finetuning the model trained on S1akh2100 BUSDB18-HQ. Since the only stems shared between MUSDB 18-HQ and Slakh2100 are 'Bass' and 'Drums, the first and second strategies can be tested only these two stems.
MUSDB 18-HQ, and finally, we train directly
F SUBJECTIVE EVALUATION
The details of the listening test explained in Figure B
20

---

Published as a conference paper at ICLR 2024
Subjective
seart valka from 1o 12 eusuare the peneral cual of song Le how mach the seper leacch as flom
heneeis betwoee raaidy, Aex ii, hoarraa iustament go soperise ber
Deginning of Test
orag
Bramplc of bast from dataGet
Prenss
Conerence Insormmnnt
Test Beginning
requestedl basa
Subjective
The task pardal must genenadon. nears that the model in given some ground arrth
nat cou sut to gana
inudio pleseent
sdingg wey each cucation,
Feak rakksse the 3 deniity
rot
eeveratee how lerd shiy greerstion viareon, wenges san euampla hun datiea and valarii lál glo incirua traaks soult pie me tlank.
M/ My) der min
Quality
Figure Snippets from the subjective evaluation form. first row relative total generation, where people were asked to evaluate 30 songs. 15 of which were generated by the mixture model and 15 by MSDM. Thirty-two people answered the survey. The second row is relative partial generation. Subjects were asked evaluate 15 songs. A random subset of sources is fixed for each song, and MSDM generates the other. The requested sourees are explicitly stated above the song (c.g., the snippet, the model has generated only the Bass stem). Twenty-one subjects answered.
21


Reviews:
[{"review_id": "Fnj3EmEyEp", "reviewer": "Reviewer_oyrq", "summary": "My review did not seem to have the right visibility, so I am re-submitting.\n\nThe authors present an approach for mixture-based diffusion modeling, complete with a novel inference approach that leverages a posterior based on Dirac delta functions and a Monte Carlo approximation of the Dirac likelihood in place of the mixture density. The authors also demonstrate that this approach can be used for generation. The authors evaluate for stem separation using Slakh2100 against established baselines, and also show quantitative and qualitative human-evaluated results for a generation task, establishing a new task and baseline in the process.", "questions": "- Do the authors have a hypothesis as to why in Table 2 performance on the indicated stem categories (i.e. bass, drums) outperforms Demucs? Is there any intuition as to the variance in results across the techniques?\n- Is there an understanding of the approach's data efficiency? How much does Slakh2100 versus MusDB alone? What about compared to other settings in Demucs?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "- The approach the authors proposed is novel and in fact quite general. Orthogonally, I am interested in understanding the potential of the approach in settings with other mixture-based diffusion.\n- The manuscript is compelling to read, well organized, and very clear. The discussion of related work is quite comprehensive.\n- Using a waveform as the representation of data generalizes the approach further, opening the door to applicability in other domains.\n- The authors provide a strong baseline for accompaniment generation with a data-rich setup (Slakh2100) which will serve to be a solid foundation. If not already planned, I would encourage the authors to release as much as they can with respect to evaluation methodology and reproducible artifacts that others can use to evaluate in a similar manner.", "weaknesses": "The results of the paper would be made stronger with more discussion of the computational footprint and details for training and inference. How does the footprint compare to Demucs or other methods? How does does computation scale with the amount of data?\n\nFurther, some brief qualitative analysis of the results might be warranted. Do the authors have an explanation for the relative performance on certain stems relative to Demucs? What are the high-level takeaways from the qualitative results in Table 3 beyond those that the reader could intuit or speculate about?\n\nSome feedback on the manuscript:\n- Section 2.1, in the last paragraph: should \"the minute\" read \"a minute\" denoting duration of the context length?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ooIKectw1y", "reviewer": "Reviewer_J4Tn", "summary": "The authors propose to tackle the source separation problem by seeing it as a separate-tracks music generation problem. \nThey model this distribution using a diffusion model.\nThis approach allows to tackle more use cases such as \"source inputation\" (accompaniment generation) using the same model.", "questions": "It would have been great to compare the inference time between the different methods, as the diffusion-based methods are likely to be order of magnitudes bigger.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The paper is well written, well-organized, self-contained and the background and reference section are comprehensive and detailed. The proposed method is sound and the authors obtain positive results. Even if not state of the start (source separation is a highly studied task), this original method shows promising results and has the advantage of being conceptually simpler and more general.\n\nThe new proposed modeling method IDSM Dirac seems relevant, clearly improves on previous methods and might be of interest in other application domains.\n\nThis method is of course more costly in terms of the amount of data required or in terms of computational resources needed. But this is addressed in the limitations section.\n\nThe appendix showcases interesting hyperparameter searches about the MSDM Dirac approach, like the impact on the constrained source.", "weaknesses": "Adding details on the correction step could make the paper even more self-contained.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "p7G4mqNYRy", "reviewer": "Reviewer_K9Yj", "summary": "This paper proposes a Multi-Source Diffusion Model (MSDM) for performing both music source separation and generation. Modeling multiple sources jointly allows the system to perform source imputation / accompaniment generation by conditioning on a partial mixture during inference. The authors demonstrate the efficacy of MDSM in both source separation and generation tasks, achieving separation performance competitive with state-of-the-art models. Moreover, by modeling sources independently and introducing a novel score function, the authors achieve separation results surpassing the state-of-the-art for certain instrument classes. While generation and imputation abilities appear to be more limited, MSDM demonstrates the strong potential of generative separation systems that model sources jointly.", "questions": "The fact that the ISDM method outperforms MSDM on certain sources (Table 2) seems contrary to the intuition that jointly modeling dependencies between sources should improve separation results. Could the authors elaborate on these results, and perhaps conjecture as to why independently modeling sources with ISDM improves (or at least does not substantially deteriorate) performance versus jointly modeling with MSDM?\n\nGiven the apparently high data requirements of MSDM, have the authors explored fine-tuning the Slakh2100 model on smaller datasets (e.g. MUSDB)? The authors mention potentially using the outputs of a source separation system to scale up the dataset (akin to SingSong), but fine-tuning also seems like an interesting avenue to explore -- especially given its popularity with diffusion models more generally.\n\nDid the authors conduct any separation or generation experiments with out-of-distribution data?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The proposed method is novel and interesting. Given recent advances in generative source separation and the apparent benefits of iterative refinement for separation [1], a diffusion-based approach seems natural.\n\nThe proposed method achieves very good separation performance using mostly off-the-shelf components, demonstrating the strength of the general approach. Additional improvements (e.g. weakly-supervised ISDM variant and Dirac score function) are well motivated in the paper.\n\nThe problem of source imputation / accompaniment generation is more relevant to many music creation workflows than full mixture generation, and is comparatively under-studied. While the accompaniment generation results here don't necessarily \"improve\" on those demonstrated in the referenced concurrent work [2] in terms of realism, the proposed method approaches the problem from a different angle (joint source separation and generation, diffusion rather than language modeling) and allows for more fine-grained control using multiple conditioning instrument classes (as opposed to singing only).\n\n[1] Ethan Manilow, Curtis Hawthorne, Cheng-Zhi Anna Huang, Bryan Pardo, and Jesse Engel. Improving\nsource separation by explicitly modeling dependencies between sources. In ICASSP 2022-2022\nIEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 291–295.\nIEEE, 2022.\n\n[2] Chris Donahue, Antoine Caillon, Adam Roberts, Ethan Manilow, Philippe Esling, Andrea Agostinelli,\nMauro Verzetti, Ian Simon, Olivier Pietquin, Neil Zeghidour, et al. Singsong: Generating musical\naccompaniments from singing. arXiv preprint arXiv:2301.12662, 2023.", "weaknesses": "I think the paper would benefit from some additional discussion of the computational and data requirements of the proposed method. Presumably separation time is linear in the number of inference steps (plus correction); it would be nice to see this explicitly compared to Demucs with and without Gibbs sampling. Similarly, if the proposed method is more data-hungry than discriminative methods such as Demucs (e.g. if the proposed method is not competitive when trained/evaluated on MUSDB), this might be worth emphasizing further. Parameter counts for each method would also be nice to see.\n\nBased on the listener study and provided listening examples, MSDM struggles to produce coherent and high-quality generations -- even when judged in the context of its synthetic training data. To my ears, it seems like tempo sometimes degrades within the model's 12-second context for unconditional generations, and for imputation generations when strongly metric signals (drums, bass) are not given as conditioning. Overall, the separation results seem much stronger than the generation results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Q2Z2I7qizY", "reviewer": "Reviewer_yjJy", "summary": "This paper proposes a unified approach (called Multi-Source Diffusion Model) to solve audio generation, source inputation and source separation. In training time, the diffusion model learns the joint distribution of the audio mixture and solves the three tasks using different inference methods. Audio generation is done via directly sampling the prior. Source inputation is done using the inpainting technique in diffusion models. The source separation is a tailored method: reconstructing audio signals under the constraint that audio mixture is the summation of each audio tracks. Experiment results show the model is successful in all three tasks.", "questions": "Q1: The separation result of ISDM and MSDM in Table 2 suggests a trade-off in terms of generation quality and separation performance. Is it confirmed? If so, why using a unified approach for source separation and generation? I understand on one hand, ISDM helps to show the diffusion method is comparable to the SOTA method. However, since MSDM is always lower than ISDM, I doubt the fundamental assumption in this paper is not valid that it is superior to use joint distribution to model generation and separation at the same time.\n\nQ2: Could you provide some demos for ISDM method (Now or after publication)?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. Originality. The idea of using a generative way to view music source separation is so natural and experiments show diffusion model can perform pretty well. The diffusion sampling process under summation constraint is borrowed from works in CV but it is a natural fit for source separation tasks.\n2. Quality: theory is presented clearly and experiments are effective.\n3. Clarity: paper is well-written. High-level idea and detail are both clear.\n4. Significance: the result is convincing according to the demo in the supplementary material.", "weaknesses": "The paper is well-written. I have some minor comments. If it can be clarified, the significance of the work will be increased.\n\n1. The idea of using a generative way to model audio and solve multiple audio tasks in one model is not a new idea. For example, we have a unified VAE approach to do source separation and transcription [1]. Of course the current work is novel but it would be better to include those studies and compare the approaches in related work.\n2. What is the novelty compared to NCSN-BASIS? Any intuition for MSDM Dirac? Is it a marginal improvement from existing methods, or not?\n3. The introduction of ISDM method is not clear. Is it a baseline method where the assumptions are obviously wrong? Is it another valid approach? Compared to MSDM, is there a trade-off in terms of generation quality and separation performance?\n4. The introduction of correction step is okay. Since it is evaluated in the experiment section, could you explain more in the paper? What is the statement to be expected prior to the experiment regarding correction step?\n\n[1] Liwei Lin, Gus Xia, Qiuqiang Kong, Junyan Jiang: A unified model for zero-shot music source separation, transcription and synthesis. ISMIR 2021: 381-388", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (oral)