MGE-LDM: Joint Latent Diffusion for Simultaneous Music Generation and Source Extraction
Authors: Yunkee Chae
Kyogu Lee
Venue: NeurIPS 2025 poster ()
Keywords: Music Generation
Source Extraction
Inpainting
Primary area: applications
TL;DR: We propose unified latent diffusion framework for simultaneous music generation, source imputation, and query-driven arbitrary source extraction.

Abstract: We present MGE-LDM, a unified latent diffusion framework for simultaneous music generation, source imputation, and query-driven source separation. Unlike prior approaches constrained to fixed instrument classes, MGE-LDM learns a joint distribution over full mixtures, submixtures, and individual stems within a single compact latent diffusion model. At inference, MGE-LDM enables (1) complete mixture generation, (2) partial generation (i.e., source imputation), and (3) text-conditioned extraction of arbitrary sources. By formulating both separation and imputation as conditional inpainting tasks in the latent space, our approach supports flexible, class-agnostic manipulation of arbitrary instrument sources. Notably, MGE-LDM can be trained jointly across heterogeneous multi-track datasets (e.g., Slakh2100, MUSDB18, MoisesDB) without relying on predefined instrument categories.

MGE-LDM: Joint Latent Diffusion for Simultaneous Music Generation and Source Extraction
Yunkee Chae'
Kyogu Lee 1,2,3
Music and Audio Research Group (MARG)
Interdisciplinary Program in Artificial Intelligence (IPAI)
2 AlIS, Department of Intelligence and Information Seoul National University (yunkimo95, kglee) @snu. ac.u.cc.k
Abstract
We present MGE-LDM, a unified latent diffusion framework for simultaneous music generation, source imputation, and query-driven source separation. Unlike prior approaches constrained to fixed instrument classes, MGE-LDM learns joint distribution over full mixtures, submixtures, and individual stems within single compact latent diffusion model. At inference, MGE-LDM enables (1) complete mixture generation, (2) partial generation (i.e., source imputation), and (3) text- conditioned extraction of arbitrary sources. By formulating both separation and imputation conditional inpainting tasks in the latent space, our approach supports flexible, class-agnostic manipulation of arbitrary instrument sources. Notably, MGE-LDM be trained jointly across heterogeneous multi-track datasets (e.g.. Slakh2 100, MUSDB 18, MoisesDB) without relying on predefined instrument categories. Audio samples are available at our project page
Introduction
Recent advances in generative modeling have significantly accelerated progress in music audio synthesis, inspired by breakthroughs in the language and vision domains. Early autoregressive models such as WaveNet [1] demonstrated the fcasibility of end-to-end waveform generation. Since then, two dominant approaches have emerged: (I) discrete-toker models, which compress raw audio into quantized codes (245) for sequence modeling [68]; and (2) diffusion-based models (TI which synthesize audio by reversing a noise corruption process in the waveform domain 12 13]. Building
on this foundation, latent diffusion models (LDMs) [14] which operate in compressed latent space, have delivered substantial gains synthesis quality and have been successfully applied music generation tasks [8).
Despite this progress, most music generation models produce single, mixed waveform, lacking access to the individual instrument stems required for remixing, adaptive arrangement, or downstream production tasks. To recover these components, audio source separation techniques aim to decompose a mixture into its constituent tracks. Discriminative approaches (19/22) learn to directly regress each source from the input mixture. achieving strong performance. contrast, generative separation models sample individual sources from learned prior 23 Recently, diffusion-basse based methods have demonstrated strong performance not only speech separation
[28 29 but also in speech
enhancement tasks 33), highlighting their potential for flexible, high-quality source recovery across audio domains.
https://yoongi42: githup Wwaittuu...o/MEEE.. 1o/MGELDM Samp /
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Xo
4e (z(),
Xiiix
fo (z.,
YA
Rmi
Generation
Encoder
Zo
Transforma
Diffusion
VAE
Kaub
YAIl Decoder
+
Purtial Generation
Xare
(Tppe Eue
CLAP
Source Extraction
VAI
Ryyc
(a) Training pipeline
(b) Inference pipeline
Figure Overview MGE-LDM. (a) Training pipeline: We train a three-track latent diffusion model on mixtures, submixtures, and sources. Each track is perturbed independently and conditioned on its corresponding timestep and CLAAP embedding. The model optimized using the v-objective, detailed in Sections 3.2 and 3.4 (b) Inference pipeline: At test time, task-specific latents either
generated or inpainted based on available context and text prompts. The resulting latents are decoded into waveforms. See Section 3.3 for details.
Recent works have explored modeling the joint distribution of multi-track stems within unified diffusion backbone, enabling mixture synthesis, accompaniment generation, and source separation within single framework (34.36). However, these approaches typically rely predefined instrument classes for each track or assume that the mixture waveform is the linear sum of its constituent stems. While the additive assumption is valid in the waveform domain, it is incompatible with the nonlinear encoder-decoder structure of latent diffusion models, limiting the applicability of such methods in compressed latent spaces.
To address these limitations, we introduce MGE-LDM, class-agnostic latent diffusion framework that jointly unifies music generation, partial generation, and arbitrary source extraction. Our approach models three interrelated latent variables: mixture, submixture, and source within a single diffusion backbone.
Given a waveform mixture ((m) and chosen source x(*), we define the submixture x(a) (m)) ((,), A pretrained encoder E maps each waveform to its latent representation:
(m) = E(z((m), 2(u) Elx:("),
2 = F(x(
We then train a diffusion model fe(=(mm) ((0), 2(0) over this joint distribution.
At inference time, diffusion--asse inpainting, conditioned on subset of {2(m), (²(), 2(+),, enables class-agnostic partial generation and text-driven source extraction without relying fixed instrument vocabularies or linear mixing assumptions. A detailed formulation appears in Section 3
Notably, training paradigm is dataset-agnostic by removing dependence instrument labels, MGE-LDM can leverage all publicly available track-wise datasets a single training run. For example, S1akh2100 10 337 provides clean, isolated stems; MUSDBI8 (3 and MoisesDB (39] include loosely labeled tracks such as other that aggregate multiple instruments. Existing frameworks 40 typically ignore such labels, they assume one-to-one correspondence between sources and tracks, making difficult to train and perform inference on mixtures containing sources outside predefined instrument classes.
36
In contrast, our three-track formulation (mixture, submixture, individual source) treats aggregated labels submixtures. This design integrates them into joint training together with fully separated stems. This dataset flexibility simplifies data collection and enhances model robustness across heterogeneous source structures.
In summary, our contributions are as follows:
We propose MGE-LDM, the first latent diffusion model to jointly address music generation, source imputation, and text-conditioned source extraction without relying on predefined instrument labels.

---

We formulate source extraction as conditional inpainting problem in the latent space. jointly modeling mixture, submixture, and source embeddings. This formulation enables class-agnostic, text-driven extraction of arbitrary stems within unified latent diffusion framework.
We propose training strategy that assigns distinct diffusion timesteps to each of the mixture, submixture, and source tracks, allowing the model to learn adaptive score functions for diverse inpainting contexts.
By eliminating dependence on fixed stem annotations, our framework ingests heteroge- neous multi-track datasets (e-g., Slakh2100, MUSDB18, MB in a unified manner, simplifying data aggregation and improving generalization to unseen instrumentation.
2 Related Work
Audio Source Separation and Extraction. Audio source separation aims to decompose polyphonic mixture into its constituent tracks, while source extraction focuses on isolating a particular target sound, often guided by metadata, text prompts, reference examples. Two dominant paradigms have emerged: discriminative models learn direct mapping from the input mixture to each target stem via regression losses, either in the waveform domain or in spectrogram representations [19/22 41 42]
In contrast, generative approaches learn probabilistic priors over source distributions and recover
individual stems via sampling 23 26 43
Recently, diffusion- based techniques have emerged powerful paradigm for audio decomposition, achieving strong results in both speech separation [28 [2 and enhancement [30 These methods
iteratively denoise a mixture under a learned score function, offering flexible and high- high-fidelity source recovery.
Query-based extraction further extends separation by conditioning the model on external cues such as class labels [44. visual signals [47 [8]. or audio exemplars [49 [5)]. Several studies have also demonstrated the effectiveness of natural language prompts for flexible, user-driven source
isolation [48] 51555). In our framework, we employ the pretrained CLAP model [56] to obtain shared
audio-text embeddings, enabling seamless, Language-guided extraction of arbitrary stems within multi-track latent diffusion architecture.
Audio Generation Models. Early neural audio synthesis methods focused on autoregressive ar- chitectures that model waveform dependencies sample by sample. WaveNet [ demonstrated the effectiveness of dilated convolutions for end-to-end generation, while SampleRNN [57] extended this with hierarchical recurrence. Subsequent work adopted adversarial objectives to improve fidelity,
using GANs generate perceptually sharp outputs [58 [59].
Parallel efforts introduced discrete-token models, where audio encoded into compact code se- quences using vector quantization (c.g.. VQ-VAE (2). Jukebox (6) models long-range dependencies over codes using Transformers [60], while recent systems enhance fidelity through residual quantiza- tion 355 and hierarchical token modeling. where coarse-to-fine code representations are generated across multiple levels 7 63) MusicGen [8] improves decoding efficiency with delayed-token generation, and Instruct-MusicGen (64] extends for targeted editing via instruction-tume -unndd prompts. In parallel, token- based masked generative modeling techniques, originally developed for vision (65], have been adapted for audio, enabling efficient non-autoreereesive synthesis and precise spectrogram
inpainting (66] 67
Diffusion-based generation emerged with DiffWave 12 and WaveGrad 13 which learn to iteratively denoise Gaussian- coorupee waveforms. These techniques have since been adapted for music-specific generation with structure and style conditioning (68 69). Latent diffusion models (LDMs) [14), which perform denoising in compressed embedding space, have further advanced generation fidelity and scalability. LDM-based audio models such AudioLDM (70] 71, MusicLDM [16), and achieve state-of-the performance. Recent frameworks like AUDIT 7 InstructME 74] explore the use of diffusion for controllable and interactive audio editing.
Stable Audio I1
18 72
Multi-Track Music Audio Modeling. Recent studies model multi-track music as a structured com- position of interdependent stems. StemGen 75 employs an iterative, non-autoreeeessive transformer over discrete tokens to generate stems conditioned on text prompts. Jen-1 Composer (76) applies latent diffusion to jointly model four canonical stems (bass, drums, instrument, melody), producing
3

---

coherent multi-track compositions. MusicGen-Shee Stem 777 combines per-stem vector quantization with an autoregressive decoder synthesize bass, drums, and aggregated other components, and supports
accompaniment generation. Diff-A-Riff 778 leverages latent diffusion co-create stems complementary to given mixtures, later extended with diffusion transformers [79].
Other studies have similarly explored mixture-conditionns stem generation 80) [1].
separate line of work focuses on joint modeling of synthesis and decomposition within a single diffusion backbone. Multi-Source Diffusion Models (MSDM) 34 model a fixed set of stems (bass, drums, gui tar, and piano) within shared diffusion framework, relying on an additive mixture assumption and Dirac colta-based posterior sampler. following the EDM formulation for ODE-based sampling [82 This line of work has since been extended in GMSDI [35]. MSG-LD [36], and others 40 83 GMSDI enables variable-stem modeling and text- batt- conditioning but remains grounded waveform-space additive mixing. MSG-L. adapts latent diffusion for four-stem modeling and classifier-free guidance [44, though it still assumes fixed instrument classes.
In contrast, our approach jointly models mixture, submixture, and source embeddings in latent space and casts both synthesis and arbirrary-soure extraction as text-conditioned inpainting tasks, offering fully class-agnostic multi-track music processing without reliance on fixed instrument vocabularies or linear mixing assumptions.
3 Method
Figure presents overview of the proposed MGE-LDM framework. We train a three-track joint latent diffusion model that learns the distribution over the joint space of mixture, submixture, and source representations, as defined in Eq. (T). During inference, the model performs various tasks by exploiting the inpainting capability of diffusion models. We first describe how training triplets are constructed, then detail how they are used for joint diffusion-basee training and inpainting- inference.
3.1 Formulating Joint Latent Representation
Let {xilier denote the set of time-domain audio stems, where the number of sources |/| may vary across mixtures depending on their instrumentation. The mixture waveform is defined as:
z (m)
Σ
IE
To construct a training example for our threc-track model, we uniformly sample index I and define:
(+() 2j. ((())
Σ F4.
iENU
yielding the triplet (m(m), (u)) (.)). We encode each element of this triplet using a pretrained variational autoencoder (VAE) 85 encoder E, resulting in latent representations 2("), 200) e RCXL where C and L denote the latent channel and temporal dimensions, respectively. This formulation naturally accommodates mixtures with a variable number of stems. Regardless of the number of instruments present, any publicly available multi-track dataset can be decomposed into mixture, submixture, and source components for joint latent modeling.
3.2 Latent Diffusion Training with Three- Track Embeddings
We build upon the Stable Audio framework 722 employing a Diffusson Transformer (DiT) backbone [66 and training the model under the v-objective [87) Below, we summarize the v-objective training procedure; full derivations and additional details are provided in Appendix A.I Let the composite latent input be defined as:
Zo
(2)
(m)
20
(u)
(x)
E R3xCxL
where + E CxL are (clean) track embeddings, with 1 E K (m, u, 8) denoting the track types mixture, submixture, and source, respectively.

---

We aim to estimate the score Ve, log (z-) across continuous noise levels T E [Tiin 1]. To this end, we perturb the clean latent variable Zo with Gaussian noise, following:
2 a, + 3,E, E~ (0,
(2)
where the noise scaling coefficients are parameterized
ar = (o,( ) B, = sin( 6 2
(3)
Here, 5 ([ 1]) is sampled from truncated uniform distribution with Tmin 0.02 for stability. denoising network fo(z,, T, c) is trained to estimate the score Vz, log (z- c) using the v- objective:
dz, (()) = Ezo.e,7 (Z, -vr12: 0, 0b,
a, - 3,20
The conditioning vector с = (c(m), d(d,, clt)) is derived using the audio branch of a pretrained CLAP encoder [56], applied to each component:
(k) am for l: K.
To enable classifier-free guidance (CFG) each ((k) is independently dropped out with probability p during training.
3.3 Inference via Conditional Sampling in Latent Space
In the image domain, inpainting refers to reconstructing missing or corrupted regions of an image by conditioning surrounding pixels. Diffusion models have demonstrated strong zero-shot inpainting capabilities, enabling arbitrary mask completion without retraining [[]] 88]. We extend this paradigm to the latent domain of music, operating embeddings. Downstream tasks are formulated as conditional generation problems, where known latents treated observed and unknown ones sampled missing components.
joint distribution of mixture, submixture, and source
In all inference modes, we condition on natural-language queries using CLAP embeddings, When text conditioning is required, we use the text branch of CLAP to produce the prompt embedding
(k) CL.AP (e (())). where (() is free-form natural language description (e.g.. "the sound of an electric guitar").
Total Generation. Let (e((m)) 2("), (x)) denote the implicit model distribution whose score we approximate with fe. To synthesize complete mixture, we condition only on the mixture prompt genittaationnn
embedding (mm) or omit all conditions for unconditional generation. We sample the mixture latent g(m) as:
2(m), 2(0),,,,,, ((0))((())))))) p=(2(m), 2((),, (())((m))) (("), ((()))
(5)
where g(w) and g(0) are auxiliary latents that are discarded. Finally, the synthesized mixture waveform ((m) is obtained by decoding ((m) through the pretrained VAE decoder D:
f (m) ())(m)).
Hereafter, we use {(k) to denote any dummy latent that is not retained during inference.
Partial Generation. Partial generation, also known as source imputation, refers to the task of gener- ating missing stems given partially observed sources. We approach this iteratively to progressively reconstruct the full mixture from the partial input.
Let (ci,..., } } be an (ordered) set of CLAP-derived text embeddings, each corresponding to a target source description to be imputed. Let =i denote the waveform mixture of the observed sources, and let (() (")) be its latent representation. We initialize the submixture latent with = and generate each missing source sequentially.
At each step j E {1....,Jl J.. sample new source latent 2(a) conditioned on the current submixture and the text embedding cz
= (m) (()) (m)) 2(1) (250)) ( (.)))
e((1)
(6)
5

---

We then update the submixture by accumulating the decoded sources:
2 E E
After iterations, we obtain the full of imputed sources 12 1j-1 and reconstruct the final mixture waveform as:
(m() x000) Σ D
Source Extraction. Text-driven extraction of an arbitrary stem is performed by conditioning on a natural- lanuual-anguage prompt. Given a prompt embedding c($), we treat the mixture latent ( as observed and inpaint the submixture and source tracks:
3(w) g(+) ~ po(2 20)) | (m)), ex(1), c(()),
(8)
where 2(4) is auxiliary prediction that is discarded. Finally, the isolated source waveform is reconstructed via ((*) = D(2())..
3.4 Track-aware Inpainting Model with Adaptive Timesteps
Conventional diffusion-baat baseed inpainting methods apply a uniform noise schedule across both observed and missing regions, failing account for their differing uncertainty characteristies [9 88) 90). In the standard setup, a denoising model fo(z,, is trained to approximate the joint score perturbed latent variable.
Following the perturbation rule and v-objective from Section 3.2
the score estimate is expressed as:
Vx, log dr (z,) Zr fa(z,, T).
Orr B,
where at and 3, cosine schedule as defined in Eq. (3) following 87 For notational simplicity, omit the conditioning vectors e in this expression. A detailed derivation is provided in Appendix A.
Recently, region-aware adaptations of diffusion inpainting, including spatially varying noise schedules [91] and per-pixel timestep conditioning in TD-Paint [92]. have demonstrated substantial improve- ments in semantic consistency by preserving fidelity in observed regions. Inspired by TD-Paint, extend this idea to three-track music audio by assigning distinct timestep conditions to each track, therehy improving inpainting quality in the latent space.
We describe our track-wing wise adaptive timestep conditioned model using general notation. Let K be set of tracks, and let N | be the number of tracks. Define the clean latent tensor and the corresponding noise levels as:
(k) ) (z)
KEEK E NXCXL T (Th)aEK [Tmin: 11%
Zo
where each The is either zero (for observed tracks) equal shared sample U((Tmin> 1]), depending on the inpainting configuration.
We define the track-wise product between vector 2, E I and latent tensor E N XCXL as:
Iy z == (Izz (k) )
and extend this notation to the cosine noise schedule terms as follows:
0r
Br (Br, )EEK.
We perturb the joint latent using track-wise noise:
2, a Zu + 3r Oe, (0,1).
(10)
where each track is independently scaled by its corresponding noise factor.

---

The denoiser fz 7) is trained to regress the velocity target under the v-objective:
VT 0 Br @ 20,
(11)
resulting in the following training loss:
[(0) = E (Z (Zr: 7) 2- 12
(12)
In setup, use N with K {m, u, s), corresponding to the mixture, submixture, and source tracks, respectively. In practice, the loss is computed only over the unknown tracks.
During training, we first sample noise level
[{[[min: 11) and set the per-track timestep vector
T I2 according to one of the following four patterns:
E ((T,T,T), (0, T,T), r), (r,0, r), (T,T,0),,
where each configuration is selected randomly for each training step.
Under this conditioning strategy, the full-noise setting T = (T, corresponds to learning the standard joint score, described in Eq. (9). In contrast, a "single-zero" pattern allows the model to learn conditional score functions for the unobserved tracks while treating the others as fixed observations.
For example, when (Tms Tux Te) (0, T, T), ), the model is trained to approximate the gradient:
b log dr (2f"), 2(()) (m))
(13)
which the joint-score formulation in Eq. cannot compute in closed form. At inference time, we clamp the observed tracks by setting their noise levels zero and apply standard reverse diffusion updates to the remaining (missing) tracks. Pseudocode and a detailed theoretical comparison with conventional inpainting methods are provided Appendix B
4 Experimental Setup
In this section, outline experimental protocol, including baseline models, datasets, and key implementation details. A comprehensive description of hyperparameters and training procedures is provided in Appendix D All baseline results re-evaluated using our test ensure consistency with our experimental setup.
Baselines. We use two recent multi-track diffusion models - MSDM 34 and MSG-LD 36 as baselines, both of which operate on fixed set of stems: bass, drums, guitar, and p ano. In addition to generative and inpainting performance, we assess source extraction capabilities against Hybrid Demucs (HDemucs) 200]. which separates the mixture into bass, drums, o other, and vocals stems, and AudioSep [53] which performs text-conditioned separation based natural language queries.
Datasets. We train and evaluate on three multi-track music datasets: Slakh2100 37 MUSDB18
38 (denoted M,), and MoisesDB (39 (denoted M,). For Slakh2100, we define two subsets: SA, containing only bass, drums, guitar. and piano stems to match the MSDM and MSG-LD setup; and Sp. which includes all remaining stems. Each dataset follows its predefined train/test split. We train models on various dataset combinations evaluate robustness under different source distributions and stem configurations.
Implementation Details. Our models use the Stable Audio backbone [72], which comprises an autoencoder and DiT-based diffusion model. To better accommodate per-track variability in the joint latent space, we replace LayerNorm 933 with GroupNorm (94] using three groups to reflect
the number of tracks.
To bridge the audio-text modality gap, we adopt stochastic linear interpolation between audio and text embeddings the source track, following prior work on multimodal fusion (54 95). Concretely, we generate the prompt "The sound of the [label]" and compute the source conditioning vector c(s) as a convex combination of the CLAP text embedding and its corresponding audio embedding, where the interpolation weight ((( 1)) is sampled randomly for each training example.
All of our models, except the one trained on the full dataset combination (S SB. M.,, M,), are trained for 200K iterations with batch size of 64, using 16 kHz audio segments of 10.24 seconds.

---

Table Total generation results. Reported scores are FAD J. computed against mixture references from each test set. Values in parentheses indicate generation results conditioned on the prompt "The sound of the bass, drums, guitar, and piano", as detailed in Section 5.1] Among our models, Ti provides fair comparison with baseline methods, as it is trained on the same dataset, while T2-Ta illustrate the effects of progressive dataset scaling. Bold values indicate the best results in each
column, and underlined values denote the best results among models trained on the same SA set.
Train Set
Test Set
Model
SA
SB
M.
M.
SA
SFall
M.
M.
MSDM
/
X
4.21
6.04
7.92
7.41
MSG-LD
/
X
X
1.38
1.55
4.61
4.26
71
/
x
X
0.47 (3.57)
1.79
6.34
5,90
MGE
T2
/
/
x
x
3.14 (2.24)
0.63
5.46
4.73
(ours)
73
X
x
/
8.80 (3.96)
6.56
2.87
1.59
74
/
6.83 (5.05)
4.22
2.78
1.47
The full combination model is trained for 320K iterations with batch size of 128. During sampling and inpainting. we apply classifier- free guidance (CFG) with a guidance scale of 2.0 and per-track dropout probability of p = 0.1. All diffusion-based samples including those from baseline models- are generated using 250 inference steps. We adopt DDIM sampling [96] for all our models, while each baseline uses its originally proposed sampling method.
5 Results
We evaluate MGE-LDM on three tasks: total generation, partial generation, and source extraction. Each result table indicates the training dataset(s) used and reports performance across multiple test sets. Unless otherwise specified, partial generation and source extraction performed using text prompts of the form "The sound of the [iabel)." Abbreviations for all stem labels are listed in Appendix Table 4 and additional ablation results are provided in Appendix E
5.1 Music Generation
Table presents FAD (Fréchet Audio Distance) 97 scores computed using VGGish embeddings 988
widely adopted metric for evaluating music generation quality. Note that the S test set contains only mixtures of bass, drums, guitar, and piano, while SFall corresponds to the full Slakh2100 test set, which includes broader and more diverse set of instruments.
We first compare our model T₁ against MSDM and MSG-LD, where all models trained on S Our model achieves the lowest FAD on the SA test set, demonstrating superior fidelity in generating standard four-stem mixtures. However, its generalization to other test sets more limited. On S which includes a broader range of instruments, MSG-LD performs slightly better than T. suggesting mild overfitting to the constrained training distribution. That said, T1 still outperforms MSDM across most test conditions.
To assess the effect of dataset extension, next train the full Slakh2100 dataset (S +S result- ing in model T2. This broader training improves performance on Spall and enhances generalization compared Ti. However, To still lags behind MSG-LD on other subsets, and its performance on SA degrades, likely due to increased variability in the training distribution.
The Ta model, trained on MUSDBI8 (M,) and MoisesDB (Mo). both containing real recordings rather than synthesized audio, achieves strong performance on their respective test sets. Despite the domain shift, it also performs competitively on 110kh2100, with results comparable to MSDM, highlighting the model's robustness to cross dooss-domain generalization. Finally, our model trained on the combined dataset comprising Slakh2100, MUSDB 18, and MoisesDB (denoted Ta), achieves the best overall results on both M,, and M,. It also outperforms Ta on Spalls benefiting from the broader training distribution.
We also report results in parentheses, which correspond to mixture generation conditioned on the text prompt "The sound of the bass, drums, guitar, and piano." " For models trained on datasets beyond SA.
8

---

Table 2: Partial generation results. Scores reported using sub-FAD 4. which measures the distance between the reference mixture and the sum given and generated sources. Each column header (e.g., B, D. G) indicates the target source being generated, conditioned on the remaining stems. Bold and underlined values follow the same convention as in Table
Model
Train Sei
SA
S
S.
D
P
BD
BG
DG
GP
BDG
BDP
BGP
DGP
Brs.
C.P.
Org.
Pipe Reed Str:
S.Lend S.Pad
MSDM
X
x
x 0.56
1.06
0.49
0.70
2.23
1,56
1.95
1.64
1333
2.31
3.19
3.53
5.72
3,86
MSG-LD
/
X
X
0.33
0.34
0.49
0.45
0.70
100
LO0
0.86
0.83
1.47
143
142
2.31
1.76
Ti /
x
x
x
1.02
1.1
L17
L19
1.5
1.29
1.25
1.69
1.65
114
1.50
1.84
1.48
1.84
1.45
0.08
0.23
3.88
5.58
1.38
4.47
1.08
MGE
(ours)
Ta /
Ts x
x
/
x
x
2.11 2.99
1.3
1.29
1.34
199
2.30
2.74
4.07
185
2.32
3,64
4
283
2.95
3:54
2.36
3.90
3.18
4.39
4.9
3.30
5.69
3.57
4.25
6.03
4.66
3.86
5.96
3.58
0.11
0.58
0.15
103
0.22
3,66
0.56
3.52
0.61
2.79
0.58
0.54
0.48
2.32
/
V
1.14
1,50
3.75
2.47
2,06
4.06
2
3.37
2.74
4.55
3.94
4.05
5.66
4.06
5.09
0.42
0.56
0.20
3.14
3,95
0.31
0.40
Table 3: Source extraction results. Metries are reported as Log-Mel LI distance J. For baseline models. scores shown only for stems included in their fixed output set. Bold and underlined values follow the same convention as in Table [
Model
Train Set
SA
SB
M.
M.
S, M.
D
Bos. C.P. Org Pipe Reed St:; S.Lead S. P.Pad
B
1
8
D
G
B.Str Perc.
x
0.90
1.53
AudioSep
1.67
2.84
3.15
2.57
2.06
1.89
131
MSDM MSG-LD
190
1:51
3.32
2.70
2.56
1.69
2.15
1.31
1.28
1.51
120
1.24
2.24
185
1966
160
1.72
1.4
2.36
2.06
Ti
/
1.28
0.66
1.27
1,07
3.22
3.07
3.13
3.11
3.30
2.77
2.68
2.30
3.50
1.91
1.35
5.15
1.61
1.10
2.86
2,68
2.03
2.94
MGE
V V.
1.68
2:71
2.69
2.16
3,43
2.16
1.84
2.33
3.07
2,44
2.31
1,93
3.55
2.14
2.15
4.66
1.86
2.11
2.28
2.18
1,93
2.36
1.10
1.04
0.83
2.29
261
185
3.68
L.76
1.1
1.07
1.02
2.25
including T2. Ta. and T4, the generated mixtures generally contain a wider variety of instruments. This can result in a distributional mismatch with the Setest set, which contains only bass, drums, guitar, and pi ano, thereby increasing the FAD due to reference-target discrepancy. To mitigate this, apply text conditioning at inference time using the above prompt constrain the generated mixture to match the reference instrument set. As shown in the results, this conditioning significantly improves FAD for T2, Ts, and Ta, yielding performance comparable to the MSDM baseline.
Table presents results for partial generation, evaluated using the sub-FAD metric. This metric computes the FAD between the original mixture and reconstruction formed by combining the given submixture with the generated stems. This evaluation protocol was introduced in prior work on music stem completion
34 36 63
On SA our model Terfforms worse than MSG-LD for single-source imputation but shows com- petitive or better performance as the number of generated stems increases. Among our models, Ti performs best on S A due domain alignment. However, for imputation tasks involving broader instrument classes in Sg, models trained on more diverse datasets perform better. Notably, the Ta model, trained solely on M,, and M,, achieves strong results on Sg despite not being exposed to Slakh2100 during training. In particular, it performs well on instruments such as organ, pi pipe, and stri ngs.
strings. The fully trained model T4 further improves performance across several SB stems, including synth lead, synth pad, and pipe.
5.2 Source Extraction
We evaluate text-queried source extraction using the Log Mel L11 distance, following MSG-LD 36 due to the inherent phase mismatch in latent-domain models that complicates waveform-doman evaluation. Table presents results for variety of stems across the SA. Sg. My, and M, test sets. Alongside MSDM and MSG-LD, we compare with two additional basclines: HDemucs 200] a strong waveform- and spectrogram-domee recent system that enables natural language-drive extraction. For fixed-stem baselines (e.g.. MSDM, MSG-LD, HDemues), report both in-distribution results and where possible (e.g., bass in M..).
separation model trained on fixed stems; and AudioSep (33.
generalization
Our model Ti, trained solely on SA performs strongly on the canonical Slakh stems (bass, drums, gui tar, pi pio),,, outperforming MSG-LD on all but bass. However, generalizes poorly to less common stems and real-world recordings. By expanding the training set to encompass the full

---

Slakh2100 dataset, model T2 achieves improved performance in categories such as chr choommato is percussion, organ, synth lead, and synth pad, demonstrating the importance of broader intra-domain coverage.
Interestingly, model Ta generalizes competitively with T2 to synthetic stems, even outperforming some stems such as dr Grums. This indicates cross-domain robustness in our latent inpainting formulation. Finally, model 74, trained on the combined dataset, exhibits robust performance across both synthetic and real-world domains. It maintains strong results on Slakh2 100, while achieving the lowest Log Mel LI scores across most stems in M., and M.. This highlights that incorporating synthetic data such as Slakh2100 alongside real recordings can enhance generalization and improve separation quality on real-world audio.
Overall, MGE-LDM delivers robust, class-agnostic extraction with strong performance across a wide range of instrument types and recording domains, highlighting its effectiveness for text-driven music source extraction both synthetic and real-world settings.
Limitations
While MGE-LDM provides a flexible, class-agnostic framework for multi-track audio modeling, several limitations remain. First, all experiments conducted using 16 kHz monaural audio, which constrains upper-frequency resolution and omits spatial cues, thereby limiting realism for high-fidelity stereo music applications. Second, the model relies on CLAP-base semantic conditioning, which introduces a modality gap between text and audio [99). This can occasionally lead to semantic drift during extraction, resulting irrelevant or hallucinated sources, particularly for stems with limited training data.
Third, although MGE-LDM reduces dependence fixed instrument classes, it still requires multi- stem supervision during training. This dependency restricts applicability to fully unlabeled or large-scale web audio collections. Fourth, training MUSDB18 alone with the same number of iterations as other configurations leads to overfitting, likely due to the limited duration (approximately 10 hours) of its training split. This highlights the challenge of achieving robust performance in low-resource multi-track settings.
Finally, our model is trained using triplets (mix, submix, source) that satisfy mix submix + source in waveform space; however, the latent diffusion process does not enforce an explicit additivity constraint for generated triplets. We believe this omission contributes directly to hallucination phenomena, where the model extracts sources absent from the mixture. Postolache et al. [27] addresses related issue by enforeing additivity in discrete VQ-VAE latent space, estimating the joint likelihood of two sources by counting codebook co-occurrences, effectively modeling p(=mix Z ). where 2, quantized latent codes. Our current pipeline, however, operates in a continuous latent space, which precludes the direct use of such discrete bin-counting methods. Adapting this latent-domain likelihood formulation to continuous spaces, for example, by designing suitable regularizers or adopting a VQ- rQ--ae- encoder with discrete diffusion [02) or MaskGiT [65]- style generation, represents a promising direction for future work.
7 Conclusion
We have presented MGE-LDM. unified class-agnostic latent diffusion framework that jointly models mixtures, submixtures, and individual sources for music generation, stem completion, and text-driven extraction. By formulating stem completion and source extraction as conditional inpainting in a shared latent space and by introducing track-dependent timestep conditioning, we overcome the limitations of fixed-class, additive mixing assumptions and achieve flexible manipulation of arbitrary instrument tracks. Empirically, MGE-LDM matches or exceeds specialized baselines on Slakh2 100 generation and separation benchmarks, while uniquely supporting zero-shot, language-guided extraction across heterogeneous multi-track datasets.
10

---

Acknowledgements
This work was supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) [No. RS-2024-0046161, 90%]] Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) [No. RS-2021- 1343, Artificial Intelligence Graduate School Program (Seoul National University). 10%] Additionally, the GPUs were partly supported by the National ITT Industry Promotion Agency (NIPA)'s nigh-peerfrrmnce computing support program 2025.
References
[1] Aaron Van Den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Semior, Koray Kavukcuoglu, et al. Wavenet: A generative model for raw audio. arXiv preprint (rXiv:1609003499 2016.
[2] Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. In Advances in neural information processing systems, volume 30, 2017.
[3] Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. SoundStream: an end-to-end neural audio codec. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 30:495-507, 2021.
[4] Alexandre Défossez, Jade Copet, Gabriel Synnaeve, and Yossi Adi. High fidelity neural audio compression. arXiv preprint arXiv:2210. 2022.
[5) Rithesh Kumar, Prem Sectharaman, Alejandro Luebs, Ishaan Kumar, and Kundan Kumar. High- figh-fidllity audio compression with improved rvggan. Advances in Neural Information Processing Systems, 36: 27980-27993, 2023.
[6] Prafulla Dhariwal, Heewoo Jun, Christine Payne, Jong Wook Kim, Alec Radford, and Ilya Sutskever. Jukebox: generative model for music. arXiv preprint arXiv:2005000341, 2020.
[7] Andrea Agostinelli, Timo Denk, Zalán Borsos, Jesse Engel, Mauro Verzetti, Antoine Caillon, Qingqing Huang, Aren Jansen, Adam Roberts, Marco Tagliasacchi, et al. MusicL.: generating music from text. arXiv preprint arXiv:2301.11325 2023.
[8] Jade Copet, Felix Kreuk, Itai Gat, Tal Remez, David Kant, Gabriel Synnaeve, Yossi Adi, and Alexandre Défossez. Simple and Controllable Music Generation. Advances in Neural Information Processing Systems, 36:47704 2023.
[9] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
[10] Jonathan Ho, Ajay Jain, and Pieter Abbecl. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840 0001668112222 2020.
[11] Yang Song, Jascha bohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021.
[12] Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis. In International Conference on Learning Representations, 2021.
[13] Nanxin Chen, Yu Zhang, Heiga Zen, Ron J Weiss, Mohammad Norouzi, William Chan. Wavegrad: Estimating gradients for waveform generation. In International Conference on Learning Representations, 2021.
[14] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolttion image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.
[15] Flavio Schneider, Ojasv Kamal, Zhijing Jin, and Bernhard Schölkopf. Moûsai: Efficient text-to-music diffusion models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8050 8068. Association for Computational Linguistics, 2024.
11

---

[16] Ke Chen. Yusong Wu, Haohe Liu, Mariannna Nezhurina, Taylor Berg- Kirk--Kriaatick and Shlomo Dubnov. Musicldm: Enhancing novelty in text-to-music generation using beat-synchronuus mixup strategies. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pages 1200-12100 IEEE, 2024.
[17] Zach Evans, CJ Carr, Josiah Taylor, Scott H Hawley, and Jordi Pons. Fast luiigg-conditione latent audio diffusion, In Forty-first International Conference on Machine Learning, 2024.
[18] Zach Evans. Julian D Parker, CJ Carr. Zack Zukowski, Josiah Taylor, and Jordi Pons. Stable audio open. In ICASSP 2025-2025 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pages 1-5, IEEE, 2025.
[19] Enric Gusó, Jordi Pons, Santiago Pascual, and Joan Serrà. On loss functions and evaluation metrics for music source separation. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pages 306 310. IEEE, 2022.
[20] Alexandre Défossez. Hybrid spectrogram and waveform source separation.
arXiv preprint
arXiv:2111.03000 2021.
[21] Yi Luo and Jianwei Yu. Music source separation with band-split mn. JEEE/ACM Transactions on Audio. Speech, and Language Processing, 31:1893-1901, 2023.
[22] Ui-Hyeop Shin, Sangyoun Lee, Tachan Kim, and Hyung-Min Park. Separate and Reconstruct: Asym- metric encoder-decoder for speech separation. In Advances in Neural Information Processing Systems, volume 37. pages
2024.
[23] Cem Subakan and Paris Smaragdis. Generative adversarial source separation. In 2018 JEEE In- ternational Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 26-30. IEEE, 2018.
[24] Qiuqiang Kong, Yong Xu, Wenwu Wang, Philip J. B. Jackson, and Mark D. Plumbley. Single-channel signal separation and deconvolution with generative adversarial networks. In Proceedings of the 28th International Joint Conference on Artificial Intelligence, page
AAAI Press, 2019.
[25] Vivek Jayaram and John Thickstun. Source separation with deep generative priors. In Proceedings of the 37th International Conference on Machine Learning, 2020.
[26] Vivek Narayanaswamy, Jayaraman J Thiagarajan, Rushil Anirudh, and Andreas Spanias. Unsupervised audio source separation using generative priors. In INTERSPEECH. 2020.
[27] Emilian Postolache, Giorgio Mariani, Michele Mancusi, Andrea Santilli, Luca Cosmo, and Emanuele Rodolà. Latent autoregressive source separation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages
2023.
[28] Bo Chen, Chao Wu, and Wenbin Zhao. Sepdiff:: ICASSP 2023-2023 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP). pages 1-5. JEEE, 2023.
Speech separation based on denoising diffusion model. In
[29] Robin Scheibler, Youna Ji, Soo-Whan Chung. Jacuk Byun, Soyeon Choe, and Min-Seok Choi. Diffusion- based generative speech source separation. In ICASSP 2023-2023 IEEE International Conference on Acoustics. Speech and Signal Processing (ICASSP), pages 1-5. IEEE, 2023.
[30] Joan Serrà, Santiago Pascual, Jordi Pons, R Oguz Araz, and Davide Scaini. Universal speech enhancement with score-based diffusion. arXiv preprint arXiv:2206. 220.022006000055 2022.
[31] Yen-Ju Lu, Zhong-QU -in Wang. Shinji Watanabe, Alexander Richard, Cheng Yu, and Yu Tsao. Conditional diffusion probabilistic model for speech enhancement. In ICASSP 2022-2022 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pages 7402 -4067 Icce, 2022.
[32] Hao Yen, François G Germain, Gordon Wichern, and Jonathan Le Roux. Cold diffussion for speech enhancement. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5. IEEE, 2023.
[33] Julius Richter, Simon Welker, Jean-Marie Lemercier, Bunlong Lay. and Timo Gerkmann. Speech enhancement and dereverberation with iiffusioo-bbsed generative models. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 31:2351-2364, 2023.
[34] Giorgio Mariani, Irene Tallini, Emilian Postolache, Michele Mancusi, Luca Cosmo, and Emanuele Rodolà. Multi- source diffusion models for simultaneous music generation and separation. In The Twelfth International Conference Learning Representations, 2024.
12

---

[35] Emilian Postolache, Giorgio Mariani, Luca Cosmo, Emmanouil Benetos, and Emanuele Rodolà. General- ized multi-source inference for text conditioned music diffusion models. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6980-6984, IEBE, 2024.
[36] Tornike Karchkhadze. Mohammad Rasool Izadi, and Shlomo Dubnov. Simultaneous music separation and generation using multi-track latent diffusion models. In ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5, IEEE, 2025.
[37] Ethan Manilow, Gordon Wichem, Prem Seetharaman, and Jonathan Le Roux. Cutting music souree separation some Slakh: A dataset to study the impact of training data quality and quantity. In 2019 JEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA), pages 45-49, IEBE, 2019.
[38] Zafar Ratii, Antoine Liutkus, Sabian-Roberr Stöter, Stylianos loannis Mimilakis, and Rachel Bittner. The MUSDBI8 corpus for music separation, December 2017. URL htRthhtps://000 .rrg 102511111111
zenodo. 1117372
[39] Igor Pereira, Felipe Araújo, Filip Korzeniowski, and Richard Vogl. MoisesDB: A dataset for source separation beyond 4-stems. International Society for Music Information Retrieval Conference, 2023.
[40] Zhongweiyang Xu, Debottam Dutta, Yu-Lin Wei, and Romit Roy Choudhury. Multi-source music generation with latent diffusion, arXiv preprint 0000::::0990009900 2024.
[41] Yi Luo and Nima Mesgarani. Conv-tasnet: Surpassing ideal time- frequency magnitude masking for speech separation. IEEE/ACM transactions audio, speech, and language processing, 27(6):1256-1266 2019.
[42] Alexandre Défossez. Nicolas Usunier, Léon Bottou, and Francis Bach. Music source separation in the waveform domain. arXiv preprint arXiv: 132244, 2019.
[43] Ge Zhu, Jordan Darefsky, Fei Jiang. Anton Selitskiy, and Zhiyao Duan. Music source separation with generative flow. IEEE Signal Processing Letters, 29:2288-2225 2292, 2022.
[44] Tsubasa Ochiai, Mare Delcroix, Yuma Koizumi, Hiroaki Ito, Keisuke Kinoshita, and Shoko Araki. Listen to what you want: Neural network- universal sound selector. In INTERSPEECH, 2020.
[45] Mare Delcroix, Jorge Bennasar Vázquez, Tsubasa Ochiai, Keisuke Kinoshita. Yasunori Ohishi, and Shoko Araki. SoundBeam: Target sound extraction conditioned on sound-class labels and enrollment clues for increased performance and continuous learning. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 31:121-133 1136, 2022.
[46] Bandhav Veluri, Justin Chan, Malek Itani. Tuochao Chen, Takuya Yoshioka, and Shyamnath Gollakota. Real-time target sound extraction. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5. IEEE, 2023.
[47] Hang Zhao, Chuang Gan, Andrew Rouditchenko, Carl Vondrick, Josh McDermott, and Antonio Torralba. The sound of pixels. Proceedings of the European conference on computer vision (ECCV), pages 570-586, 2018
[48] Hao-Wen Dong, Naoya Takahashi, Yuki Mitsufuji, Julian McAuley, and Taylor Berg- Kirkpatrick. CLIPPSe:: Learning text-queried sound separation with noisy unlabeled videos. In Proceedings of International Conference on Learning Representations (ICLR), 2023.
[49] Jie Hwan Lee, Hyeong Scok Choi, and Kyogu Lee. Audio query-based music source separation. In International Society for Music Information Retrieval Conference, 2019.
Aleong-Seok
[50] Qiuqiang Kong, Ke Chen, Haohe Liu, Xingjian Du, Taylor Berg- Kirkpatrick, Shlomo Dubnov, and Mark D Plumbley. Universal source separation with weakly labelied data, arXiv preprint arXiv:2305.07444 2023.
[51] Kevin Kilgour, Beat Gfeller, Qingging Huang, Aren Jansen, Scott Wisdom, and Marco Tagliasacchi. Text-driven separation of arbitrary sounds. In INTERSPEECH, 2022.
[52] Xubo Liu, Haohe Liu, Qiuqiang Kong, Xinhao Mei, Jinzheng Zhao, Qiushi Huang, Mark D Plumbley, and Wenwu Wang. Separate what you describe: Language queried audio source separation. arXiv preprint arXiv:2203.1 /5147, 2022.
[53] Xubo Liu, Qiuqiang Kong, Yan Zhao, Haohe Liu, Yi Yuan, Yuzhuo Liu, Rui Xia, Yuxuan Wang. Mark D Plumbley, and Wenwu Wang. Separate anything you describe. IEEE/ACM Transactions on Audio, Speech. and Language Processing, 2024.
13

---

[54] Hao Ma, Zhiyuan Peng, Xu Li. Mingjie Shao. Xixin Wu, and Ju Liu. CLAPSep: Leveraging contrastive pre-trained model for multi-modal query-condtioneed target sound extraction. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2024.
[55] Hao Ma. Zhiyuan Peng. Xu Li, Yukai Li, Mingjie Shao, Qiuqiang Kong, and Ju Liu. Language- queried target sound extraction without parallel training data. In ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5. TEFE, 2025.
[56] Yusong Wu, Ke Chen, Tianyu Zhang. Yuchen Hui, Taylor Berg-Kingg King-akkattrikk, and Shlomo Dubnov. Large- scale contrastive language-audhe pretraining with feature fusion and Syyoord-o-ccation augmentation. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5, IEEE, 2023.
[57] Soroush Mehri, Kundan Kumar, Ishaan Gulrajani, Rithesh Kumar, Shubham Jain, Jose Sotelo, Aaron Courville, and Yoshua Bengio. SampleRNN: An unconditional end-to-end neural audio generation model. In International Conference on Learning Representations, 2017.
[58] Chris Donahue, Julian McAuley, and Miller Puckette. Adversarial audio synthesis. In International Conference on Learning Representations, 2019.
[59] Jesse Engel, Kumar Krishna Agrawal, Shuo Chen, Ishaan Gulrajani, Chris Donahue, and Adam Roberts. GANSynth: Adversarial neural audio synthesis. In International Conference on Learning Representations, 2019.
[60] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[61] Zalán Borsos, Raphaël Marinier, Damien Vincent, Eugene Kharitonov, Olivier Pietquin, Matt Sharifi, Dominik Roblck, Olivier Teboul, David Grangier, Marco Tagliasacchi, et al. Audiolm: a language modeling approach to audio generation. JEEE/ACM transactions audio, speech, and language processing, 31:2523-2553 25233, 2023.
[62] Felix Kreuk, Gabriel Synnaeve, Adam Polyak, Uriel Singer, Alexandre Défossez, Jade Copet, Devi Parikh, Yaniv Taigman, and Yossi Adi. AudioGen: Textually guided audio generation. In The Eleventh International Conference on Learning Representations, 2023.
[63] Chris Donahue, Antoine Caillon, Adam Roberts, Ethan Manilow, Philippe Esling, Andrea Agostinelli, Mauro Verzetti, Ian Simon, Olivier Pietquin, Neil Zeghidour, et al. Singsong: Generating musical accompaniments from singing. arXiv preprint arXiv: 2023.
[64] Yixiao Zhang, Yukara Ikemiya, Woosung Choi. Naoki Murata, Marco A Martinez-Ramírez Liwei Lin, Gus Xia, Wei-Hsiang Liao, Yuki Mitsufuji, and Simon Dixon. Instruct- MusiucGn: Unlocking text- to-music editing for music language models via instruction tuning. arXiv preprint arXiv:2405. 18386, 2024.
[65] Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman. Maskgit: Masked generative image transformer. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pages 11315-11325. 2022.
[66] Hugo Flores Garcia, Prem Seetharaman, Rithesh Kumar, and Bryan Pardo. Vampnet: Music generation via masked acoustic token modeling. In International Society for Music Information Retrieval Conference, 2023.
[67] Marco Comunità, Zhi Zhong, Akira Takahashi, Shigi Yang. Mengjie Zhao, Koichi Saito, Yukara Ikemiya, Takashi Shibuya, Shusuke Takahashi, and Yuki Mitsufuji. Speemaskgit: Masked generative modeling of audio spectrograms for efficient audio synthesis and beyond. In International Society for Music Information Retrieval Conference, 2024.
[68] Qingqing Huang, Daniel S Park, Tao Wang. Timo I Denk, Andy Ly, Nanxin Chen, Zhengdong Zhang, Zhishuai Zhang, Jiahui Yu, Christian Frank, al. Noise2music: Text-conditioned music generation with diffusion models. arXiv prepriat arXiv:2302.039) 2023.
[69] Shih-Lun Wu, Chris Donahue, Shinji Watanabe, and Nicholas Bryan. Music controlnet: Multiple time-varying controls for music generation. JEEE/ACM Transactions on Audio, Speech, and Language Processing, 32:2692-277 2024.
14

---

[70] Haohe Liu, Zehua Chen, Yi Yuan, Xinhao Mei, Xubo Liu, Danilo Mandic, Weawu Wang, and Mark D. Plumbley. AudioLDM: text-to-audio generation with latent diffusion models. In Proceedings of the 40th International Conference on Machine Learning. ICML'23, 2023.
[71] Haohe Liu, Yi Yuan, Xubo Liu, Xinhao Mei. Qiugiang Kong, Qiao Tian, Yuping Wang, Wenwu Wang. Yuxuan Wang, and Mark D Plumbley. AudioLDM 2: Learning holistic audio generation with self- supervised pretraining. JEEE/ACM Trunsactions on Audio, Speech, and Language Processing, 2024.
[72] Zach Evans, Julian D Parker, CJ Carr, Zack Zukowski. Josiah Taylor, and Jordi Pons. Long-form music generation with latent diffusion. International Society for Music Information Retrieval Conference, 2024.
[73] Yuancheng Wang. Zeqian Ju, Xu Tan, Lei He, Zhizheng Wu, Jiang Bian, et al. Audit: Audio editing by following instructions with latent diffusion models. Advances in Neural Information Processing Systems, 01771400-71377 2023.
[74] Bing Han, Junyu Dai, Weituo Hao, Xinyan He, Dong Guo, Jitong Chen, Yuxuan Wang, Yanmin Qian, and Xuchen Song, InstructME: an instruction guided music edit framework with latent diffusion models. In Proceedings of the Thirty-Third International Joint Conference Artificial Intelligence, IJCAI '24, 2024.
[75] Julian D Parker, Janne Spijkervet, Katerina Kosta, Furkan Yesiler, Boris Kuznetsov, Ju-Chiang Wang. Matt Avent, Jitong Chen, and Due Lc. Stemgen: A music generation model that listens. In ICASSP 2024-2024 IEEE International Conference Acousties, Speech and Signal Processing (ICASSP), pages 1116- 166-11201 1120. IEEE, 2024.
[76] Yao Yao, Peike Li, Boyu Chen, and Alex Wang. Jen-1 composer: A unified framework for high-fidelity multi-track music generation. Proceedings of the AAAI Conference on Artificial Intelligence, pages 14459-144 14467, 2025.
[77] Simon Rouard, Robin San Roman, Yossi Adi, and Axel Roebel. MusicGen-Stem: Multi-stem music generation and edition through autoregressive modeling. In ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5. IEEE, 2025.
[78] Javier Nistal, Marco Pasini, Cyran Aouameur, Maarten Grachten, and Stefan Lattnrr Diff-a-riff: Musical accompaniment co-creation via latent diffusion models. International Society for Music Information Retrieval Conference, 2024.
[79] Javier Nistal, Marco Pasini, and Stefan Lattner. Improving musical accompaniment co-creation via diffusion transformers. NeurlPS 2024 Workshop on Al-Driven Speech, Music, and Sound Generation (Audio Imagination). 2024.
[80] Marco Pasini, Maarten Grachten, and Stefan Lattner. Bass accompaniment generation via latent diffusion. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1166-1170. IEEE, 2024.
[81] Giorgio Strano, Chiara Ballanti, Donato Crisostomi, Michele Maneusi, Luca Cosmo, and Emanuele Rodolà. Stage: Stemmed accompaniment generation through prefix-based conditioning. International Suciety for Music Information Retrieval Conference, 2025.
[82] Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. Advances in neural information processing syxtems, 35-26565-26577, 2022.
[83] Tornike Karchkhadze, Mohammad Rasool Izadi, Ke Chen, Gerard Assayag, and Shlomo Dubnov. Multi- Track Musicl DM: Towards versatile music generation with latent diffusion model. arXiv preprint arXiv:2409.02845 2024.
[84] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. NeurlPS Workshop on Deep Generative Models and Downstream Applications, 2021.
[85] Diederik P Kingma, Max Welling, et al. Auto- encoding variational bayes, 2013.
[86] William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4195-4205, 2023.
[87] Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. In International Conference on Learning Representations, 2022.
[88] Jascha Sohl- Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learning, pages 2256-2265. pmlr, 2015.
15

---

[89] Andreas Lugmayr. Martin Danelljan, Andres Romero, Fisher Yu. Radu Timofte, and Luc Van Gool. Repaint: Inpainting using denoising diffusion probabilistic models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11461-11471, 2022.
[90] Ciprian Corneanu, Raghudeep Gadde, and Aleix M Martinez. Latentpaint: Image inpainting in latent space with diffusion models. In Proceedings of the IEEF/CVF winter conference on applications of computer vision. pages 4334-4343, 2024.
[91] Sora Kim, Sungho Suh, and Minsik Lee. RAD: Region-aware diffusion models for image inpainting. arXiv preprint arXiv:2412.0911), 2024.
[92] Tsiry Mayet, Pourya Shamsolmoali, Simon Bernard, Eric Granger, Romain HÉRAULT. and Clement Chatelain. TD-Paint: Faster diffusion inpainting through time aware pixel conditioning. In The Thirteenth International Conference on Learning Representations, 2025.
[93] Jimmy Lei Ba. Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint urXiv:1607.06495 2016.
[94] Yuxin Wu and Kaiming He. Group normalization. In Proceedings of the European conference on computer vision (ECCV), pages 19,19,222 2018.
[95] Timo Lüddeckee and Alexander Eeker. Image segmentation using text and image prompts. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 7086-7096, 2022.
[96] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In International Conference on Learning Representations. 2021.
[97] Kevin Kilgour, Mauricio Zuluaga, Dominik Roblek, and Matthew Sharifi. Fr), 'echet audio distance: A metric for evaluating music enhancement algorithms. In INTERSPEECH, 2018.
[98] Shawn Hershey. Sourish Chaudhuri, Daniel PW Ellis, Jort F Gemmeke, Aren Jansen, R Channing Moore, Manoj Plakal, Devin Platt, Rif A Saurous, Bryan Seybold, et al. CNN architectures for large-scale audio classification. In 2017 ieee international conference on acoustics, speech and signal processing (icassp), pages 131 135. IEEE, 2017.
[99] Victor Weixin I Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Y Zou. Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning. Advances in Neural Information Processing Systems, 15:17122-17625, 2022.
[100] Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and Baining Guo. Vector quantized diffusion model for text-to- image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10696-10706 2022.
[101] Andrew Campbell, Joe Benton, Valentin De Bortoli, Thomas Rainforth, George Deligiannidis, and Arnaud Doucet. A continuous time framework for discrete denoising models. Advances in Neural Information Processing Systems, 35:28266 28279, 2022.
[102] Aaron Lou, Chenlin Meng, and Stefano Ermon. Discrete diffusion modeling by estimating the ratios of the data distribution. In Proceedings of the 41st International Conference on Machine Learning, ICML'24, 2024.
[103] Colin Raffel. Learning-baat -baee methods for comparing sequences, with applications to audio-to-min -iiid alignment and matching. Columbia University. 2016.
[104] Xun Huang and Serge Belongic. Arbitrary style transfer in real-time with adaptive instance normalization. Proceedings of the IEEE international conference on computer vision, pages 1501 11110 2017.
[105] Ruben Ciranni, Giorgio Mariani, Michele Mancusi, Emilian Postolache, Giorgio Fabbro, Emanuele Rodolà, and Luca Cosmo. COCOLA: Coherenco-oriented contrastive learning of musical audio rep- resentations. In ICASSP 2025-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5. IEEE, 2025.
[106] Florian Grötschla, Ahmet Solak, I Luca A Lanzendörfer, and Roger Wattenhofer. Benchmarking music generation models and metrics via human preference studies. In ICASSP 2025-2025 IEEE International Conference on Acousties, Speech and Signal Processing (ICASSP), pages 1-5. IEEE, 2025.
[107] Azalea Gui, Hannes Gamper, Sebastian Braun, and Dimitra Emmanouilidou. Adapting frechet audio distance for generative music evaluation. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1331-1333 IEEE, 2024.
16

---

[108] Yichen Huang, Zachary Novack, Koichi Saito, Jiatong Shi, Shinji Watanabe, Yuki Mitsufuji, John Thickstun, and Chris Donahue. Aligning text-to-music evaluation with human preferences. arXiv preprint arXiv:2503.16660 2025.
[109] Marco Pasini. Stefan Lattner, and George Fazekas. Music2Latent: Consistency autoencoders for latent audio compression. International Society for Music Information Retrieval Conference. 2024.
[1 [0] Marco Pasini, Stefan Lattner, and György Fazekas. Music2Latent2: Audio compression with summary embeddings autoregressive decoding. In ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1 1-5. IEEE, 2025.
[111] Junghyun Koo, Yunkee Chae, Chang-Bin Jeon, and Kyogu Lee. Self-refining pseudo labels for music source separation with noisy labeled data. International Society for Music Information Reirieval Conference, 2023.
[112] Dmitry Bogdanov, Minz Won, Philip Tovstogan, Alastair Porter, and Xavier Serra. The MTG-Jamendo dataset for automatic music tagging. In Machine Learning for Music Discovery Workshop. International Conference on Machine Learning (ICMI 2019), Long Beach, CA, United States, 2019. URL http
//hd1 handdlee net 022 42015
[113] Michačl Defferrard, Kirell Benzi, Pierre Vandergheynst, and Xavier Bresson. FMA: A dataset for music analysis. International Society for Music Information Retrieval Conference, 2016.
17

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction clearly describe the scope contributions the paper. We propose a novel three-track latent diffusion model that jointly addresses multi-track music genera- tion, source imputation, and text-conditing conditioned extraction. These capabilities are accurately summarized in the abstract and elaborated in itemized form in the introduction (please Section [ align with the methods and experiments presented throughout the paper.
The claims
Guidelines:
The answer NA means that abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Please see Section
Guidelines:
The answer NA means that paper has no limitation while the answer means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust results are violations these assumptions (e.g.. independence assumptions, noiseless settings, model well-speciffication asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested a few datasets with few runs. In general, empirical results often depend implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If l'applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty conceriin limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: Please see Appendix A.2 and B
Guidelines:
18

---

The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should he numbered and cross-referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Please see Section 3 4 and Appendix D
Guidelines:
The answer NA means that the paper does not include experiments.
the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution novel architecture, describing the architecture fully might suffice, if the contribution is specific model and empirical evaluation, it may be necessary to either make possible others to replicate the model with same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results. access to hosted model (e.g.. in the of large language model), releasing of model checkpoint, or other means that are appropriate to research performed.
While NeurIPS does not require releasing code, the conference does require all submissions provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) If the contribution is primarily architecture clearly and fully.
model architecture, the paper should describe the
(c) If the contribution is a new model (e.g.. a large language model), then there should either be way to access this model for reproducing the results way to reproduce the model (e.g.. with open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access the model is limited in way (e.g.. registered users), hut should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code. with sufficient instructions to faithfully reproduce the main experimental results, as descrihed in supplemental material?
Answer: [Yes]
Justification:
We released the full training code and all model checkpoints under an open-source
license.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines
gui des.dee//coooo Pesss/Coeeubnn ssi onpol cyl for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
19

---

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (httpss//nips... cc/ public/gui wetpgeieeeeeeeeeeeeeeeeeeeeeeeee
wwi.nnisssssssiiiooiooooii for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for new proposed method and baselines. If only a subset experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code permitted.
6. Experimental selting/details
Question: Does the paper specify all the training and test details (e.g.. data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: See Scction and Appendix D
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined other appropriate informa- tion about the statistical significance of the experiments?
Answer: [NA]
Justification: We did not perform repeated runs due to limited computational resources. While we do not report error bars confidence intervals, the paper includes extensive evaluation across datasets, stem types, and tasks, with per-stem metrics and consistent trends observed across conditions. These provide empirical support for the reliability of our findings. Detailed results are included Section and Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, statistical significance tests, least for the experiments that support the main claims the paper.
The factors variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report I-sigma error bars, but one should state it. The authors should preferably report 2-sigma error har than state that they have 96% CI, if the hypothesis of Normality of errors not verified.
asymmetric distributions, authors should be careful not show in tables or figures symmetric error bars that would yield results that are out of range (c.g. negative error rates).
crror bars are reported tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
Experiments compute resources
Question: For each experiment, does the paper provide sufficient information the computer resources (type of compute workers, memory, time execution) needed reproduce the experiments?
Answer: [Yes]
20

---

Justification: We report our computational setup in Appendix However, do not report execution time due to variability across models and hardware conditions.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory
storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer:
[Yes]
Justification: The authors have carefully reviewed the NeurlPS Code of Ethics and confirm that the research complies with its guidelines.
Guidelines:
The answer means that authors have not reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain special circumstances that require deviation from Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss hoth potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [Yes]
Justification: As the proposed method involves generative modeling of music, we address potential broader impacts in Appendix
Guidelincs:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has societal impact why the paper does not address societal impact.
Examples of negative socictal impacts include potential malicious unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploy- ment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it For example, is legitimate point out that an improvement the quality generative models could used to generate deepfakes for disinformation. On other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional unintentional) misuse the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition to attacks, mechanisms for monitor- ing misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g.. pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
21

---

Justification: Our models are trained on relatively small, open-source multi-track dataset, which limits the potential for misuse.
Guidelines:
The answer NA means that paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are license and terms use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All third-part party codebases and datasets used in this work are open-source and publicly available.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset. For scraped data from particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, (paperappsssssaaa www.swttthhsseo com/ cat aset s has curated licenses for some datasets. Their licensing guide can help determine the license
dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it changed) should provided.
this information not available online, the authors are encouraged reach out to asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: released all code, training checkpoints, and inference scripts, accompanied by well-structured documentation detailing installation, usage, and reproduction steps.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their sub- missions via structured templates. This includes details about training. license, limitations.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sereenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The study daes not include any crowdsourced data collection experiments involving human participants..
Guidelines:
22

---

The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then much detail as possible should he included in the main paper.
According to the NeurlPS Code of Ethics, workers involved data collection, curation, other lahor should be paid at least. minimum wage in country data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: This paper do not contain any crowdsoureing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. you obtained IRB approval. you should clearly state this the paper.
recognize that procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applica- ble), such as the institution conducting the review.
16. Declaration LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original. or non-standard component of the
methods in this research? Note that if the LLM is used only for writing, editing. or formatting purposes does not impact the core methodology, scientific rigorousness, or originality of the rescarch, declaration is not required.
Answer: [NA]
Justification: We did not involve any large language models in the development of the core methods.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM what should should not be described.
for 2ttp:/eennnee/2222/1121
23

---

A
Background
this section, we review three foundational components that support our method: (1) the v-objective score- matching formulation for diffusion models [7], which underpins our latent modeling, and (2) the canonical inpainting algorithm introduced by Song et al. | and (3) fürther refined by Lugmayr et al. [89.
A.1 y-Objective Diffusion
We adopt the continuous-time diffusion framework proposed by Salimans al. applied in recent music generation models [T5] 17 72)
87 which has been widely
The forward perturbation kernel is defined
g(z-/zo) = (/zz; 20 321)
(14)
where:
a, = cos(
B+ = =in((>>, @r 2 TE [,1].
7
Instead of predicting the noise e as in DDPMs
10 the v-objective formulation introduces a velocity target:
Oz.
@co((er)
sin((): sin( (- )0 cus( a.e 3,
Dr DΦp
zn 0ø,
(15)
dø,
and trains the denoiser fo T) directly regress this velocity. Recovering zo and
Using the definition of v, and recalling the perturbation process defined in Eq.
3, Or Zo 3
(16)
rearrange the recover the clean latent by:
sin()r) cos( ) V+
sin(ør) (z, cos(ór) (zo) Ur. cos(
sin (0+) cos(0r) cos' ( sin(Ø-)ou
(sin"(0,) + co: (d:))z cos(2r )2, sin(d,) )0r1
thus get
Zo a, V..
(17)
Similarly, the noise vector can expressed as:
3,2, + ax
(18)
Score function approximation. The marginal score Vx, 42g (2,) timestep (or noise level) is approxi- mated as:
Vx, (oggr(z)) 0,20(27) Zr
87
az a, Br folz,, 7) 2,
83
S2rr fo
3
--, 8,
folz,,
where (0(z.) = a,2, (ofo from the Eq. 17
DDIM sampling, We adopt the DDIM sampling (96) to generate samples in a non-stochastic, deterministic
manner. Given a latent 2., each reverse step is computed as:
ú, fo(2r,7) 7)
20 arZr 0r
3. + a+ 0-
7, 0 + Briee,
where 7' 7 taken from linearly spaced decreasing schedule from to 0. We use 250 inference steps in all experiments.
24

---

A.2 Canonical Inpainting in Score-Based Models
The core idea behind inpainting in score- boore-baeed generative models is estimate the score of the unknown region conditioned on the known region [1]
Let K denote the set of tracks. Suppose a subset K observed (i.e., known), and K denote the complement, i.c. the unobserved tracks we aim inpaint. Define z" {2( Jven and z := (2 her.
The goal is to approximate the conditional score:
Ver log dr (z,
(19)
This conditional gradient is generally intractable for a score model trained only on joint marginals. However, following Song et al. [11 can approximate it via:
/ Qr (z. 2e dz
2: 1- (20)
/
9- 27 2o )+ (2 | ddz
(u
9- (z. wa 20))
E (ufl
9r (z. (z.)
(20)
gr (z. (2.))
(21)
where 20 q-(2f12)) Ni2 a-z0, (21) is noised sample of the known region. Accordingly, the conditional score can be approximated as:
V log qr (z. 20) = V.r log d (zf12.))
V.F log do (zz af
where [x.:; 2!" denotes composite latent vector, such that the known region is replaced by 2f while the unknown region remains as 2f. adopting the same notation as Song et al. [...
This approximation enables zero-shot inpainting without requiring retraining: at each diffusion timestep, a noised version of the known latents is sampled, concatenated with the current estimate of the unknown latents, and passed to the score model. The resulting gradient is then applied to update only the unknown region. This process is repeated throughout the reverse diffusion trajectory.
A.3 RePaint
Lugmayr al. proposed RePuint 89 resampling-basee mechanism that improves score-based inpainting by repeating diffusion process across multiple forward reverse cycles. Their key insight that, in conventional inpainting (as described in Eq. (2I). the sampled noise for the known region independent of the generated (inpainted) region. This lack of synchronization can lead to semantic inconsistencies and disharmony between the known and unknown parts of the sample.
To address this, RePaint introduces resampling mechanism during generation. At each denoising timestep, the algorithm alternates between one reverse diffusion step and one forward diffusion step, repeating this cycle times. These micro-steps refine the sampling distribution and can have the effect of partially marginalizing over the known region at noise level r in Equation (2I). thereby reducing the approximation error inherent in conditional score estimation. This iterative resampling procedure improves consistency but incurs higher computational cost, each denoising step requires multiple forward-reverse passes, making RePaint significantly more expensive than standard inpainting methods.
While originally proposed DDPM- bascd models, RePaint can adapted to velocity- -based objectives as used our framework. We apply this adaptation in the sampling procedure deseribed Algorithm [ Setting the resampling count U 1 recovers the canonical single- inpainting method described in Appendix A.2 A.2
B Adaptive Timestep Conditioning and Score Approximation
using a single sampled estimate of the known latents. This corresponds to a hhgh-variance Monte Carlo estimate of the expectation over (zf) 22 ). which may lead to instability especially at high noise levels.
Conventional inpainting approaches approximate the conditional score Eq. 19
By contrast, the adaptive timestep model proposed in Section 3.4 inspired by TD-Paint 92 circumvents this marginalization by training the model to directly approximate the conditional score.
25

---

Algorithm Inpainting using the RePaint approach.
Input:
Number of timesteps T;
Re-denoising steps reverse step U;; Noise schedule {7, Hi=o with ar, Pr;
Binary mask m (1 for known, 0 for unknown); Known clean (masked) latents zg k
Denoiser network fo Mer ~ ~N,00, r)
1:
2: for == T. do
u I....,
4:
DDIM sampling step
5:
Ür. + folzT,, Ti) io + ?T, 3rg Dr
6:
é t 3r,
3r, + ar, Dt.
8:
- + 0 30 Br:
9:
Sample the known regions
10:
N(0, 1) if clse known - 07;- 20
11:
known Br.-
12:
Combine known and generated regions b Reapply forward process
13:
-T + m @ z
m) TT
14:
15:
if u < U and / > then
zr, ~ N ar11 2 (1 1-
of
)1)
16:
at
18:
end for
19:
end for
20:
return Zro
Following the notation in Section 3.4 and Appendix dataset. Then, using alternative factorization:
A.2
we assume = [K6; 20) is clean sample from the
9- (2
9. (et xo 120 ddx
Or (2. (z. 2 (x(dd (a ddxo
Baoe (2- 1xf x00)
(22)
= go (2=12b 20) = (2 (2=12))
(23)
= -N a, 26, 121)
(24)
where the approximation assumes that z ~ (xxx (z))) is available from the dataset. Unlike the marginalization- based approximation in Eq. (1 this expression introduces sampling noise during inference, thereby reducing variance.
From this, the conditional score be written
Ver log (-
a 2
Zr
(25)
83
a Zo (z 2 7T)
zf
23
(26)
82
Qt
fo(z)
2n
)r,
(27)
3,
where fo(-)r denotes the output corresponding the unknown region. The per-track timestep vector /! defined as:
if e
for each & E K,
(28)
0. E
26

---

Algorithm 2 Inpainting using adaptive timestep approach
Input:
Number of timesteps T;
Re-denoising steps reverse step U;; Noise schedule {7, Hi=o with ar, Pr; Binary mask m (1 for known, 0 for unknown); Known clean (masked) latents zō kmow; Denoiser network fo
1: (0.
2: Tr Tr(1 m)
3: Zrr m2z mowwa + (1 (1-m)E
4: for i ...... do
5:
Partial DDIM sampling unknown region
Ūt; fo( 7:)
7:
2n t ar, 2r Br, D+;
8:
+ Br,ZT, + Cit, DT;
271-1 t- 0.,-, 20 + 1.._12
10:
TI-1 T--1 m)
11: ZT f- muko
known
+ (1 m1) 2,
unknown
12:
end for
13: return Zro
as in Section 3.4 region. This allows the denoiser to explicitly learn the conditional score z. avoiding the need for stochastic marginalization and improving accuracy in conditional inpainting tasks. The full sampling procedure is detailed Algorithm
The model is trained using the velocity objective in Eq. 12 restricted to the unknown
C Iterative Generation
In addition to the one-stage mixture generation described in Section MGE-LDM also supports an iterative, dem--by-ttem synthesis procedure. This approach constructs full mixture by sequentially generating individual sources, leveraging the partial generation mechanism each step.
Let I = (c y be an (ordered) set of CLAP embeddings corresponding to the desired instrument description. At the first iteration (i = 1), we generate an initial source latent 2 by sampling with the model conditioned only on the prompt (s), g(m) g(w) 2(() (=(2(m)) (m) (1)), (+) (00 (.c(())
set 2(11 2(*) as the initial submixture latent. For each subsequent iteration 2,.., [Z], we follow the itcrative imputation strategy partial generation, treating the current submixture as the accumulated sum of decoded sources from the previous steps.
Finally, the full mixture waveform is constructed by decoding and summing the generated source latents:
D(
preliminary evaluation of this iterative procedure presented in Appendix E
D Experimental Details
This appendix provides a comprehensive description of datasets, baseline implementations, model architecture, and training settings used throughout our experiments.
D.1 Datasets
We train and evaluate on three multi-traed track music datasets: S1akh2100 37
MUSDBIS 38 and MoisesDB [39
All datasets consist of mixture tracks paired with isolated stem recordings. A summary of stem abbreviations is provided Table
27

---

Table Abbreviations of instrument stems. The table lists all abbreviations used throughout the paper along side their corresponding full instrument labels, grouped by dataset.
Common
MoisesDB
Brs.
C.P.
Org.
Pipe
Read
S.Lend
S.Pad
Batr
Inst.
bass druns guitar plano vocals breas
chronatic percuasion
organ pipe reed at strings ings
synth
lead
synth
pad
scrings
bowed
perousaiio in
Slakh2100 derived from the Lakh MIDI Dataset v0.1 [03]] and contains synthesized tracks rendered with sample-base based virtual instruments. It comprises 2100 songs divided into training (1500), validation (375), and test (225) splits, totaling approximately 145 hours of audio. It includes a wide variety of instrument classes (c.g.. bass, drums, gui tar, pi anc, sgg,ngs, synth pad, etc.). We adopt the naming SA to denote subset containing only bass, drums, quitar, and piano, the four classes used by MSDM and MSG-LD, and Se to denote the complementary subset of the remaining stems. We follow official dataset splits provided by S1akh2100 for training, validation, and testing.
MUSDB18 18 consists of 150 real-world music recordings with four stems: drums, bass, other, and voca ls. We use all 100 tracks from the official training split for training and the 50-track test split for evaluation. The total datasct length is approximately 10 hours.
Mo comprises 240 songs (14 hours total) contributed by 47 artists across 12 genres. Each stem in the song is annotated with two-tier stem taxonomy. Each track decomposed into constituent sources and annotated using a two-level hicrarchical taxonomy of stem classes. We aggregate all second- tracks into their corresponding top-level classes. Among the 11 stem classes, we evaluate only the 7 unambiguous stems (e.g., bass, percussi io, vocals, etc.). For evaluation, we randomly sample 24 tracks (10%) the test set and the remaining tracks for training.
Data Construction. We train our model using randomly constructed 3-track tuples (mix, sub, src). A source stem randomly selected from available stems, and the remaining stems are aggregated into a submixture. We select non-silent segments from the source track whenever possible, allowing up to 10 random resampling attempts per instance. The same temporal offset is applied across all stems to ensure alignment. For generation evaluation, sample 300 random segments per test set. For source extraction, we sample between 150 and 700 non-silent segments per instrument class. All audio downsampled 16 kHz.
D.2 Baseline Implementations
All baseline metrics are recomputed on our test splits for fair comparison. The following models are used:
We use the official implementation and pretrained checkpoint Since MSDM operates at 22 kHz, we upsample our 16 kHz test audio for inference and downsample the output back to 16 kHz..
MSDM 34
MSG-LD 36
: As no checkpoint is publicly released, we reproduce the model by retraining it from the official codebase
HDemucs 20
We train a 16 kHz version of Hybrid Demucs using the demuca_ Larnuuc__lghttrrrr
implementation
We evaluate using the publicly available implementation and checkpoint provided by the authors Since AudioSep operates at a sampling rate of 32 kHz, we upsample all test audio from 16 kHz to 32 kHz before inference and subsequently downsample the separated outputs back to 16 kHH for evaluation consistency.
AudioSep 53
D.3 Model Architecture and Training
Autoencoder. We adopt the VAE-based architecture from Stable Audio (7).. with downsampling ratio of 2048, yiclding 7.8125 Hz latent resolution and 64 latent channels. We train the autoencoder all training subsets from Slakh2100, MUSDB18, and MoisesDB using 16 kHz mono audio for 600K steps with a batch size of 16.
Diffusion Model. In practice, the three latent representations are concatenated along the channel dimension, such that the input the diffusion model becomes Concat[z(m) z(*), 2(+) E 33Cx1. We DiT backbone 86 with 24 transformer blocks with 48 heads, and a projected latent dimension of 1536 (3 tracks x $12
https:///githuubccc///kr///
28

---

Table Source extraction performance of RePaint-based methods vs. MGE-LDM. Metrics are reported as Log-Mel LI distance J. T indicates the number of reverse timesteps, and U specifies the number of denoising operations per reverse step (i.e., U-1 intermediate resampling steps). The case U = corresponds to the canonical single-sample conditional score approximation [[]] as described Appendix [.2 MGE-LDM uses adaptive timestep conditioning without resampling.
S
S
Model
T
U
G
Avg.
Brs.
Org.
Pipe
Reed
S.Lead S.Pad
Avg.
MGE
250
1.68
2.71
2.69
2.16
2.31
3.43
2.16
1.84
2.33
3.07
2.44
2.31
1.93
2.48
(ours)
250
1
2.00
3.20
3.15
2.83
2.79
4.75
2.47
2.79
4.27
4.65
3.06
3.73
2.80
3.56
RePaint
2.75
2.91
2.68
2.55
4.33
2.37
6.67
3.84
4.27
2.82
3.64
2.58
3.81
50
5
1.80
2.43
2.83
2.55
2.40
3.89
2.32
2.57
3.35
3.81
2.64
3.56
2.42
3.07
-aaed
1.77
2.28
2.80
2.51
2.34
3.79
2.31
2.50
3.15
3.71
2.66
3.44
2.40
2.99
89
250
1.90
2.65
2.94
2.65
2.53
4.23
2.41
2.71
3.82
4.27
2.84
3.66
2.59
3.31
250
4
1.79
2.34
2.83
2.59
2.38
3.95
2.34
2.66
3.42
3.88
2.69
3.55
2.45
3.12
each). Timestep embeddings are prepended to the input vector of the transformer. CLAP embeddings for each track are processed by independent projection layers (without weight sharing) produce scale and shift parameters AdalN-style conditioning [104. These are applied group-wiss wise GroupNorm within each DIT layer to modulate the corresponding track-specific activations. Text embeddings are obtained from CLAP [56 using the checkpoint mus c_ audi oset_ _ase__epchh__ _1_p_________ _5__sc__________________________ 90 o0.11 14. pt via the a on-onn -- library Our
implementation builds upon the official stable -aud i c-tools 1. repository from Stability A and the training framework from fri endl y-stable a-aaaleeedddeee o-toola 6000 GPU (48 GB of memory).
[ All models were trained on single NVIDIA RTX
E
Ablation Study
This section presents ablation experiments designed to further analyze the key components of our framework. Unless otherwise specified, all models are trained on the combined Sa dataset. Each of our models is evaluated with the same configuration as in Section 5 using T = 250 denoising steps during sampling.
E.1
Comparison with Canonical Inpainting Methods
We assess the effectiveness of adaptive timestep conditioning strategy by comparing it against two prior approaches: the canonical one sample conditional score approximation (Appendix A.2 and the RePaint method 89 (Appendix A.3)) rabpot5 the results of the source extraction task.
In RePaint, U denotes the number denoising steps performed per reverse timestep: one denoising step followed by U-1 forward (resampling) steps. As a result, the total number of denoising steps hecomes T x U during the full inpainting process. Note that setting U = recovers the canonical single-sample estimator in Eq. 21
We observe that, for fixed number of denoising steps, using fewer timesteps T with more resampling cycles U generally improves performance, confirming observations the original RePaint paper. We hypothesize that repeated resampling helps stabilize conditional generation by mitigating the noise mismatch between observed and unobserved regions, particularly at high noise levels, where observed latents contain little informative content and single-samplc approximations of Eq. (2T) become highly unreliable. While this approach does not yield precise marginal score estimate, it heuristically improves inpainting quality through localized refinement.
Interestingly, also observe that RePaint configurations with larger total denoising steps such as T 250, U 2 and both SA and SB. This suggests that, for inpainting tasks, accurately modeling the conditional score at each timestep is more critical than simply increasing the number of reverse steps. As RePaint approximates the conditional score by marginalizing primarily through increased resampling (U), not longer trajectories (1))
250,
consistently underperform compared T 25, U = 10, across all stems
perturbed conditions via resampling, performance benefits are observed
Nevertheless, adaptive timestep model outperforms all RePaint variants across both datasets, with the sole exception of drums SA. By directly leamingg track-specific conditional scores during training. our method eliminates the need for inference-tima marginalization, resulting in lower variance and improved reconstruction quality.
29

---

Table Total generation performance across modeling variants. Metrics are reported as FAD 4. All models are trained on SA+SS Sp. The baseline model uses uniform (non-adaptive) timesteps across all tracks. MGE variants apply adaptive timestep conditioning and test the impact of normalizations and CFG dropout rates. Values in parentheses indicate generation conditioned on the text prompt "The sound of the bass, drums, guitar, and piano".
Testset
Model
SA
SFall
M,.
M
Non-adaptive
3.26 (2.00)
0.79
5.12
4.51
MGE (adaptive)
3.14 (2.24)
0.63
5.46
4.73
w/o GroupNorm
3.48 (2.44)
0.76
5.61
4.88
CFG dropout 0.5
3.12 (2.67)
0.58
5.43
4.82
Table 7: Source extraction performance under different architectural and CFG settings. Metrics are reported as Log-Mel LI distance All models are trained on S Sg- GN and LN denote GroupNorm and LayerNorm, respectively. p indicates the classifier-free guidance (CFG) dropout rate applied to each track's conditioning vector, and s refers to the CFG guidance scale.
SA
SB
Norm.
P
8
B
D
G
P
Avg.
Brs.
C.P.
Org.
Pipe
Reed
Str.
S.Lead
S.Pad
Avg.
GN
0.1
2.0
1.68
2.71
2.69
2.16
2.31
3.43
2.16
1.84
2.33
3.07
2.44
2.31
1.93
2.43
LN
0.1
2.0
1.67
4.22
2.65
2.15
2.67
3.42
2.35
1.97
2.40
3,45
2.51
2.32
2.03
2.55
GN
0.5
2.0
1.78
1.96
2.62
1.96
2.08
3.37
2.22
1.97
2.36
2.89
2.44
2.07
1.89
2.40
GN
0.1
1.0
1.67
2.79
2.70
2.05
2.30
3.24
2.23
1.85
2.25
2.88
2.28
2.27
1.87
2.35
GN
0.1
4.0
1.77
2.49
2.79
2.27
2.33
3.64
2.17
1.91
2.42
3.20
2.66
2.31
2.06
2.54
GN
0.1
8.0
1.93
2.49
2.93
2.41
2.44
4.35
2.28
2.01
2.57
3.47
3.27
2.42
2.30
2.83
A potential concern is whether optimizing for adaptive tinestep-conditional inference might degrade generation quality when using uniform timestep schedules across tracks. To assess this, we evaluate our adaptive timestep model with uniform timestep vector (r, 7). which corresponds the total generation task, and compare to baseline trained with non-adaative, shared timesteps.
As shown Table 6 comparing the non-adaptive uniform timestep baseline model and our model, both models achieve comparable FAD scores, indicating that timestep adaptation preserves generation performance under uniform scheduling while providing significant advantages for inpainting tasks.
E.2 Additional Design Ablations
We additionally investigate the impact of various modeling and training choices, including normalization strategies and classifier-fre guidance (CFG) dropout rates.
Table[iincluee results from models trained with LayerNorm instead of GroupNorm. following original DiT architecture, as well as a variant using a higher CFG dropout rate of p = 0.5. We observe that GroupNorm slightly outperforms LayerNorm across all test sets, supporting the use of track-wise normalization in our multi-track setting. Regarding CFG dropout, increasing the dropout rate improves unconditional generation performance, particularly
SA and Sg. However, when conditioned on the prompt (values in parentheses), the model trained with p = 0.5 performs worse, suggesting that overly aggressive dropout may impair semantic conditioning for total mixture generation.
We further examine how modeling and training design choices, such as normalization layers, classifier-free guidance (CFG) dropout probability. and CFG scale, affect extraction performance and report the results in Table When comparing normalization strategies, GroupNorm consistently matches or outperforms LayerNorm across most stems, demonstrating the advantage of modeling track-wise statistics in our multi architecture. This observation aligns with the trends seen in mixture generation results. For CFG dropout, a higher dropout probability (p = 0.5) leads to improved performance compared to the default p = 0.1, suggesting that stronger stochastic conditioning is beneficial during source extraction. While this differs from the trend observed in mixture generation (T). the discrepancy may be explained by the fact that, in extraction, non-target tracks are effectively treated as unconditioned. This makes overall performance more sensitive to model's ability to generalize in the presence of dropout. We also evaluate various CFG scales (1. 2, 4, 8). scale of yields the best performance overall, although scales 2 and 4 remain competitive. Performance degrades at scale 8, indicating that overly strong guidance can impair extraction quality.
30

---

Table 8: Preliminary results for iterative stem-wise generation. Metrics are reported as FAD 4. Evaluation is conducted using model trained exclusively on SA. Each column header indicates the generation order of stems (e.g., BDGP denotes bass drums- > gu (eiitta-)) piano).
SA
Total Gen.
BDGP
BDPG
DRGP
DBPG
DGBP
DPGB
GPBD
GPDB
PDGB
PGBD
MGE
0.47
0.60
0.66
0.78
0.75
0.70
0.77
0.61
0.66
0.63
0.60
Table 9: Partial generation results evaluated with COCOLA [00]] score 1. Higher values indicate stronger coherence between the given mixture and the generated stems, complementing sub-FAI by emphasizing musical consistency. "Ground truth" values correspond to the COCOLA scores computed between the given mixture and the ground-truth accompaniment from the dataset. Bold values indicate the best results in each column (excluding Ground truth), and underlined values denote the best results among models trained on the same SA set.
Model
Train See
$A
Sy
M,,
B
D
BD
BG
BP
DG
DP
GP
BDG
BDP
BGP
IXP
Brs.
C.P.
Org.
Pipe
Road
So.
S.leaal S.Pal
MSDM MSGLLD X T.
X
57.14
1188
57.76
10
45
96
57.6
$4.
$0.67
49.40
55.75
43:25
40.
52.82
47.89
58.15
50.58
58.55
$8.30
15
63
56.29
49
47:51
57.94
45.70
41
$4.19
46.08
3321
55.0
59.75
WATT
03:43
56.16
S545
51,90
46
13.46
5677
49.73
03.30
56.44
59.2
6233
61.11
63.1
3856
MGE (Ouns)
61.29 51.43
62.61
61.35
79
60.94
59.
51.2
49.39
60.21
46.06
42.81
56.73
47 64.32
65.21
65.46
62.31
64.31
62.33
64.52
61.10
55.00
46.7
57.29
59.54
$3.90
18
56.09
45.66
45
91
46
10
63.15
51.64
45.55
M323
54.24
56
70
00 18
56.50
60.6
62.00
61:00
5673
49.65
50.58
61.06
46.92
57,
88
57.81
50.60
49.93
58.32
46.33
45.39
54.02
46 86
63.94
64.47
63.77
59
62.39
66.00
58.52
Gronn muth
6011 53.55
56,33
57.16
30-42
59.94
56.99
53.11
52.29
57.90
47.06
44.32
56,07
30.18
64.58
64.34
6485
63.38
64.75
63:20
66.29
62.16
E.3 Iterative Generation Variants
Table D presents preliminary results for the iterative generation procedure described in Appendix C applied to model trained on A.. The task involves sequentially generating the four canonical stems (bass, drums guitar, and piano) in various orders.
Across all tested permutations, iterative generation produced higher FAD scores compared to one-stage mixture generation, indicating a degradation in perceptual quality. Nonetheless, iterative generation may offer utility in settings that require fine-grained, source-specific control.
An interesting trend observed: generation sequences that began with drumsconsistently resulted in poorer performance relative to other orderings. This suggests that the model may be more effective at first establishing harmonic or melodic content before aligning rhythmic elements. While this observation is speculative, it highlights potential inductive bias in the model that warrants further investigation, particularly in scenarios beyond the four- nuu-inntrummnt configuration.
F Extended Evaluation with Alternative Metries
To complement the quantitative results presented Section we provide additional evaluations using alternative metrics that better capture musical coherence and perceptual quality. While sub-FAD is widely adopted in
accompaniment generation research [34] 36 63 76 81). it is considered suboptimal for evaluating musical coherence, since it primarily measures glohal timbral similarity rather than the harmonic and rhythmic interplay hetween stems [T5). To address this limitation, Ciranni et al. [105] introduced the COCOLA metrie, which quantifies harmonic and rhythmic coherence through contrastive learning of musical audio representations.
We report the COCOLA results Table to complement our main findings. The metric computed between the given mixture and the generated stems, and we additionally report the "Ground truth" values computed between the same mixture and the corresponding reference accompaniment from the dataset. Among models trained on Sa. our Ti variant outperforms the baselines most instrument categories, except for guitar generation. Larger-scale models (T2-Ta) show similarly strong or even superior coherence, in several cases approaching or surpassing ground-truth levels. Since COCOLA emphasizes musical coherence rather than timbral accuracy with respect to reference stems, it provides mcaningful complement to sub-FAD, offering more balanced view of partial generation quality.
Because the VGGish-based FAD has been shown to correlate poorly with human perception in recent studies [06 we further report FADCLAP-MA and sub-FADeLAP.MA, computed using CLAP embeddings from the music_audio www.uusssstuuueeeee.ueuueeuuesss cosst__ppch____________e_______e 1eaeh_____s_____________________ 20... 14. pt checkpoint, which achieves the highest correlation with subjective ratings according to Grötschla et al. [106). 100 total generation results evaluated with FAD where Ti achieves the best overall performance among models trained on SA. For the combined SA Sp reference set. T2 yields the lowest FAD scores, while Ts performs best on M., and M.. aligning with
31

---

Table 10: Total generation results evaluated with FADCLAP-MA
Train Set
Test Set
Model
S
SB
M.
M.
SA
Srall
M
M.
MSDM
X
X
.329
.290
.767
.744
MSG-LD
/
x
X
.191
.312
.658
.697
Ti
/
x
x
110 (.450)
.285
.637
.677
MGE
T2
/
/
x
.368 (.359)
.099
.578
.621
(ours)
73
x
/
.731 (.669)
.632
.222
.165
71
/
/
.563 (.712)
.426
.235
.186
Table 11: Partial generation results evaluated with Sub-FADeLAPSAR 4.
Train Set
SASgM, M.
D
P
BD
BG
BP
DG
DP
GP
BDG
BDP
DGP
DGP Brs. C.P. Org. Pipe Reed Str. S.Load S.Pad
MSDM MSG-LD
x
/
x
x
x
071
070
076
077
130
135
117
121
157
185
188
2211
.190
.40
170
133
134
175
200
164
218
327
3335"
(ours)
Ta
Ts
x
.178
202
356
253
307
402
340
368
320
477
.45
469
517
.632
087
.62
144
327
090
330
.079
.660
Ti
.170
187
367
251
253
408
308
350
269
455
450
436
.51
486
312
002
107
055
.340
208
085
.663
their respective training domains. The sub-FADeLAP-M results for partial-generation, presented in Table exhibit trends consistent with Table 2 with MSG-LD achieving the best scores on most stem combinations. Together, these additional metrics provide more comprehensive and reliable assessment of both perceptual quality and cross coss-domain generalization in partial and total music generation.
G
Future Work
Future extensions of MGE-LDN include scaling to higher-resolutee formats such as 44.1 kHz stereo audio, enabling richer timbral detail and spatial fidelity. In particular, this can be achieved by leveraging high-que uehhquuaity latent representations recently developed for the music domain [109 [10). To reduce the modality gap in text- conditioned extraction, fine-tuning on curated audio-text datasets like MusicCaps (7) promising direction. Given minimal reliance on precise stem houndaries, MGF-IDMM is naturally suited for incorporating weakly which may expand training diversity.
or noisily labeled multi-track data 39 11
Another promising avenue is to pre-train MGE-LDM on large-scale mixture-only corpora such as MTG-Jamendo [122 or the Free Music Archive [I3 to learn general audio priors for mixture tracks, followed by tie-ttunng multi-track datasets for source-aware generation. This two-stage training strategy expected enhance generative quality and improve generalization.
We also plan extend MGE-LDM to text-based music editing tasks, drawing inspiration from recent instruction- guided frameworks such as AUDIT 73), latent inpainting capabilities and InstructME, (74), and Marruct-MusiGGenn (64), Leveraging MGE-LDM's generation, this extension could enable user-directed operations such as instrument replacement and style transformation via natural language prompts, building upon the model's unified training scheme and class-agnostic design.
H Spectrogram Examples of Generated Samples
We present Mel-spectrogram visualizations of generated audio samples across the three primary tasks: total generation, partial generation (imputation), and source extraction. All examples produced by MGE-LDM trained on combined Slakh2100 (S ASB). MUSDB18 (M.), and MoisesDB (M.) datasets.
We note that the model is capable of generating vocals present in the training data. Although MGE-LDM does not currently support fine- omn-ggrined control over vocal generation, this points to promising direction for future work, such as incorporating explicit vocal prompts segment-leve level control for more expressive and structured multi-track modeling.
unconditional setting, as vocal sstems
32

---

T sub
4096
2048
1024
512
0
4096
2048
1024
512
0
4096
2048
1024
512
4096
2048
1024
0
4096
2048
1024
512
0
4096
2048
1024
512
1.5
4.5
7.5
9
1.5
3
4.5
7.5
1.5
4.5
7.5
9
Time
Time
Time
Figure 2: Total generation examples. Each sample displays Mel-spectrograms of the mixture, submixture, and source tracks, all generated simultaneously by MGE-LDM. The mixture track is used to evaluate the total generation output.
33

---

Given
2
apch
Est. A.Y
4096
Drums
2048
1024
512
0
4096
Bass
2048
1024
512
0
4096
Guitar
2048
1024
512
4096
Plano
2048
1024
512
0
4096
Brass
2048
1024
512
0
4096
Organ
2048
1024
512
0
Strings
4096
2048
1024
512
1.5
3
4.5
7.5
1.5
3
4.5
7.5
1.5
3
4.5
7.5
Time
Time
Time
Figure Source imputation examples. Each row illustrates source inpainting results by MGE-LDM, conditioned on the text prompt "The sound of the [label]". The middle column shows the provided context mixture (submix), the rightmost column the generated source, and the leftmost column is the recombined mixture of the submix and generated source. While some stems are imputed accurately, others fail due data imbalance during training.
34

---

Given 2
Est. &
G.T.
Drums
4096
2048
1024
512
0
4096
Bass
2048
1024
512
0
Guitar
4096
2048
1024
512
Piano
4096
2048
1024
512
0
Brass
4096
2048
1024
512
0
Organ
4096
2048
1024
512
Strings
4096
2048
1024
512
1.5
4.5
7.5
0
1.5
3
4.5
6
7.5
9
1.5 3
4.5
6
7.5
9
Time
Time
Time
Figure 4: Source extraction examples. Source extraction results produced by MGE-LDM. condi- tioned on the text query "The sound of the [label)" ". The leftmost column shows the input mixture, middle column is the extracted source predicted by the model, and the rightmost column ground-truth source. We observe that extraction quality may degrade for underrepresented classes such as strings, and in some cases, the model hallucinates unrelated instruments or incorrect timbres.
35

---

I Ethics Statement
This work introduces a class-agnostic generative framework multi-track music modeling. trained exclusively on publicly available datasets (Slakh2100, MUSDB18, and MoisesDB). While the model enables flexible music generation, source imputation, and source extraction, it also carries potential risks, such unauthorized manipulation. misuse in derivative content, generation of audio resembling copyrighted material. To mitigate these concerns, we commit to releasing the model and code under license with clear usage guidelines. emphasizing responsible research and ethical creative applications.
36


Reviews:
[{"review_id": "AcvuKlFwwT", "reviewer": "Reviewer_ShCW", "summary": "The paper introduces MGE-LDM, a latent diffusion model that simultaneously addresses music generation, source imputation, and text-driven source separation without relying on predefined instrument classes. Unlike prior models constrained to fixed stem definitions, MGE-LDM learns a joint distribution over mixtures, submixtures, and individual sources in a compressed latent space. By framing both source extraction and stem completion as conditional inpainting tasks, the model can handle arbitrary instruments and enables language-guided manipulation using CLAP embeddings. It further introduces track-wise adaptive timestep conditioning during training to enhance inpainting fidelity, allowing the model to distinguish between observed and missing components. Empirical evaluation demonstrates competitive performance across synthetic and real-world benchmarks in total generation, partial generation, and source extraction; an accompanying demo website confirms the competitiveness of the approach.", "questions": "- Can you quantify or mitigate the limitations of CLAP-based conditioning? CLAP is conceptually appealing for language-guided extraction, but its granularity and semantic specificity are limited. Do you observe consistent issues with ambiguous or overlapping prompts (e.g., \"strings\" vs. \"synth pad\")? Have you explored alternative or complementary embeddings, or prompt disambiguation strategies?\n\n- What are the primary failure modes, and how often do they occur?", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "This paper presents a timely contribution to the field of music generation and source separation by proposing a latent diffusion model that, similarly to recent methods, unifies multiple tasks in a coherent framework. The technical foundation is solid, and empirically the model is clearly capable of high-fidelity generation and source manipulation, with performance that is competitive with or better than strong baselines. One of the key strengths lies in its use of submixtures as an intermediate representation, which elegantly works around the need to know the exact number or type of stems present in a mixture. This design choice improves scalability and robustness, especially when working with heterogeneous real-world datasets where source labels are often noisy or aggregated. The use of CLAP embeddings for conditioning is another good idea. While CLAP is not flawless, its integration here is conceptually compelling. Finally, I appreciated the adaptation of TD-Paint's per-region timestep conditioning to the audio domain, which is both technically interesting and practically useful in improving latent inpainting fidelity.\n\nAn aspect I find worthy of praise is the possibility to move beyond the common and somewhat restrictive four-stem modeling (bass, drums, guitar, piano/other) seen in many prior works. By demonstrating the model's ability to handle a wider range of instruments including synths, brass, pipe, and others, it takes an important step toward more realistic and musically rich applications, even if the results for these less common stems are not yet exceptional. This direction is significant because it reflects the diversity of real-world music and shows a commitment to broader applicability.\n\nThe paper is generally well-organized, though some sections (e.g. the method) can be dense and may benefit from additional high-level intuition or summarization. Nonetheless, the figures and detailed mathematical exposition help make the approach reproducible. The joint modeling of mixture, submixture, and source in the latent space, combined with conditional inpainting, represents a meaningful departure from previous work that tends to rely on fixed-class assumptions or waveform-domain additivity.\n\nKey weaknesses include limitations in generalization and fidelity for non-canonical instruments, but these are also clearly outlined by the authors. This suggests the model struggles with finer details or less frequent classes, and points to room for improvement in generalization to rare or underrepresented sources. But I don't see this as a critical drawback. Further, as already mentioned, while the method is mathematically rigorous, its clarity suffers from heavy notation and long explanations without always grounding the reader in intuitive takeaways. The presentation could be tightened to emphasize conceptual contributions before diving into formalism.\n\nOverall, I believe the paper meets the bar for NeurIPS, given its improvements upon prior related work published in such venues.", "was_revised": true, "final_justification": "The authors clarified my concerns and included additional results that I explicitly asked for. Having also gone through the other reviews + rebuttal, I can see that several other experiments address all the other major concerns raised by the other reviewers. I think the authors did an excellent job for this rebuttal. Therefore, I confirm my recommendation to accept the paper for presentation at NeurIPS.", "rebuttal": "We thank Reviewer ShCW for the thoughtful and constructive feedback and are delighted that you recognize our work as a timely contribution to the field. Below, we address each of your points:\n\n> *\"The paper is generally well-organized, though some sections (e.g. the method) can be dense and may benefit from additional high-level intuition or summarization. /  while the method is mathematically rigorous, its clarity suffers from heavy notation and long explanations without always grounding the reader in intuitive takeaways. The presentation could be tightened to emphasize conceptual contributions before diving into formalism.\"*\n\n**Authors' response:**\nWe agree that the Method section can feel dense. To make it more accessible, we will revise the method section to include a conscise, high-level overview outlining the key ideas in plain language before describing mathematical details.\n\n> *\"Q: Can you quantify or mitigate the limitations of CLAP-based conditioning? CLAP is conceptually appealing for language-guided extraction, but its granularity and semantic specificity are limited. Do you observe consistent issues with ambiguous or overlapping prompts (e.g., \"strings\" vs. \"synth pad\")? Have you explored alternative or complementary embeddings, or prompt disambiguation strategies?\"*\n\n**Authors' response:**\nYou’re correct that CLAP’s semantic granularity can lead to ambiguity. \nTo illustrate, we ran a small experiment on mixtures containing strings or synth‑pad and measured Log‑Mel L1 distances between the true isolated stems and the model’s output under different text prompts:\n\n| Cond. inst. \\ Ref inst. | strings | synth_pad |\n|:------------------------|--------:|----------:|\n| strings                 |    2.57 |      2.12 |\n| synth_pad               |    2.25 |      2.06 |\n\nIn an ideal setting, the lowest errors would lie on the diagonal, but here we see that asking for “synth pad” on a mixture with strings yields a lower L1 distance (2.25) than asking for “strings” (2.57). \nListening confirms that the model often reproduces the correct pitch contour but with a synth‑like timbre, which is a hallucinatory behavior we have noted in our Limitations section. This highlights an important direction for future work in improving prompt‑based source separation (which we also discuss with the reviewer W8on).\n\nWe also intend to explore more recent embedding models such as M2D‑CLAP [1] to enhance semantic precision, investigate negative prompting strategies (e.g. “strings, not synth pad”) to explicitly suppress unwanted classes.\nWe also can try leverage CLAP’s audio‑branch embeddings from short reference clips to anchor the model and reduce ambiguity. Although we have not yet quantified these ideas, we believe they could meaningfully improve disambiguation.\n\n\n> *\"Q: What are the primary failure modes, and how often do they occur?\"*\n\n**Authors' response:**\nEmpirically, we observe three primary failure modes.\nFirst, the model sometimes fails to extract a source and outputs near‑silence instead of the intended stem. \nSecond, with ambiguous prompts the model often hallucinates sounds that aren’t present in the mixture, as discussed above. This issue seldom arises for clear prompts like “the sound of vocals,” where the model correctly outputs silence when vocals are absent.\nThird, the model struggles to capture the fine-grained detail of isolated single notes on piano and guitar: while chords and percussive strokes are generally well reproduced, individual notes often vanish or become blurred. \nWe believe these issues stem from a relative scarcity of fine-detail, single-source examples in our training data. \nTo address them, we plan to augment our dataset with additional isolated‑note samples and experiment with increased loss weighting on fine‑detail reconstruction.\n\n\n**Reference**\n\n[1] Niizumi, D., et al. “M2D‑CLAP: Masked modeling duo meets CLAP for learning general‑purpose audio‑language representations.” Interspeech 2024."}, {"review_id": "7Cg0v0VVnm", "reviewer": "Reviewer_W8on", "summary": "This paper follows the line of research set up by MSDM (Mariani et al. 2024), where a diffusion model is trained on stem-separated musical data (of fixed instrument classes) and is able to perform simultaneously full track generation, accompaniment generation given conditioning tracks and source separation. MSDM was generalised in GMSDI (Postolache et al. 2024), where text conditioning is introduced and by MSG-LD (Karchkhadze et al. 2025), which sets the diffusion model in a latent space, improving quality. MSG-LD though keeps the number of instrument classes fixed like MSDM and cannot generate arbitrary classes via text conditioning. Doing all tasks in the latent domain is difficult especially because of source separation, which requires linearity in the likelihood function (that's why GMSDI works in the waveform domain only). In this paper, the authors propose MGE-LDM, a method that aims to improve on both limitations, defining a latent diffusion model (via Stable Audio Open) that can be text conditioned and can solve all three tasks simultaneously. They train over triples composed by $(z^{(m)}, z^{(u)}, z^{(s)})$, where $z^{(m)}$ is a latent mixture, $z^{(u)}$ is a latent sub-mixture (the mixture minus a source) and $z^{(s)}$ is a source. At the same time they condition on CLAP embeddings of the form $(c^{(m)}, c^{(u)}, c^{(s)})$. The three tasks at inference time: \n- Total generation: sample the triple and decode $z^{(m)}$.\n- Partial generation: inpaint $(z^{(m)}), z^{(s)})$ given $z^{(u)}$ and decode $z^{(s)}$.\n- Source separation: inpaint $(z^{(u)}), z^{(s)})$ given $z^{(m)}$ and decode $z^{(s)}$.\n\nThe authors also introduce track-individual noising in order to improve the inpainting mechanism. The authors benchmark their model against MSDM and MSG-LD reaching good results (especially on source separation), with improved generalisability of the model.\n\n**References**\n- Mariani, Giorgio, et al. \"Multi-Source Diffusion Models for Simultaneous Music Generation and Separation.\" ICLR 2024.\n- Postolache, Emilian, et al. \"Generalized multi-source inference for text conditioned music diffusion models.\" ICASSP 2024.\n- Karchkhadze, Tornike, Mohammad Rasool Izadi, and Shlomo Dubnov. \"Simultaneous music separation and generation using multi-track latent diffusion models.\" ICASSP 2025.", "questions": "- Line 162: Do we have a different $p$ on each $c^{(k)}$? \n- Line 176: I would say: \"whose score we approximate with $f_\\theta$\". \n- Line 270: \"Results\" instead of \"Result\"\n- Line 293: This makes sense since FAD cannot capture the distribution shift.\n- Line 297: I think the authors could improve on these results with negative conditioning of unwanted tracks (see GMSDI, Fig. 2).\n- Line 630: Before giving the expression of  the velocity, the expression of $z_t$ in function of $\\epsilon$ and $z_0$ should be given.\n- Line 631: The $0$ subscript should be not bold.", "limitations": "yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n- It is interesting to see the multi-track music generation research direction being improved, especially given the presented model it is general enough to let one train over multiple stem-separated datasets at the same time. This is a great departure from models with fixed classes (MSDM, MSG-LD).\n\n- Quantiative evaluation is thorough, comparing to MSDM and MSG-LD in a plethora of tasks. Would have been cool to compare to GMSDI given the commonality of text conditioning, but such paper did not release code / checkpoints so I agree it is a thorough task. (still one key metric is missing for partial generation, see Weaknesses #1)\n\n- The track-aware inpainting method is very interesting. Also seeing the comparison with RePaint in Table 5 of Appendix D and seeing the presented inpainting method as the best performing, gives a strong indicative that this should be the default mechanism for multi-track generation (at least in the absence of a constraining latent likelihood, see Weaknesses #2).\n\n- Cool to see a discussion about iterative generation in Appendix C. Having the individual separated tracks of a mixed track was one of the rationales of  MSDM so it is interesting to see one is not losing that feature (it is actually generalised).\n\n- Overall the paper is great to read and a good contribution in the (controllable) music generation domain. \n\n**Weaknesses**\n\n\n- As the author acknowledge in the limitations section, we still depend on labels in a loose way when training.\n\n- A limitation of the paper is that partial generation metrics are computed on sub-FAD, a metric which, while having served well as an initial foray in evaluating accompaniments models (e..g, in SingSong, in MSDM, etc.), there is empirical evidence (Ciranni et al. 2025) that it is not optimal for such a task, favouring randomly mixed stems over natural combinations. The authors should also compute metrics such as COCOLA score (Ciranni et al. 2025), which are better suited for the task. Also in that paper one finds a benchmark on a conditional model based on Stable Audio Open, released publicly, and trained on MusDB and MoisesDB, which shares the diffusion architecture of the presented paper: it would be interesting to see how the presented joint model compares to the conditional one.\n\n- While the authors acknowledge that the nonlinearity of the latent space makes it difficult to constrain the sources in a mixture and they opt out for their approach, I believe that the three entries of a triple are not necessarily constrained to sum well in the data (waveform) domain. This could have a negative impact in the performance of the model. Thus, finding the best way to solve compositional music generation in the latent domain it still remains an open problem, and maybe a more constrained approach would be given, for example, by learning latent-domain likelihood functions like in (Postolache et al. 2023). The authors could discuss this in the concluding remarks of the paper or in limitations.\n\n**References**\n\n- Ciranni, Ruben, et al. \"Cocola: Coherence-oriented contrastive learning of musical audio representations.\" ICASSP 2025.\n- Postolache, Emilian, et al. \"Latent autoregressive source separation.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 37. No. 8. 2023.", "was_revised": true, "final_justification": "I keep my full accept on this paper! This is a great improvement in the area of compositional audio generation, finally having a well-defined stem-level generative model in a latent domain. I am glad the authors introduced evaluations on COCOLA score, showcasing improved stem coherence with respect to previous baselines.", "rebuttal": "We sincerely thank Reviewer W8on for the through and insightful feedback.\nWe appreciate your recognition of our contritbutions to multi-track music generation, and we address your comments below.\n\n\n> *\"A limitation of the paper is that partial generation metrics are computed on sub-FAD, a metric which, while having served well as an initial foray in evaluating accompaniments models (e..g, in SingSong, in MSDM, etc.), there is empirical evidence (Ciranni et al. 2025) that it is not optimal for such a task, favouring randomly mixed stems over natural combinations. The authors should also compute metrics such as COCOLA score (Ciranni et al. 2025), which are better suited for the task. Also in that paper one finds a benchmark on a conditional model based on Stable Audio Open, released publicly, and trained on MusDB and MoisesDB, which shares the diffusion architecture of the presented paper: it would be interesting to see how the presented joint model compares to the conditional one.\"*\n\n**Authors' Response**: \nYou're correct that sub-FAD alone can bias toward random mixtures rather than musical coherence.\nIn response to your suggestion, we computed the COCOLA score [1] on exactly the same partial-generation outputs reported in Table2 of the main paper.\nThe results are shown below, with the highest COCOLA score per column bolded, and among models trained on the $\\mathbf{S}_A$ set (MSDM, MSG-LD, or our T1), the best result underlined.\n\n| Model | Trainset        |    **B** |  **D** |     **G** |     **P** |    **BD** |    **BG** | **BP** |    **DG** |    **DP** |    **GP** |   **BDG** |   **BDP** |   **BGP** |   **DGP** |\n| ---------------- | -------- | -----------: | --------: | --------: | --------: | --------: | --------: | --------: | --------: | --------: | --------: | --------: | --------: | --------: | --------: |\n| MSDM             | $\\textbf{S}_A$         | $57.74$ |     $51.87$ |     $57.06$ |     $56.10$ |     $45.96$ |     $57.64$ |     $54.48$ |     $50.67$ |     $49.40$ |     $55.75$ |     $43.25$ |     $40.46$ |     $52.82$ |     $47.09$ |\n| MSG‑LD           | $\\textbf{S}_A$         |        $58.18$ |     $50.58$ |     $\\underline{58.55}$ |     $58.30$ |     $46.12$ |     $57.63$ |     $56.29$ |     $49.11$ |     $47.51$ |     $57.94$ |     $43.70$ |     $41.33$ |     $54.19$ |     $44.98$ |\n| $\\mathcal{T}_1$   | $\\textbf{S}_A$         |        $\\underline{60.93}$ | $\\underline{\\mathbf{53.71}}$ |     $58.09$ |     $\\underline{59.78}$ |     $\\underline{\\mathbf{49.77}}$ |     $\\underline{60.43}$ | $\\underline{\\mathbf{59.16}}$ | $\\underline{\\mathbf{52.95}}$ |     $\\underline{\\mathbf{51.90}}$ |     $\\underline{60.03}$ |     $\\underline{46.28}$ |     $\\underline{43.46}$ | $\\underline{\\mathbf{56.77}}$ |     $\\underline{\\mathbf{49.73}}$ |\n|\n| $\\mathcal{T}_2$   | $\\textbf{S}_A$+$\\textbf{S}_B$      |    $\\mathbf{61.29}$ |     $51.43$ | $\\mathbf{62.61}$ | $\\mathbf{61.35}$ |     $48.79$ | $\\mathbf{60.94}$ |     $59.12$ |     $50.82$ |     $49.39$ | $\\mathbf{60.21}$ |     $46.06$ |     $42.81$ |     $56.73$ |     $47.01$ |\n| $\\mathcal{T}_3$        | $\\textbf{M}_u$+$\\textbf{M}_o$    |        $55.40$ |     $44.76$ |     $57.29$ |     $59.34$ |     $43.90$ |     $55.18$ |     $56.09$ |     $45.68$ |     $45.42$ |     $55.91$ |     $44.10$ |     $43.35$ |     $51.44$ |     $43.53$ |\n| $\\mathcal{T}_4$       | $\\textbf{S}_A$+$\\textbf{S}_B$ +$\\textbf{M}_u$+$\\textbf{M}_o$|        $56.73$ |     $49.65$ |     $60.58$ |     $61.06$ |     $46.92$ |     $57.88$ |     $57.82$ |     $50.66$ |     $49.93$ |     $58.32$ |     $\\mathbf{46.53}$ |     $\\mathbf{45.39}$ |     54.02 |     46.86 |\n|\n| **Ground Truth** | –          |        $60.11$ |     $53.55$ |     $56.33$ |     $57.16$ | $50.42$ |     $59.74$ |     $58.59$ | $53.11$ | $52.29$ |     57.80 | $47.06$ | $44.32$ |     $56.07$ | $50.18$ |\n\n---\n---\n---\n---\n\n| Model            | Trainset                                           | **Brs.** | **C.P.** | **Org.** |  **Pipe** |  **Reed** | **Str.** | **S.Lead** | **S.Pad** |\n| :--------------- | :------------------------------------------------- | --------: | ----------: | --------: | --------: | --------: | ----------: | ---------------: | -------------: | \n| T1    | $\\mathbf{S}_A$  | $63.20$ |   $56.44$ | $59.52$ | $60.59$ | $62.35$ | $61.11$ |   $63.74$ | $58.66$ |\n| T2  |$\\textbf{S}_A$+$\\textbf{S}_B$ | $64.32$ |   $\\textbf{65.21}$ | $\\textbf{65.46}$ | $62.31$ | $\\textbf{64.31}$ | $62.33$ |   $64.52$ |     $\\textbf{61.10}$ |\n| T3 | $\\mathbf{M}_u$+$\\mathbf{M}_o$ | $60.23$ |   $54.74$ | $56.70$ | $60.18$ | $56.50$ | $60.66$ |   $62.03$ |  $61.00$ |\n| T4  | $\\textbf{S}_A$+$\\textbf{S}_B$+$\\mathbf{M}_u$+$\\mathbf{M}_o$ | $\\textbf{65.94}$ |   $64.10$ | $64.47$ | $\\textbf{63.77}$ | $59.49$ |  $\\textbf{62.39}$ |   $\\textbf{65.02}$ |     $58.52$ |\n|\n| **Ground Truth** | –  | $64.58$ |   $64.34$ | $64.85$ | $63.38$ | $64.75$ |  $63.20$ |   $65.29$ |     $62.16$ |\n\n---\n\nThe results show that our MGE-LDM variants achieve the highest coherence across nearly all stem combination compared to the baselines (MSDM and MSG-LD), often matching or even approaching the ground-truth levels.\n\nBecause COCOLA emphasizes musical coherence rather than timbral accuracy to reference stems, we believe it complements sub-FAD and provides a more balanced view of partial generation quality. \nWe will integrate these COCOLA results either alongside sub-FAD in the main manuscript or in the Appendix, as you suggested.\n\n(For reference, the COCOLA score (Harmonic) of stable audio + control net reported by the COCOLA paper [1] is 57.34, though it is not directly comparable to our results since it uses a different evaluation set.)\n\nOnce again, thank you for this valuable suggestion.\n\n> *\"While the authors acknowledge that the nonlinearity of the latent space makes it difficult to constrain the sources in a mixture and they opt out for their approach, I believe that the three entries of a triple are not necessarily constrained to sum well in the data (waveform) domain. \nThis could have a negative impact in the performance of the model. \nThus, finding the best way to solve compositional music generation in the latent domain it still remains an open problem, and maybe a more constrained approach would be given, for example, by learning latent-domain likelihood functions like in (Postolache et al. 2023). \nThe authors could discuss this in the concluding remarks of the paper or in limitations.\"*\n\n**Authors' Response**:\nYou rightly point out that, although we train on triplets satisfying $mix = submix + source$ in waveform space, our latent-diffusion model does not enforce an explicit additivity constraint for generated triplets.\nThis issue is critical, as we believe it directly related to the “hallucination” phenomenon, in which the model extracts sources that aren’t actually present in the mixture.\nPostolache et al. [2] offer an elegant solution by enforcing additivity in a discrete VQ-VAE latent space:\nthey estimate the joint likelihood of two sources by counting exact codebook occurences (bin-counting), effectively modeling $p(z_{mix}|z_{src_1}, z_{src_2})$, where $z$ are quantized latent codes. \nThis bin-counting approach serves as a powerful regularizer to enforce a summation constraint in the latent domain. \nUnfortunately, our current pipeline relies on a continuous latent space, which precludes direct application of discrete bin counts.\n\nNonetheless, we believe adapting this latent-domain likelihood approach to our model would be highly promising.\nWe could either extend it to a continuous latent space by designing appropriate regularizers, or convert our encoder to a discrete representation, such as using VQ-VAE based encoder and discrete diffusion (or MaskGIT) for generation to enable bin counting in the latent space.\n\nWe thank the reviewer for this valuable suggestion, and we will discuss these directions in the Limitations and Future Work sections as promising approaches to reduce hallucinations and improve compositional consistency.\n\nWe also appreciate your helpful line-by-line suggestions and will update the manuscript accordingly. \nIn particular, your idea of negative conditioning to supress unwanted tracks is very valuable and will be explored in our future work.\n\n**References**:\n\n[1] Ciranni, Ruben, et al. \"Cocola: Coherence-oriented contrastive learning of musical audio representations.\" ICASSP 2025.\n\n[2] Postolache, Emilian, et al. \"Latent autoregressive source separation.\" AAAI 2023."}, {"review_id": "eHkiGPOmit", "reviewer": "Reviewer_cGKq", "summary": "The authors introduce MGE-LDM, a unified latent diffusion framework for simultaneous music generation, source imputation, and text-driven source separation. Unlike prior methods that depend on fixed instrument classes or assume additive mixing in the waveform domain, MGE-LDM operates in a shared latent space and models the joint distribution over mixtures, submixtures, and individual sources. By formulating both stem completion and source extraction as conditional inpainting tasks and leveraging CLAP embeddings for text conditioning, the model enables class-agnostic manipulation of arbitrary instruments.", "questions": "- Why are the authors only training on 16 kHz? As far as I know the Stable Audio pipeline supports up to 44.1 kHz stereo.\n- Why are the authors using an outdated version of Demucs? The Hybrid Demucs is significantly better compared to the original.\n- Why does the demo page only have comparisons in the Slakh section but not for Moises/MusDB? Especially for the source extraction, which can be easily compared to existing methods.", "limitations": "The authors have a section on limitations in the appendix. I suggest they move this to the main text.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "While none of the individual components of the proposed architecture are novel, the combination of diffusing stems in the latent space based on clap conditioning signals (and therefore not requiring explicit stem labels) seems novel.\n\n\nUnfortunately, I’m not convinced by the results. The empirical study using VGGish FAD is poor, as multiple previous works have shown that VGGish is a suboptimal embedding model for music [1-3]. Additionally, while I appreciate the authors having trained various models (T1-4), I don’t see how it makes sense to compare these against a baseline only trained on a single subset of the data. While the results on synthetic data (slakh) seem decent, the model underperforms when listening to real music (moises, musdb). The biggest issue I have is the missing human evaluation, as the FAD numbers don’t show the full picture.\n\n\n[1] Gui, Azalea, et al. \"Adapting frechet audio distance for generative music evaluation.\" ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2024.\n\n\n[2] Grötschla, Florian, et al. \"Benchmarking Music Generation Models and Metrics via Human Preference Studies.\" ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2025.\n\n[3] Huang, Yichen, et al. \"Aligning Text-to-Music Evaluation with Human Preferences.\" arXiv preprint arXiv:2503.16669 (2025).", "was_revised": true, "final_justification": "Even though paper does not show substantial improvements over prior work, the approach contains novelty and should be interesting to the audience of NeurIPS.", "rebuttal": "We sincerely thank Reviewer cGKq for your careful reading and constructive comments. Below, we address your main concerns:\n\n> *\"Unfortunately, I’m not convinced by the results. The empirical study using VGGish FAD is poor, as multiple previous works have shown that VGGish is a suboptimal embedding model for music [1-3].\"*\n\n**Authors' Response**:\nWe appreciate your feedback and agree that VGGish-based FAD is not the ideal metric for music generation. \nWe initially adopted it for its widespread use as in SingSong [4] and MSDM [5], but we acknowledge its limitations.\nWe additionally calculated the FAD using CLAP-MA embeddings (LAION-CLAP checkpoint `music_audioset_epoch_15_esc_90.14`), which Grötschla et al. [2] have shown best correlation with human perception.\n\nBelow, Table A reports the total-generation results using FAD-CLAP-MA.\n\n---\n\n*Table A: Total generation result with FAD-CLAP-MA $\\downarrow$*\n|Model | Trainset | $S_A$ | $S_{full}$ | $M_u$ | $M_o$ |\n| ---------- | -----| ------- | -------- | -------- | -------- | \nMSDM | $S_A$ | 0.329 | 0.290 | 0.767 | 0.744 |\nMSG-LD | $S_A$ | 0.191 | 0.312 | 0.658 | 0.697 |\n$T_1$ | $S_A$ | **0.110** (0.450) | 0.285 | 0.637 | 0.627 |\n$T_2$ | $S_A$+$S_B$ | 0.368 (0.359) | **0.099** | 0.578 | 0.621 |\n$T_3$ | $M_u$+$M_o$ | 0.731 (0.669) | 0.632 | **0.222** | **0.165** |\n$T_4$ | $S_A$+$S_B$+$M_u$+$M_o$ | 0.563 (0.712) | 0.426 | 0.235 | 0.186 |\n\n---\n\nIt shows a similar relative ranking to Table 1 of the main paper, with lower absolute values.\nOne notable difference is that our $T_1$ model achieves the lowest FAD-CLAP-MA score compared to the baselines, which is a clear contrast to the VGGish-based results in Table 1 of the main paper.\n\nWe also computed the FAD-CLAP-MA for partial generation task, where the results are shown below in Table B.\n\n---\n\n*Table B: Partial generation result with sub-FAD-CLAP-MA $\\downarrow$*\n\n|Model | Trainset | B | D | G | P | BD | BG | BP | DG | DP | GP | BDG | BDP | BGP | DGP |\n| ------- | -----| ------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- |\n| MSDM | $S_A$ | 0.103 | 0.100 | **0.066** | 0.096 | 0.227 | 0.189 | 0.252 | 0.135 | 0.191 | 0.302 | 0.275 | 0.357 | 0.555 | 0.403 |\n| MSG-LD | $S_A$ | **0.071** | **0.070** | 0.076 | **0.077** | **0.130** | **0.135** | **0.143** | **0.117** | **0.121** | **0.157** | **0.185** | **0.188** | 0.241 | 0.190 |\n| $T_1$ | $S_A$ | 0.140 | 0.170 | 0.153 | 0.154 | 0.184 | 0.175 | 0.181 | 0.200 | 0.189 | 0.184 | 0.221 | 0.218 | **0.190** | 0.214 |\n| $T_2$ | $S_A$+$S_B$ | 0.259 | 0.329 | 0.251 | 0.264 | 0.420 | 0.302 | 0.362 | 0.380 | 0.361 | 0.337 | 0.500 | 0.517 | 0.422 | 0.460 |\n| $T_3$ | $M_u$+$M_o$ | 0.178 | 0.202 | 0.356 | 0.253 | 0.307 | 0.402 | 0.340 | 0.368 | 0.320 | 0.477 | 0.445 | 0.469 | 0.632 | 0.484 |\n| $T_4$ | $S_A$+$S_B$+$M_u$+$M_o$ | 0.170 | 0.187 | 0.367 | 0.251 | 0.253 | 0.405 | 0.308 | 0.350 | 0.289 | 0.455 | 0.450 | 0.436 | 0.571 | 0.486 |\n---\n---\n---\n\n|Model| Trainset | Brass | C.P. | Org. | Pipe | Reed | Str. | S.Lead | S.Pad |\n| :---------- | :--------- | --------: | ----------: | --------: | --------: | --------: | -------: | ---------: | --------: |\n| $T_1$ | $S_A$  | **0.278** |  0.086 | 0.069 | 0.327 | 0.462 | 0.201 |   0.335 | 0.118 |\n| $T_2$ | $S_A$+$S_B$ | 0.425 |   **0.071** | 0.129 | 0.327 | 0.330 | 0.296 |   0.123 |     0.313 |\n| $T_3$ | $M_u$+$M_o$ | 0.361 |   0.087 | **0.062** | **0.045** | **0.090** | **0.087** |   **0.079** |  **0.060** |\n| $T_4$ | $S_A$+$S_B$+$M_u$+$M_o$ | 0.312 |   0.072 | 0.107 | 0.057 | 0.340 | 0.208 |   0.085 |     0.063 |\n---\n\n\nThe overall relative pattern of the Table B closely matches Table 2 of the main text, with MSG-LD achieving the best scores on most stem combinations.\nWe believe these additional metrics offer a more reliable assessment of partial generation performance.\n\nWe also agree that FAD alone cannot capture musical coherence, so we also evaluated the COCOLA score [6] for partial generation ***(please refer to the table and discussion in our response to Reviewer W8on)***.\n\n\n> *\"Additionally, while I appreciate the authors having trained various models (T1-4), I don’t see how it makes sense to compare these against a baseline only trained on a single subset of the data.\"*\n\n**Authors' Response**:\nYou are correct that only the baselines (MSDM and MSG-LD) and our $T_1$, each trained on the same $S_A$ set, are directly comparable.\nWe included $T_2$~$T_4$ to illustrate how training on larger and more diverse datasets affects performance, since dataset-agnostic scaling is rarely evaluated in prior work and no existing baselines cover these settings. \nWe hope these results will serve as useful reference points for future research.\n\nFor additional comparability, we also trained Hybrid Demucs on the $S_A$ set. The corresponding results are shown in Table C below to strenghten the baseline comparison.\n\n---\n\n*Table C: Source extraction results with Hybrid Demucs.\nColumn headers indicate the target stem and its corresponding set.*\n\n|Model| Trainset |B ($S_A$) | D ($S_A$) | G ($S_A$) | P ($S_A$) | V ($M_u$) | B ($M_u$) | D ($M_u$) | V ($M_o$) | B ($M_o$) | D ($M_o$) | G ($M_o$) | P ($M_o$) |\n| ------- | -----| ------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- |\n|Demucs | $M_u$ | 1.29 | 0.92 | - | - | 1.59 | **1.67** | 1.29 | 0.89 | **1.48** | 1.08 | - | - |\n|HDemucs | $M_u$ | 1.49 | 0.90 | - | - | **1.50** | 1.99 | 1.53 | **0.83** | 1.71 | 1.1 | - | - |\n|HDemucs | $S_A$ | **0.83** | **0.57** | **0.78** | **0.79** | - | 1.87 | **1.22** | - | 1.61 | **1.01** | **1.45** | **0.99** |\n|$T_1$ \t| $S_A$ | 1.28 | 0.66 | 1.27 | 1.07 | 3.80 | 1.91 | 1.33 | 5.15 | 1.61 | 1.10 | 2.86 | 2.68 |\n\n---\n\n> *\"The biggest issue I have is the missing human evaluation, as the FAD numbers don’t show the full picture.\"*\n\n**Authors' Response**:\nWe agree that human evaluation is crucial for assessing generated music quality.\nAccordingly, we conducted a listening study on a total generation task, and the results are shown in Table D below.\n\n---\n\n*Table D: Mean Opinion Score (MOS) for the total‐generation task, rated from 1 (bad) to 5 (excellent). Two aspects were evaluated: Quality (how much the snippet sounds like noise versus a real song) and Coherence (consistency across generated stems). A total of 24 subjects participated, each listening to 8 segments per model. Scores are reported with 95% confidence intervals.*\n\n|Model | Ground truth | MSDM | MSG-LD | $T_1$ | $T_2$ | $T_3$ | $T_4$ |\n| ------ | ----- | ----- | ------ | ------- | ------- | ----- | ------- |\n|Quality| 3.79±0.18 | 2.05±0.16 | 3.80±0.17 | 3.63±0.17 | 3.92±0.16 | 3.81±0.16 | 3.92±0.16 |\n|Coherence| 3.75±0.19 | 2.06±0.18 | 3.70±0.17 | 3.54±0.18 | 3.99±0.15 | 3.75±0.17 | 3.79±0.15 |\n---\n\n\nThe results indicate that $T_1$ is slightly less preferred than MSG-LD, whereas $T_2$~$T_4$ receive higher preference ratings, demonstrating that expanding the training dataset can benefit the perceptual quality of generated music.\n\n\n> Q:*\"Why are the authors only training on 16 kHz? As far as I know the Stable Audio pipeline supports up to 44.1 kHz stereo.\"*\n\n**Authors’ Response:**\nWe initially chose 16 kHz mono to match the baseline models and ensure a fair comparison. We agree that 16 kHz mono is not sufficient for high-fidelity music generation (as noted in our Limitations section). Having demonstrated the validity of our framework at 16 kHz, we plan to extend MGE-LDM to 44.1 kHz stereo in future work.\n\n> Q:*\"Why are the authors using an outdated version of Demucs? The Hybrid Demucs is significantly better compared to the original.\"*\n\n**Authors’ Response:**\nThank you for pointing this out. \nWe have attached results for Hybrid Demucs (HDemucs) in *Table C* above.\nAll models were trained to convergence. \nIn our experiments, HDemucs results have not shown significant improvements over standard Demucs, so we initially reported the original Demucs comparison in the main paper, consistent with the MSDM paper. However, we agree that including HDemucs in the main tables is more appropriate and will update the manuscript accordingly.\n\n\n> Q:*\"Why does the demo page only have comparisons in the Slakh section but not for Moises/MusDB? Especially for the source extraction, which can be easily compared to existing methods.\"*\n\n**Authors’ Response:**\nThank you for your feedback regarding the demo page.\nOn the “Slakh Full” demo page, we initially showcased only $T_4$ results, assuming that adding other references would not be informative given the diversity of instruments. \nHowever, we agree that this limits comparability. \nAccordingly, we have now added Demucs inference examples alongside our model under “Source Extraction (Full) / MUSDB18 & Moises\".\nWe hope this makes the demo more informative and directly comparable.\n\n> *\"The authors have a section on limitations in the appendix. I suggest they move this to the main text.\"*\nThank you for the suggestion. We will move the Limitations section into the main text to ensure it is clearly visible to readers.\n\nWe appreciate your valuable feedback and will incorporate these changes in the final manuscript to enhance its clarity and rigor.\n\n\n**References**\n\n[1] Gui et al. \"Adapting frechet audio distance for generative music evaluation.\" ICASSP 2024\n\n[2] Grötschla et al. \"Benchmarking Music Generation Models and Metrics via Human Preference Studies.\" ICASSP 2025\n\n[3] Huang et al. \"Aligning Text-to-Music Evaluation with Human Preferences.\" arXiv:2503.16669, 2025.\n\n[4] Donahue et al. \"Singsong: Generating musical\naccompaniments from singing.\" arXiv:2301.12662, 2023.\n\n[5] Mariani et al. \"Multi-source diffusion models for simultaneous music generation and separation.\" ICLR 2024\n\n[6] Ciranni et al. \"Cocola: Coherence-oriented contrastive learning of musical audio representations.\" ICASSP 2025."}]

Author rebuttal:

Meta-review: The paper proposes MGE-LDM, a latent diffusion model for multi-track music generation, source imputation, and text-conditioned source separation without relying on fixed stem categories. The framework models mixtures, sub-mixtures, and sources jointly in a latent space and frames both extraction and completion as conditional inpainting, with adaptive timestep conditioning to improve fidelity.

All reviewers agree on the significance of the methodological contribution, which allows handling mixtures beyond the standard four-stem pipelines and offers a scalable, stem-agnostic approach. The ability to exploit heterogeneous datasets during training is also an important contribution of this work.

One reviewer remains unconvinced by the experimental evaluation, noting that the proposed method does not always outperform competitors, particularly in human evaluation. This concern was partially addressed by the introduction of the COCOLA metric.

Even if the performance is not always superior, the clear methodological advance, the ability to unify multiple tasks in a scalable latent framework, and the new experiments presented during the rebuttal make this work a clear step forward in this line of research.
Decision: Accept (poster)