reviews: 
Comments: This paper proposes a Multi-Source Diffusion Model (MSDM) for performing both music source separation and generation. Modeling multiple sources jointly allows the system to perform source imputation / accompaniment generation by conditioning on a partial mixture during inference. The authors demonstrate the efficacy of MDSM in both source separation and generation tasks, achieving separation performance competitive with state-of-the-art models. Moreover, by modeling sources independently and introducing a novel score function, the authors achieve separation results surpassing the state-of-the-art for certain instrument classes. While generation and imputation abilities appear to be more limited, MSDM demonstrates the strong potential of generative separation systems that model sources jointly.
Strengths: The proposed method is novel and interesting. Given recent advances in generative source separation and the apparent benefits of iterative refinement for separation [1], a diffusion-based approach seems natural.

The proposed method achieves very good separation performance using mostly off-the-shelf components, demonstrating the strength of the general approach. Additional improvements (e.g. weakly-supervised ISDM variant and Dirac score function) are well motivated in the paper.

The problem of source imputation / accompaniment generation is more relevant to many music creation workflows than full mixture generation, and is comparatively under-studied. While the accompaniment generation results here don't necessarily "improve" on those demonstrated in the referenced concurrent work [2] in terms of realism, the proposed method approaches the problem from a different angle (joint source separation and generation, diffusion rather than language modeling) and allows for more fine-grained control using multiple conditioning instrument classes (as opposed to singing only).

[1] Ethan Manilow, Curtis Hawthorne, Cheng-Zhi Anna Huang, Bryan Pardo, and Jesse Engel. Improving
source separation by explicitly modeling dependencies between sources. In ICASSP 2022-2022
IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 291–295.
IEEE, 2022.

[2] Chris Donahue, Antoine Caillon, Adam Roberts, Ethan Manilow, Philippe Esling, Andrea Agostinelli,
Mauro Verzetti, Ian Simon, Olivier Pietquin, Neil Zeghidour, et al. Singsong: Generating musical
accompaniments from singing. arXiv preprint arXiv:2301.12662, 2023.
Weaknesses: I think the paper would benefit from some additional discussion of the computational and data requirements of the proposed method. Presumably separation time is linear in the number of inference steps (plus correction); it would be nice to see this explicitly compared to Demucs with and without Gibbs sampling. Similarly, if the proposed method is more data-hungry than discriminative methods such as Demucs (e.g. if the proposed method is not competitive when trained/evaluated on MUSDB), this might be worth emphasizing further. Parameter counts for each method would also be nice to see.

Based on the listener study and provided listening examples, MSDM struggles to produce coherent and high-quality generations -- even when judged in the context of its synthetic training data. To my ears, it seems like tempo sometimes degrades within the model's 12-second context for unconditional generations, and for imputation generations when strongly metric signals (drums, bass) are not given as conditioning. Overall, the separation results seem much stronger than the generation results.
Recommendation: 8: accept, good paper
Reviewer confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness/correctness: 4 excellent
Clarity: 3 good