A Data-Driven Prism: Multi-View Source Separation with Diffusion Model Priors
Authors: Sebastian Wagner-Carena
Aizhan Akhmetzhanova
Sydney Erickson
Venue: NeurIPS 2025 poster ()
Keywords: Diffusion
Source Separation
Score-Based
Linear Mixing Models
Expectation-Maximization
Unsupervised Methods
Primary area: deep_learning
TL;DR: Diffusion models disentangle latent sources from multiple, noisy, incomplete observations without source-specific assumptions, yielding generative priors and posterior inference across synthetic tasks and real galaxy data.

Abstract: In the natural sciences, a common challenge is to disentangle distinct, unknown sources from observations. Examples of this source separation task include deblending galaxies in a crowded field, distinguishing the activity of individual neurons from overlapping signals, and separating seismic events from the ambient background. Traditional analyses often rely on simplified source models that fail to accurately reproduce the data. Recent advances have shown that diffusion models can directly learn complex prior distributions from noisy, incomplete data. In this work, we show that diffusion models can solve the source separation problem without explicit assumptions about the source. Our method relies only on multiple views, or the property that different sets of observations contain different linear transformations of the unknown sources. We show that our method succeeds even when no source is individually observed and the observations are noisy, incomplete, and vary in resolution. The learned diffusion models enable us to sample from the source priors, evaluate the probability of candidate sources, and draw from the joint posterior of our sources given an observation. We demonstrate the effectiveness of our method on a range of synthetic problems as well as real-world galaxy observations.

A Data-Driven Prism: Multi-View Source Separation with Diffusion Model Priors
Sebastian Wagner-Careea wwggurcarooedliaiiioaassitttttte org
Aizhan Akhmetzhanova 1.3,+ aakhmetzhanovaßg harvard. edu
Sydney Erickson* aydney3@staanford.om
Center for Computational Astrophysics, Flatiron Institute
Center for Data Science, New York University
3 Department of Physics, Harvard University
4 Department of Physics, Stanford University
Abstract
A common challenge in the natural sciences is to disentangle distinct, unknown sources from observations. Examples of this source separation task include de- blending galaxies in crowded field, distinguishing the activity of individual neu- rons from overlapping signals, and separating seismic events from an ambient background. Traditional analyses often rely on simplified source models that fail accurately reproduce the data. Recent advances have shown that diffusion mod- els can directly learn complex prior distributions from noisy, incomplete data. In this work, we show that diffusion models can solve the source separation prob- lem without explicit assumptions about the source. Our method relies only on multiple views, or the property that different sets of observations contain differ- ent linear transformations of the unknown sources. We show that our method succeeds even when no source is individually observed and the observations are noisy, incomplete, and vary in resolution. The learned diffusion models enable us to sample from the source priors, evaluate the probability of candidate sources, and draw from the joint posterior of the source distribution given observation. We demonstrate the effectiveness of our method on range of synthetic problems well as real-world galaxy observations.
1 Introduction
For scientific data, pristine, isolated observations are rare: images galaxies come blended with other luminous sources [1-3], electrodes measuring brain activity sum multiple neurons [4-6], and seismometers registering earthquakes contend with constant seismic background [7, 8]. Addi- tionally, the observations often incomplete and collected by heterogeneous each with unique resolutions. The corrupted data rarely directly usable. Instead, leveraging these datasets for scientific discovery requires solving source separation problem either learn the un- known source prior [9, 10] or constrain the posteriors for individual sources given an observation [1, 7J. In this work, address the general challenge of multi-view source separation (MVSS).
of instruments,
Most source separation methods, including ICA-based methods [11-13], non-negative matrix fac- torization methods [14-16]. and template-fittee fitting methods [17-19], require strong prior assumptions
Equal contribution.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

about the sources. Similarly, most deep-learning based methods require access to samples from the source priors to generate training sets [20-25]. When the source distributions are well- understood, this poses a degeneracy: isolating and measuring the source signals requires source prior, but constraining the source prior requires isolated measurements of the sources.
Alternatively, some source separation methods assume a known mixing process and thereby relax the need for source prior [26-29]. To break the degeneracies between the sources, these methods rely on distinct collections of observations, or views, with each view offering a different linear mixture of the underlying sources. These works focus on contrastive datasets, where the goal to separate a signal that is enriched in target view compared to a background view. While relevant for a number of scientific datasets, these source separation methods are either limited in their expressivity [28, 29] or are not designed for incomplete data [26]. Additionally. the contrastive assumption fails in domains where no source is ever individually measured.
Recent work has shown that score- based diffusion models [30] can serve as expressive Bayesian priors. Notably, once a diffusion model prior is trained, it enables effective posterior sampling for Bayesian inverse problems [31-39]. In the setting of noisy, incomplete observations, embedding diffusion models within an pnnectaaiionmaaimizattie prior [40]. In this work, we extend the use of diffusion model priors to MVSS. By leveraging the ability to sample joint diffusion posteriors over independent sources, our method directly learns prior for each source. The main contributions of our method are:
framework can be used to learn an empirical
Generalist method for multi-view source separation: Our method is designed for any MVSS problem that is identifiable and linear. We show experimentally that our method works even when the data is incomplete, noisy, and varies in dimensionality. Additionally, our method does not require contrastive examples and succeeds even if every source is present in every observation.
Source priors and posteriors: Our method results in independent diffusion models for each source. This affords all of the sampling and probability density evaluation benefits diffusion models.
State-of-the-art (SOTA) performance: Our method outperforms existing methods on the con- trastive MVSS problem despite having a more generalist framework.
2 Problem Statement
Consider noisy observation y of view a E { Nvvews) which is composed of a linear mixture of distinct sources x with 8 {1,..., N.). The exact mixture of each source is given by matrix A that depends on the view, a, the source, B, and the specific sample, iq. This model can be formalized as:
yf.
ad e to
X
+ nh.
where nf, NO Ey,). The a subscript on the sample index highlights that sample indices between views are unrelated. Importantly, source draws are not shared between views.
Goal: Given samples of noisy observations in each view, aim to infer the individual prior dis- tributions p(x()) of each source (xx) 1 Access to these source priors then allows to perform source separation by sampling from the joint posterior p( (x Hy?? A.
Dimensionality: Unlike traditional source separation, the dimensionality of the observation is deter- mined by the specific view: y -17 E dd and n E pdd.dd,, Similarly, the source dimensionality can vary between sources, x Rdd, leading to a mixing matrix whose dimensionality is deter- mined by the view and source, Aad E Rd xd3, No assumption is placed on the relative magnitude of the da and d3 values, although for many applications d3 2 da V a, 3.
Source Independence: We assume that each source conditionally independent of the other sources, allowing us to factorize the prior distribution: p([x3) 13=1) = =I p(x*). Similarly, we 0
assume independence between the sources and mixing matrices: p(x1 1(x²1A,
p(x²) A VBB,8' a.

---

Mixing Matrix and Incomplete Data: In contrast to blind source separation, we will assume that the mixing matrices, a/ are known. However, while the dimensionality of the mixing matrices are fixed for each view and source, the specific matrix can differ between samples ia. For the purposes of this work, we will consider any non-invertible linear transformation to generate incomplete data.
Identifiability: Not all choices of mixing matrices and dimensionalitiis will lead to a unique so- lution for the prior distributions. When a unique solution does not exist, any MVSS method will converge to a set of source distributions that accurately describe the data but may not match the true distributions. Therefore, we will assume identifiability throughout this work.
3 Related Works
Multi-view Source Separation: Research on MVSS problems has focused on the contrastive set- ting. In this setting, there are two views, the background view that contains only the background source and the target view that contains both the background source and the target source:
bkg
blg
hkg, 7/2 yJ
targ
x
bkg
Xj
targ
17
targ
(2)
Contrastive latent variable models (CLVM; (28]) define two sets of latent distributions in lower- dimensional space, one for the background source and one for the target source. The latent variables are mapped to the observed space either with linear model (CLVM- (CL.M-LLnnar) or through non-linear transformation parameterized by a neural network (CLVM VAE). The parameters controlling the transformation from the low- www-imennsionl latent space the observation space are optimized through an expectation pnreectationmmaxinne maximization or variational inference approach. The CLVM method can generate posterior and prior samples for both sources, and it can be adapted to incomplete data.
Contrastive principal component analysis (CPCA; [27]), and its probabilistic extension (PCPCA; [29]), attempt to find vectors that maximize the variance in the target view without explaining the variance in the background view. They do so by introducing a pseudo-data covariance matrix C Curz "Cag. with 7 a tunable hyperparameter. PCPCA can only sample from the target source posterior and prior, but it can be adapted to incomplete data.
Contrastive variational autoencoder models (CVAE; (26)) are an alternative formulation of the CLVM-VAE method?. In contrast to VAE, CVAE uses two encoders shared across views: one produces background latents and the other produces target latents. The concatenated latents are fed shared decoder, with the target latents multiplied by zero for the background decoding task. In the original formulation, the CVAE model never outputs the target source during training, only the target observation. This allows for non-lineat mixing of target and source, but makes extending the model to incomplete data challenging when x,, and x,
targ do not share a mixing matrix.
Diffusion Models: Diffusion models [30, 41-44] seek reverse a known corruption process in order to be able to generate samples from a target distribution. In the continuous-tine time framing, samples from the target distribution, 2o p(æo), are corrupted through diffusion process governed by the stochastic equation:
dx; = f(x, t)dt + g())dw..
(3)
where f(x,,1) is known as the drift coefficient, g(() is known as the diffusion coefficient, and w is generated through standard Wiener process. The time coefficient t ranges from 0 to 1, with xo being the original samples and x, being the fully diffused samples. This induces a conditional distri- bution of the form p(x, Xn) = (x, (a, E.). where a and E, can be derived from our drift and diffusion coefficients [45]. The forward stochastic differential equation (SDE) has corresponding
reverse SDE [46]:
dx: = [[((x,,) [ log)(x))] di + g())dw)
(4)
where w is the standard Wiener process with time reversed. This reverse SDE evolves sample from the fully diffused distribution p(x:) back to the original data distribution p(Xo). Equation 4 requires access to the score function, Vx, log p(x:), which is approximated by a neural network
We arbitrarily use CLVM-VAN versus CVAE to distinguish between Severson et al. [28] and Abid et al. [26].
3

---

trained via score matching on samples from the forward diffusion process [30, 47, 48). Training and sampling from a diffusion model requires selecting an SDE parameterization [30, 42, 44, 491, score matching objective [30, 42, 43, 45], and a sampling method for the reverse SDE [30, 42, 44, 45].
We adopt the variance exploding parameterization for the SDE [43], the denoiser parameterization from Karras et al. [45] for the score matching approach, and the predictor-corrector (PC) algorithm as our sampling method [30]. The denoiser parameterization approximates E[xo Xx] by minimizing
the objective:
(5) Here, dx(X,, 1) is our denoiser model with parameters 0. We use loss weighting term, A(t), to ensure all time steps equally prioritized. Note the denoiser returns the expectation value, which can be directly converted to the score via Tweedie's ' formula [50].
[()) = (x.) (xx,))) x
Posterior Sampling: Once trained. a diffusion model can be used as Bayesian prior for condi- tional posterior sampling [30]. Specifically, the score function in the reverse SDE is replaced by the posterior score function:
(6) where is our observation, and x. log p(y|x;) is the score of the likelihood. The prior score in Equation 6 given by the trained diffusion model, but evaluating the likelihood score with respect to an arbitrary i requires solving:
Vx, p(x:l) = Vx, log)(x;) + Vx. log p(y(()))))
p(y|x))= p(y|xx)))) (((p) (xppp//xxxx/xx)))p/pp )xx-
Many methods have been proposed for evaluating the conditional score [51]. Of particular interest are methods that propose an approximation to the right-most conditional distribution in Equation 7 [34, 36, 38-40J. In general, these methods use multivariate Gaussian approximation:
p(xo|xx) R N (xo [oo]//xoxxx [x]], V[xo|x:))
When the observation function is defined by the linear matrix A and the likelihood is Gaussian, p(y Xo) = (y Axxo E,), this approximation yields an analytic solution for the likelihood score:
Xx, log ~ Vx, [xx] A (Ey + AVIx x. (y AE(xx]
(9)
The moment matching posterior sampling (MMPS; [40]) approximation leads to the best sampling when compared on linear inverse problems. The trick behind MMPS is to use Tweedie's variance formula:
V/xo/x,] E, x, E[xp] Xt].
(10)
While the Jacobian, [[x in Equation 10 would be extremely costly materialize, the MMPS method avoids instantiating the matrix by the use of the vector-Jacobian product combined with a conjugate gradient solver [52] for the inverse in Equation 9.
Expectation Maximization: Expectation-maxiie maximization (EM) is a framework for finding the max- imum likelihood estimate for model parameters, 0, in the presence of hidden variables [53]. EM builds a sequence of model parameters 0o, 0,,..., 0x that monotonically improve the likelihood of the data². We adopt the Monte Carlo EM (MCEM) framework from [40]. We embed a diffusion model prior into an MCEM framework where the hidden variables are the true signals, x, and the
observations are the noisy, linear transformations of the signal, y Ax + m. The following two steps of the framework are then repeated until convergence:
Expectation (E): Given the current diffusion model parameters Gg. sample from diffusion posterior for each observation: X; ~ qo, (x, y,, A.). Here qo, (x,ly,, A,) is the distribution given by sampling from the reverse SDE in Equation 4 while using the posterior score from Equation 6 and the denoiser model do, (x,,).
Maximization (M): Given the set of posterior samples for the full dataset, {x₂ } maximize the data likelihood with respect to the model parameters 0 to get 0 In practice, since the mixing matrix and noise covariance are fixed, the denoising score matching objective (Fquation 5) can be used as a surrogate.
Note that the MCEM framework used in this work does not give this theoretical guarantee.

---

4 Methods
Our goal is learn the prior distribution p(x²) of each source {x* ) given the observations, {y?, ), known mixing matrices, A. and known noise covariance E (see Section 2). The prior distribution for each source 5 will be parameterized by a variational distribution qas (xx). defined by a denoiser diffusion model, dga (xe () with parameters 0P_ We use an EM framework to iteratively maximize the likelihood of the set of diffusion model parameters θ₂ = { 1 where k indexes the EM round. We summarize the full method, DDPRISM, in Algorithm I provided in Appendix A. All of the code reproduce our method and experiments has been made public*.
Maximization Step: For the M step we want to maximize the expected log- log-liiklihood of the full set of diffusion model parameters with respect to the data with u, (xx) (x?) ..... (x )T
1+1 = arg max Epty", "piy",(A'")) (A [log geguuo y" [A)
(11)
arg max Epty" (Aad)) goo (oly""(AA0)] (wo y", {AaH))
(12)
arg max Eply (AA8)) Egog (
[A Σ log qa= (x])]
(13)
where ge (uo, y", {)* is the full joint distribution of the observation, mixing matrices, and sources under the diffusion model parameters. In getting from Equation 12 to Equation 13 we have dropped all the terms independent of θ and taken advantage of the independence of the sources. Since each distribution qa (x²) is independent of the others, Equation 13 reduces to optimizing each diffusion model separately expectation value over p(y", (A ()) by drawing examples from the dataset, so all that remains is defining how to sample from the joint posterior distribution ge, ([x*)y> Hy (A (Aa))) set of diffusion model parameters Ø₁
the samples from its corresponding source. We can take an
given the current
Expectation Step: We want to sample from the joint distribution ge, (fx* Hy AP). To do we need the joint posterior score:
Vu, log ge, (u y {A Vu, log ge, (u.) + Vu, log qe, (y" Uu (A*9)).
(14)
Because our sources independent, the first term the right- hight-hand side of Equation 14 simplifies to:
Vu, un log ge, ge,
(ux) Σ Vn, log Log (x²))
(15)
3
which is simply the sum of the individual diffusion model scores. The remaining likelihood term in Equation 14 is given by:
gea (y lu, (A 1) = /
ply (A )) II qo, (=g(<²²) drg- f p..
(16)
3
To solve this we employ the MMPS approximation [40], wherein each conditional distribution R (x is approximated by its first and second moments. Since the conditional distributions
ge (4 are now Gaussian, Equation 16 is simply N, analytic Gaussian convolutions. The final likelihood
score approximation is then:
V E[x] (AA
(A
Vu, log 90, (y {A 1)
N: N. (A
Eg + Σ A A (A
(y Σ B
A od E
(x1))
(17)
'Code: https://gittub..cml// com////aaggarrrrrree/////1111
5

---

Posterior
Prior
Method
PQM
FID
PSNR
SD
PQM f
FID
SD
ID Manifold: Cont. 2 Sources
PCPCA [29]
0.0
9.35
7.69
0.0
7.91
CLVM Linear [28]
0.0
9.58
5.80
0.0
5.86
CLVM VAE [28]
0.0
17.15
1.81
0.0
2.91
DDPRISM- [54]
0.0
12.66
3.96
0.0
3.92
DDPRISM-Joint [Ours]
0.26
38.27
0.35
0.01
0.37
ID Manifold: Cont. 3 Sources
PCPCA [29]
0.0
6.89
12.57
0.0
10.22
CLVM Linear [28]
0.0
11.64
2.03
0.0
2.16
CLVM VAE [28]
0.0
13.09
2.22
0.0
1.82
DDPRISM- [54]
0,0
9.50
4.50
0.0
4.53
DDPRISM-Joint [Ours]
0.0
19.78
0.75
0.0
0.78
ID Manifold: Mix. (fmiz
0.1)
DDPRISM-Gibbs [54]
0.0
17.69
1.84
0.0
1.81
DDPRISM-Joint [Ours]
0.001
24.15
0.05
0.0
0.04
GMNIST: Cont. Full-Resolution
PCPCA [29]
0.0
22.3
18.99
0.0
176.0
CLVM Linear [28]
0.0
101.3
13.30
0.0
139.9
CLVM VAE [28]
0.0
18.87
14.56
0.0
57.67
DDPRISM-Joint [Ours]
1.00
1.57
25.60
0.20
20.10
GMNIST: Cont. Downsampled
PCPCA [29]
0.0
121.7
14.08
0.0
115.4
CLVM Linear [28]
0.0
199.5
12.16
0.0
211.4
CLVM VAE [28]
0.0
1008.0
8.48
0.0
737.0
DDPRISM-Joint [Ours]
0.94
2.36
19.73
0.06
12.63
Table 1: Comparison of metries between methods and baselines for all the experiments shown in the paper. For each metric, the arrow indicates whether larger (1) or smaller (L) values are optimal. Our method sets matches the state-of-the-are for all combinations of experiments and baselines. The posterior metrics are calculated using posterior samples and the true source signals. Since these samples independent, is possible get large positive PQMass p-values.
Note that the computational cost of Equation 17 scales linearly with the number of sources. As with regular MMPS, the Jacobian can be avoided through the of the vector-Jacobian product, and the gradient of the variance is ignored. We note that a similar joint posterior equation was concurrently derived by Stevens et al. [25] for removing structured noise using diffusion models.
an alternative to directly sampling the joint posterior, also possible to use Gibbs sampling method. We derive an extension of the Gibbs diffusion algorithm presented in Deeueig- et al. [54] for MVSS in Appendix B. We note that converging to the posterior requires large number of Gibbs sampling rounds for source distributions with complex structure, thereby rendering the Gibbs sampling approach computationally infeasible for most problems.
Gibbs Sampling:
Contrastive MVSS Simplification: For the contrastive MVSS problem, each new view introduces one new source. theory, this problem solved by the generic EM method we have presented. In practice, it can be useful to train the diffusion models sequentially, optimizing 0' on observations from view a 1, optimizing @ on observations from view a with 0 held fixed, and so on. This limits the computational cost by reducing the number of source models in the joint sampling for all the final view. However, it discards the information about source & present views a B. We summarize this simplified method in Appendix A.

---

qolx joint/Gibbs
4.0
fri 0.0
fiix 0.1
3.5
frix 0.5
Mean Sinkhom Divergence
3.0
fnix A varies
(x(x3) Joint
Gibbs, frix = 0.1, 8x compute
2.5
2.0
ge(x²)
Gibbs
1.5
1.0
Ge Joint
0.5
0.0
Last Truth EM Lap
109
10
Lap=1
Lap=8 8
EM Lap
Figure 1: Comparison of posterior samples for our joint sampling method and the Gibbs sam- pling method [54] the ID manifold problem. Both methods are equivalent for the first source. The plots show the evolution of the marginals for the first and second dimension of the specified source distribution. The last EM lap for sources 2, are 16, 32, 64 respectively.
Figure 2: Comparison of the mean Sinkhorn di- vergence for different fmix on the mixed ID man- ifold problem. Also shown are the Gibbs sam- pling method with eight times as many compu- tations per EM lap and our method when fmi 1.0 and A depends on 3. Even for large mix- ing fractions, method can accurately learn the two distinct underlying source distributions.
Results
We present five experimental setups that demonstrate the effectiveness of our method. The first four experiments are variations of two synthetic problems previously explored in the literature [26-28, 40]. The final experiment is the wallwwrrld scientific task of separating galaxy light from random light, demonstrating the viability of our method on complex scientific data. Timing comparisons for all experiments can found Appendix G.
5.1
Oine-Dmmnsiionl Manifold Experiments
In this pair of experiments, our sources, x are drawn from distinct, one-dimensional manifolds embedded in R Our observations, y lie in RS and are generated by a random linear projection, A E R whose rows are drawn from the unit sphere, $t, In addition, we add isotropic Gaussian noise with standard deviation o, 0.01 for all views. This setup follows previous work [40], although we now add multiple sources to our observations.
Contrastive MVSS: For the contrastive experiment, each view introduces mixing matrix shared among all the sources: A Ai, ca with aa 1 if and c otherwise. We set Nview = N₂ and generate a dataset size 216 for each view.
source, and the
For our joint diffusion sampling approach, train three denoiser models, dgs (x,, t), each consist- ing of a multi-layer perceptron. We employ the simplified contrastive MVSS algorithm described in Section 4. We train the the sampling we use the PC algorithm with 16,384 predictor steps. each with one corrector step (PC step). The initial posterior samples are drawn using Gaussian prior whose parameters are optimized through a short EM loop. We also train the Gibbs sampling approach with the same de- noiser architecture and the same number of EM laps. To keep the computational costs (compute") on par with our joint diffusion approach, we do 64 Gibbs rounds per expectation step and reduce the number PC steps to 256. Because the Gibbs approach performs poorly, we only run it up to the second view. We also compare to PCPCA, CLVM-Linear, and CLVM-VAE, We provide additional
1, diffusion models for 16, 32, 64 EM laps respectively, For
We approximate compute the number of denoiser and vector-Jacobian product evaluations required.

---

experimental details on the diffusion parameters, generation of the random manifolds, and baselines in Appendix C.
As shown in Figure 1, our method learns the first two source distributions nearly perfectly despite the linear projection to lower dimension and the presence of noise. The third source distribution is also learned, although the final posterior samples are not as sharp. This is to be expected: later sources are only observed together with all the previous sources, making them harder sample. We compare the Sinkhorn divergence [55], PQMass p-value [56], and peak signal-to-noise ratio (PSNR) for our method and the baselines in Table 1. Our method compares favorably, outperforming all the baselines on both source distributions across all the metrics.
Mixed MVSS: For the mixed experiment, each source is present in every view. The mixing matrix given by: A A., ca with ca3 = 1 if 3 = and ca3 fmix otherwise. We set Nviews N₂ = and generate a dataset of size 216 for each view. We consider four different mixing fractions fmix E {0.0, 0, 0.5, 0.93 When fiix = 1.0 the problem is fully degenerate and therefore not identifiable (see Appendix C). For comparison, we also present a mixed experiment with A drawn separately, meaning that every source is fully present in every view but with a different mixing.
For the joint sampling and Gibbs sampling approach, we use the same denoiser models and initial- ization procedure as the Contrastive MVSS problem. However, because the Gibbs sampling was not able to Iearn either source distribution with equivalent compute, we instead used 64 Gibbs rounds and 2048 PC steps. This means that each EM lap for the Gibbs sampling is eight times as expensive. We provide additional experimental details in Appendix C.
In Figure 2 we compare the Sinkhorn divergence averaged over both source distributions as a func- tion of EM laps. We find that our method can learn the underlying source distributions with high accuracy up to fmix = 0.5. For fmix = 0.9 our method continues to improve its estimate of the source distributions as the EM laps progress, but it does not converge. If the mixing matrix varies between the sources, we can reconstruct the source distributions even with fmix = 1.0. By compar- ison, Gibbs sampling for fmix 0.1 converges much more slowly despite requiring eight times as much compute per EM lap.
5.2 Grassy MNIST Experiments
For this pair of experiments, we use the Grassy MNIST dataset first presented in [27]. The dataset is a contrastive MVSS problem which consists of two views: the first containing random 28 x 28 crops of grass images from ImageNet [57], and the second containing linear combination of grass images with 0 and 1 MNIST digits [58]. In addition, we add a small amount of Gaussian noise to each observation (ay 0.01). For both experiments, view and 13,824 for the linear combination of digits and grass.
generate 32,768 observations for the grass
Full-Resolution In the full-resolution experiment, we set A! A = 1 for 8 = 1 (grass) and A!² 0, A 0.5 x II for 3 2 (MNIST). Two example observations can be seen in Figure 3. For our denoiser models, dgo (x, l). we use a U-Net architecture [42, 59] with attention blocks [60] and adaL.- norm modulation [61]. We employ the simplified contrastive MVSS algorithm described in Section 4, and we initialize our posterior samples using a Gaussian prior whose parameters are optimized through 32 EM laps. We train the grass and MNIST diffusion models for 64 EM laps. For the sampling we use the PC algorithm with 256 PC steps. We compare to PCPCA, CLVM-Linear, and CLVM-VAE but omit Gibbs sampling since its computational cost makes it impractical for this problem. We provide additional experimental details in Appendix D.
In Table 1 we compare the FID scores [62], the PSNR, and the PQMass value the posterior MNIST digit samples across the entire dataset of the MNIST + grass view. For the FID score, we use trained MNIST classifier in place of the Inception-v3 network. We also report the FID score and PQMass p-value on samples from the learned priors. In addition, in left-hand side of Figure 3 show example posterior draws for our method, PCPCA, and CLVM-VAE.
Our method visually returns the closest posterior samples to the ground truth, and outperforms the baselines across all the metrics. The prior samples also outperform the baselines on both PQMass p- value and FID. To better understand the source of this improvement, we run ablation studies over the
For fei 0.0 we no longer have source separation problem, but include this setup as limiting case.
8

---

Full-Resouution
Downsampled
Observation
DDPRISM
PCPCA
CLVM-VAE
Truth
CLVM-VAE
PCPCA
DDPRISM
Observation
No Pasterior
No Postericr
Figure 3: Comparison of posterior samples for two example observations in Grassy MNIST exper- iment. The observations are on the far left and right, the true input sources are in the middle, and draw from DDPRISM [ours], CLVM-VAE [28], and PCPCA [29] for both the full-resolution and downsampled case is shown in between. PCPCA cannot sample the grass posterior, and CLVM- Linear is omitted for brevity. Our joint diffusion model returns the best reconstruction of both sources, with mear-perfect posterior samples in the full-resolution case.
model architecture, the number of EM laps, the number of initialization laps for the Gaussian prior, the dataset size, and the number of sampling steps. The full ablation study details and results can be found in Appendix H. Overall, the method is fairly insensitive changes in these hyperparameters. Only extreme choices, such as replacing the U-Net with a small MLP (FID=49.03), conducting only 2 laps of EM (FID=96.85) removing the Gaussian prior initialization entirely (FID=10.41), or using only 16 PC sampling steps (FID=5.41) 41) appear to meaningfully reduce performance across our metrics. The one exception is reductions in the dataset size, where using 1/4th, 1/16th, and 1/64th of the dataset leads to an FID of 4.64, 10.19, and 38.34 respectively.
Downsampled: In the downsampling experiment, resolution, one third are 2x downsampled, and one third are 4x downsampled. Otherwise the dataset size and underlying source distributions are kept the same. An example observation be in Figure 3. We compare to PCPCA, CLVM-L.neat and CLVM-VAF, and use the same configurations the full-resolution experiment for all four methods. We provide additional details in Appendix D.
third of observations at full-
In Figure we show example posteriors for 2x downsampling, and Table we report the FID score, the PSNR, and the PQMass p-value for the posterior samples across the dataset. We also report the FID score and PQMass p-value of draws from the prior distribution. As with the full- resolution dataset, our method visually returns the closest posterior samples to the ground truth for both sources and is SOTA across the baselines. Notably, while both of the CLVM methods and PCPCA struggle with the downsampled images, method returns visually plausible digits and comparable metric performance to the full-resolution experiment.
5.3 Galaxy Images
In astronomical images, instrumental noise, cosmic rays, and random foreground and background objects along the line of sight contaminate observations ("random" light). Separating the flux of these contaminants from the target object is contrastive MVSS problem. There are two source populations: random light and galaxies. We also get two views: random sightlines that are uncor- related with galaxies, and targeted sightlines built from catalog. The targeted sightlines contain both the galaxy and random light. To build our views we use archival Hubble Space Telescope ob- servations of the COSMOS survey [63]. For the galaxy view, we select targets using the Galaxy Zoo Hubble object catalog [64], and for the random view we make cutouts at random locations in

---

2.20
Observation
Uppar Thangle Colorbar
0015
Random Source
Pusterior Draw
66/0
Lower Triangle Colorbar
Galaxy Source
Posterfor Draw
+0.00
Figure 4: Observations of galaxy images along with posterior samples for the random and galaxy source using method. Images split into high-contrast (upper region) and low-contrast (lower region) colormaps to highlight the range of features. The galaxy source captures the central light while the small-scalc fluctuations and the uncorrelated light is separated into the random source.
the COSMOS field. Each cutout 128 128 pixels. For our denoiser models, dæø [x] t), we use the same U-Net architecture for Grassy MNIST, but change the model depth and size to model the larger images. We employ the simplified contrastive MVSS algorithm, and we train the random diffusion model for 32 EM laps and the galaxy diffusion model for 16 EM laps. For the sampling use the PC algorithm with 256 PC steps. We provide additional experimental details in Appendix E.
In Figure 4 we show sample galaxy observations along with random and galaxy source posterior sample for each observation. While we do not have access to the ground truth, a visual inspection of the source posteriors samples show that our model is effectively identifying the central galaxy light and separating it from the random light. Notably, the background around the galaxy light appears to be nearly flat at zero. We make the full dataset of 79k pristine galaxy images publicly available'. Generating the full dataset required hours on four NVIDIA H100 GPUs.
Discussion and Limitations
We present DDPRISM, data-driven framework for tackling general MVSS problems using diffu- sion model priors. To our knowledge, it is the first method to provide a unified solution for linear MVSS problems, achieving state-of-the-an performance across diverse experiments. We further demonstrate that DDPRISM delivers high-quality source separation on a complex real-world astro- physical dataset.
Despite these advances, the framework has important limitations. First, is restricted to linear source combinations, which excludes nonlinear generative processes like occlusion. The Gaussian noise assumption can be relaxed through the inclusion of an additional "noise" source, but only an extra view is available. We also assume exact knowledge of the mixing matrix, whereas scientific applications often involve probabilistic rather than deterministic mixing. These assumptions limit generality and motivate extensions that relax linearity, Gaussianity, and deterministic mixing.
Computationally, our method requires expensive sampling that compounded by the EM-style training. This places limits on the resolution, dataset size, and number of sources that can be feasi- bly modeled. Our baselines are far cheaper, albeit at the cost sample quality. Performance also degrades with smaller datasets, creating tension between the benefits of large datasets and the com- putational demands of the method. Replacing our initialization method with random initializations degrades sample quality, suggesting that clever initialization methods may improve convergence and alleviate computational bottlenecks. Nevertheless, DDPRISM establishes diffusion-based MVSS as promising tool for disentangling structured signals across scientific domains.
"https://doo wtt.:o......////ooeee...oooo 17159988
10

---

Acknowledgments and Disclosure of Funding
We would like thank Shirley Ho, David Spergel, and Francisco Villaeseusa-Navaroo for insight- ful discussions during the development of this project. The computational resources used in this work were provided by the Flatiron Institute, division of the Simons Foundation. The work of SWC supported by the Simons Foundation. AA acknowledges financial support from the Flat- iron Institute's Predoctoral Program, and thanks the LSST-DA Data Science Fellowship Program, which is funded by LSST-DA, the Brinson Foundation, the WoodNext Foundation, and the Research Corporation for Science Advancement Foundation; her participation in the program has benefited this work. SE acknowledges funding from NSF GRFP-2021313355 and the Stanford Data Science Scholars Program.
References
[1] P. Melchior et al. "SCARLET: Source separation in multi-band images by Constrained Matrix Factorization". In: Astronomy and Computing 24 (2018), p. 129.
[2] Simon Samuroff al. "Dark Energy Survey Year 1 results: the impact of galaxy neighbours on weak lensing cosmology with IM3SHAPE". In: Monthly Notices of the Royal Astronomi- cal Society 475.4 (2018), 8), pp. 4524-4543.
[3] James Bosch al. "The Hyper Suprime-Cam Software Pipeline". In: Publications of the Astronomical Society of Japan 70 (2018), S5.
[4] Michael S Lewicki. "A review of methods for spike sorting: the detection and classification
of neural action potentials". In: Network: Computation in Neural Systems 9.4 (1998), R53,
[5] Gaute T. Einevoll et al. "Modelling and analysis of local field potentials for studying the function of cortical circuits". In: Nature Reviews Neuroscience 14.11 (2013), pp. 770-785.
[6] Marius Pachitariu et al. "Kilosort: realtime spike-sorting for extracellular electrophysiology with hundreds of channels". In: bioRxiv (2016). eprint: https / www biorxiv. org 00000/01181... full. ful...df.
content/ear wwweeeeeteeeereeeeeeeeeeeeeellee hate//arrlyyyyy1 20//2016/666
[7] Yanping Liu et al. "An Amplitude- Amplitude-Preserved Time FaeeueFrreeyyyyye Peak Filtering Based on Em- pirical Mode Decomposition for Seismic Random Noise Reduction". In: IEEE Geoscience and Remote Sensing Letters 11.5 (2014), pp. 896-900.
[8] Tong Shen et al. "A Strong Noise Reduction Network for Seismic Records". In: Applied Sciences 14.22 (2024).
[9] François Lanusse al. "Deep generative models for galaxy image simulations". In: Monthly Notices of the Royal Astronomical Society 504.4 (2021), pp. 5543-5555.
[10] Euclid Collaboration et al. "Euclid preparation. XIII. Forecasts for galaxy morphology with the Euclid Survey using deep generative models". In: Astronomy & Astrophysicss 657 (2022), A90.
[11] Pierre Comon. "Independent component analysis, a new concept?" In: Signal processing 36.3 (1994), pp. 287-314.
[12] A. Hyvarinen. "Fast and robust fixed- point algorithms for independent component analysis".
In: IEEE Transactions on Neural Networks 10.3 (1999), pp. 626-634.
[13] Francis R Bach and Michael I Jordan. "Kerell independent component analysis". In: Journal
of Machine Learning Research 3 (2002), pp. 1-48.
[14] Daniel D. Lee and H. Sebastian Seung. "Learning the parts of objects by non-negative matrix factorization". In: Nature 1.755.655566 (1999), pp. 788-791.
[15] Patrik O Hoyer. nega-neeative matrix factorization with sparseness constraints". In: Journal of Machine Learning Research 5 (2004), pp. 1457-1469.
[16] Paris Smaragdis. "Non-negative matrix factor deconvolution; extraction of multiple sound sources from monophonic inputs". In: International Conference on Independent Component Analysis and Signal Separation. Springer. 2004, pp. 494-499.
[17] C. L. Bennett et al. "First- Year Wilkinson Microwave Anisotropy Probe (WMAP) Obser- vations: Foreground Emission". In: Astrophysical Journal Supplement Series 148.1 (2003),
pp. 97-117.
[18] Felix Franke et al. "Bayes optimal template matching for spike sorting-combining fisher dis- criminant analysis with optimal filtering". In: Journal of Computational Neuroscience 38 (2015), pp. 459.
11

---

[19] G. Kovács et al. "A box-fitting algorithm in the search for periodic transits". In: Astronomy & Astrophysics 391 (2002), pp. 369-377.
[20] Ashish Bora et al. "Compressed sensing using generative models". In: International Confer- ence on Machine Learning. PMLR. 2017, pp. 537-546.
[21] Zhuo Chen et al. "Deep attractor network for single-microphone speaker separation". In: 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). 2017,
pp. 246-250.
[22] Daniel Stoller et al. Wave-u-net: mulli-scale neural network for end-to-end audio source separation. 2018. arXiv: 1806. 03185
[23] Muhammad Asim et al. "Invertible generative models for inverse problems: mitigating repre- sentation error and dataset bias". In: International Conference on Machine Learning. PMLR. 2020, pp. 399-409.
[24] Jay Whang et al. "Solving inverse problems with flow-based noise model". In: International Conference on Machine Learning. PMLR. 2021, pp. 11 11146-11157.
[25] Tristan SW Stevens et al. "Removing Structured Noise using Diffusion Models", In: Trans- actions on Machine Learning Research (2025).
[26] Abubakar Abid and James Zou. Contrastive variational autoencoder enhances salient fea- tures. 2019, arXiv: 04601 [cs.LG].
[27] Abubakar Abid et al. "Exploring patterns enriched in dataset with contrastive principal component analysis". In: Nature Communications 9, 2134 (2018).
[28] Kristen A Severson et al. "Unsupervised learning with contrastive latent variable models". In: Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 33. 01. 2019, pp. 4862- 4869.
[29]
Didong Li al. "Probabilistic contrastive dimension reduction for case-control study data". In: The Annals of Applied Statistics 18.3 (2024), pp. 2207-2229.
[30] Yang Song al. "Score-Basse Based Generative Modeling through Stochastic Differential Equa- tions". In: International Conference on Learning Representations. 2021.
[31] Babjat Kawar et al. "SNIPS: Solving Noisy Inverse Problems Stochastically". In: Advances in Neural Information Processing Systems. Ed. by M. Ranzato al. Vol. 34. Curran Associates, Inc., 2021, pp. 21757-21769.
[32] Yang Song et al. "Solving Inverse Problems in Medical Imaging with Score-Based Generative Models". In: International Conference on Learning Representations. 2022.
[33] Bahjat Kawar et al. "Denoising Diffusion Restoration Models". In: Advances in Neural In- formation Processing Systems. Ed. by S. Koyejo et al. Vol. 35. Curran Associates, Inc., 2022, pp. 23593-23606.
[34] Hyungjin Chung et al. "Diffusion Posterior Sampling for General Noisy Inverse Problems". In: International Conference on Learning Representations. 2023.
[35] Berthy T. Feng et al. "Score-Bas Based Diffusson Models as Principled Priors for Inverse Imag- ing". In: Proceedings the IEEE/CVF International Conference on Computer Vision (ICCV). 2023, pp. 10520-10531.
[36] Yuanzhi Zhu et al. "Denoising Diffusion Models for Plug-and-Play Image Restoration". In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops. 2023, pp. 1219-1229.
[37] Ronan Legin et al. "Beyond Gaussian noise: A Generalized approach to likelihood analysis with non-Gaussian noise". In: The Astrophysical Journal Letters 949.2 (2023), p. L41.
[38] Benjamin Boys et al. "Tweedie Moment Projected Diffusions for Inverse Problems". In: Transactions on Machine Learning Research (2024).
[39] Jiaming Song et al. "Pseudoinverse-guiee guidedd diffusion models for inverse problems". In: Inter- national Conference on Learning Representations. 2023.
[40] François Rozet et al. "Learning Diffussin Priors from Observations by Expectation Maxi- mization". In: Advances in Neural Information Processing Systems. Ed. by A. Globerson et al. Vol. 37. Curran Associates, Inc., 2024, pp. 77647-87682.
[41] Jascha
et al. "Deep unsupervised learning using nonequilibrium thermody- namics". In: International Conference on Machine Learning. PMLR. 2015, pp. 2256-2265.
12

---

[42] Jonathan Ho et al. "Denoising Diffusion Probabilistic Models". In: Advances in Neural In- formation Processing Systems. Ed. by H. Larochelle al. Vol. 33. Curran Associates, Inc., 2020, pp. 6840-6851.
[43] Yang Song and Stefano Ermon. "Generative Modeling by Estimating Gradients of the Data Distribution". . In: Advances in Neural Information Processing Systems. Ed. by H. Wallach et al. Vol. 32. Curran Associates, Inc., 2019.
[44]
Jiaming Song et al. "Denoising Diffusion Implicit Models". In: International Conference on Learning Representations 2021.
[45] Tero Karras et al. "Elucidating the Design Space of Diffusion-Based Generative Models", In: Advances in Neural Information Processing Systems. Ed. by S. Koyejo et al. Vol. 35. Curran Associates, Inc., 2022, pp. 26565-26577
[46] Brian D.O. Anderson. "Reverse-time diffusion equation models". In: Stochastic Processes and their Applications 12.3 (1982), pp. 313-326.
[47] Aapo Hyvärinen and Peter Dayan. "Estimation of non-normalized statistical models by score matching". In: Journal of Machine Learning Research (2005), pp. 695-709.
[48] Pascal Vincent. "A connection between score matching and denoising autoencoders". In: Neu- ral computation 23.7 (2011). pp. 1661-1674.
[49] Alexander Quinn Nichol and Prafulla Dhariwal. "Improved denoising diffusion probabilistic
models". In: International Conference on Machine Learning. PMLR. 2021, pp. 8162-8171. [50] Bradley Efron. "Tweedie's 's formula and selection bias". In: Journal of the American Statistical
Association 106.496 (2011), pp. 1602-1614.
[51] Giannis Daras et al. A survey on diffusion models for inverse problems. 2024. arXiv: 2410.
00083 [ca. LG].
[52] Magnus R Hestenes, Eduard Stiefel, al. "Methods of conjugate gradients for solving linear systems". In: Journal of Research of the National Bureau of Standards 49.6 (1952), pp. 409- 436.
[53]
Arthur P Dempster et al. "Maximum likelihood from incomplete data via the EM algorithm". In: Journal of Royal Statistical Society 39 (1977), pp. 1-22.
[54]
David Heurtel-Depeiges et al. "Listening to the noise: Blind Denoising with Gibbs Diffusion".
In: International Conference on Machine Learning. 2024.
[55] Lénaic Chizat et al. "Faster Wasserstein Distance Estimation with the Sinkhorn Divergence". In: Advances in Neural Information Processing Systems. Ed. by H. Larochelle et al. Vol. 33. Curran Associates, Inc., 2020, pp. 2257-2269.
[56] Pablo Lemos et al. "PQMass: Probabilistic Assessment of the Quality of Generative Models using Probability Mass Estimation". In: International Conference on Learning Representa- tions. 2025.
[57] Olga Russakovsky et al. "ImageNet Large Scale Visual Recognition Challenge". In: Interna- tional Journal of Computer Vision 155.3 (2015), pp. 211-252,
[58] Y. Leeun et al. "Gradient-based" learning applied to document recognition" In: Proceedings of the IEEE 86.11 1 (1998), pp. 2278-2322
[59] Olaf Ronneberger et al. "U-Net: Convolutional Networks for Biomedical Image Segmen- tation". In: International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer. 2015, pp. 234-241.
[60] Ashish Vaswani et al. "Attention is All you Need". In: Advances in Neural Information Pro- cessing Systems. Ed. by I. Guyon et al. Vol. 30. Curran Associates, Inc., 2017.
[61] William Peebles and Saining Xie. "Scalable Diffusion Models with Transformers". In: Pro- ceedings of the IEEE/CVF International Conference Computer Vision (ICCV). 2023, pp. 4195-4205.
[62] Martin Heusel et al. "GANs Trained by Two Time-Scale Update Rule Converge to Local Nash Equilibrium". In: Advances in Neural Information Processing Systems. Ed. by I. Guyon et al. Vol. 30. Curran Associates, Inc., 2017.
[63] A. M. Koekemoer al. "The COSMOS Survey: Hubble Space Telescope Advanced Camera for Surveys Observations and Data Processing". In: Astrophysical Journal Supplement Series
172.1 (2007). pp. 196-202.
13

---

[64] Kyle W. Willett et al. "Galaxy Zoo: morphological classifications for 120 000 galaxies in HST legacy imaging". In: Monthly Notices of the Royal Astronomical Sociery 464.4 (2017), pp. 4176-4203.
[65] Friedemann Zenke and Tim P. Vogels. "The Remarkable Robustness of Surrogate Gradient Learning for Instilling Complex Function in Spiking Neural Networks". In: Newral Compu- tation 33.4 (2021), pp. 899-925.
[66] Stefan Elfwing et al. "Sigmoid-weighte linear units for neural network function approxi- mation in reinforcement learning". In: Neural Networks 107 (2018). Special issue on deep reinforcement learning, pp. 3-11.
[67]
Jimmy Lei Ba et al. Layer Normalization. 2016. arXiv: 1607. , 06450 Setat.ML]
[68]
Takuya Akiba et al. "Optuna: A next-generation hyperparameter optimization framework". In: Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 2019, pp. 2623-2631.
[69]
Ethan Perez et al. "Film: Visual reasoning with general conditioning layer". In: Proceedings of the AAAI conference on artificial intelligence. Vol. 32. 2018.
[70]
Robin Rombach et al. -High-Resolution Image Synthesis With Latent Diffusion Models". In: Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition
(CVPR). 2022, pp- 10684-10695.
[71] Zhendong Wang et al. "Patch Diffusion: Faster and More Data-Efficient Training of Diffusion Models" In: Advances in Neural Information Processing Systems. Ed. by A. Oh et al. Vol. 36. Curran Associates, Inc., 2023, pp. 72137-72154.
[72] Zhaoyu Zhang al. "Training Diffusion--assee Generative Models with Limited Data". In: International Conference on Machine Learning. 2025.
[73] Jiwan Hur et al. "Expanding Expressiveness of Diffusion Models With Limited Data via Self-Distillation Based Fine-Tuning". In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). 2024, pp. 5028-5037.
14

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? Answer: [Yes]
Question:
The method is tested in several different settings. Performance is compared against multiple previous works.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: See Section 6 for discussion of limitations. Main points include the assump- tions of linear observation model, deterministic mixing, and Gaussian noise model, and the high computational cost of the method.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are discussed in the paper.
The authors are encouraged to create separate" "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The au- thors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the ap- proach. For example, facial recognition algorithm may perform poorly when image resolution is low images taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to ad- dress problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
15

---

Answer:
[NA]
Justification: While we include a number equations the paper, there are no formal theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theo- rems.
The proofs can either appear the main paper or supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be comple-
mented by formal proof' provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclu- sions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Datasets used for the experiments are either previously established (ID Man- ifolds, Grassy MNIST), in the galaxy images case, are made public. The methods are described in sufficient detail to be reproduced, and the source code has been made public.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary either make it possible for others replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all sub- missions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results way produce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case thors are welcome to describe the particular way they provide for reproducibility. In the case of closed-souree models, it may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
16

---

5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Code, for both the method and for simulating/ querying the datasets, is pub- licly available.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines htttps://nips.coo cc/
mublic//giids/Codeeeee ceabbbmmssioooooccii.. for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: //nips.. cc/
wttpcccpppcccccccccccccccccccccc public/ wwwicccgguiu/eeieeeeeieeo Seaedddde/oediiuieee Chte//oooduubmii ssi onPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: The most important training details are included in the text, with additional details provided in appendices.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should presented in the core the paper to level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropri- ate information about the statistical significance of the experiments?
Answer: [No]
Justification:
We compare performance using point statistics.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
17

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (c.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state The authors should prefer- ably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Runtime and computational resources for the experiments in the paper detailed in Appendix G.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't t make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: The authors have reviewed the guidelines and verify that this work abides by the Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This paper does not work with datasets with societal impact. The method does not have immediate runway for malicious usage.
Guidelines:
18

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact spe- cific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular upplications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitiga- tion strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: All experiments are performed on datasets that are low risk. Two of the datasets are simulated, the third one uses astronomical images.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by re- quiring that users adhere usage guidelines restrictions access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Use of existing assets mentioned and relevant papers are cited in the main text. Further details, such as location of repositories and licenses, are included in supple- mental materials.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URI.
19

---

The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the pack- age should be provided. For popular datasets, anperswithcode com datastsss has curated licenses for some datasets. Their licensing guide can help determine the li- cense of dataset.
For existing datasets that are re re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documenta- tion provided alongside the assets?
Answer: [Yes]
The code base for the method is publicly available. For the galaxies images experiment, we make the dataset of denoised images with galaxy light publicly available well for further use by the astrophysics community.
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
For crowdsourcing experiments and research with human subjects, does the pa- per include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Question:
Answer: [NA]
Justification:
This paper does not involve crord-sourcing or human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contri- bution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, cura- tion, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowd-sourcing or human subjects.
Guidelines:
20

---

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research conducted, IRB approval (or equiva-
lent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: Our usage of LLMs falls under the category of: "spell checkers and grammar suggestions, programming aid for editing purposes".
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to LLM policy (https: pittps:/neeurppr http//eeeppp./pe/ee//ee//e for what should or should not be described.
21

---

Algorithm 1: MVSS WITH JOINT DIFFUSION
(y2, (A N, ΣΦ Nias to 1==11 ta Jozl number of sources N,, number of views
Input: Dataset D
Nviews, initial denoiser parameters O { number of EM rounds K Output: Trained diffusion model priors with denoiser parameters O for k < 0 to K do foreach (y (Aa)) D
D do
8=1 ~ gei ((x²) {xx)) (A // E step using equation 14
end
Oc+1 arg maxe [Ei, Eg log gos (x. (x..) // и step using equation end return Ø
Algorithm 2: SIMPLIFIED CONTRASTIVE MVSS WITH JOINT DIFFUSION
Nazs Input: Dataset D {yr, (A. B=11 Eg J==1 A 0 if a, number of sources N,,
number of views Niiews N.. initial denoiser parameters Op 10 100) number of
EM rounds K Output: Trained diffusion model priors with denoiser parameters Oμ for a <- 1 to N do
for & 0 to K do foreach (y (A..)
ag tu B=1 ~ 9(00)
Dig
E with a = a do
)a
[x",, (A // E step using equation 14
end
0 = arg max [Ei, log gou (x. )] // M step for only source a end
end
return O
A Joint Sampling Algorithms
In Algorithm we present an algorithmic summary of our MVSS method using joint sampling- This method conducts joint EM training for all sources simultaneously. In Algorithm 2 we present a summary the contrastive MVSS simplification. This method trains each source sequentially under the assumption that view sources B {1,2) 2). and so on (see Section 4). While Algorithm 2 discards the information about source 3 for views f 3, it reduces the computational complexity.
only contains source 3 1, view = 2 only contains
B Gibbs Sampling
Gibbs sampling with diffusion models, originally proposed for blind denoising [54], can be easily extended the full MVSS problem. We start by selecting a source By. Given the 0 source realizations for {x0 St ,,,, , observation y set of known mixing matrices (A 1 noise covariance D0, then the posterior score for the remaining source becomes:
13=1" and the
V x logp(x; (ogpxx/'y" ly, (Aa3IM 1 {x( }344,, =V
log ((x,') +
logp(y Go (A
(18)
22

---

For conciseness, introduce the observation residual r = E 31480 A 3' then write the remaining likelihood term in Equation 18 as:
Xg.
We can
8, ply x (Aa0)
N. 1 1==1 {xx )))+8)) = (x)))) 1x ((xx ) jdx
(19)
which is identical to Equation 7 with the observation replaced by the residual. Since Gibbs sampling fixes all but one source time, posterior evaluation reduces to traditional posterior sampling with diffusion prior with the observation replaced by the residual. An implementation of the Gibbs sampling procedure for MVSS can be found the provided code.
C One-Dimensional Manifold Experiment Details
We generate random one-dimensional manifolds following the steps outlined in [65]. In all experi- ments in Section 5.1, the smoothness parameters for the manifolds corresponding the first, second, and third source distributions are set to 3, 4, and 5 respectively. Similarly, we use the same denoiser architecture for each source and each experiment. The denoiser is a multi-layer perceptron (MLP) composed of 3 hidden layers with 256 neurons and SiLU activation function [66], followed by a layer normalization functiin [67]. The denoiser conditioned on noise and noise embeddings are generated with the sinusoidal positional encoding method [60].
For the metrics, we use 16384 samples for the Sinkhorn divergence evaluation, 1024 samples for the PQMass evaluation, and 16384 samples for the PSNR evaluation. Note that we use the same number of samples from the true distribution and the prior / posterior distribution for all metrics. Metrics on the posterior samples are calculated by comparing to the true source value for the corresponding observation. For the PQMass evaluation, parameters.
1000 tessellations and otherwise keep the default
C.1 Contrastive MVSS
In the contrastive MVSS experiment, we use the simplified contrastive MVSS algorithm to train the denoiser models. We train the 3 = 1, 2, denoiser models for 16, 32, and 64 EM laps respectively. Following Rozet et al. [40], we reinitialize the optimizer and learning rate after each EM lap while keeping the current denoiser parameters. The MLP takes as input concatenated vector of the diffused sample at time t and the corresponding noise embedding. We summarize other training hyperparameters in Table 2.
For the Gibbs sampling approach we use the same dataset (up f = 2), denoiser architectures and training hyperparameters parameters are set to 64 Gibbs rounds with 256 PC steps, and we maintain one corrector step per predictor step. The number of PC steps is lowered that the number of denoiser and vector- Jacobian product evaluations per EM lap are roughly equivalent for Gibbs and joint sampling.
for the joint posterior sampling approach. The posterior sampling
For both the Gibbs and joint sampling approaches, the denoisers are initialized using samples from Gaussian prior. Since we are using our contrastive algorithm, only the diffusion model for the current source, 3, is replaced by the Gaussian prior, and the remaining diffusion models for sources 3' < 33 are kept to the optimum from the previous views (see Algorithm 2). The posterior is then sampled as usual. To optimize the parameters of this Gaussian prior, we use the same EM procedure as for the diffusion model. The final Gaussian prior used to generate an initial set posterior samples which are used to train the initial diffusion model, deg (Xt, t).
For the baselines, we use modified implementations built for incomplete data. For PCPCA, minimize a loss function defined across our two views, 3 = bkg, targ:
Nar
L(W, 4))
Σ
log det (Ci) + (y. wrng
A larg 11)
C, (y lurg
A
Larg
(20)
Nias
log (Dj) + (y A 1 DJ1 (y - A A.
+
2
23

---

Parameter
Contrastive MVSS
Mixed MVSS
MLP Parameters
Activation
SiLU 64
SiLU
Time Embedding Features
128
Training Parameters
Optimizer
Adam Linear
Adam
Scheduler
Linear
Initial Learning Rate
10-3
10-4
Final Learning Rate
10-6
10-5
Gradient Norm Clipping
1.0
1.0
Optimization Steps per EM Lap
65,536
65,536
Batch Size
1024
1024
Gaussian Initialization EM Laps
16
8192
Sampling Parameters
Noise Minimum
10-3
5 x 10-3
Noise Maximum
10
1.5 x 10
Conjugate Gradient Denominator Minimum
0.0
10-3
Conjugate Gradient Regularization
0.0
10-3
Corrector (PC) Steps
16.384
16,384
Corrections per PC Step
PC
10-1
8 x 10-2
Table 2: Hyperparameters for denoiser training and sampling on the one-dimensional manifold experiments.
where y is the learnable mean parameter for the target source. The views, bkg yy. are the same as those defined in Section 3 but with the mixing matrix A,t applied to the sources underlying y and the mixing matrix A applied the sources underlying y: y.. Here, y is tunable parameter that controls the relative importance of the variations in the background and target data. The dependence on the weights, W, comes from the two covariance matrices given by:
targ
C A thT W WWT
(A
lung
+ +
(21)
D, = A WWW
A big
+ofI.
(22)
where o; is the standard deviation of the observation noise. Note that we only optimize the weights and the mean, since the noise is known. We initialize the weights using the empirical covariance derived from source values given by multiplying the observations with the pseudo-inverse of the mixing matrix. We find that this smart initialization considerably improves the performance of PCPCA on incomplete data.
For CLVM-Linear and FLVM-VAE, we explicitly account for the mixing matrix in both the encod- ing and decoding steps. For CLVM-Linear, the encoded distribution is given by small modification to the original equations for the latent variable distributions:
bkg
(0 S
ST
(23)
11
d ;
bke
T (y) bKs)
(24)
Syjoint
(o M MM
(25)
join
1 M (y.lin p
Ijoini
(26)
bkg where Σ and j4, are the covariance and mean for the latents of y bkg The covariance and mean sojni
and i4 correspond to the joint latents tke z m for (yiiint [ bkg
joint
TJ. The

---

Parameter
Grassy MNIST
Galaxy
U-Net Parameters
Channels per Level
(32, 64, 128) (2, 2,
(64, 128, 236, 256, 512) (2, 2, 2, 2) (0, 0, 4, 8, 16) 0.1 SiLU
Residual Blocks per Level
Attention Heads per Level
(0, 0, 4)
Dropout Rate
0.1
Activation
SiLU
Training Parameters
Optimizer
Adam
Adam
Scheduler
Cosine Decay 10-3
Cosine Decay 10-5
Initial learning rate
Gradient Norm Clipping
1.0
1.0
Optimization Steps per EM Lap
4096
4096
Batch size
1920
64
Gaussian Initialization EM laps
32
4
Sampling Parameters
Noise Minimum
10-4
10-2
Noise Maximum
10a
10a
Conjugate Gradient Denominator Minimum
10-2
10-3
Conjugate Gradient Regularization
10-6
10-2
Conjugate Gradient Error Threshold
10
10a
Predictor-Corret (PC) Steps
256
64
Corrections per PC Step
1
PC
10 10-2
10-1
EMA decay
0.999
0.995
Table 3: Hyperparameters for denoiser training and sampling for the Grassy MNIST experiments and the Galaxy experiment. During sampling, conjugate gradient calculations whose total residuals exceed the conjugate gradient error threshold are recalculated using the denominator minimum and regularization.
matrices S, W are the background and target factor loading matrix in CLVM, M the con- catenated factor loading matrix, M =S' W For CLVM-VAF, the encoded distribution is calculated by explicitly passing the mixing matrix to the encoder network. For both CLVMM Linear and CLVM-VAE, the decoder outputs the complete source signal and the variational loss is calculated by first transforming the sources using the known mixing matrices. The optimization is done by maximizing the evidence lower bound as described in [28].
For PCPCA, CLVM-Linear, and CLVM-VAL we optimize the hyperparameters using 100 point sweep with the default Bayesian optimization used in optuna [68]. The results are reported using the best hyperparameters for each method on the posterior Sinkhorn divergence for three sources. For PCPCA this is 0.3 and 5 latent dimensions using a linear learning rate from 10-3 3 to 0.0 over 10 epochs For Ci.VM-Linee source latents, with a batch size of 1024, a cosine learning rate initialized to 10-1, and 1024 epochs of training. For CLVM-VAF VAE, the encoder architecture were multi-layer perceptrons composed of hidden layers with 256 neurons and dropout rate of 0.1 followed by SiLU activation function and a layer normalization function. The decoders follow the same structure. The CLVM-VAE was trained with batch size of 1024, a cosine learning rate initialized to 5 x 10-4, and 1024 epochs of training.
this was a dimensionality of for the background latents and 5 for the
C.2
Mixed MVSS
In the mixed MVSS experiment, we use Algorithm to train the diffusion models. We train for 70 EM laps and reinitialize the optimizer and learning rate after each EM lap while keeping the current
More iterations hampered performance in the hyperparameter sweep.
25

---

Encoder
Decoder
FID
MLP
MLP
18.87
U-Net (full-depth)
MLP
20.
U-Net (1 hidden channel)
MLP
45.37
U-Net (full-depth)
U-Net (full-depth)
388.26
U-Net (1 hidden channel)
U-Net (1 hidden channel)
390.27
Table 4: Evaluation of different CLVM-VAL encoder-decoder architectures for MVSS trained on
Grassy MNIST.
denoiser parameters. To condition on the time embedding, the output of each dense layer is passed through a FiLM layer [69]. We summarize other training hyperparameters in Table 2.
For the Gibbs sampling, only the sampling parameters number of Gaussian EM laps is reduced to 16, and we use 512 Gibbs rounds with 256 PC steps. As result, each EM lap of the Gibbs model is eight times expensive as the joint sampling laps, so we only run 13 laps of EM.
changed. To improve performance, the
For both the Gibbs and joint sampling approaches, the denoisers are initialized using samples from Gaussian prior. Unlike for the contrastive algorithm, all the diffusion models are replaced by the Gaussian prior for initialization. The posterior is then sampled usual. To optimize the parameters of these Gaussian priors, we use the same EM procedure as for the diffusion model. Note that the resulting Gaussian prior will differ by source. The final Gaussian priors used to generate initial set of posterior samples which are used to train the initial diffusion models, dng (xe, t) for all B.
C.2.1 Identifiability
In the case where frs 1, the mixed MVSS problem cannot be solved. Consider the vector z x² + x² and the trivial vector z² = 0. Then, for any observation ia, we can write:
y Ai (x!_ +
x²
+ ni
(27)
Ai, i + 2 + m
(28)
8(z2). Since we have constructed x' and x² lie on ID manifolds, know p(z?) * p(x') and p(z²) * p(x²). We have found two new sources that match our observations even though one of the sources is guaranteed to not have the same distribution either of the original sources. Therefore, there is not a unique solution to the source priors when fine 1.
However, p(z²)
D Grassy MNIST Experiments
The MNIST dataset is available under a CC BY-SA 3.0 license [58]. The ImageNet dataset made available for non-commercial purposes [57]. We generate grass images by taking random 28 x 28 pixel crops from ImageNet images with the grass label. We use different set of random crops for each view. For digits, images, we normalize the pixel values to the range [0, 1].
images with the 0 and 1 label. For both the grass and MNIST
We use the same U-Net denoiser architecture for each source and each experiment. The denoiser and training parameters are presented Table 3. For sampling, we use an exponential moving average of the model weights. The full sampling and U-Net code is provided with out codebase. For the initialization, we use a Gaussian prior optimized via EM as described in Appendix C.
For the metrics, we use 8192 for the FID evaluation, 512 samples for the PQMass evaluation, and 8192 samples the PSNR evaluation. We use the same number of samples from the true dis- tribution and the prior/ posterior distribution for all metrics. Metrics on the posterior samples are calculated by comparing to the true source value for the corresponding observation. For the PQMass evaluation, use 1000 tessellations and otherwise keep the default parameters.
26

---

EM Lap
EM Lap 2
EM Lap
EM Lap 8
EM Lap 16
Figure 5: Random samples of galaxy posteriors across EM iterations in the galaxy-image experi- ment. Early iterations fail to isolate galaxy light and show strong small-scale fluctuations. By iter- ation 4, small-scale fluctuations 16, the uncorrelated light is assigned to the random posterior component, leaving nearly noiseless, isolated galaxy posteriors. As in Figure 4, images are split into high-contrast and low-contrast col- ormaps to highlight the range of features.
separated but residual uncorrelated light remains. By iteration
D.1 Grassy MNIST Baselines
We optimize the PCPCA, CLVM-Linear, and CLVM-VAE hyperarameess using a 100 point sweep with the default Bayesian optimization used in optuna. The results we present are using the best hyperparameters on the FID for the full-resolution experiment.
For PCPCA, we the traditional algorithm the full-resolution experiment. On the downsampled experiment we fit the PCPCA parameters on the full-resolution subset and then sample using the equation for incomplete data. The PCPCA tunable parameter is set 7 0.39 and we latent dimensions.
For CLVM-Linear and CLVM-VAE, use the traditional algorithm the full-resolution exper- iment. On the downsampled experiment we follow the same procedure outlined in Appendix C. For CLVM-Linear,
set the dimensionality of the background latents to 265 and the dimensional- ity of the target latents to We use a cosine learning rate with initial value 1.2 x 10 trained over the dimensionality of the background
batches of 1920 images for 2 steps. For CLVM-VAL VAE, latents to 380 and the dimensionality of the target latents to 15. The cosine learning rate is used again, but with initial value of 2 10-4, 1, batch size of 256, and total of 214 steps.
The CLVM-VAE method uses an MI.P with three hidden layers of size 70 for the encoder and decoder. The number of hidden layers was optimized during the hyperparameter sweep, but the
27

---

Training Time
Inference Time / Sample
ID Manifold
GMNIST
Galaxy Images
ID Manifold
GMNIST
Galaxy Images
Method
Full Res.
Full Res.
PCPCA [29]
s
m
< 0.1 ms
1,5 ms 1
CLVM- Linear [28]
1.5
10 m
0.1 ms
CLVM VAE [28]
18 m
33 m
< 0.1 ms
1 mss
DDPRISM-Joint [Ours]
32 h
68 h
48
22 ms
90 ms
1.5
Table 5: Comparison of computational costs for our method and baselines for three experiments shown in the paper: contrastive ID manifold experiment with source distributions, full-resolution Grassy MNIST experiment, and the galaxy experiment. Training time refers to the amount of time each method takes train its corresponding model. Inference time per sample is the time method takes to obtain single posterior sample for an observation, given trained model. ID Manifold and Grassy MNIST experiments were run A100 GPUs, and Galaxy Image experiments were run on H100 GPUs.
encoder and decoder architectures were selected as part of separate ablation study whose results are summarized in Table 4. the ablation study, we compared three encoder choices:
A fully connected MLP.
A full-depth U-Net identical to the "'dwnssmplingg" half of U-Net used for our diffusion models without skip connections.
A convolutional neural network equivalent to our U-Net implementation with hidden channel (no downsampling).
We also compared two decoder choices:
A fully connected MLP.
A convolutional neural network equivalent to our U-Net implementation with hidden channel (no upsampling)
We found that more complex architectures negatively impacted performance, and that the best per- formance was achieved with simple MLP decoder and encoder.
E Galaxy Images Experiment
We query data hosted by the Mikulski Archive for Space Telescopes (MAST), which is available in the public domain. We retrieve data files using the astroquery package, which has a 3-clause BSD style license. We generate our galaxy images by querying 128 x 128 pixel cutouts centered on the Galaxy Zoo Hubble Catalog [64]. We generate our random fields by making 128 x 128 cutouts at random coordinates within the larger COSMOS exposures. This results in 78,707 galaxy images and 257,219 random images. We apply three normalizations in this order: (1) we pass the images through aresinh transform with scaling of 0.1. (2) scale the data by factor of 0.2, and (3) we clip the maximum absolute pixel value to 2.0. These three transformations help preserve the morphological features in the brightest sources while stabilizing the diffusion model training.
We use the same U-Net denoiser architecture for the galaxy and random source. The denoiser and training parameters are presented in Table 3. For sampling, we use exponential moving average of the model weights. For initialization, we use Gaussian prior optimized via EM as described in Appendix C. All of the code required to reproduce this experiment can be found in the DDPRISM codebase. For completeness, Figure we show the evolution of the galaxy posterior samples as function of EM laps.
28

---

Posterior
Prior
GMNIST Full Res.
PQM f
FID
PSNR
PQM t
FID 0
Model Architecture
MLP
0.05
49.03
17.93
0.
215.36
U-Net, small
1.00
2.47
25.28
0.06
15.38
U-Net (default)
1.00
1.57
25.60
0.20
20.10
Training Length (EM laps)
2
0.
96.85
16.62
0.
199.70
8
1.00
0.04
27.15
0.14
17.35
32
1.00
2.26
25.66
0.08
27.96
64 (default)
1.00
1.57
25.60
0.20
20.10
Initialization Laps
0 (random initialization)
0.97
10.41
23.35
0,01
22.22
4
1.00
0.00
26.80
0.15
24.33
16
1.00
0.00
27.02
0.21
6.31
32 (default)
1.00
1.57
25.60
0.20
20.10
Dataset Size
Full Dataset (default)
1.00
1.57
25.60
0.20
20.10
1/4th Dataset
1.00
4.64
23.67
0.14
21.36
1/16th Dataset
0.99
10.19
20.97
0.07
15.16
1/64th Dataset
0.0
38.34
15.34
0.0
36.55
Sampling Steps
16
0.87
5,41
21.01
0.
30.20
64
1.00
0.88
25.02
0.24
3.58
256 (defaul()
1.00
1.57
25.60
0.20
20.10
1024
1.00
0.59
26.50
0.08
7.18
Table 6: Ablation study of individual components and parameters of our method on the full- resolution grassy MNIST experiment. For each metric, the arrow indicates whether larger (f) or smaller (J) values are optimal. The values used for the Grassy MNIST (GMNIST) experiment in Section 5.2 denoted as default values. While most of the default values correspond to the optimal performance, we find that decreasing the length of the training and using fewer rounds of Gaussian EM improve the overall performance of the method.
Additional Diffusion Model Details
For our diffusion models, we use the preconditioning strategy from Karras et al. [45]. Our variance exploding noise schedule is given by:
(1) = exp(log(Tm)) (min)) + (log(Amax) log (amin)) (),
(29)
with minimum noise min and maximum noise mmxx During training, we sample the time parameter t from beta distribution with parameters a = 3, 3 3.
G Runtime and Computational Cost
We provide a detailed comparison of the computational costs between our method and baselines in Table 5. All timing was done on NVIDIA A100 GPUs with the exception of the galaxy images experiment that was run on H100s. Both ID manifold experiments used one A100 (40GB) GPU, both Grassy MNIST experiments used four A00 (40GB) GPUs, and the Galaxy Image experiments used four H100 (80GB) GPUs. Our method is more computationally expensive than the baselines, almost entirely due cost of sampling from the diffusion model. However, this computational cost comes with significant improvements in performance and sample quality.
29

---

H
Ablation Studies
In order to clarify the importance of the individual components of our method, we conduct series of ablation studies which are summarized in Table 6. We explored the effect of varying the following components on the performance of the method:
Diffusion model architecture: We find the model architecture to be important for perfor- mance: using an MLP-based architecture (5 hidden layers with 2048 hidden features per layer) gives poor results across all metrics. However, scaling down the UNet model (chan- nels per level: (32,64,125 (16,32), residual blocks per level: (2.2), embedding features: 64-+16 attention head moved up one level) does not degrade performance appre- ciably.
Training length: We observe that the model achieves good performance after as few EM iterations and that longer training leads to slight degradation in performance.
Initialization strategy: The number of Gaussian EM laps used to generate the initial sam- ples (and train the initial diffusion model) has minimal impact on performance. Only start- ing from randomly initialized model considerably reduces the performance of the method.
Training dataset size: We train our method using 1/4th, 1/16th, and 1/64th of the origi- nal grass and MNaST datasets. The 1/16th and 1/64th runs given 1/4th as many EM laps as the original training to account for the smaller dataset, but all other hyper- parameters unchanged. There is clear degradation in performance the grass and MNIST datasets are reduced in size. However, even with 1/16th of the original dataset (2048 grass images, 512 MNIST digits) our method still outperforms the baselines run on the full dataset. We note that there exists a few strategies for improving diffusion model performance in the low-data regime [45, 70-73J.
Sampling steps: steps: Increasing the number of predictor steps used to sample from the posterior generally improves the performance of our method. However, we find that the method performs well even with the number of sampling steps reduced to 64.
The overall robustness of our method to most ablation highlights that its effectiveness is driven by the ability to directly sample from the posterior given our current diffusion model prior. Because the likelihood is often constraining, this enables us to return high-quality posterior samples even when the prior specified by our diffusion model is not an optimal fit the empirical distribution. As evidence for this point, we note the large gap in performance between posterior and prior samples on the Grassy MNIST experiments (see Table 1).
30


Reviews:
[{"review_id": "puhx9aoWwN", "reviewer": "Reviewer_zJxX", "summary": "This paper proposes a diffusion model-based approach for multi-view source separation, where multiple noisy observations contain different linear mixtures of unknown sources. Unlike prior methods relying on contrastive setups or simple priors, the authors use diffusion models to directly learn source priors and perform posterior sampling. Integrated into an EM framework, the method iteratively refines source estimates without assuming isolated source observations. Experiments on synthetic and real-world data show strong performance over existing MVSS methods, particularly in noisy, incomplete, and heterogeneous settings.", "questions": "- The proposed EM framework with diffusion priors is computationally expensive, especially in posterior sampling. Can you provide quantitative comparisons of runtime and memory usage against baselines? How feasible is your method for larger-scale problems (like high-resolution astronomy datasets)?\n\n- The method currently assumes linear mixing and additive Gaussian noise. Can your framework be extended to nonlinear observation models, or do you plan to explore this in the future?\n\n- Diffusion models are known to be data-hungry. How does your method perform when the number of observations is small? Could you provide experiments in low-data regimes or discuss ways to mitigate overfitting in such cases?", "limitations": "Yes.", "rating": 5, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The proposed framework is general. The paper proposes a general MVSS solution applicable to various scientific domains, without relying on contrastive assumptions or isolated source data. The use of diffusion priors is also novel, the paper introduces diffusion models as flexible source priors for source separation, enabling expressive modeling of complex distributions.\n\n- The paper presents strong results in challenging settings, which demonstrates state-of-the-art results on both synthetic and real-world datasets, especially under noisy and incomplete data.\n\n- The paper has clear theoretical foundation, the method is grounded in a principled EM framework with posterior sampling, and provides solid mathematical formulation.\n\n- The results show that the approach working across different problem scales, from low-dimensional manifolds to real astronomical images.\n\nWeaknesses:\n- The EM iterations and diffusion-based sampling are computationally expensive, making scalability to large datasets questionable.\n\n- The work requires large datasets to train the diffusion priors effectively; unclear how well it performs in low-data regimes.", "was_revised": true, "final_justification": "The authors have satisfactorily addressed my pre-rebuttal concerns on computational cost by providing detailed runtime and memory comparisons against baselines, and clarifying feasibility for large-scale datasets.They have conducted additional experiments to assess performance under limited data, showing graceful degradation and still outperforming baselines in most settings, which resolves my concern. The limitation regarding linear mixing and Gaussian noise assumptions remains, but the authors provided a clear discussion of possible extensions and future directions.\n\nGiven the strong empirical results, thorough rebuttal clarifications, I maintain my positive assessment and recommendation for acceptance.", "rebuttal": "We would like to thank the reviewer for their thoughtful feedback. Below we address the concerns and questions brought up within the review.\n\n### Runtime and Computational Cost\n\nWe provide a detailed comparison of the computational time for each method in the table below:\n\n|||Training Time||¦||Inference Time per Sample||\n|-|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|**Method**|**1D Manifold**|**GMNIST Full Res.**|**Galaxy Images**|¦|**1D Manifold**|**GMNIST Full Res.**|**Galaxy Images**|\n|PCPCA [1]|5s|1m|--|¦|<0.1ms|1.5ms|--|\n|CLVM - Linear [2]|1.5m|10m|--|¦|<0.1ms|1ms|--|\n|CLVM - VAE [2]|18m|33m|--|¦|<0.1ms|1ms|--|\n|DDPRISM-Joint [Ours]|32h|68h|48h|¦| 22ms| 90ms|1.5s|\n\nCLVM-Linear denotes a contrastive latent-variable model with a linear transformation from observation to latents, and CLVM-VAE replaces the linear model with a neural network encoder / decoder. All timing was done on NVIDIA A100 GPUs. As the reviewer notes, our method is more computationally expensive than the baselines, almost entirely due to the cost of sampling from the diffusion model (since sampling must be done both during the EM loop and at inference time). However, this computational cost comes with significant improvements in performance. Below is the comparison of our model to the baselines across several experiments and metrics:\n\n|||Posterior|||||Prior||\n|-|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|**Dataset / Method**|**PQM ↑**|**FID ↓**|**PSNR ↑**|**SD ↓**||**PQM ↑**|**FID ↓**|**SD ↓**|\n|**1D Manifold: Cont. 2 Sources**|||||||||\n|PCPCA [1]|0.0|--|9.35|7.69|¦|0.0|--|7.91|\n|CLVM - Linear [2]|0.0|--|9.58|5.80|¦|0.0|--|5.86|\n|CLVM - VAE [2]|0.0|--|17.15|1.81|¦|0.0|--|2.91|\n|DDPRISM-Gibbs [3]|0.0|--|12.66|3.96|¦|0.0|--|3.92|\n|DDPRISM-Joint [Ours]|**0.26**|--|**38.27**|**0.35**|¦|**0.01**|--|**0.37**|\n|**1D Manifold: Cont. 3 Sources**||||||||||\n|PCPCA [1]|0.0|--|6.89|12.57|¦|0.0|--|10.22|\n|CLVM - Linear [2]|0.0|--|11.64|2.03|¦|0.0|--|2.16|\n|CLVM - VAE [2]|0.0|--|13.09|2.22|¦|0.0|--|1.82|\n|DDPRISM-Gibbs [3]|0.0|--|9.50|4.50|¦|0.0|--|4.53|\n|DDPRISM-Joint [Ours]|0.0|--|**19.78**|**0.75**|¦|0.0|--|**0.78**|\n|**1D Manifold: Mix. ($f_{mix}=0.1$)**|||||||||\n|DDPRISM-Gibbs [3]|0.0|--|15.48|2.52|¦|0.0|--|2.43|\n|DDPRISM-Joint [Ours]|**0.001**|--|**24.15**|**0.05**|¦|0.0|--|**0.04**|\n|**GMNIST: Cont. Full Resolution**|||||||||\n|PCPCA [1]|0.0|22.30|18.99|--|¦|0.0|176.0|--|\n|CLVM - Linear [2]|0.0|101.3|13.30|--|¦|0.0|139.9|--|\n|CLVM - VAE [2]|0.0|18.87|14.56|--|¦|0.0|57.67|--|\n|DDPRISM-Joint [Ours]|**1.00**|**1.57**|**25.60**|--|¦|**0.20**|**20.10**|--|\n|**GMNIST: Cont. Downsampled**|||||||||\n|PCPCA [1]|0.0|121.7|14.08|--|¦|0.0|115.4|--|\n|CLVM - Linear [2]|0.0|199.5|12.16|--|¦|0.0|211.4|--|\n|CLVM - VAE [2]|0.0|1008.0|8.48|--|¦|0.0|737.0|--|\n|DDPRISM-Joint [Ours]|0.0|**60.34**|**17.09**|--|¦|0.0|**79.43**|--|\n\nPSNR is the peak signal-to-noise ratio, PQM is the pqmass metric introduced in [4], FID is the FID score [5] using our MNIST classifier, and SD is the sinkhorn divergence. Note that we’ve replaced the CVAE [6] baseline with CLVM-VAE [2] since the latter is a superset of the former and achieves better performance. The substantive gap between our method and the baselines justifies the additional computational resources required to train and sample the diffusion model.\n\nIn terms of the feasibility for large datasets, the galaxy images experiment serves as a good baseline. The HST images we use are much higher resolution that ground-based observations [7,8] and equivalent in resolution to other current space missions [9]. Training the random and galaxy images took 48 GPU hours, and sampling from the posterior of an individual observation takes 1.5 seconds. For the ~80k galaxy images in the catalog, this amounts to under 34 GPU hours to draw one posterior sample for each galaxy. This compute budget is realistic for an academic lab.\n\nWe have added a section to the appendix that includes the full timing comparison above and point to that section in our discussion and limitations section and in the results for each experiment. \n\n### Performance as a Function of Dataset Size \n\nWe agree with the reviewer that, by using diffusion models to fit our prior distributions, our method will be sensitive to dataset size and likely perform poorly in the low data regime. Some work has explored solutions to reduce dataset size requirements, including data augmentation [11], patch-wise diffusion training [12], latent-space diffusion models [10, 13], and transfer learning [14]. It would be possible to incorporate these methods during our maximization step.\n\nTo better quantify our dependence on the dataset size, we ran a set of ablation studies on our performance on the full-resolution Grassy MNIST experiment. We train our method using 1/4th, 1/16th, and 1/64th of the original grass and grass+MNIST datasets. The 1/16th and 1/64th runs are given 1/4th as many EM laps as the original training to account for the smaller dataset, but all other hyperparameters are unchanged:\n\n|||Posterior|||||Prior||\n|-|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|**GMNIST Full Res.** |**PQM ↑**|**FID ↓**|**PSNR ↑**|**SD ↓**| |**PQM ↑**|**FID ↓**|**SD ↓**|\n|Full Dataset|**1.00**|**1.57**|**25.60**|--|¦|**0.20**|20.10|--|\n|1/4th Dataset|1.00|4.64|23.67| -- |¦|0.14|21.36|--|\n|1/16th Dataset|0.99|10.19|20.97|--|¦|0.07|**15.16**|--|\n|1/64th Dataset|0.0|38.34|15.34|--|¦|0.0|36.55|--|\n\nAs the reviewer suggested, there is clear degradation in performance as the grass and MNIST datasets are reduced in size. However, even with 1/16th of the original dataset (2048 grass images, 512 MNIST digits) our method still outperforms the baselines run on the full dataset. We have added the dataset scaling study to the appendix and referenced it in the Discussion and Limitations section. We have also added a brief discussion of strategies that can be used to improve the performance of diffusion models in the low-data regime.\n\n### Linear Mixing and Gaussian Noise Assumptions\n\nThe reviewer is correct that our current method assumes both a linear observation model and Gaussian noise. As it stands, our framework cannot easily be extended to the non-linear case without substituting our joint sampling for Gibbs sampling. It is likely possible to generalize our method further under a reasonable set of assumptions, but we reserve that for future work.\n\nThe Gaussian noise assumption can be relaxed within our framework. It would require access to “noise-only” views or the ability to sample from a known noise distribution. One can consider the noise as an additional source and pre-train its corresponding diffusion model with the noise samples. From there, the rest of our machinery operates identically to the experiments presented in the paper, with the weights of the noise diffusion model frozen. However, the linear mixing assumption must be preserved. Noise that depends on the signal (for example Poisson noise from photon counts) cannot be modeled in this way. We leave the extension to non-linear noise models for future work. \n\n### Additional notes\n\n* For the posterior metrics, we are comparing posterior samples to the original signal, meaning that they are not independent draws. Therefore, it is possible to get large positive pqmass values and large negative sinkhorn divergence values. The posterior metrics are meant to compare the reconstruction produced by different methods,  and should not be interpreted in absolute terms. We have added a sentence to the main body of the paper to clarify this.\n* To ensure fair comparison, we ran a hyperparameter sweep for the baseline methods. Similarly, we made small architectural changes to improve the diffusion models used in our method and reran all experiments. Therefore, some of the metrics quoted in the table changed from our original submission.\n\n[1] Li, Didong, et al. \"Probabilistic contrastive principal component analysis” (2020)\n[2] Severson, Kristen A., et al. “Unsupervised learning with contrastive latent variable models” (2019)\n[3] Heurtel-Depeiges, David, et al. \"Listening to the noise: Blind denoising with Gibbs diffusion\" (2024)\n[4] Lemos, Pablo, et al. \"Pqmass: Probabilistic assessment of the quality of generative models using probability mass estimation\" (2024)\n[5] Heusel, Martin, et al. \"GANs trained by a two time-scale update rule converge to a local Nash equilibrium\" (2017)\n[6] Abid, Abubakar, et al. “Contrastive Variational Autoencoder Enhances Salient Features” (2019)\n[7] Koekemoer, Anton M., et al. \"The COSMOS survey: Hubble space telescope advanced camera for surveys observations and data processing\" (2007)\n[8] Ivezić, Željko, et al. \"LSST: from science drivers to reference design and anticipated data products\" (2019)\n[9] Gardner, Jonathan P., et al. “The James Webb Space Telescope Mission” (2023).\n[10] Rombach, Robin, et al. \"High-resolution image synthesis with latent diffusion models\" (2022)\n[11] Karras, Tero, et al. \"Elucidating the design space of diffusion-based generative models\" (2022)\n[12] Wang, Zhendong, et al. \"Patch diffusion: Faster and more data-efficient training of diffusion models\" (2023)\n[13] Zhang, Zhaoyu, et al. “Training Diffusion-Based Generative Models with Limited Data” (2025)\n[14] Hur, Jiwan, et al. \"Expanding expressiveness of diffusion models with limited data via self-distillation based fine-tuning\" (2024)"}, {"review_id": "udiu1XD0WT", "reviewer": "Reviewer_BBqy", "summary": "This paper proposes a method that should work under a general MVSS framework, where the sources are independent, the observations are noisy, and are incomplete and vary in resolution. The paper learns a diffusion model for each source and each of these separate diffusion models can be used to perform sampling and evaluate likelihoods. The main algorithm consists of an EM algorithm where the expectation step consists of sampling posterior using the reverse SDE and the maximization step consists of updating the model weights. For sampling, the authors propose an improved version of Gibbs sampling that reduces the sampling time. The experiments consist of using synthetic 1D manifold data, grassy MNIST dataset in which the objective is to separate the MNIST digits from the background, and astronomy images where the goal is to separate galaxies from random background light. The results show that the proposed method outperforms previous SOTA methods in accuracy and fidelity metrics.", "questions": "1 Are there some other relevant comparison methods that could be used to further confirm the efficacy of the method, such as more classical methods? It would also be useful to show runtimies of these methods, as in my experience diffusion model based methods tend to have a much slower runtime but produce higher quality images. It would be interesting to see the tradeoff between runtime and image quality.\n\n2 Can any theoretical analysis be performed, either to bound the sampling error, show convergence toward distribution, or other theoretical guarantees? \n\n3 In Table 2, the FID scores of the comparison methods seem extremely bad. Are there any weaknesses of those comparison methods that make them bad, and is there any way to adjust these or other existing methods to at least obtain reasonable results?\n\n4 How does the method perform when certain parameters are adjusted? It would be useful to have some ablation studies that study the effects of individual components.\n\n5 The paper assumes that the observations are linear combinations of the sources. Could the proposed methods be adapted in any way to handle cases where the observations are nonlinear combinations? Similarly it appears that the noise is assumed to be Gaussian in the experiments, but could the method be applied if the noise was Poisson, like it is in many real world applications?\n\nOverall, the paper proposes an interesting method that shows promising results on a relevant problem and I am happy to revise my score should the concerns be addressed.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths: The proposed method appears to be fairly flexible and can handle noisy, incomplete, and varying resolution data. Leveraging diffusion models which are known to learn priors very well, the paper describes how to combine them with EM to provide a data-driven solution to the problem. The paper demonstrates visually and quantitatively how the method is applied to real world problems in different fields, and the writing is clear and easy to follow. \n\nWeaknesses: While the experiments performed and their corresponding settings are diverse, they are fairly sparse in detail. In particular, the tables of experimental results are small: more comparison methods could be included, including with more classical and non-deep learning methods. Furthermore, a greater range of experimental parameters for each setting could be observed: for example, different noise levels, image resolutions, etc. Also, a lack of ablation studies makes it difficult to determine whether each component of the model and algorithm helped improve the results significantly, or whether careful hyperparameter tuning is required to achieve stable results. Finally, there is a lack of theoretical analysis for the proposed algorithms.", "was_revised": true, "final_justification": "After reading the rebuttal, considering the additional experiments and results, and reading the other reviewer comments, I feel most of my concerns have been addressed so I increased my score.", "rebuttal": "We would like to thank the reviewer for their detailed feedback. Below we address the concerns and questions brought up within the review:\n\n### Additional Comparisons\n\nWe agree that the work would benefit from more baselines, metrics, and a fuller exploration of the baselines. To address this we ran three baselines, PCPCA, CLVM-Linear, and CLVM-VAE, on all the contrastive experiments. To our knowledge, these are the best baselines available. We have run the Gibbs sampling baseline where computationally feasible. Where applicable, we track PSNR, the peak signal-to-noise ratio, PQM, the pqmass metric introduced in [1], FID, the FID score [2] using our MNIST classifier, and SD, the Sinkhorn divergence.\n\n|||Posterior|||||Prior||\n|-|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|**Dataset / Method**|**PQM ↑**|**FID ↓**|**PSNR ↑**|**SD ↓**||**PQM ↑**|**FID ↓**|**SD ↓**|\n|**1D Manifold: Cont. 2 Sources**|||||||||\n|PCPCA [3]|0.0|--|9.35|7.69|¦|0.0|--|7.91|\n|CLVM - Linear [4]|0.0|--|9.58|5.80|¦|0.0|--|5.86|\n|CLVM - VAE [4]|0.0|--|17.15|1.81|¦|0.0|--|2.91|\n|DDPRISM-Gibbs [5]|0.0|--|12.66|3.96|¦|0.0|--|3.92|\n|DDPRISM-Joint [Ours]|**0.26**|--|**38.27**|**0.35**|¦|**0.01**|--|**0.37**|\n|**1D Manifold: Cont. 3 Sources**||||||||||\n|PCPCA [3]|0.0|--|6.89|12.57|¦|0.0|--|10.22|\n|CLVM - Linear [4]|0.0|--|11.64|2.03|¦|0.0|--|2.16|\n|CLVM - VAE [4]|0.0|--|13.09|2.22|¦|0.0|--|1.82|\n|DDPRISM-Gibbs [4]|0.0|--|9.50|4.50|¦|0.0|--|4.53|\n|DDPRISM-Joint [Ours]|0.0|--|**19.78**|**0.75**|¦|0.0|--|**0.78**|\n|**1D Manifold: Mix. ($f_{mix}=0.1$)**|||||||||\n|DDPRISM-Gibbs [5]|0.0|--|15.48|2.52|¦|0.0|--|2.43|\n|DDPRISM-Joint [Ours]|**0.001**|--|**24.15**|**0.05**|¦|0.0|--|**0.04**|\n|**GMNIST: Cont. Full Resolution**|||||||||\n|PCPCA [3]|0.0|22.30|18.99|--|¦|0.0|176.0|--|\n|CLVM - Linear [4]|0.0|101.3|13.30|--|¦|0.0|139.9|--|\n|CLVM - VAE [4]|0.0|18.87|14.56|--|¦|0.0|57.67|--|\n|DDPRISM-Joint [Ours]|**1.00**|**1.57**|**25.60**|--|¦|**0.20**|**20.10**|--|\n|**GMNIST: Cont. Downsampled**|||||||||\n|PCPCA [3]|0.0|121.7|14.08|--|¦|0.0|115.4|--|\n|CLVM - Linear [4]|0.0|199.5|12.16|--|¦|0.0|211.4|--|\n|CLVM - VAE [4]|0.0|1008.0|8.48|--|¦|0.0|737.0|--|\n|DDPRISM-Joint [Ours]|0.0|**60.34**|**17.09**|--|¦|0.0|**79.43**|--|\n\nTo ensure fair comparison, we ran an extensive hyperparameter sweep for the baseline methods. Similarly, we made small architectural changes to improve the diffusion models used in our method and reran all experiments. We did not run hyperparameter sweeps on our method due to time constraints for the rebuttal, but will do so for the camera-ready paper. Note that some of the metrics quoted in the original submission have changed, including improvements in the PCPCA performance that concerned the reviewer. However, there remains a substantial gap between the performance of the baselines and our model.\n\n### Runtime and Computational Cost\n\nWe provide a detailed comparison of the computational cost for each method in the table below:\n\n|||Training Time||¦||Inference Time per Sample||\n|-|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|**Method**|**1D Manifold**|**GMNIST Full Res.**|**Galaxy Images**|¦|**1D Manifold**|**GMNIST Full Res.**|**Galaxy Images**|\n|PCPCA [3]|5s|1m|--|¦|<0.1ms|1.5ms|--|\n|CLVM - Linear [4]|1.5m|10m|--|¦|<0.1ms|1ms|--|\n|CLVM - VAE [4]|18m|33m|--|¦|<0.1ms|1ms|--|\n|DDPRISM-Joint [Ours]|32h|68h|48h|¦| 22ms| 90ms|1.5s|\n\nAll timing was done on NVIDIA A100 GPUs. Our method is more computationally expensive than the baselines, almost entirely due to the cost of sampling from the diffusion model. However, this computational cost comes with significant improvements in performance and sample quality. In our ablation studies below we also compare the performance as a function of EM laps which can be used to control the tradeoff between training time and sample quality for our method.\n\nWe added a section to the appendix that includes the full timing comparison above and point to that section in our discussion and limitations section and in the results for each experiment. \n\n### Ablation Studies\n\nPer the reviewer’s suggestion, in order to clarify the importance of individual components of our method, we conducted an ablation study. We explored the effect of the following components on the performance of the method: (1) diffusion model architecture; (2) training length (in terms of the number of EM iterations), (3) initialization strategy, and (4) dataset size:\n|||Posterior|||||Prior||\n|-|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|**GMNIST Full Res.**|**PQM ↑**|**FID ↓**|**PSNR ↑**|**SD ↓**||**PQM ↑**|**FID ↓**|**SD ↓**|\n|**Model Architecture**|||||||||\n|MLP|0.05|49.03|17.93|--|¦|0.|215.36|--|\n|UNet, small|1.00|2.47|25.28|--|¦|0.06|**15.38**|--|\n|UNet, default|**1.00**|**1.57**|**25.60**|--|¦|**0.20**|20.10|--|\n|**Training Length (EM laps)**|||||||||\n|2|0.|96.85|16.62|--|¦|0.|199.70|--|\n|8|**1.00**|**0.04**|**27.15**|--|¦|0.14|**17.35**|--|\n|32|1.00|2.26|25.66|--|¦|0.08|27.96|--|\n|64 (Default)|1.00|1.57|25.60|--|¦|**0.20**|20.10|--|\n|**Initialization (# of Gaussian EM laps)** |||||||||\n|0 (Random)|0.97|10.41|23.35|--|¦|0.01|22.22|--|\n|4|**1.00**|**0.00**|26.80|--|¦|0.15|24.33|--|\n|16|**1.00**|**0.00**|**27.02**|--|¦|**0.21**|**6.31**|--|\n|64 (Default)|**1.00**|1.57|25.60|--|¦|0.20|20.10|--|\n|**Dataset Size**|||||||||\n|Full Dataset|**1.00**|**1.57**|**25.60**|--|¦|**0.20**|20.10|--|\n|1/4th Dataset|1.00|4.64|23.67| -- |¦|0.14|21.36|--|\n|1/16th Dataset|0.99|10.19|20.97|--|¦|0.07|**15.16**|--|\n|1/64th Dataset|0.0|38.34|15.34|--|¦|0.0|36.55|--|\n\nWe find that model architecture is important for performance: using an MLP-based architecture (5 hidden layers with 2048 hidden features per layer) gives poor results across all metrics. However, scaling down the UNet model (channels per level: (32,64,128)->(16,32), residual blocks per level: (2,2,2)->(2,2), embedding features: 64->16, attention head moved up one level) does not degrade performance appreciably. \n\nWe also find that the model achieves good performance after as few as 8 EM iterations and that longer training leads to a slight degradation in performance. In the table comparing our performance to the baselines we report the results for the model trained for the default number of EM laps. \n\nFor dataset size, we train our method using 1/4th, 1/16th, and 1/64th of the original grass and grass+MNIST datasets. The 1/16th and 1/64th runs are given 1/4th as many EM laps as the original training to account for the smaller dataset, but all other hyperparameters are unchanged. There is clear degradation in performance as the grass and MNIST datasets are reduced in size. However, even with 1/16th of the original dataset (2048 grass images, 512 MNIST digits) our method still outperforms the baselines run on the full dataset. \n\nSurprisingly, the number of Gaussian EM laps used to generate the initial samples (and train the initial diffusion model) has minimal impact on performance. Only starting from a randomly initialized model considerably reduces performance.\n\nWe added these ablation studies to the appendix, referred to them in Section 5.2, and discussed them in Section 6. If the reviewer would like additional ablations, we can incorporate them in the camera-ready version.\n\n### Linear Mixing and Gaussian Noise Assumptions\n\nAs it stands, our framework cannot easily be extended to the non-linear case without substituting our joint sampling for Gibbs sampling. It is likely possible to generalize our method further under a reasonable set of assumptions about the form of the mapping (i.e. a small perturbative expansion around linear), but we leave this for future work. We think the linear assumption is a reasonable constraint for the current paper given that our method is already more flexible than previous baseline methods.\n\nThe Gaussian noise assumption, however, can be relaxed within our framework. It would require access to “noise-only” views or the ability to sample from a known noise distribution. Essentially, one can consider the noise as an additional source and pre-train its corresponding diffusion model with the noise samples. From there, the rest of our machinery operates identically to the experiments presented in the paper, with the exception that the weights of the noise diffusion model are never updated. The only limitation is that the linear mixing assumption must be preserved. That is, noise that depends on the signal (for example, Poisson noise mentioned by the reviewer) cannot be modeled in this way. \n\n### Theoretical Bounds\n\nTo our knowledge, there are three relevant convergence guarantees for this method. First, in the limit of perfect optimization, infinite training examples, and a sufficiently expressive architecture, the score-based diffusion model is guaranteed to converge to the correct score [6,7]. In the limit of infinite sampling steps, our sampling method is guaranteed to sample from the learned prior [8,9]. Finally, substituting the prior score for the posterior score in Equation 6 is guaranteed to return posterior samples. However, we do not meet the requirements for any of these guarantees due to finite training dataset, finite sampling steps, and limitations of the MMPS approximation. We are unfortunately not aware of any relevant convergence bounds.\n\n[1] Lemos, Pablo, et al. (2024)\n[2] Heusel, Martin, et al. (2017)\n[3] Li, Didong, et al. (2020)\n[4] Severson, Kristen A., et al. (2019)\n[5] Heurtel-Depeiges, David, et al. (2024)\n[6] Hyvärinen, A. (2005)\n[7] Song, Yang et al. (2019)\n[8] Sohl-Dickstein, Jascha, et al. (2015)\n[9] Song, Yang, et al. (2020)"}, {"review_id": "zHewaj9J06", "reviewer": "Reviewer_G8fL", "summary": "The paper introduces a multi-view source separation (MVSS) method using diffusion model priors. The method learns on data where observations consist of different linear transformations of the unknown sources. The approach works by training separate diffusion models for each source, facilitating sampling and probability estimation. The method is able to handle noisy, incomplete, and observations with varied resolution without requiring explicit source assumptions. It employs an expectation-maximisation framework with moment matching posterior sampling (MMPS) to learn source priors and outperforms existing methods like PCPCA and CVAE in synthetic and real-world datasets. Experimental results demonstrate state-of-the-art performance on one-dimensional manifold and Grassy MNIST datasets, as well as real-world galaxy image separation from the COSMOS survey.", "questions": "- Why does their diffusion-based approach work so effectively for separating linear mixtures of sources? This is not fully addressed, e.g. in terms of specific mechanisms related to diffusion models or architectural inductive biases. While the role of diffusion models as expressive priors and the importance of the multi-view setup is emphasised, but the work lacks a detailed explanation of how the diffusion process or the architectural choices aid the separation of sources. A more explicit discussion of the interaction between the linear mixing model, the diffusion process, and the architectural biases would strengthen the response to this question. Furthermore ablations could reinforce any conclusions or speculations on this.\n- Could the authors add additional results from other real-world experiments (beyond galaxy separation)? This could help demonstrate the general utility and applicability of the method and broaden impact. E.g. perhaps showing demonstrating results on cell separation in neuroscience or sound source separation etc.\n - Could the work be applied in situations where the mixing matrix is unknown? And have the authors considered whether the method could be meaningfully extended to datasets with non-linear mixtures of sources? Could the method be meaningfully applied in such scenarios, or are there constraints in the assumptions made that preclude this? Such extensions could significantly broaden the applicability of the approach across diverse scientific and real-world domains.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The paper introduces a novel method that uses score-based diffusion models for multi-view source separation, bypassing the need for explicit source assumptions. The approach is innovative as it allows for flexible modelling of complex source distributions, with the potential to disentangle signals in diverse scientific applications. And it leverages the capability of diffusion models to sample complex joint posteriors.\n\nThe paper is well-written, with a clear structure, and presents technical concepts and experimental results in a coherent manner. This work has the potential to advance transform source separation tasks across scientific domains, such as astronomy, neuroscience, and seismology, where disentangling mixed signals is critical for scientific progress. The paper demonstrates its method’s effectiveness primarily through a limited set of experiments, including synthetic 1D manifold and Grassy MNIST datasets and one real-world galaxy image separation task. This limits the diversity of tested scenarios and may not fully showcase the method’s generalisability across varied real-world applications. Additionally, the absence of explicit ablation studies, such as testing variations in the diffusion model architecture, sampling strategy, or EM framework components, limits insights into the specific contributions of each element, weakening the understanding of why the method excels for linear mixtures.\n\nWhile effective, their method is computationally intensive and assumes linear observation models, limiting its applicability to datasets with non-linear characteristics. The authors discuss and address the computational limitations, including heavy compute requirements and being data hungry. They also provide guidance on where existing techniques would be better suited. But the main paper does not provide a straightforward quantification of what the extra burden is relative to other methods.", "was_revised": true, "final_justification": "The authors addressed most of my and other reviewer concerns raised. Importantly the authors provided additional baselines and evaluation metrics, ablations results, computational cost analyses, and also shared additional discussion on the effectiveness and limitations of their approach. However, the authors were unable to provide results on other real-world datasets beyond galaxy observations due to time constraints. While adding such results would further extend the scope and impact of this interesting method, the work is now solid enough rate it Accept.", "rebuttal": "We would like to thank the reviewer for their detailed comments and feedback. Below we address the concerns and questions brought up within the review:\n### Additional Results\nRunning on a fourth dataset was not feasible for the rebuttal, but we ran a wider set of baselines and evaluation metrics to better compare the models performance:\n|||Posterior|||||Prior||\n|-|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|**Dataset / Method**|**PQM ↑**|**FID ↓**|**PSNR ↑**|**SD ↓**||**PQM ↑**|**FID ↓**|**SD ↓**|\n|**1D Manifold: Cont. 2 Sources**|||||||||\n|PCPCA [1]|0.0|--|9.35|7.69|¦|0.0|--|7.91|\n|CLVM - Linear [2]|0.0|--|9.58|5.80|¦|0.0|--|5.86|\n|CLVM - VAE [2]|0.0|--|17.15|1.81|¦|0.0|--|2.91|\n|DDPRISM-Gibbs [3]|0.0|--|12.66|3.96|¦|0.0|--|3.92|\n|DDPRISM-Joint [Ours]|**0.26**|--|**38.27**|**0.35**|¦|**0.01**|--|**0.37**|\n|**1D Manifold: Cont. 3 Sources**||||||||||\n|PCPCA [1]|0.0|--|6.89|12.57|¦|0.0|--|10.22|\n|CLVM - Linear [2]|0.0|--|11.64|2.03|¦|0.0|--|2.16|\n|CLVM - VAE [2]|0.0|--|13.09|2.22|¦|0.0|--|1.82|\n|DDPRISM-Gibbs [3]|0.0|--|9.50|4.50|¦|0.0|--|4.53|\n|DDPRISM-Joint [Ours]|0.0|--|**19.78**|**0.75**|¦|0.0|--|**0.78**|\n|**1D Manifold: Mix. ($f_{mix}=0.1$)**|||||||||\n|DDPRISM-Gibbs [3]|0.0|--|15.48|2.52|¦|0.0|--|2.43|\n|DDPRISM-Joint [Ours]|**0.001**|--|**24.15**|**0.05**|¦|0.0|--|**0.04**|\n|**GMNIST: Cont. Full Resolution**|||||||||\n|PCPCA [1]|0.0|22.30|18.99|--|¦|0.0|176.0|--|\n|CLVM - Linear [2]|0.0|101.3|13.30|--|¦|0.0|139.9|--|\n|CLVM - VAE [2]|0.0|18.87|14.56|--|¦|0.0|57.67|--|\n|DDPRISM-Joint [Ours]|**1.00**|**1.57**|**25.60**|--|¦|**0.20**|**20.10**|--|\n|**GMNIST: Cont. Downsampled**|||||||||\n|PCPCA [1]|0.0|121.7|14.08|--|¦|0.0|115.4|--|\n|CLVM - Linear [2]|0.0|199.5|12.16|--|¦|0.0|211.4|--|\n|CLVM - VAE [2]|0.0|1008.0|8.48|--|¦|0.0|737.0|--|\n|DDPRISM-Joint [Ours]|0.0|**60.34**|**17.09**|--|¦|0.0|**79.43**|--|\n\nPSNR is the peak signal-to-noise ratio, PQM is the pqmass metric introduced in [4], FID is the FID score [5] using our MNIST classifier, and SD is the sinkhorn divergence. Note that we’ve replaced the CVAE [6] baseline with CLVM-VAE [2] since the latter is a superset of the former and achieves better performance. Across the experiments, we find that our method substantially outperforms the baselines.\n\n### Runtime and Computational Cost\n\nIn addition, we provide a detailed comparison of the computational time for each method in the table below:\n\n|||Training Time||¦||Inference Time per Sample||\n|-|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|**Method**|**1D Manifold**|**GMNIST Full Res.**|**Galaxy Images**|¦|**1D Manifold**|**GMNIST Full Res.**|**Galaxy Images**|\n|PCPCA [1]|5s|1m|--|¦|<0.1ms|1.5ms|--|\n|CLVM - Linear [2]|1.5m|10m|--|¦|<0.1ms|1ms|--|\n|CLVM - VAE [2]|18m|33m|--|¦|<0.1ms|1ms|--|\n|DDPRISM-Joint [Ours]|32h|68h|48h|¦| 22ms| 90ms|1.5s|\n\nAll timing was done on NVIDIA A100 GPUs. Our method is more computationally expensive than the baselines, almost entirely due to the cost of sampling from the diffusion model. However, this computational cost comes with significant improvements in performance and sample quality. In our ablation studies below we also compare the performance as a function of EM laps, which can be used to control the tradeoff between training time and sample quality for our method. \n### Effectiveness on Linear Source Separation\nThe effectiveness of our method is driven by the ability to directly sample from the posterior given our current diffusion model prior (Eq.17). Because the likelihood is often constraining, this enables us to return high-quality posterior samples even when the prior specified by our diffusion model is not an optimal fit to the empirical distribution. As evidence for this point, we note the large gap in performance between posterior and prior samples on the Grassy MNIST experiments.\n### Ablation Studies\nPer reviewer’s suggestion, in order to clarify the importance of individual components of our proposed method, we conducted an ablation study. In particular, we explored the effect of the following components on the performance of our method: (1) diffusion model architecture; (2) training length (in terms of the number of EM iterations), (3) initialization strategy, and (4) dataset size:\n|||Posterior|||||Prior||\n|-|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|**GMNIST Full Res.**|**PQM ↑**|**FID ↓**|**PSNR ↑**|**SD ↓**||**PQM ↑**|**FID ↓**|**SD ↓**|\n|**Model Architecture**|||||||||\n|MLP|0.05|49.03|17.93|--|¦|0.|215.36|--|\n|UNet, small|1.00|2.47|25.28|--|¦|0.06|**15.38**|--|\n|UNet, default|**1.00**|**1.57**|**25.60**|--|¦|**0.20**|20.10|--|\n|**Training Length (EM laps)**|||||||||\n|2|0.|96.85|16.62|--|¦|0.|199.70|--|\n|8|**1.00**|**0.04**|**27.15**|--|¦|0.14|**17.35**|--|\n|32|1.00|2.26|25.66|--|¦|0.08|27.96|--|\n|64 (Default)|1.00|1.57|25.60|--|¦|**0.20**|20.10|--|\n|**Initialization (# of Gaussian EM laps)** |||||||||\n|0 (Random)|0.97|10.41|23.35|--|¦|0.01|22.22|--|\n|4|**1.00**|**0.00**|26.80|--|¦|0.15|24.33|--|\n|16|**1.00**|**0.00**|**27.02**|--|¦|**0.21**|**6.31**|--|\n|64 (Default)|**1.00**|1.57|25.60|--|¦|0.20|20.10|--|\n|**Dataset Size**|||||||||\n|Full Dataset|**1.00**|**1.57**|**25.60**|--|¦|**0.20**|20.10|--|\n|1/4th Dataset|1.00|4.64|23.67| -- |¦|0.14|21.36|--|\n|1/16th Dataset|0.99|10.19|20.97|--|¦|0.07|**15.16**|--|\n|1/64th Dataset|0.0|38.34|15.34|--|¦|0.0|36.55|--|\n\nWe find that the model achieves good performance after as few as 8 EM iterations and that longer training leads to a slight degradation in performance. In the table comparing performance to baselines we report the results for the model trained for the default number of EM laps. For dataset size, we train our method using 1/4th, 1/16th, and 1/64th of the original grass and grass+MNIST datasets. The 1/16th and 1/64th runs are given 1/4th as many EM Laps as the original training to account for the smaller dataset, but all other hyperparameters are unchanged. There is clear degradation in performance as the grass and MNIST datasets are reduced in size. However, even with 1/16th of the original dataset (2048 grass images, 512 MNIST digits) our method still outperforms the baselines run on the full dataset. Surprisingly, the number of Gaussian EM laps used to generate the initial samples (and therefore train the initial diffusion model) has minimal impact on performance. Only starting from a randomly initialized model considerably reduces performance.\n\nWe also find that model architecture has significant impact on performance: using an MLP-based architecture (5 hidden layers with 2048 hidden features per layer) gives poor results across all metrics. However, scaling down the UNet model (channels per level: (32, 64, 128) -> (16, 32), residual blocks per level: (2, 2, 2) -> (2, 2), embedding features: 64 ->16, attention head moved up one level) does not degrade performance appreciably. \n\nHowever, the inductive bias of the UNet architecture alone is unlikely to be the cause of our model’s strong performance. In our extended baseline comparison above we contrast our performance with a CLVM implementation that uses neural networks as encoders and decoders. After running an extensive hyperparameter sweep, we found that using MLPs over UNet-style convolutional encoders / decoders led to a substantial improvement in performance. If the UNet inductive bias was essential, it would be surprising to find the CLVM did not benefit from it. We added a summary of this discussion in Section 6.\n### Linear Mixing and Known Mixing\nWe agree with the reviewer that the principle limitation of our method is that it requires linear mixing. As it stands, our framework cannot easily be extended to the non-linear case without substituting our joint sampling for Gibbs sampling. It is likely possible to generalize our method further under a reasonable set of assumptions, but we cannot explore those extensions within the timeframe of the rebuttal. However, we think the linear assumption is a reasonable constraint for the current work given that our method is already more flexible than previous baseline methods. For example, we make no source assumptions, do not require contrastive observations, and can (theoretically) separate an arbitrary number of sources within a single observation.\n\nIf we place no constraints on the mixing matrix, the problem becomes fully degenerate. Blind source separation methods must make strong assumptions about the sources in order to break this degeneracy [7]. However, it could be possible to incorporate a known prior distribution on the mixing matrix. One could then approximate each evaluation of Eq. 17 with a single Monte Carlo sample, and rely on the large dataset size and multiple training epochs to expose the model to the full mixing matrix distribution. Unfortunately, we were unable to run an experiment along these lines in the time allotted to the rebuttal.\n\n### Additional notes\n\n* To ensure fair comparison, we ran a hyperparameter sweep for the baseline methods. Similarly, we made small architectural changes to improve the diffusion models used in our method and reran all experiments. Therefore, some of the metrics quoted in the table changed from our original submission.    \n\n[1] Li, Didong, et al. \"Probabilistic contrastive principal component analysis” (2020)\n\n[2] Severson, Kristen A., et al. “Unsupervised learning with contrastive latent variable models” (2019)\n\n[3] Heurtel-Depeiges, David, et al. \"Listening to the noise: Blind denoising with Gibbs diffusion\" (2024)\n\n[4] Lemos, Pablo, et al. \"Pqmass: Probabilistic assessment of the quality of generative models using probability mass estimation\" (2024)\n\n[5] Heusel, Martin, et al. \"GANs trained by a two time-scale update rule converge to a local Nash equilibrium\" (2017)\n\n[6] Abid, Abubakar, et al. “Contrastive Variational Autoencoder Enhances Salient Features” (2019)\n\n[7] Hyvärinen, Aapo, et al. \"Independent component analysis: algorithms and applications\" (2000)"}, {"review_id": "ZcT8OeBHJc", "reviewer": "Reviewer_FLez", "summary": "This paper proposes a novel diffusion-based multi-view source separation algorithm. The authors introduce an EM algorithm that alternates between estimating the separation results and updating the diffusion model parameters, gradually improving both the separation quality and the accuracy of the prior modeling. In the E-step, a diffusion-based posterior sampling method is employed to infer the separated sources from the observations. In the M-step, the separated results are used as an estimate of the clean samples to update the diffusion parameters. The effectiveness of the proposed method is validated on both synthetic and real-world datasets.", "questions": "While I am not an expert in this area, I'm curious about the data requirements for training diffusion models. It is well known that diffusion models typically require large-scale training data. In practical multi-view source separation (MVSS) tasks, is it realistic to expect such a volume of data to be available? The reviewer notes that in this paper, most synthetic datasets reach the scale of 10k samples, even though the data is relatively low-dimensional (e.g., 5D vectors or 28×28 images). Can this scale of data be available in real-world MVSS scenarios?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths:\n\n- The paper is clearly written and easy to follow.\n- The simulation experiments suggest that the proposed algorithm consistently outperforms the baselines across all scenarios. (I would like to hedge this claim as I am not entirely familiar with the domain.)\n- The problem tackled in this paper is meaningful, and the proposed solution is well-motivated and clearly structured.\n\n### Weaknesses:\n\n- One concern from the reviewer is about the initialization of the diffusion model. According to the provided pseudocode and description, the algorithm first performs posterior sampling using the diffusion model to obtain an initial separation result, which is then used to update the diffusion model in subsequent EM steps. However, if the initial diffusion model is randomly initialized and does not model the prior well, how can the first posterior sampling step be reliable? In other words, if the diffusion model starts from random weights, I believe the posterior sampling algorithm would generate meaningless noise, making it difficult to proceed with the following M and E steps. The reviewer would appreciate it if the authors could clarify how the diffusion model is initialized or pre-trained, or point out if there is a misunderstanding about the algorithmic pipeline. This is important for assessing the feasibility and strengths of the proposed method.\n- The reviewer suggests adding an ablation study beyond the toy example to evaluate how increasing or decreasing the number of EM iterations affects modeling performance. This could further enhance the understanding of the proposed approach.\n- Minor error: Line 217: it should be $\\mathbb{R}^{3\\times 5}$ instead of $\\mathbb{R}^{3x5}$.\n\nThe reviewer looks forward to further discussion.", "was_revised": true, "final_justification": "I believe the authors have clearly addressed the concerns regarding the Diffusion Model initialization and dataset size, particularly clarifying that the size of real-world datasets is sufficient to support effective training of Diffusion Models, which ensures the feasibility of the proposed method. I have also read the other reviewers' comments and the authors’ responses, and I think the additional experiments and discussions have substantially improved the quality of the paper. I raise my score to 5, and I believe the methods and findings in this paper could be useful to the NeurIPS audience.", "rebuttal": "We would like to thank the reviewer for their detailed comments and feedback. Below we address the concerns and questions brought up within the review:\n### Initialization\n \nAs the reviewer points out, the results of EM can depend heavily on the quality of the initialization. EM reaches a local but not necessarily global optimum. To mitigate this issue, we follow [1] and start by running an EM procedure that optimizes the parameters of a simple Gaussian diffusion model (mean and covariance). These approximate samples are then used to train our initial full diffusion model, which we then plug into our EM loop. We agree that exploring the importance of this initialization procedure helps contextualize our results, so we ran an ablation study on the Grassy MNIST experiment where we test random initialization compared to initialization using a Gaussian diffusion model optimized over 4, 16, and 64 (the default) EM laps. Below we show the results of our ablation study:\n\n|||Posterior|||||Prior||\n|-|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|**GMNIST Full Res.**|**PQM ↑**|**FID ↓**|**PSNR ↑**|**SD ↓**||**PQM ↑**|**FID ↓**|**SD ↓**|\n|**Initialization (# of Gaussian EM laps)** |||||||||\n|0 (Random)|0.97|10.41|23.35|--|¦|0.01|22.22|--|\n|4|**1.00**|**0.00**|26.80|--|¦|0.15|24.33|--|\n|16|**1.00**|**0.00**|**27.02**|--|¦|**0.21**|**6.31**|--|\n|64 (Default)|**1.00**|1.57|25.60|--|¦|0.20|20.10|--|\n\nWe find that the number of Gaussian EM laps used to generate the initial samples (and therefore train the initial diffusion model) has minimal impact on performance. Only starting from a randomly initialized model considerably reduces performance. While this result may seem surprising, we believe it can be explained by our ability to directly sample from the posterior given our current diffusion model prior (Eq.17). Because the likelihood is often constraining, this enables us to return high-quality posterior samples even when the prior specified by our diffusion model is not an optimal fit to the empirical distribution. As evidence for this point, we note the large gap in performance between posterior and prior samples on the Grassy MNIST experiment. This means that even if the initial prior fit is poor, the resulting posterior samples will likely not be random noise. We included this ablation study in the appendix of our work and pointed to it in Sections 5.2 and 6. We also reworked the introduction of our method in Section 4 and the pseudocode to highlight the use of the Gaussian denoiser initialization.\n \n### EM Laps\n \nWe agree that a full exploration of the performance of our model as a function of EM laps is important, especially given the higher computational cost of our model compared to the baseline methods. Beyond the performance on the one-dimensional manifold problem shown in Figures 1 and 2, we also added a comparison of our model’s performance on the Grassy MNIST experiment as a function of EM laps to the appendix. We summarize the results below:\n \n|||Posterior|||||Prior||\n|-|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|**GMNIST Full Res.**|**PQM ↑**|**FID ↓**|**PSNR ↑**|**SD ↓**||**PQM ↑**|**FID ↓**|**SD ↓**|\n|**Training Length (EM laps)**|||||||||\n|2|0.|96.85|16.62|--|¦|0.|199.70|--|\n|8|**1.00**|**0.04**|**27.15**|--|¦|0.14|**17.35**|--|\n|32|1.00|2.26|25.66|--|¦|0.08|27.96|--|\n|64 (Default)|1.00|1.57|25.60|--|¦|**0.20**|20.10|--|\n\nWe find that the model achieves good performance after as few as 8 EM iterations and that longer training leads to a slight degradation in performance. In the table comparing performance to baselines we report the results for the model trained for the default number (64) of EM laps. \n\nFor completeness, we also added a Figure to the appendix showing how the visual performance of the galaxy experiment varies as a function of EM Laps. However, given the restrictions of the rebuttal, we cannot reproduce that here.\n \n### Dataset Size\n \nThe reviewer brings up a valid concern about the scale of the dataset required to successfully fit a diffusion model to a distribution. Some previously explored solutions to reduce the dataset size requirement include data augmentation [2], patch-wise diffusion training [3], latent-space diffusion models [4,5], and transfer learning [6]. It would be possible to incorporate these methods during our maximization step.\n \nTo better quantify our dependence on the dataset size, we ran a set of ablation studies on our performance on the full-resolution Grassy MNIST experiment. We train our method using 1/4th, 1/16th, and 1/64th of the original grass and grass+MNIST datasets. The 1/16th and 1/64th runs are given 1/4th as many EM Laps as the original training setup to account for the smaller dataset, but all other hyperparameters are unchanged:\n \n|||Posterior|||||Prior||\n|-|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|**GMNIST Full Res.**|**PQM ↑**|**FID ↓**|**PSNR ↑**|**SD ↓**| |**PQM ↑**|**FID ↓**|**SD ↓**|\n|Full Dataset|**1.00**|**1.57**|**25.60**|--|¦|**0.20**|20.10|--|\n|1/4th Dataset|1.00|4.64|23.67| -- |¦|0.14|21.36|--|\n|1/16th Dataset|0.99|10.19|20.97|--|¦|0.07|**15.16**|--|\n|1/64th Dataset|0.0|38.34|15.34|--|¦|0.0|36.55|--|\n \nThere is a clear degradation in performance as the grass and MNIST datasets are reduced in size. However, even with 1/16th of the original dataset (2048 grass images, 512 MNIST digits) our method still outperforms the baselines run on the full dataset. We added the dataset scaling study to the appendix and referenced it in the Discussion and Limitations section.\n \nIn terms of whether these dataset sizes are realistic for real world applications, we think the galaxy experiment serves as a relevant example. Using just the HST COSMOS field, we have millions of random cutouts and ~80k galaxy image examples. If we incorporate ground-based telescopes or future space-based telescopes, we can access orders-of-magnitude more random and galaxy datapoints [7].\n\n### Additional notes\n\n* To ensure fair comparison, we ran a hyperparameter sweep for the baseline methods. Similarly, we made small architectural changes to improve the diffusion models used in our method and reran all experiments. Therefore, some of the metrics quoted in the tables above changed from our original submission.\n\n[1] Rozet, François, et al. \"Learning diffusion priors from observations by expectation maximization\" (2024)\n\n[2] Karras, Tero, et al. \"Elucidating the design space of diffusion-based generative models\" (2022)\n\n[3] Wang, Zhendong, et al. \"Patch diffusion: Faster and more data-efficient training of diffusion models\" (2023)\n\n[4] Zhang, Zhaoyu, et al. “Training Diffusion-Based Generative Models with Limited Data” (2025)\n\n[5] Rombach, Robin, et al. \"High-resolution image synthesis with latent diffusion models\" (2022)\n\n[6] Hur, Jiwan, et al. \"Expanding expressiveness of diffusion models with limited data via self-distillation based fine-tuning\" (2024)\n\n[7] Mellier, Y., et al. \"Euclid. I. Overview of the Euclid mission\" (2024)"}]

Author rebuttal:

Meta-review: The paper received uniformly positive reviews with scores of 4, 5, 5, 5 (mean: 4.75). Reviewers praised the novelty of introducing diffusion priors for multi-view source separation (zJxX, BBqy, G8fL, FLez), the principled EM-based framework (zJxX, G8fL), strong empirical results across both synthetic and real-world datasets (zJxX, BBqy, G8fL), and clarity of presentation (BBqy, FLez). Initial concerns on scalability, evaluation breadth, ablations, and initialization were effectively addressed in the rebuttal with additional experiments, runtime analyses, and expanded discussion of limitations.

After thoroughly reviewing the paper, the reviews, and the authors’ rebuttal, the AC concurs with the reviewers’ highly positive consensus and therefore recommends the paper for **acceptance**.

For the camera-ready version, the authors should ensure all clarifications and additional results from the rebuttal are incorporated into the main paper and supplementary material. In particular:

**1. Evaluation (BBqy, G8fL):** Integrate the extended baseline comparisons and expanded metrics in the rebuttal directly into the main text.

**2. Ablations and initialization (FLez, BBqy, G8fL):** Summarize the ablation results on architecture, EM iterations, initialization strategies, and dataset size to clarify the robustness of the approach.

**3. Computational cost (zJxX, G8fL, BBqy):** Include the runtime and memory comparisons with baselines, along with the discussion of scalability to large-scale datasets (e.g., astronomy).

**4. Limitations and scope (zJxX, G8fL):** Retain the discussion on linear mixing and Gaussian noise assumptions, and highlight possible extensions or constraints explicitly in the conclusion.
Decision: Accept (poster)