Title: Score-Based Diffusion Modeling for Nonparametric Empirical Bayes in Heteroscedastic Gaussian Mixtures
Venue: NeurIPS 2025
Accepted: yes
Keywords: empirical Bayes

Gaussian location mixture model

g-modeling

heteroscedastic noise

diffusion processes

score matching
Abstract: We propose a generalized score-based diffusion framework for learning multivariate Gaussian mixture models with homoscedastic or heteroscedastic noise. Our goal is to nonparametrically estimate the latent location distribution and denoise the observations. 
Departing from the conventional maximum likelihood approach, we reinterpret each observation as a temporal slice of a family of stochastic diffusion processes. This modeling choice enables a principled characterization of the additive noise structure and supports a multi-step denoising procedure grounded in reverse-time dynamics. We introduce a score-based objective that explicitly models the latent distribution and accommodates observation-specific noise covariances.
Theoretically, we establish that the score estimation error with $n$ independent observations achieves a near-parametric error rate of $\frac{\mathrm{polylog}(n)}{n}$, 
improving upon existing results in the diffusion literature. Empirically, our method outperforms the nonparametric maximum likelihood estimator in both density estimation and denoising fidelity,  especially in high-dimensional settings.
These findings suggest a promising direction for integrating nonparametric empirical Bayes with diffusion-based generative modeling for latent structure recovery.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a method to integrate nonparametric empirical Bayes with score-based diffusion generative  models. The goal is to estimate an unknown prior distribution from noisy observations. In doing so, the paper addresses both statistical and computational limitations of the classical approach, which combines the nonparametric maximum likelihood estimator (NPMLE) and Tweedie-based posterior denoising. The proposed method generalizes and improves upon existing approaches, which either do not account for heteroscedastic measurement noise or lack theoretical guarantees in dimensions higher than $1$. \n\nThe key idea is to reinterpret each noisy observation as a time-1 snapshot from a latent stochastic diffusion process. This perspective naturally induces a denoising score matching objective for explicitly learning the prior distribution $G^*$. The resulting optimization problem is infinite-dimensional and nonconvex. The posterior mean is then estimated by combining this approach with a multi-step Tweedie-type empirical Bayes denoising procedure. \n\nAssuming access to the minimizer of the non-convex optimization problem, the paper provides theoretical guarantees on the statistical performance of the method. Finally, numerical experiments on synthetic and real-world datasets are presented to validate the theory, with particular attention paird to robustness as the dimension of the problem varies.\n\nSTRENGTHS AND WEAKNESSES:\n**Strenght:** \n\nThe paper is well-written and provides both a theoretical treatment and numerical experiments that illustrate the theory.  One interesting feature of the analysis is that the error bounds highlight the interplay between the number of observations and the dimensionality of the problem – something that previous literature didn’t address in this context. Finally, some assumptions are needed to make the analysis clean; these are carefully discussed with reference to prior literature.\n\n\n**Weakness:** \n\nTo my understanding, the paper does not provide strong algorithmic guarantees. To me, this is not a major issue, since the authors are transparent about their assumptions; however, these limitations should be discussed explicitly in a “Limitations” section, which is currently missing. For example, assuming solvability of the nonconvex problem is, I believe (though I am happy to change my mind based on the authors’ response), quite strong, as the optimization can easily get stuck in a local minimum. It would also be interesting to incorporate in the error analysis of Theorem 1-2 the fact that $\\widehat{G}$ may not be a good  approximation of the minimizer of the nonconvex objective. \n\nI also noticed a typo on line 182: a repeated “that”.\n\nQUESTIONS:\n- I appreciate a lot how the paper is organized. However, I wonder why the authors do not emphasize more strongly the result regarding the interplay between $n$ and $d$ in the presentation, since dimension plays a role also in the experiments. \n\n- I am confused by the example about the over-shrinkage bias. Isn't the reduced variance simply a consequence of observing $X$, which provides information and hence reduces uncertainty in the posterior estimation? \n\n- How difficult would it be to quantitatively incorporate the optimization error of the nonconvex estimator into the statistical analysis of the method?\n\nLIMITATIONS:\nSee Weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper propose Score-based Diffusion with G-Modeling for non-parametric empirical Bayes in multivariate heteroscedastic Gaussian‐location mixtures. The key idea is to reinterpret each noisy observation as the time-1 snapshot of a forward SDE whose latent initial state is drawn from an unknown prior G. This perspective yields a reverse-time ODE (Eq. 7/9) that supplies a multi-step denoising rule generalising Tweedie’s one-shot formula. For the homoscedastic case the authors prove that the plug-in estimator achieves near-parametric Fisher-risk and Hellinger error O(polylog(n)/n) and Wasserstein error O(1/logn) (Thm 1). They extend the guarantees to heteroscedastic noise (Thm 2). Two practical solvers are described: a discrete-support EM-style optimiser and a continuous normalising-flow parameterisation. Experiments on synthetic and real-world data show improvement over NPMLE baselines.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. They paper proposes a reverse-time denoising procedure and score-matching objective that yield near-parametric Fisher-risk and Hellinger error rates, improving on prior diffusion-model analyses as well as NPMLE estimators. \n\n2. Practical denoising advantage – Multi-step reverse denoiser corrects the classic Tweedie over-shrinkage and demonstrates tangible MSE gains in experiments.\n\n3. Empirical results – On synthetic and real data, SDGM achieves lower Fisher divergence and negative log-likelihood than NPMLE and PEM baselines, supplementing their theory results.\n\nWeaknesses\n1. Requires compact support, well conditioned heteroscedastic covariances, but it is not clear why the assumption holds in practice.\n\n2. Error bounds carry $log^d(n)$ factors; the actual exponent is unspecified, so guarantees may become impractical for large $d$.\n\n3.Each gradient step touches all n observations (or all obsevations in the batch); the paper provides no wall-clock or memory benchmarks, leaving practical scalability unclear. In context of Diffusion models, there was a recent [paper ](https://arxiv.org/pdf/2310.12395) which tried to come up with a non-parametric form of the score function, but they faced certain challenges when compared with neural networks regarding scalability and overfitting. I am curious to see if the rich representation power of neural networks, that has been applied very successfully with diffusion models for score representation, could be applied here as well.\n\n4. Experiments do not compare against recent diffusion-based mixture learners mentioned in the manuscript. It would help to state why these methods don't apply to the authors' experiments directly.\n\nQUESTIONS:\nSee weaknesses above for points about the paper which I am concerned about. I would appreciate clarifications on them to increase my score.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposed a non-parametric empirical Bayes approach, which is based on a generalized score-based diffusion framework, for learning multivariate Gaussian mixture models with homoscedastic or heteroscedastic noise. The main idea is to view each observation as temporal slices of a family of stochastic diffusion process, which enables a multi-level score-based objective for learning the latent location distribution. Theoretically, a near-parametric error rate was established for the score estimation error. They also show that the proposed method outperforms the non-parametric maximum likelihood estimator in both density estimation and denoising fidelity.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n\n- The use of score matching, rather than the classical NPMLE, to estimate the prior within the empirical Bayes framework via G-modeling is novel and insightful.\n- The introduction of a multi-step denoising procedure based on reverse-time diffusion dynamics offers a principled generalization of the traditional one-step Tweedie estimator, mitigating over-shrinkage bias.\n- The theoretical analysis is rigorous and well-developed, providing near-parametric guarantees for score estimation and extending to the more realistic heteroscedastic noise setting.\n\nWeakness:\n\n- While the overall methodological development is well-presented, the paper lacks a clear explanation of how the marginal density $f_{G,\\Sigma_i}$  computed in practice. Given its central role in both the score matching objective and the denoising procedure, a more explicit description would be helpful for reproducibility and understanding.\n- The rationale for preferring score matching over nonparametric maximum likelihood estimation (NPMLE) is not fully justified. As mentioned around Line 233, the convergence rate achieved by score matching merely matches that of NPMLE. Moreover, it seems plausible that NPMLE could also be applied to perturbed observations across multiple noise levels, as in the multi-step denoising setting. Since score matching still relies on computing the marginal score, it is not immediately clear what practical advantages it offers in this context. A deeper comparison—both theoretical and empirical—would strengthen the case for using score-based methods.\n\nTypo:\n- Line 69: scoring matching  -> score matching\n\nQUESTIONS:\n-\tIn the real-world astronomy experiment, what constitutes an astrochemically meaningful structure in the denoised output? Is the goal to recover scientifically interpretable features, or is sharpness alone the main criterion for success? Some clarification on the downstream relevance would help assess the practical value of the method.\n-\tThe constants hidden in Theorems 1 and 2 are stated to depend on the dimension ddd. Could you specify how this dependence scales (e.g., linearly, polynomially, or exponentially)? Also, how does the method behave in higher dimensions? The experiments are limited to $d=19$ -- do you anticipate scalability to much higher dimensions?\n-\tHow is the upper bound $T$ for the diffusion time chosen in practice? Does its value affect the convergence rate or the bias-variance tradeoff in the score estimation? Any practical heuristics or theoretical guidance would be appreciated.\n\nLIMITATIONS:\nThe main limitations are the computation of the marginal density $f$. Additionally, the paper lacks discussion and empirical validation of performance in higher-dimensional settings, limiting insight into the method’s scalability.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a novel framework that integrates score-based diffusion models with G-modeling for empirical Bayes estimation, particularly targeting heteroscedastic Gaussian mixture models. The authors reinterpret each noisy observation as a terminal state of a diffusion process, enabling the estimation of a nonparametric prior distribution $G^*$ without assuming parametric forms.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper delivers a technically sound and mathematically sophisticated contribution to both diffusion modeling and empirical Bayes literature.The use of score matching in G-modeling is novel and theoretically grounded, with convergence results rigorously derived. As far as I know, this is the first work to extend score-based methods to a heteroscedastic Gaussian mixture setting with provable statistical guarantees.\n\nThe only weakness I see is that the main theoretical result (Theorem 1 in Section 3) is not fully interpretable in its current form: the constant dependencies (on $\\sigma$) in the bound are not made explicit, making it difficult to assess practical implications or generality. Also, the assumptions are not verified or illustrated by the experiments. See details below in the Question part.\n\nI would raise my score if the authors are able to (1) clarify the structure and implications of Theorem 1 with precise dependencies and (2) confirm whether the assumptions are satisfied in the presented synthetic and real-data examples.\n\nQUESTIONS:\nI have some questions regarding Theorem 1:\n\n1. The requirement to target distribution is only 'compact support'. If we have better property, like $G^*$ has strongly-convex score, how will the bound in Theorem improve?\n\n2. The noise is supposed to be lower bounded by $\\underline{\\sigma}$. Do you require $\\underline{\\sigma}$ to be strictly positive? Or how the result depends on the condition number of $\\Sigma_i$? Also, can you verify these assumptions by some examples in section 5?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-03"}