Title: Beyond Scores: Proximal Diffusion Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: Generative models

Diffusion models

Proximal operators

Backward discretization
Abstract: Diffusion models have quickly become some of the most popular and powerful generative models for high-dimensional data. The key insight that enabled their development was the realization that access to the score---the gradient of the log-density at different noise levels---allows for sampling from data distributions by solving a reverse-time stochastic differential equation (SDE) via forward discretization, and that popular denoisers allow for unbiased estimators of this score. In this paper, we demonstrate that an alternative, backward discretization of these SDEs, using proximal maps in place of the score, leads to theoretical and practical benefits. We leverage recent results in _proximal matching_ to learn proximal operators of the log-density and, with them, develop Proximal Diffusion Models (`ProxDM`). Theoretically, we prove that $\widetilde{\mathcal O}(d/\sqrt{\varepsilon})$ steps suffice for the resulting discretization to generate an $\varepsilon$-accurate distribution w.r.t. the KL divergence.
Empirically, we show that two variants of `ProxDM` achieve significantly faster convergence within just a few sampling steps compared to conventional score-matching methods.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes to use proximal optimization techniques to speed up sampling in diffusion models. Two variants of the algorithm is proposed, one uses full backward discretization, and one uses a hybrid of current and previous point. Motivated by the algorithm, the authors also proposed proximal loss functions to train the \"proximal score\", giving the data-dependant proximal operator used during sampling. The authors analyzed the convergence properties, discussed relations to the usual forward discretization, and verified performance on synthetic and real image datasets.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n1. The method is well-motivated by research in proximal-optimization, and is well-developed into two plausible versions. \n1. The sampling method then motivates a new loss function, leading to the a new proximal score matching technique that is novel to my knowledge. \n2. Writing of the paper is of exceptionally high quality. The main text is accessible to a wide audience, has the perfect amount of technicality, with connections to more conventional methods clearly discussed. I did not fully checked the proofs in Appendix (yet).\n3. The experimental results show promising performance at small number of function evaluations. I cannot wait to see more adoption of this method. \n4. I also appreciate that the authors spent enough passage discussing implementation details, including the crucial reparameterization of the model. I find these non-trivial details very useful and educational, highlighting the practical focus of the authors, in addition to their analytical rigor.\n\nWeaknesses:\nThere is hardly any weaknesses of large concerns (except the proofs in Appendix which I haven't checked). I point out a few minor suggestions.\n1. The authors discussed the benefits of the MAP estimate from proximal points vs the MMSE denoiser (which is provides a lot of nice insights). However, the experiments did not support the claims. I would have thought that the synthetic dataset has a sharp score field at low noise, and hence the authors would show that the good performance comes from using the MAP instead of MMSE. It would be great to use this experiment to further develop this point. \n2. I am a little confused by the definition of $f_\\theta(\\cdot;t,\\lambda)$, I think the reader can figure it out almost by themselves, but it would be nice to provide a quick derivation (e.g. in the extra page if accepted). \n3. Would be nice to provide some intuition on the loss of Fang et al., if there is any. I think this is a very cool reference to read, but for self-contained purposes do give wide audience some high-level ideas. \n4. It would be nice to visualize the score field of the proximal model. I think this would help us understand the differences and connections between the two frameworks, and give readers more visuals to help understanding and interpreting the method.\n\nIf I have time I will read more into the Appendix, and provide more feedback. I put my confidence low for now (sorry for the limited review). If the authors could address the points above and provide more results (the suggestions do not require any large-scale experiments on new datasets), I am willing to further increase the score.\n\nQUESTIONS:\n1. Why can't you let $\\zeta$ go to zero? Is the schedule used in the CIFAR-10 experiments (which I appreciate the honesty) from previous work? How important is this parameter? \n\n2. Why does the synthetic dataset composed of very well separated points? What if you use a distribuion where the outline of the picture is more connected, i.e. almost on a low-d manifold? This would provide a case where the score field becomes low-d, and hence help show advantage to MMSE learning? \n\n3. Why can't the FID at high function evaluation beat the normal forward diffusion? Please discuss on this, as people still care about the quality. \n\n4. Why is the backward method missing in the CIFAR-10 experiment?\n\nLIMITATIONS:\nLimitations are well-discussed. Although addressing the performance saturation at high NFE would be helpful.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes an alternative approach to training diffusion models - based on *proximal matching* instead of score matching. The key idea behind their approach is to use a *backward discretization* of the reverse stochastic process  for the sampling (in place of the standard forward discretization used by score matching methods).\n\nThey show that the backward discretization update can be interpreted as a proximal step w.r.t the log-density, and provide an algorithm \"ProxDM\" (with two variants for two kinds of discretizations) to generate samples using this proximal operator (analogous to the sampling algorithm used by score-based models like DDPMs).\n\nThey show how to train a neural network to learn this proximal step (\"proximal-matching\"). Their method involves sampling different times and regularizer strengths (t, $\\lambda$ pairs) in order to to handle multiple step sizes at sampling time, and they also propose appropriate parametrization and loss weighting to balance the training loss contribution of different times / regularizer strengths.\n\nThey also prove (under assumption that training algorithm converges to oracle proximal operator) that both variants of their ProxDM sampling algorithm converge to the true distribution, at a faster rate (fewer sampling steps) than score-matching methods.\n\nFinally, they provide experimental verification using toy datasets (MNIST and CIFAR10) that ProxDM generates better samples than score-matching methods for small number of steps.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. Excellent presentation, nice theory, very easy and enjoyable read especially for a mathematically dense paper.\n2. The idea of using proximal operators to do sampling in diffusion models is certainly novel, and the motivation behind it (faster convergence) is very compelling.\n3. The authors clearly explain the salient differences between their method and the score-based method, and also provide convenient mental models for comparing them (e.g. interpretation as MAP denoiser vs MMSE denoiser).\n4. The paper does a great job closing the loop from theory to practically useful methods (evidenced by their detailed analysis/experiments which revealed that the hybrid discretization method is better in practice even though the fully backward discretization was better in theory). They didn't shy away from discussing practical considerations (sampling of t and lambda, loss weighting) and limitations (minimum steps requirement for the fully backward method).\n\nWeaknesses:\nMore experimental results could have been provided (e.g. using large image datasets like CelebA) to better justify the practical utility of these methods for \"very high quality\" image generation. In this regime (large image + high quality), I would have liked to know if either of the two ProxDM variants give a significant reduction in number of steps compared to score-based methods. It would also be interesting to see if the theoretical minimum on number of steps for the fully backward discretization is even a concern in this regime.\n\nQUESTIONS:\n(nit) The text says that you include pseudo-code for proximal-matching training in the appendix but I couldn't find it, could you please add it (or else remove this statement)?\n\nMy main questions are about comparison with score-based methods:\n\n1. [Sampling of (t, $\\lambda$) pairs during training]: since the choice of $\\lambda$ depends on the number of sampling steps N that we wish to use during inference, ProxDM requires training the network with, say, 10x more ( t, $\\lambda$  ) pairs in order to support 10 different Ns compared to a single fixed N. Is this an inherent limitation of ProxDM compared to score-based methods like DDPM (which as I understand requires sampling only t during training, and then any number of sampling steps can be used at inference)?\n\n2. For score-matching, the training objective is actually equivalent to minimizing the variational bound on the negative log likelihood of the training data. Does the ProxDM training objective also have such an equivalent formulation?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new method for improving the discretization for backwards processes for diffusion models. Inspired by proximal techniques in optimization, they derive a proximal method-based solver for the backwards process of the OU evolution, as well as a hybrid method, loosely based on forward-backwards methods in optimization theory. Their main theoretical result is a set of error bounds for these methods, given L2-accurate proximal operators. Under some reasonable assumptions on the distribution, they show that the proximal method achieves error eps with O(d / eps) steps, and the hybrid method requires O(d / sqrt(eps)).\n\nSTRENGTHS AND WEAKNESSES:\nQuality: The technical content and ideas presented in the paper are very nice. The approach taken by the paper is very natural in retrospect given the success of proximal methods for optimization, and the bounds they obtain are relatively clean, and assuming learning accurate proximal estimators is about as difficult as learning score functions, achieve better error rates in terms of eps, although of course this is not directly comparable.\n\nClarity: The paper is well-written and very easy to read.\n\nSignificance: This method in this paper could yield new, improved algorithms for solvers for diffusion models, which are a very important area of study. \n\nOriginality: The idea of using such proximal methods, while clearly inspired by optimization, is very nice in this setting.\n\nQUESTIONS:\nOne feedback I have is that it would be nice to have a more complete comparison to the previous non-asymptotic rates for how many timesteps are necessary, given L2 accurate score functions. I believe the best rates are currently given by [1]; notably, their dependence on their eps is 1/eps^2, which maybe worth noting (although again these results are not quite comparable). \n\n[1] Nearly d-Linear Convergence Bounds for Diffusion Models via Stochastic Localization. Benton et al., ICLR 2024.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nProximal Diffusion Models propose learning the proximal operator of the reverse‐time SDE in diffusion generative models, rather than estimating its score (gradient of the log probability). This operator directly produces the MAP denoised output under a Gaussian corruption model, simplifying each sampling step. The authors introduce a “proximal matching” objective to train this network and back it up with rigorous convergence guarantees, showing that Proximal Diffusion Models reaches high‐quality samples in fewer iterations than traditional score‐based methods.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- The paper is well-structured and easy to follow.\n- The derivations are comprehensive and, to the best of my knowledge, mathematically sound.\n- The proofs are rigorous, and the theoretical convergence guarantees are clearly presented.\n- The experiments compellingly demonstrate superior sample quality with fewer sampling steps.\n\nWeaknesses\n- Beyond sampling efficiency, no clear advantage over standard diffusion models.\n- Experiments rely solely on FID versus sampling steps, offering a limited view of performance.\n- On larger datasets (e.g., CIFAR-10), traditional diffusion at full timesteps still achieves better FID than the proximal approach.\n- Comparative analyses of training efficiency and convergence behavior are absent.\n\nQUESTIONS:\n- Proximal matching trains a MAP denoiser under Gaussian corruption (i.e., effectively targeting the mode of the posterior) whereas score‐based diffusion methods optimize a variational bound that (at least loosely) increases the model’s log‐likelihood, this can be more explicit with MLE diffusion weighting. Do you have any theoretical guarantees that minimizing the proximal matching loss will also improve—or even converge to—the true data log‐likelihood? In other words, is there an ELBO‐style bound or other analysis showing that training with a MAP denoiser yields maximum‐likelihood consistency?\n- The paper reports faster convergence, but is that improvement limited to the sampling phase, or does it extend to faster convergence during training as well? \n- With current trends of one-pass models such as consistency models, distillation methods, shortcut models, and flow models, what are the immediate benefit of proximal diffusion models? Are the main benefits faster inference? If so, those aforementioned methods already \"straighten\" the gradient trajectory—whether along the denoising path or the velocity field, allowing for a single one pass inference rendering a proximal operator unnecessary.\n- Overall, the paper is well written and engaging, but unless you can demonstrate potential innovation for future diffusion works or significant benefits beyond sampling speed, which many recent methods already address, I remain a little skeptical, though I am open to further discussion.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-07"}