Title: Learning normalized image densities via dual score matching
Venue: NeurIPS 2025
Accepted: yes
Keywords: energy-based models

image dimensionality

score matching

diffusion models

manifold hypothesis

negative log-likelihood
Abstract: Learning probability models from data is at the heart of many machine learning endeavors, but is notoriously difficult due to the curse of dimensionality. We introduce a new framework for learning \emph{normalized} energy (log probability) models that is inspired by diffusion generative models, which rely on networks optimized to estimate the score. We modify a score network architecture to compute an energy while preserving its inductive biases. The gradient of this energy network with respect to its input image is the score of the learned density, which can be optimized using a denoising objective. Importantly, the gradient with respect to the noise level provides an additional score that can be optimized with a novel secondary objective, ensuring consistent and normalized energies across noise levels. We train an energy network with this \emph{dual} score matching objective on the ImageNet64 dataset, and obtain a cross-entropy (negative log likelihood) value comparable to the state of the art. We further validate our approach by showing that our energy model \emph{strongly generalizes}: log probabilities estimated with two networks trained on non-overlapping data subsets are nearly identical. Finally, we demonstrate that both image probability and dimensionality of local neighborhoods vary substantially depending on image content, in contrast with conventional assumptions such as concentration of measure or support on a low-dimensional manifold.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper has two related but distinct contributions.\n\nFirstly, the paper proposes a framework called “Dual Score Matching” which is an extension of the original Score Matching (DSM). On top of traditional DSM, the authors used the matching of “time-score”s which is a quantity similar to score but the gradient is with respect to time $t$.\n\nSecondly, the authors did not model the score directly, instead they model “normalised energy” which is almost equivalent to log-probability. They came up with a new architecture that models energy as a conservative field.\n\nIn the experiment section, denoising performance and log-likelihood comparisons have been made. Additionally, a significant part of the experiment section focuses on analysing the image distribution using the learned model.\n\nSTRENGTHS AND WEAKNESSES:\nThe authors have investigated a more accurate score matching by introducing a time score component. The authors claim that this modification lead to better denoising and log-likelihoods. While improving the fundamentals of score matching is commendable, the paper has several flaws in terms of novelty, presentation, focus and evaluations.\n\n- It is quite unclear to me why the time score matching is necessary. There is not much reasoning or justification given to support usage of the time score. Is it only to improve the estimated values at the network output? The time score does not seem to have any explicit usage in the paper (e.g. in log-likelihood calculation) — am I wrong in this assessment? The only piece of reasoning I found in the paper is (L42) “[it] .. ensure consistency of the energy estimates across noise levels”. What does this even mean? There is no explanation of this sentence. Why would a score network *without* the time matching part be NOT consistent across noise levels?\n- Authors claim the second objective (i.e. the time score matching) is novel (L42; “second is novel”) but clearly there is a 2022 paper by Kristy Choi et al. (it is cited indeed) that proposes the time score and the matching objective. There is no comparison or discussion about this paper.\n- I do not quite understand how section 3 of the paper is related to dual score matching, which is the central theme of the paper. Computing entropy, log density distribution, intensity range, dimensionality — these seem to be related more to the dataset and does not have much to do with the model or the algorithm presented in the paper. I'm sure any other model that computes log-likelihood and trained properly would behave the same way.\n- I'm quite doubtful about the usage of *conservative* energy models. In the Diffusion literature, it is quite an accepted fact that conservative energy models are NOT superior to unconstrained score models. The authors accepted that their finding is (L162) “*contrary to*” the results of Salimans & Ho [2021], where they attributed this to the specific choice of architecture. However, there is no ablation regarding the choice of architecture. Also, it is unfair to compare with Salimans & Ho [2021] as they investigated generation/sampling performance but this paper did not.\n- The authors also claimed that the (L178) “.. unique advantage of our method is that .. it is fast [for NLL computation] ..”, which seems to be an over-claim. The reason it is fast has not much to do with the specific parameterisation or the matching algorithm, but the specific choice of NLL calculation algorithm that they adopted from Karczewski et a. [33]. These important details only appear in appendix rather than mentioning clearly in the main paper.\n- There are a lot of approximations made in the theoretical calculations and justifications. On one hand you argue that your method is better due to “homogeneity” property but on the other hand you say that the network is *only approximately* (L508 appendix) homogeneous. On one hand, you argue about your energy functions being “normalised”, only to reveal later that it is only an approximation. Also, the explanation in appendix A.3 is quite hand-wavy, non-rigourous and does not conclude anything concrete. In general, it seems that the authors are quite uncertain about their rationals/claims.\n\nOverall, I am do not particularly see much novelty (or proper justifications for them) convoluted by the fact that the many rationals are quite uncertain and not concrete. Also, the paper lacks proper experimentation to validate the claims.\n\nQUESTIONS:\nMajor questions:\n\n- Since you are modelling the energy using a neural network but still matching the “score”, you require a “double backpropagation” (once with respect to space and then parameters). It is quite evident from Eq. 43 of appendix C.2. This is clearly a computational drawback but not discussed in the paper (except one small line in the limitation section).\n- With regard to the “normalisation” in Eq. 9, do you need to invoke the network twice for every forward pass (i.e. one for a given $t$ and another for $t_{max}$) ? If so, it is also a computational drawback. Please clarify.\n- I feel like the two parts of the paper are sort of unrelated. Is it necessary to consider energy models in order to use dual score matching? Could you have done dual score matching with score models (one model for space and another for time)? There are no ablation or justification for these questions.\n\nMinor questions:\n\n- In several places in the paper it says \"gradient of the energy is the score”, which is technically untrue. The score is the *negative of the* gradient of the energy. Isn’t that true ?\n    \n    $p(x) = e^{-U(x)}/Z \\implies \\log p(x) = -U(x) - \\log Z \\implies \\nabla_x\\log p(x) = -\\nabla_x U(x)$\n    \n- A similar issue is with the term “normalised energy (log prob)” which is also technically not true. Log-probability is the “normalised *exponentiated* negative energy”. It seems that authors used certain terms in a very loose sense.\n- Equation 2 & 3 may have a typo — shouldn't it be $\\frac{x-y}{t}$ and not $\\frac{y - x}{t}$ ?\n\nLIMITATIONS:\nMentioned limitation. But they are rather big drawbacks of the paper and should be discussed in the main paper.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a method for learning normalized energy-based models by extending score matching to account for both spatial and temporal gradients. Leveraging the structure of diffusion models, the authors propose a novel dual score matching objective that enables direct estimation of energy values rather than just their gradients. The method is demonstrated on a synthetic toy example and ImageNet64, showing consistent energy estimates and competitive negative log-likelihood performance. The paper also explores the geometry of the learned energy landscape.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The paper tackles a key limitation of score-based models - their inability to estimate normalized densities -and proposes a direct way to recover normalized energies using both space and time gradients in diffusion.\n2. Learning a normalized energy opens the door to likelihood-based evaluation, OOD detection, and deeper understanding of image statistics — things that score-based models can’t directly do.\n3. On ImageNet64, the proposed method achieves negative log-likelihood (NLL) scores that are competitive with several strong baselines, including flow-based models\n4. The paper includes a detailed empirical study of the learned energy geometry, revealing properties like variation in local dimensionality and log-probability distributions across image content.\n\nWeaknesses:\n1. The paper lacks theoretical guarantees that minimizing the proposed objective leads to an accurate approximation of the true log-density, which makes some of the empirical findings (e.g., observed geometric properties) harder to fully trust.\n2. The paper evaluates only on ImageNet64 (and on a synthetic example). Including additional datasets would help demonstrate the generality and robustness of the proposed method.\n3. While the paper focuses on likelihood and geometric analysis, it would be valuable to demonstrate downstream applications (e.g., out-of-distribution or outlier detection) where normalized energy estimates are especially useful.\n\nQUESTIONS:\n1. One of the main motivations for learning normalized energy is enabling tasks like OOD detection or uncertainty estimation. Have you tried applying your model in one of these settings? Even a small experiment could help illustrate the practical value beyond likelihood evaluation.\n2. The method is only tested on ImageNet64 (and a toy example). It would be helpful to know whether it generalizes to other datasets. Even a single additional benchmark could make the results feel more robust\t.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

comments: SUMMARY:
The authors propose a framework to learn a time-dependent energy function for a distribution given a set of images. This paper presents a dual score matching framework that enables direct learning of normalized energies using spatial score matching, based on the standard denoising target, and temporal score matching, a novel target, based on the gradient with respect to the noise level.

STRENGTHS AND WEAKNESSES:
The submission is written very clearly, especially the introduction, which does a great job of motivating the problem and explaining the core idea. In my opinion, this paper is a very good fit for NeurIPS. One of the main strengths is the deep analysis of different aspects of the learned image distribution, where the authors go beyond just reporting likelihood scores. These insights are really interesting for the community.

Small weaknesses:
- The example in Figure 1 needs a bit more explanation. It's not immediately clear why single score matching fails there, maybe adding a short sentence in the main text or figure caption would help.
- In Figure 3, it would be helpful to mention directly that Figures 6 and 7 (from the supplement) show the images corresponding to the marked arrows.

QUESTIONS:
- Will you publish code of your experiments?
- Could you please expand on why single score matching fails in the toy Gaussian mixture example in Figure 1? A short 1-2 sentence  explanation would be sufficient.
-  Can you comment on generalization to larger-scale images?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 4
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a framework for learning normalized energy-based models directly from image data, leveraging techniques from score-based diffusion models. Particularly, a dual score matching objective is proposed to match gradients of the energy network with respect to both space and noise level. The authors further propose a novel architecture for energy estimation by computing the inner product of an input image with the output of a score network. The resulting model is evaluated on ImageNet64 dataset and achieves comparable results to other methods on NLL.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The proposed dual score matching objective is interesting and looks effective for energy function learning.\n2. The proposed model achieves a good result on ImageNet64 in terms of NLL. In addition, extensive experiments and analyses are provided to illustrate the comprehensive ability of the proposed method.\n3. The overall paper is well-written and organized.\n\nWeaknesses:\n1. Compared to common diffusion objective, the proposed dual score matching requires to learn two objectives. The training process might be slower. Can you provide the convergence speed comparison between the DSM-only loss and the DSM-TSM joint loss.\n2. In the experiments, it would be great to also apply the method to higher resolution images such as ImageNet $256 \\times 256$ or CelebA. In addition, can you also report FID to evaluate the sample quality?\n3. In Table 2, it would be better to add at least one more energy-based models (EBMs). And adding one figure to compare the training stability between different EBMs would further increase the reliability of the proposed model.\n\nQUESTIONS:\n1. In the combined loss (equation (8)), why you normalize the two losses with dimensionality $d$? Especially there is no $d$ in the original DSM loss.\n2. Can the proposed method be applied to conditional generative models?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-02"}