Title: SD-KDE: Score-Debiased Kernel Density Estimation
Venue: NeurIPS 2025
Accepted: yes
Keywords: Kernel Density Estimation

Diffusion models

Score based models

Synthetic Data

Probabilistic modeling
Abstract: We propose a method for density estimation that leverages an estimated score function to debias kernel density estimation (SD-KDE).
In our approach, each data point is adjusted by taking a single step along the score function with a specific choice of step size, followed by standard KDE with a modified bandwidth.
The step size and modified bandwidth are chosen to remove the leading order bias in the KDE, improving the asymptotic convergence rate.
Our experiments on synthetic tasks in 1D, 2D and on MNIST, demonstrate that our proposed SD-KDE method significantly reduces the mean integrated squared error compared to the standard Silverman KDE, even with noisy estimates in the score function.
These results underscore the potential of integrating score-based corrections into nonparametric density estimation.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nKernel density estimation is a classical density estimation technique which yields properly normalised nonparametric density estimates. Score based methods match only the score function (gradient of log density), which does not require the normalising constant, and therefore does not yield a density estimate in a classical sense. The authors propose to use information in (an estimate of) the score function to assist in KDE, yielding properly normalised score debiased KDE (SD-KDE). This is achieved through updating the original data samples in one step of the gradient of the score function. A theoretical result shows improved estimation error compared with naive KDE, as well as optimal bandwidth and step size selection, in the case where the exact score function is known. Experiments are demonstrated on a range of low dimensional settings, showing good benefits above classical KDE.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n- Algorithm 1 is very elegant and clean. It is easy to read and implement. The intuitive description provided by the authors about using the score to debias the samples is helpful.\n- I liked that the authors showed two demonstrations of approximate score functions. One involves an initial KDE and then using the score, and the other involves the more standard neural estimator via Hyvarinen score or Fisher divergence. (Unforunately, the neural estimator seems to perform not so well, but this is a separate point). \n\n**Weaknesses:**\n- The presentation of theorem 1 could be improved, in my opinion.\n    - First, there is no need for the first equation, as it is just substituting step 1 into step 2 of algorithm 1, which is already referenced in the theorem text. \n    - Second, I believe \"asymptotically optimal\" is more precisely \"the smallest AMISE\". Please correct me if I'm wrong.\n    - Third, \"Let $\\hat{s}$ be the exact score function of $p$.\" could more precisely be stated as \"Let $\\hat{s}$ be the exact score function of $p$, i.e. $\\hat{s} = s$\". \n- There is a gap between Corollary 1 and a notion of \"error\" obtained by standard scoring algorithms. Unfortunately, I can't really see how corollary 1 is useful in its current form. It would be nicer if we could relate the AMISE to some error in the score estimate (e.g. the error in the Hyvarinen matching). This I see as the biggest weakness of this paper. This is acknowledged in the limitations section, by the authors. Would it be possible to upper bound the first term in the RHS of corollary 1?\n- Figure 16 is qualitative and it is difficult to assess its meaning. I am not convinced that the images on the right are less representative than the images on the left (I don't have any suggestions for improvement, unfortunately).\n- The proof of theorem 1 reveals that some conditions are missing from the statement of theorem 1. Namely, the kernel has to have a convergent Taylor series (satisfying Taylor's theorem)\n\n**Questions:**\n- Does one (or should one) use the same datapoints for both the score estimate and the SD-KDE, or should you use different points? Would it make sense to first estimate the score using an independent pretraining set? Why or why not, and did you try both ways?\n\nMinor:\n- There are three counts of \"the theorem 1\". Recommend changing to \"Theorem 1\", to retain consistency throughout. \n- Line 79 uses the nonstandard notaiton $\\mathcal{N}(0, \\sigma)$, which maybe should be $\\mathcal{N}(0, \\sigma^2)$. \n- I believe \"std\" in the legends in Figure 2 should be $\\sigma$, however this is not mentioned in the text.\n\nQUESTIONS:\nI am currently borderline between reject and accept. I err on the positive side (believing in advance that the authors will give a reasonable rebuttal) and give accept for this review. I am unlikely to raise my score due to the main weakness of corollary 1, unless the authors can somehow improve the result or apply it to a concrete setting. Actionable summary of my weaknesses:\n\n- Please discuss whether something more practical can be salvaged from Corollary 1. For example, is it possible to upper bound the integrated RHS? Are there any takeaways for practical score estimation?\n- Please answer my question about whether independent training set is used for score matching versus the KDE step. Include any modifications you might make to an updated manuscript.\n- Please address the minor comments.\n\nLIMITATIONS:\nYes.\n\nLimitations are pointed out in the paper, and agree with the major weakness in my review above. I am not concerned about negative societal impact for this broad application-agnostic paper.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-11"}

comments: SUMMARY:
This paper debiased KDE using an estimated score function.

STRENGTHS AND WEAKNESSES:
The primary weakness is the lack of novelty and motivation. KDE is non-parametric and not scalable. In its comfort zone, there are many ways to improve the performance of a standard KDE without invoking diffusion models, and of course the silverman baseline. Theorem 1 is also highly idealized.

QUESTIONS:
I don't quite see a clear path for me to raise my score.

LIMITATIONS:
yes.
TITLE: Official Review
CLARITY: 2
IMPACT: 1
ORIGINALITY: 1
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents SD-KDE (Score-Debiased Kernel Density Estimation), a method that enhances classical KDE by moving each sample point along the estimated score function before applying KDE with an adjusted bandwidth. This single-step correction cancels the leading-order bias and improves the MISE. The authors provide a complete theoretical analysis and demonstrate consistent performance improvements on synthetic 1D/2D data and MNIST.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths  \n\n- Reduces KDE bias using score-guided data shifts, improving estimation accuracy.  \n- Achieves provable MISE improvement.\n- Experimental results confirm strong gains across settings and noise levels.  \n- Performs well even when the score function is noisy or estimated from data.  \n- Iterative application enables further bias reduction without requiring a score oracle.  \n- Bridges score-based generative modeling and nonparametric density estimation.\n\nWeaknesses\n\n- Theoretical results require access to the exact score function, which is rarely available.  \n- No empirical comparison with modern KDE variants beyond Silverman’s rule. For instance, recent approaches such as Fast KDE with density matrices and Random Fourier Features (Gallego et al., 2022) and the diffusion-based KDE method diffKDE with optimal bandwidth approximation (Pelz et al., 2023) are not considered.   \n- Step size $\\delta$ and bandwidth $h$ may require careful tuning; robustness not analyzed.  \n- High-dimensional and structured data domains remain largely untested.  \n- MNIST results are mostly visual; lacks quantitative metrics for density quality.\n\nQUESTIONS:\n- How does SD-KDE perform in high-dimensional spaces with limited data and noisy scores?\n- How does SD-KDE compare with neural density estimators like normalizing flows or autoregressive models in terms of accuracy and efficiency?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper provides a new kernel based density estimate method that leverages the \"score\" (the gradient of the log-density, which shows up in diffusion models).  Each data sample is moved in the direction of the score (essentially denoising the samples), then a KDE is built on the updated points.  The paper shows:\n   - this significantly increases the rate of estimation from O(n^{-4/(d+4)}) to O(n^{-8/(d+8)})\n   - shows improvement in practice in 1 and 2 dimensions (and perhaps on MNIST)\n   - if one can only estimate the score, then one can get nearly as much improvement.\n\nSTRENGTHS AND WEAKNESSES:\nThe O(n^{-4/(d+4)}) convergence result (of Silverman 1984) is classic, and so improvement on this is very exciting.  Directly improving it would merit strong acceptance.  \nThe paper falls a bit short of this, since it uses the score as an oracle in the result, and then bounds error in estimated score, and shows how to estimate it in practice (which still shows improvement empirically on Silverman's result, but not as much as with the oracle).  \nThe paper does not provide an improvement in density estimate without knowing this oracle or having some known bound on the accuracy of the score.  \n\nRegardless, I find this as very important progress on a classic and very fundamental result, one in which the line of work was thought to have been perhaps closed.  \nI suspect this will soonafter lead to future work which will close this loop on this approach -- I wish it was in this paper.  \n\n\nSo there are many important strengths to the paper, but here are some relatively minor weaknesses:\n\n - The paper would be improved with better early-on discussion (and mathematical definition) of the score function.  It may be fairly mysterious to those only familiar with classic density estimate results.  \n\n - Results in 1, 2 dimensions where the results are more meaningful is ok, but it would be useful to show results for a variety of settings in high-dimensions, other than the simple qualitative MNIST experiment.  \n\n - The proof is informal in a few places. \n   *  \" we can show the h^3 terms in bias vanish\"\n   *  \"thus the bias is O(h^4)\" then a few lines later \"The error due to bias is O(h^8) ...\"\n\nAs such I was not able to verify the proof.  I think I see how this works, and I suspect I could figure it out -- but for a result like this, I expect the paper to connect these parts much more carefully.\n\nQUESTIONS:\nIs it possible to obtain a stronger total bound on the convergence of the density estimate, without assuming the score oracle, but rather estimating the score and using that (as shown to work empirically)?  Are there obvious reasons why this is non-trivial?\n\nLIMITATIONS:\nyes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-02"}

comments: SUMMARY:
The paper presents a simple modification of the classical Silverman Kernel Density Estimation (KDE) method. It relies on having access to the score function of the data generating process (which is generally impossible) but empirical results show that a learnt surrogate (through a diffusion model) also performs very well. Importantly, the authors provide asymptotic results that demonstrate an improvement in performance over Silverman's method in the ideal case of a known score function.

STRENGTHS AND WEAKNESSES:
Strengths:
- Solid theoretical result;
- Practical algorithm for a real world problem;
- Empirical evaluation supports theory.

Weaknesses:
- Computational costs, particularly when the score needs to be learnt;
- Theoretical results require exact knowledge of the score, the sensitivity to errors in score estimation is unknown;
- The empirical evaluation is mostly on toy data/ well used data sets.

QUESTIONS:
Standard KDE is super-fast; what is the empirical scaling of the method? Also, learning a score requires a fairly large number of points, can you experiment to see what is the behaviour of the learnt density as you vary this parameter? Finally, I am left somewhat uncertain as to what is the advantage of this approach to directly using a generative model such as diffusion, particularly since the score needs to be learnt training a diffusion model. Overall though I enjoyed the paper, I think it is good work.

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 4
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-04