SD-KDE: Score-Debiased Kernel Density Estimation
Authors: Elliot L Epstein
Rajat Vadiraj Dwaraknath
Thanawat Sornwanee
John Winnicki
Jerry Weihong Liu
Venue: NeurIPS 2025 poster ()
Keywords: Kernel Density Estimation
Diffusion models
Score based models
Synthetic Data
Probabilistic modeling
Primary area: probabilistic_methods
TL;DR: SD-KDE improves on previous methods for kernel density estimation by using a noisy estimate of the score function.

Abstract: We propose a method for density estimation that leverages an estimated score function to debias kernel density estimation (SD-KDE).
In our approach, each data point is adjusted by taking a single step along the score function with a specific choice of step size, followed by standard KDE with a modified bandwidth.
The step size and modified bandwidth are chosen to remove the leading order bias in the KDE, improving the asymptotic convergence rate.
Our experiments on synthetic tasks in 1D, 2D and on MNIST, demonstrate that our proposed SD-KDE method significantly reduces the mean integrated squared error compared to the standard Silverman KDE, even with noisy estimates in the score function.
These results underscore the potential of integrating score-based corrections into nonparametric density estimation.

SD-KDE: Score-Debiased Kernel Density Estimation
Elliot L. Epstein
Rajat Dwaraknath"
1+
Thanawat Sornwanee
John Winnicki
Jerry Weihong Liu
Sttanood University, Stanford, CA 94305, USA
(epsteine, rajatvd, tsornwanee, winnicki, ju150)@staaford.com
Abstract
We propose a novel method for density estimation that leverages an estimated score function debias kernel density estimation (SD-KDE). In our approach, each data point is adjusted by taking single step along the score function with a specific choice of step size, followed by standard KDE with modified bandwidth. The step size and modified bandwidth are chosen to remove the leading order bias in the KDE, improving the asymptotic convergence rate. Our experiments on synthetic tasks in ID, 2D and on MNIST, demonstrate that our proposed SD-KDE method significantly reduces the mean integrated squared error compared to the standard Silverman KDE, even with noisy estimates in the score function. These results underscore the potential of integrating score-base based corrections into nonparametric density estimation.
Introduction
Kernel density estimation (KDE) (Rosenblatt 1956 Parzen 1962) is a widely used nonparametric method for estimating unknown probability density function from finite set of data points. The classical KDE effectively smooths the data by convolving with a kernel function, such as the Gaussian kernel, and then normalizing the result to obtain density estimate. KDE finds application in diverse fields such as anomaly detection, clustering (Campello et al. 2013), data visualization (Scott
2012), nonparametric statistical inference (Guerre al. 2000 Zhang al. 2008), and dynamical systems (Hang et al. 2018).
The classical KDE suffers from well-known bias-variance trade-off, controlled by the choice of ker- nel bandwidth (Silverman) 1986). Larger bandwidths lead to smoother estimates with lower variance but higher bias, while smaller bandwidths yield more variable estimates with lower bias (Rosenblat) 1956 Parzen 1962). This trade-off is particularly damaging in cases with highly variable density functions, where the bias dominate the estimation error.
Recent advances in score-based generative modeling and diffusion processes have demonstrated the power of using the score function-the gradient of the log-density- reverse forward process of noise injection, effectively reconstructing the underlying data distribution (Ho et al. (020). Notably, methods such diffusion models, (Song & Ermon 2019) provide robust estimates the score function even in complex, high-dimensiooal settings, without requiring density estimation.
matching (Hyvärinen & Dayan 2005) and its deep learning extensions,
In this work, we investigate whether incorporating knowledge of the score function into the KDE framework allows us to push the Pareto frontier of the bias-variance trade-off. We propose method
Equal contribution.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

to debias the KDE using the score function to improve density estimation accuracy. Specifically, our method adjusts each data point by taking small step the direction of the estimated score, and then performs KDE with modified bandwidth, as illustrated in Figure] I Intuitively, taking step along the score sharpens the sample distribution, which counteracts the smoothening effect of applying the KDE. We find that with a carefully chosen combination of step size and KDE bandwidth, we remove the leading order bias in the KDE, resulting in a more accurate, debiased density estimate. Crucially, SD-KDE also works with empirical scores obtained directly from vanilla KDE (via the gradient of the log the KDE density estimate); no learned diffusion model is required.
In summary, our contributions are the following:
1. We propose Algorithm D our method for score-debiased kernel density estimation (SD- KDE).
2. We provide asymptotically optimal bandwidth and step size selection for Algorithm (Theorem D. achieving the asymptotic mean integral square error (AMISE) of order
O (n ())45)) instead of the O (n=4/((+))) achieved by standard KDE (Silverman) 1986).
3. In Section 3] we numerically corroborate theoretical results 1D and 2D synthetic datasets and observe strong agreement with the asymptotic scaling identified in Theorem
2 Method and Theoretical Results
Algorithm 1 Score-Debiased Kernel Density Estimation
Require:
Data {= score estimator 6, kernel K, KDE bandwidth h, score step size
1: Take single step along the score function: 3; +- I. 58(x;) for i 1,...,n
2: Compute the debiased kernel density estimate: p(x) ahd E K
Theorem (Optimal Bandwidth and Step Size selection for Algorithm Dp. Let faile be i..dd samples from a smooth density p in Rdd Let be the exact score function of p. Let K be a symmetric kernel with mean O, covariance uu K(u)du L, and convergent Taylor series. The debiased kernel density estimate P obtained by running Algorithml I with bandwidth h and step size ö is given by
-x; + 58(2;))
p(x) The
K
The asymptotically optimal bandwidth and step size for Algorithm 1 are given by
2
hops O (n (o-//tts). 8ept
The resulting debiased kernel density estimate P satisfies
[ (B(x) - -0)))((())))))))))()))) dx = 0 (n-8/(4t)).
MISE: :=
We include the detailed proof of Theorem in Section
Corollary 1. If the estimate score 8(-) is not equal to the actual score s(-), the bandwidth is h, and the stepsize is 4e as in Theorem! 1 then the bias is given by
2
E [p(x) - p(z)] = - [(5(x) s(x)) Vp(x) + (())))
The proof for this corollary directly follows the proof for Theorem

---

Target
Data: X
KDE
O
SD-KDE
h2 8, X: + S s(,,) 2
O
Score: s(x) V (x)
Figure 1: First schematic diagram on SD-KDE. The estimation objective is to estimate the target distribution pdf. In the conventional setting, we have finite samples from the distribution (blue box). Using only this information, we can perform KDE to estimate the probability density function. However, we have access to the score function, we can combine the data points and score function to get SD-KDE By fixing the kernel bandwidth to be h, will get that the vanilla KDE and SD-Ke 3D-KDE have a pointwise variance of order O (nta) However, SD-KDE reduces the pointwise bias from Oh?) to O(h*) per Theorem
Discussion. Theorem demonstrates that, when score oracle is available, one can eliminate the asymptotically dominant term, thereby reducing the bias from the conventional order O (h²) to
O (ha) Although a higher-order sernel-such as the effective spline kernel described by Silverman (1984) similarly achieve similar bias reduction, it typically introduces regions where the estimated density assumes negative values. This drawback poses a significant practical challenge, the numerical normalization of the resulting probability density function is computationally intractable (Song & Ermon 2019).
We note that method flexibly allows variety of kernels be used for KDE, since the only require- ment for the kernel is in symmetricity and covariance structure, both of which can conveniently
satisfied (Chen 2017).
Although Theorem requires the knowledge of the score function, we observe empirically that small discrepancy of the estimated score function and the underlying score function under some level may only have minimal effect on the performance. See Section for more details.
Experiments
3.1 ID Synthetics
Experimental setup. We test the empirical performance of the SD-KDE method on density estimation of ID Gaussian mixture models, and include similar analysis for Laplace mixture models
We open-source implementation of SD-KDE at / htt.pp///ggbbboo////ooeeeeeeeeee

---

Geaeka Mierae
Greeia Minee
Groeia Mature
137
KIF
Figure 2: MISE error as a function of for the three Gaussian mixtures for Silverman vs. SD-KDE. Each point is the average MISE over 50 random seeds. The slopes inside each subplot are fitted regression lines log-log scale, indicating how quickly each method's error decays as n increases.
in Appendix A We sample data from three mixtures, where
p(x) NN oi) + (1- (1-x) (2)02)
and each mixture's 's parameters (I,, 141, F1, (42, 02) are outlined in Table [ We compare the SD- KDE method with a baseline based on the classical Silverman KDE, using Silverman's 's bandwidth formula (Silverman 1986), given by h = 0.9 min(ō, IQR/1111 - -1/5, where IQR is the inter- quartile range. To investigate how sensitive our method to the estimation accuracy of the score function, we test the performance of our method when only given access to noisy score function estimate, e.g. we observe S(x) = S(x) + where S(x) is the score function and e ~ (O, o²) for given standard deviation σ. Performance is evaluated with mean integrated squared error (MISE).
Most of the experiments in the paper were conducted on a Linux cluster with 5 NVIDIA RTX A6000 GPUs, each with 49140 MB memory, rumning CUDA Version 12.5. The cluster has 256 AMD EPYC 7763 64-Core Processor CPUs. Some experiments were also conducted on a MacBook Air (2022) equipped with an Apple M2 chip and 16 GB of unified memory. All experiments took less than hour to run.
SD-KDE is robust to noisy score function estimate. In Figure we show the MISE of the SD-KDE (as function of the number of observed samples, n), with varying degree of added noise and compare to the Silverman KDE. Each point in the plot represents an average over 50 seeds. We that the SD-KDE method has significantly better asymptotic scaling than the Silverman baseline, up to a score function noise level with a = Even in the presence highly noisy score function, find the SD-KDE method provides significant gain. We also display the fitted regression slope associated with each line, along with the theoretical asympototic convergence rate of n 8/9 We note the close tracking between the SD-KDE asymptotic decay (-0.85 for mixture 1, and -0.93 for mixture 3) compared with the theoretical predicted decay (= 8/99 -0866). For mixture 2, all models have weaker performance due to the challenging mixture shape, indicating that larger n is needed reach the theoretical decay rate. For n = 5 x 10 the SD-KDE has order of magnitude smaller MISE error on average across 50 seeds compared with the Silverman method.
SD-KDE consistently beats Silverman baseline. Figure we examine the consistency of the performance gains across multiple data seeds for n 100. We observe that the SD-KDE method is consistently better than the Silverman baseline; for mixtures and 2, SD-KDE method outperforms for all 100 samples, and for the third mixture, it is better in 95% of samples.
SD-KDE with the Empirical Score We now relax the assumption that the Score function is given, rather, we use Silverman KDE to approximate the score function, and then apply SD-KDE based on this estimated score. We call this method Emp-SD-KDE. Figure4shove that Emp-SD- SDDE method greatly improves on the standard Silverman KDE, without any assumptions on knowing the true score function. Figure 5 shows how the empirical score is used.

---

Cemila Viduas
Natues
Glacao Mour
Figure 3: Histogram of MISE difference of the SD-KDE method and the Silverman method, for n 100 samples and 50 random seeds per mixture. The SD-KDE method is consistently having lower MISE than the Silverman baseline; for mixtures I and 2, SD-KDE method outperforms for all 100 samples, and for the third mixture, it is better in 95% of samples.
Greviaa Mimrao
Licawiax Mittzoe
13
137
Dog mik)
Shames KDC
Day SDADE
Figure 4: MISE error as function of n for each of the three gaussian mixtures. For each point, we compute the MISE with 50 random seeds per mixture. Each subplot plots the mean integrated squared error function of n. The legend compares Silverman KDE, Emp-SD-KDE (estimating the score from the data), and SD-KDE (ground truth score). The slopes inside each subplot are fitted regression lines in log-log scale indicating how quickly each method's error decays increases.
3.2 2D Syntheties
We present preliminary results on 2D synthetic tasks, spiral distribution (Figure 6 and mixture of Gaussians (Figure 15). following Liu et al. (2020); Grathwohl et al. (2019).
In Figure 6 we compare the 2D Silverman method to SD-KDE for the spiral distribution. We compare the accuracy of our method using the true score function to using an estimate of the score function obtained by training a denoising diffusion probabilistic model (DDPM) from scratch on the training data. For the diffusion model architecture, we use a 3-layer MLP with hidden dimension 512, and we train the model with Adam for 1500 steps. We use 1000 diffusion steps during training. Using the true score function, our proposed method outperforms the Silverman method both qualitatively (via visual assessment) and quantitatively, as measured by the MISE. When employing the score estimated from the diffusion model, our method achieves performance comparable to that of the Silverman method. We attribute this discrepancy with the method under the true score parameter primarily to challenges encountered during the training of the diffusion model rather than any inherent limitations of the method itself, particularly given the accuracy observed when using the true score.
3.3 Iterated SD-KDE: Incremental Improvements to KDE
In a 1D Gaussian mixture experiment, we examine an iterative application of SD-KDE to further improve the density estimate. We work with a gaussian mixture centered at +t0.5, each with standard deviation 0.2, 0.3 with weights 0.7 and 0.3. For this experiment, we will sample 1000 points and hold the bandwidth constant at 0.15. We start with a vanilla Gaussian-kerno KDE fit to the mixture data and compute its the closed form solution or approximation of its score, then apply SD-KDE (one
5

---

Estimated Score: (x)
SD-KDE (Estimated)
Empirical
Misture
p(x)
ndd Σ*
8
Variance (²% mna)
O(h'E +h)
Figure 5: Second schematic diagram on SD-KDE. In the case where score function is not available. We can use proxy score function from proxy distribution. In the example, this is the mixture of original distribution with some Gaussian distribution. We can also estimate score from data points. If the estimated score and the actual score are close enough as in the corollary T. then one can attain a better result with SD-KDE compared to vanilla KDE.
Salvermen
SD-KDE (True)
SD-KDE (Diftusion) 4-1000, R-D.116 MISE=0.000583
True Density
C-1000, R-0.144
n-1080, D-0.118
n=1000
MISE=0.000000
MISE-9.000459
2-
2-
1
0
c
-2
Figure 6: Comparison of the true 2D spiral density vs. Silverman and SD-KDE. For SD-KDE, we evaluate with both the true and learned (diffusion model) score. SD-KDE outperforms Silverman with the oracle score function, and achieves comparable performance even using noisy score.
score-based correction step, with scale 0.015 decaying at a rate of 0.15 at each iteration) to generate surrogate points that remove the leading-order bias. The resulting debiased KDE serves the baseline for the next iteration, where we recompute the score and apply SD-KDE again; each successive iteration thus leverages shows the method when one iteration is taken. Intuitively, since the first SD-KDE step cancels the dominant bias term, subsequent iterations can target smaller remaining discrepancies, progressively aligning the estimated density more closely with the true distribution. As shown in Figure 7 repeated application SD-KDE yields closer alignment between the estimated and true probability densities and corresponding reduction in KL divergence and mean integrated squared error (MISE) with each iteration. Notably, while single SD-KDE iteration often captures the majority of the improvement in simpler mixture scenarios (additional iterations confer negligible benefit), more complex multi-modal cases or smaller sample regimes benefit from multiple iterations, albeit with diminishing returns. These results illustrate how SD-KDE could be used to directly improve upon KDE without training a separate score oracle (which can often difficult to train). Similar to the previous sections, we include similar analysis for Laplace mixture models in Appendix
accurate score estimate correct residual higher-order biases. Figure

---

Table 1: Parameters for the three univariate Gaussian mixtures used in our experiments. Each mixture
follows the generic form p(x) = T (x μl, of) + (1 πT Nx 12, oz)
Mixture
P1
01
142
02
0.4
-2.
0.5
2.0
1.0
2
0.3
-2.0
0.4
4.0
1.5
0.5
0.0
0.4
1.5
1.5
3.4 MNIST Dataset
In this study, we follow a similar experimental setup to Liu et al. (2020) and explore the relation- ship between generated image quality and estimated density using the MNIST dataset-a widely recognized benchmark comprising 70,000 grayscale images (28 x 28 pixels) of handwritten digits (LeCun & Cortes 2010). We trained DDPM this dataset and, by selecting the lowest diffusion timestep (t 1), obtained an estimate of the score function for individual images. Using this score, apply SD-KDE in latent space assess the realism of generated images. We ranked generated images from highest to lowest estimated probability density, visualized in Figure T6 The images with higher density appear more realistic and are correlated with higher quality.
3300.00
Left to right: (a) Density estimates obtained by vanilla KDE (blue) and by SD-KDE after one to four score-debiased iterations (warm colors). The surrogate samples produced by the final iteration (red, = 1000) visibly sharpen the bimodal structure relative to the raw data (blue, n = 1000). (b) The corresponding score functions converge toward the analytic score (black dashed), illustrating progressive removal of higher-order bias. (c) Kullback-Leii Leibler divergence falls by more than a factor of three after the first correction and attains its minimum at the second iteration before mild over-correction appears. (d) Monte-Care MISE (mean integrated square error over 200 replicates) mirrors the KI. trend, confirming that small number of SD-KDE steps yields the best bias-variance trade-off for this 1D Gaussian mixture.
Figure 7:
4 Proof of Theorem 1
Proof. First, we decompose the MISE into the bias and the variance terms as
MISE / (E [p(:)] p(=))² dr / (E [p(x)]] E [(x))))) do
Bias [xxx))² dx + /Var [p(x)) de,
where the variance term Var [p(x)] = E [}(x} - E {xx)) and the bias term Bias [p(x)] p(x) E [px)]. The variance term is given by
x x (X-(xx 58 58()))
Var [p[)]] - Var Td
h
since j((x) is a sum i.i.d. terms. Using Taylor expansion at the kernel around
a=x yields
h
K r (X + 53( (X))
x h
X
x -
K
x(x)
VK
+
h
h
h
h2
7

---

The variance is dominated by the leading order term, which gives
Var [p(x)] Var where we used the standard KDE variance result for the leading term. Now, we analyze the bias term.
K
X
82
=
82 nhd nha42)
1
n hd
n/?+a
Bias Elj((x)] p(a).
We write the expectation p(x:) as
1
(X + (8(x))
} /
hd K x (y + +o0(())) ply)dy.
1
(0-
E
E
K
hd
h
h
We substitute
=- to obtain
T [+(x))]
= /
K u
8(x huu) pla hu)du.
(1)
h
Taylor expansion will yield that
1²2 22" V
plx hu) =p(x)= Iuu Vo(x))() +
and that
K
3(x hu)
8 5(x) + du V +0()))(((0))
K
8² s(x) 2 V2K(()((o)) + (5h
82 83 +
K(u)= ((x) (Vx))>((()) + 8u VV())KK(()>
+ 8h
21
h 113
Substitute these expansions into Equation T.. and expand the product. We consider each term separately.
K(u)p(x) integrates to p(x) by the definition of K.
2. ((x) ( (K()p((r)) integrates to 0 since K is symmetric and decays to 0 at infinity.
3. K(u) (-hu" Vp(z)) integrates to 0 by the symmetry of K.
h² 4. K(u) (T integrates to Z Tr (V²p(z) J uu K())du) = - P2p(z).
5. (3()) T VK(u)- huu Vp(x)). Integrate by parts on VK(u) to obtain
83(2) /
(P)))VV(()dd 85(x) (-f K(w)V(os)d)) -08((z) Vp(z).
Using ((x) V log p(z), we have
(Vp()) -8((r) Vp(x) - (ogg((()) Vp(x) - |p(x))1
p(x)
Using standard multivariable calculus identity, have
|p(z)]] - (bge))
8
p(z)
6. V(z))u"V Again, after integration by parts, we obtain
of VVV = 5 p(x) (V)()))))(((())))))))))))))))()) (V5(x)) du = 6p(x))r (V8(x)) = 5p(x)) (loo))(((

---

T 7. p(z) 2x (2x) V (u) integrates to O.
Using smoothness, we then have that
1:2 E [ p(z) 2ー
0 8h 133
8² 63
2
Now, by choosing
A²
2 we make the leading term zero, and the bias E [p(x)] - p(x)
O h + 8h + b = O (h*). Using the standard KDE argument (symmetry of K and
decay to 0 at infinity), ) we can show that h terms in the bias also vanish. Thus, the bias is O (h²).
Moreover, note that h- so the variance term Var [f(=)] = O (wbet)
For optimal error scaling, we balance the bias and the leading variance terms. The error due to bias is O (1²), ), and the leading error due to variance is O
Balancing these terms, we obtain hopt = 0 (n (n-1// 1/(++)).
Finally, the MISE is O (h$) O (n (n(8/(4)))).
5 Additional Discussion
Connections to Langevin dynamics. We note that the algorithm is an analog to the continuous time Langevin dynamics, which uses the score function and yields that the stochastic differential equation
1 dx, 2 (X,) dt dB,
(2)
will have the stationary distribution according to the probability distribution function p. which corresponds to the score function (Song & Ermon 2019] [Song et al., (020). Our work can be viewed as one-step Euler-Maruyama discretization of the Langevin dynamics to estimate the location-shifted kernel from the sample points. This ensures both tractability as well as the benefit of bias-reduction as seen the main theorem (Theorem TT.. To our knowledge, this the first approach that employs Langevin dynamics to inform position-based debiased kernel density estimator.
Bridging Score-Based and Sample-Based Density Estimation. While the paper (Song al. 2020) suggests a formulation of the flow ODE as an evolution of the density function from an approximate posterior. This approach is prior-free and the flow maps scaled Gaussian distribution to the data distribution. However, this process does not utilize the availability of samples and relies solely on the score estimate. Since this scheme requires spatial and temporal discretization for density estimation, it is computationally less feasible due to the curse of dimensionality.
Many works in non-parametria methods (ie. KDE, histogram) (Silverman) 1986; Rosenblatt 1956 Parzen 1962, Scott 1979 Lugosi & Nobell 1996) and neural-based density estimation (Liu et al. 2021 Magdon-Ismail & Atiya 1998; Rezende & Mohamed| 2015) Dinh et al., 2016 [Berg al.
2018 use the sample points for the density estimation, but do not incorporate score function in the density estimation framework.
promising future direction to consider a multi-step discretization of the Langevin dynamics to obtain asymptotically superior debiasing. Using higher order discretization schemes is also an interesting avenue that we are currently exploring. The multi-step approach introduces more challenges, including non- non- -on-Gaussianity of the final kernel, since it will be convolution of multiple Gaussian kernels with different score-dependent shifts.
Conclusion
In this work, we demonstrate that incorporating score information can asymptotically improve density estimation accuracy. We propose method for score-debiased kernel density estimation that achieves O (n-88/(6 8/(d+8) convergence rate in mean integrated squared error, improving upon the classical O (n-1/((+1)) rate of standard KDE. Our experiments these theoretical predictions and show that the method remains effective even when using noisy score estimates, suggesting practical applicability beyond settings where the true score is known.
variety of synthetic datasets validate

---

Limitations. A key limitation of our proposed method is that the theoretical performance guaran- tees for SD-KDE require access to an exact score oracle, which is typically unavailable in practical scenarios. Although our empirical results demonstrate that accurate score estimates obtained from state-of-the-art methods (such as score matching or diffusion models) still provide significant per- formance improvements, these estimation methods themselves can be computationally expensive, particularly in nigh-dimensional settings or with large datasets. Future work might explore more efficient score estimation techniques or approximate methods that retain the benefits of SD-KDE while reducing the associated computational overhead.
References
Rianne den Berg, Leonard Hasenclever, Jakub M Tomezak, and Max Welling. Sylvester normalizing flows for variational inference. arXiv preprint arXiv: 05649, 2018.
Ricardo J. G. B. Campello, Davoud Moulavi, and Joerg Sander. Density-basee clustering based on hierarchical density estimates. In Jian Pei, Vincent S. Tseng, Longbing Cao, Hiroshi Motoda, and Guandong Xu (eds.), Advances in Knowledge Discovery and Data Mining, pp. 160-172, Berlin, Heidelherg, 2013. Springer Berlin Heidelberg. ISBN 978-3-642-37456-2.
Yen-Chi Chen. A tutorial on kernel density estimation and recent advances. Biostatistics & Epidemi- ology, 1(1):161-187, 2017.
Laurent Dinh, Jascha Narain Sohl-Dickstein, and Samy Bengio. Density estimation using real nvp. ArXiv, abs/ abs/1605.08803, 2016. URL https://api semanticscholar. wwwpssschhhaaroooorooooooo I
8768364
Will Grathwohl, Ricky T. Q. Chen, Jesse Bettencourt, and David Duvenaud. Scalable reversible generative models with free-form continuous dynamics. In International Conference on Learning Representations, 2019. URL http:stt////peeeeeeeeee wwt.:/neee/eeeeeee wowrooodnnnndneeeaeaeee
Emmanuel Guerre, Isabelle Perrigne, and Quang Vuong. Optimal nonparametric estimation of first-price auctions. Econometrica, 68(3):525-57) --74, 2000.
Hanyuan Hang, Ingo Steinwart, Yunlong Feng, and Johan A.K. Suykens. Kernel density estimation for dynamical systems. Journal of Machine Learning Research, 19(35):1-49, 2018. http
//jmlr.
1191199999999911111 html
Jonathan Ho, Ajay Jain, and Picter Abbeel. Denoising diffusion probabilistic models. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS '20, Red Hook, NY, USA, 2020. Curran Associates Inc. ISBN 9781713829544
Aapo Hyvärinen and Peter Dayan. Estimation of normaazlll statistical models by score matching. Journal of Machine Learning Research, 6(4), 2005.
Yann LeCun and Corinna Cortes. MNIST handwritten digit database. 2010. URL http://yann
lecun cww. hetp//ddbb/mmiimee
Qiao Liu, Jiaze Xu, Rui Jiang, and Wing Hung Wong. Roundtrip: A deep generative neural density
estimator. 2020. doi: 10. 00.1773/nas.2100344
Qiao Liu, Jiaze Xu, Rui Jiang, and Wing Hung Wong. Density estimation using deep generative neural networks. Proceedings of the National Academy of Sciences, 118(15)ee1013344111 2021. doi: 00.1073/paa.2101344118 URL htttp:///wwt www. pnas, wttpa//////o///////..00000 pnas 2101344118
Gábor Lugosi and Andrew Nobel. Consistency of data-driven histogram methods for density estimation and classification. The Annals of Statistics, 24(2):687-706, 1996.
Malik Magdon-Ismail and Amir Atiya. Neural networks for density estimation. Advances in Neural Information Processing Systems, 11, 1998.
Emanuel Parzen. On estimation of probability density function and mode. Annals of Mathemut- ical Statistics, 13:1065-1076, 1962. URL. htt.:https:///e //api. semanticscholar www.aarr.rrrooooo Arrg/orrusID.
122932724
10

---

Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. In International conference on machine learning, pp. 1530-1538. PMLR, 2015.
Murray Rosenblatt. Remarks on Some Nonparametric Estimates a Density Function. The
Annals of Mathematical Statistics, 27(3):832 837, 1956. doi: 10. 10.1214/0mm/11177729999 URL
https://doi /ttp://///oo.oo//1..o. 1214 20214/20mm1177721119009
David Scott. On optimal and data-based histograms. Biometrika, 66(3):605-610) 1979.
David W. Scott. Multivariate density estimation and visualization. 2012. URL hattps://pp
semanticscholar, wrg/CorpusIDD.......o 1253508
Bernard W Silverman. Spline smoothing: the equivalent variable kernel method. The annals of Statistics, pp. 898-916, 1984.
Bernard W Silverman. Density Estimation for Statistics and Data Analysis, volume 26. CRC Press, 1986.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456. 2020.
Dongling Zhang, Yingjie Tian, and Peng Zhang. Kernel-based nonparametric regression method. In 2008 IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology, volume 3, pp. 410-413. IEEE, 2008

---

A Synthetic 1D experiments
Figure and Figure To shows the fitted densities for different noise levels of the SD-KDE method, as well as the Silverman baseline, for 200 samples for three different Gaussian (and Laplace respectively) mixture models, with parameters outlined Table I In Figure [1] we examine the consistency of the performance gains for the SD-KDE method over the Silverman baseline for mixture Laplace densities. The Laplace mixtures use the same location and scale parameters as Gaussian Mixture, given in T
Next, show the scaling in for density estimation task for Laplace Mixtures. Figure shows the results. In Figure [2] and [3] we show a visualization of the score function and the densities for both the Gaussian and Laplace Mixtures.
Che pocke Mistrro
Carmles Mir
Gresinn Mistrre
1:3
1.0
HER
Figure 8: Drawing 200 samples from each of the three Gaussian mixtures in equation 3.1 The dashed black line is the true PDF, while the colored lines represent the estimated PDFs.
Luplacs
Since Deg
wald
Sarmia KDC
SD400 01-2
50400
Figure 9: MISE error as function of for each of the three gaussian mixtures. For each point, we compute the MISE with 50 random seeds per mixture. Each subplot plots the mean integrated squared error as function n. The legend compares Silverman KDE SD-KDE at multiple noise settings. The slopes inside each subplot are fitted regression lines in log- scale indicating how quickly each method's error decays as n increases.
B Synthetic 2D mixture of Gaussians
In Figure 15 mixture of Gaussians ground-truth density, compare the Silverman method with SD-KDE.
C MNIST Dataset Image
The following figure depicts the ordering of generated images based on estimated probability density values.
12

---

Saplas Mittern
Ladann Natues
012
Figure 10: Drawing n = 200 samples from each of the three Laplace mixtures in equation 3.1 The dashed black line is the true probability density function, while the colored lines represent the estimated probability density functions.
11300 31B80200-0000
Figure 11: Histogram of MISE difference of the SD-KDE method and the Silverman method, for 100 samples and 50 random seeds mixture. A positive value in the plot indicates that the SD-KDE method performed better for that seed. We observe that SD-KDE consistently performs better than the Silverman method over multiple random seeds.
0.01
0.2
015
0.0
Figure 12: In each subplot, we plot the Gaussian mixture's 's density (blue, left axis) and the log-density derivative (score) in red (right axis).
Midter Logher
Manoe Sapinn
1.2
0.3
Figure 13: In each subplot, we plot the Laplace mixture's density (blue, left axis) and the log-density derivative (score) in red (right axis).
13

---

Exclrtie Secad)
Diaggie R3
Dusaken
2122
Left right: (a) Density estimates obtained by vanilla KDE (blue) and by SD-KDE after one to four score-debiased iterations (warm colours). The surrogate samples produced by the final iteration (red, (blue, 1000). (b) The corresponding score functions converge toward the analytic score (black dashed), illustrating progressive removal of higher-order bias. (c) Kullback-Leibler divergence falls by more than factor of three after the first correction and attains its minimum at the second iteration before mild over-correction appears. (d) Monte-Carke MISE (mean integrated square error over 200 replicates) mirrors the KL trend, confirming that small number of SD-KDE steps yields the best bias-variance trade-off for this ID Laplacian mixture.
Figure 14:
1000) visibly sharpen the bimodal structure relative the raw data
Silverman KDE
Score-Debler Deblased KDE r=1000, h=0.365 MISE=0.000570
True Density
n=1000, h=0.382 MISF=00.00287
n=1000
4
2-
2-
2-
-2-
-2
-2
-4
-4
-2
-2
-2
Figure 15: Comparison true 2D mixture of Gaussians density vs. the Silverman method and our SD-KDE method using the true score. Given the oracle score function, SD-KDE outperforms Silverman MISE by nearly an order of magnitude.
NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims in the abstract and the introduction match the theoretical and experimental results. The three main claims are clearly stated in the introduction, and then proved/shown empirically in the remaining sections.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long clear that these goals not attained by the paper.
14

---

Highest Density Value
Lowest
2
2
2
3
3
3
3
7
7
7
Figure 16: Generated MNIST images of digits 2, 3, and are displayed in descending order of estimated probability density as determined by score-based KDE, The ordering illustrates that images with higher probability density estimates exhibit more realistic features.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We have limitation section which addresses the strong assumptions made, and the difficulties that come with these assumptions. See Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
15

---

Justification:
Yes, we have single theorem and that has a correct proof.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Yes, the details of the experiments are all laid out in the document. Any datasets which were used are either publicly available or easy to generate by following the the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
16

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: Yes, we provide code for all experiments in the supplementary material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ peabls..c/publli/guuueeeese CoodeSubeeieeeeee ssi iessonPolll mollicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All the parameters, such as the noise level or the bandwidth were specified. All the training details, such the dataset and which parameters for mixture models were used, were specified.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification:
We include error bars in key Figures such as Figure
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
17

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Yes, we explicitly provided the computing hardware and runtime in 3.1
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: Yes, the paper conforms with the NeurlPS Code Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- cration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: The paper on a novel method for density estimation. very far removed from any societal concerns.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
18

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All assets are original or explicitly mentioned/properly cited.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
19

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Any of the training data which is not public is synthetic, and we detailed clearly how to reproduce it. The code is documented in the aforementioned released version.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
20

---

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [Yes]
Justification: An LLM used in the ideation phase in the proof of the theorem.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non standtaddard components.
Please refer to our LLM policy (https: for what should or should not be described.
21


Reviews:
[{"review_id": "ToFdmqpbWs", "reviewer": "Reviewer_ftbi", "summary": "The paper presents a simple modification of the classical Silverman Kernel Density Estimation (KDE) method. It relies on having access to the score function of the data generating process (which is generally impossible) but empirical results show that a learnt surrogate (through a diffusion model) also performs very well. Importantly, the authors provide asymptotic results that demonstrate an improvement in performance over Silverman's method in the ideal case of a known score function.", "questions": "Standard KDE is super-fast; what is the empirical scaling of the method? Also, learning a score requires a fairly large number of points, can you experiment to see what is the behaviour of the learnt density as you vary this parameter? Finally, I am left somewhat uncertain as to what is the advantage of this approach to directly using a generative model such as diffusion, particularly since the score needs to be learnt training a diffusion model. Overall though I enjoyed the paper, I think it is good work.", "limitations": "Yes", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- Solid theoretical result;\n- Practical algorithm for a real world problem;\n- Empirical evaluation supports theory.\n\nWeaknesses:\n- Computational costs, particularly when the score needs to be learnt;\n- Theoretical results require exact knowledge of the score, the sensitivity to errors in score estimation is unknown;\n- The empirical evaluation is mostly on toy data/ well used data sets.", "was_revised": true, "final_justification": "The discussion has clarified some important concerns. I still think this paper introduces novel ideas to a classical field and I would like to recommend acceptance.", "rebuttal": "We would like to thank reviewer ftbi for their thoughtful review, which has made our paper better.\n\n“Standard KDE is super-fast; what is the empirical scaling of the method?”\n SD-KDE inherits the runtime profile of standard KDE. The only extra work is evaluating a score nnn times (once per query/sample). In many common settings this overhead is negligible—either because the score has already been computed or cost is low compared to standard KDE, or because the score is estimated directly from the KDE—so the effective empirical scaling is the same as standard KDE. Importantly, SD-KDE is independent of any particular score–estimation technique; the wall-clock time is that of standard KDE plus the cost of the practitioner’s chosen score oracle.\nTo make this concrete, we will include in the revision: (i) a scaling plot comparing wall-clock time vs. number of queries for standard KDE and SD-KDE across our datasets, and (ii) a minimal timing script so reviewers can reproduce the results. These additions transparently show that SD-KDE’s runtime matches standard KDE up to the n-fold score evaluations described above.\n\n\n\n\n“Also, learning a score requires a fairly large number of points, can you experiment to see what is the behaviour of the learnt density as you vary this parameter?” \nOn “learning a score requires many points” and sensitivity to data size: thank you for raising this. SD-KDE does not necessarily require a learned score: in some of our experiments the score was taken directly from the KDE (via its gradient), so the method could work with exactly the same data requirements as standard KDE. The score oracle is modular; if one chooses to learn a score, that is optional and orthogonal to SD-KDE itself.\nThat said, we agree that it is useful to see sensitivity when a learned score is used. In the revision, we will add an ablation that varies the number of training points for the score estimator while keeping the KDE fixed. Concretely, we subsample the score-training set and fit a standard score estimator on each subset, and report SD-KDE quality versus (i) pure KDE and (ii) SD-KDE with the KDE-derived score. We typically observe that performance improves smoothly with more score-training data and plateaus once score variance is below the KDE’s own smoothing error; importantly, even at small training sizes SD-KDE remains competitive because the KDE already provides a usable score. We will include the sensitivity plot and a minimal script for reproducibility in the revision. \n\n“Finally, I am left somewhat uncertain as to what is the advantage of this approach to directly using a generative model such as diffusion, particularly since the score needs to be learnt training a diffusion model.”\nWe appreciate the question. SD-KDE does not require training a diffusion model: the method works out-of-the-box with a score obtained directly from the KDE (via its gradient), so its data and compute requirements match standard KDE. When a learned score is available, SD-KDE offers a different value proposition from training a full generative model: it converts any score oracle–analytic, pretrained, or lightweight–into a normalized, likelihood-evaluable density with the calibration and interpretability of KDE. In contrast, diffusion models are optimized for sample quality and typically do not provide tractable likelihoods without additional, costly machinery. Crucially, our theory shows that density error is controlled by score error under the stated metric and that this error is still controlled under a score estimate. Hence an approximate or mismatched score can still improve over plain KDE when it is “good enough”–and SD-KDE reduces to standard KDE if not. This modularity is practically useful: (i) many workflows already produce a score as a byproduct (e.g., from a previously trained model), which SD-KDE can reuse with no retraining; (ii) in domains like tabular or scientific data, training a high-capacity diffusion model is often unnecessary, while SD-KDE yields gains at the cost and sample size of KDE. \nA practical advantage of SD-KDE is that it can reuse off-the-shelf scores. For instance, a score pre-trained on generic natural images can serve as a useful prior when our target dataset consists only of dog images. While the dog-specific score will differ, natural images share strong regularities—edges, textures, and sparse multi-scale structure (e.g., curvelet-like representations in the sense of Donoho) and broader manifold structure—so the generic score is a reasonable approximation. Under our theory, density error tracks score error; thus an approximate but “good enough” score can yield clear gains over plain KDE, and if it isn’t good enough SD-KDE defaults to standard KDE.\nWe hope this clarifies that SD-KDE is complementary to diffusion models: it leverages a score if one is available and delivers a calibrated, normalized density with competitive accuracy.\n\n“Theoretical results require exact knowledge of the score, the sensitivity to errors in score estimation is unknown.”\nThank you for the opportunity to clarify this point. While Theorem 1 is stated for an oracle score, Corollary 1 directly addresses the estimated-score case and quantifies sensitivity. Under the same bandwidth and step size as Theorem 1, the corollary gives an explicit bias expression whose leading term is proportional to the size of the score error (and its first spatial derivative), with higher-order terms suppressed. In plain language: the bias grows linearly with the score error, and the added MISE term grows with the square of that error. Thus SD-KDE is stable to misspecification, and approaches the oracle behavior as the score improves."}, {"review_id": "GKMJL8gaxr", "reviewer": "Reviewer_f5yx", "summary": "This paper provides a new kernel based density estimate method that leverages the \"score\" (the gradient of the log-density, which shows up in diffusion models).  Each data sample is moved in the direction of the score (essentially denoising the samples), then a KDE is built on the updated points.  The paper shows:\n   - this significantly increases the rate of estimation from O(n^{-4/(d+4)}) to O(n^{-8/(d+8)})\n   - shows improvement in practice in 1 and 2 dimensions (and perhaps on MNIST)\n   - if one can only estimate the score, then one can get nearly as much improvement.", "questions": "Is it possible to obtain a stronger total bound on the convergence of the density estimate, without assuming the score oracle, but rather estimating the score and using that (as shown to work empirically)?  Are there obvious reasons why this is non-trivial?", "limitations": "yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "The O(n^{-4/(d+4)}) convergence result (of Silverman 1984) is classic, and so improvement on this is very exciting.  Directly improving it would merit strong acceptance.  \nThe paper falls a bit short of this, since it uses the score as an oracle in the result, and then bounds error in estimated score, and shows how to estimate it in practice (which still shows improvement empirically on Silverman's result, but not as much as with the oracle).  \nThe paper does not provide an improvement in density estimate without knowing this oracle or having some known bound on the accuracy of the score.  \n\nRegardless, I find this as very important progress on a classic and very fundamental result, one in which the line of work was thought to have been perhaps closed.  \nI suspect this will soonafter lead to future work which will close this loop on this approach -- I wish it was in this paper.  \n\n\nSo there are many important strengths to the paper, but here are some relatively minor weaknesses:\n\n - The paper would be improved with better early-on discussion (and mathematical definition) of the score function.  It may be fairly mysterious to those only familiar with classic density estimate results.  \n\n - Results in 1, 2 dimensions where the results are more meaningful is ok, but it would be useful to show results for a variety of settings in high-dimensions, other than the simple qualitative MNIST experiment.  \n\n - The proof is informal in a few places. \n   *  \" we can show the h^3 terms in bias vanish\"\n   *  \"thus the bias is O(h^4)\" then a few lines later \"The error due to bias is O(h^8) ...\"\n\nAs such I was not able to verify the proof.  I think I see how this works, and I suspect I could figure it out -- but for a result like this, I expect the paper to connect these parts much more carefully.", "was_revised": true, "final_justification": "The paper makes exciting progress on a classic and important problem.  This is very unexpected.  However it is conditioned on oracle access to the \"score\" function.  This seems to be easy estimate, but it is not clear in theory, and the emperical experiments (which show improvement) are limited.  \n\nIt may be an important paper, but some questions remain.", "rebuttal": "We would like to thank reviewer f5yx for their thoughtful review, which has made our paper better.\n\nW1(score definition): We thank the reviewer for suggesting to allocate more space early on in the paper to formally define the score, and some key properties and applications. We will update this for the camera ready paper.\n\nW2(informal steps in proof):\nThe error is h^4 and in a few line after becomes h^8 because h^4 is the bias, and h^8 is the mean square error contributed from the bias (so it is (h^4)^2 = h^8). \n\nI understand that the vanishing of the h^3 term is vague but to expand all the term is actually quite laborious, though possible. We can also do that, but it does not provide much intellectual benefit for the reader. We will include the expansion in the appendix of the camera ready paper.\n\nQ1(stronger bound without assuming score estimate): The analysis of KDE (as well as SD-KDE) relies on the separability of MSE into bias term and variance term. The bias term (before squared) is linear, making it tractable. If we want to include the error from score, the error will go in to the bias term in a non-linear manner (before expectation), so it will become intractable."}, {"review_id": "VwQ6qUaIvN", "reviewer": "Reviewer_etfG", "summary": "The paper presents SD-KDE (Score-Debiased Kernel Density Estimation), a method that enhances classical KDE by moving each sample point along the estimated score function before applying KDE with an adjusted bandwidth. This single-step correction cancels the leading-order bias and improves the MISE. The authors provide a complete theoretical analysis and demonstrate consistent performance improvements on synthetic 1D/2D data and MNIST.", "questions": "- How does SD-KDE perform in high-dimensional spaces with limited data and noisy scores?\n- How does SD-KDE compare with neural density estimators like normalizing flows or autoregressive models in terms of accuracy and efficiency?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths  \n\n- Reduces KDE bias using score-guided data shifts, improving estimation accuracy.  \n- Achieves provable MISE improvement.\n- Experimental results confirm strong gains across settings and noise levels.  \n- Performs well even when the score function is noisy or estimated from data.  \n- Iterative application enables further bias reduction without requiring a score oracle.  \n- Bridges score-based generative modeling and nonparametric density estimation.\n\nWeaknesses\n\n- Theoretical results require access to the exact score function, which is rarely available.  \n- No empirical comparison with modern KDE variants beyond Silverman’s rule. For instance, recent approaches such as Fast KDE with density matrices and Random Fourier Features (Gallego et al., 2022) and the diffusion-based KDE method diffKDE with optimal bandwidth approximation (Pelz et al., 2023) are not considered.   \n- Step size $\\delta$ and bandwidth $h$ may require careful tuning; robustness not analyzed.  \n- High-dimensional and structured data domains remain largely untested.  \n- MNIST results are mostly visual; lacks quantitative metrics for density quality.", "was_revised": true, "final_justification": "**Concerns:** Exact score assumption, few baselines, tuning sensitivity, low-dim focus.  \n**Response:** Clarified non-learned score use, added MNIST confidence metric and 10D Gaussian mixture, explained $\\delta$–$h$ relation and robustness, committed to modern KDE comparisons.  \n**Outcome:** Maintained borderline accept.", "rebuttal": "We would like to thank reviewer for their thoughtful review, which has made our paper better.\n\nW1(Theoretical results require exact score): We believe that it may be possible to extend the analysis in the current work to situations where we don’t know the exact score, this will be the topic of follow up research.\nW2(Visual results on MNIST): In our revision, we have added quantitative evaluations to better support the MNIST results. Specifically, we use Classifier confidence: The average top-1 confidence, \\frac{1}{N} \\sum_{i=1}^{N} \\max_{k} \\, P(y = k \\mid \\text{image}_i),  from a pretrained MNIST classifier on generated samples, which serves as a proxy for sample realism. This is more clearly showing the advantage of our method.\nW3(Tuning of step size and band width): Once the band-width is set, there is a fixed formula for the step size based on the band width. We observed that using a similar constant on the band width term as in the Sliverman method works well without being very sensitive to the exact constant value. We will perform an in-depth analysis in the camera ready version of the paper.\nW4(comparisons with other KDE methods): We want to thank the reviewer for this valuable suggestion, we will compare the SD-KDE method with the more recent methods for the camera ready paper.\n\nQ1(high dimensional setting): Thank you for the suggestion, we have included 10 dimensional gaussian mixture results for the camera ready paper. \nQ2(efficiency comparison with normalizing flow and autoregressive modeling): Compared with these neural approaches, SD-KDE offers a much simpler fitting scheme, where each point is moved in the direction of the score and then applying standard KDE."}, {"review_id": "hcrLqR9btz", "reviewer": "Reviewer_cUSv", "summary": "This paper debiased KDE using an estimated score function.", "questions": "I don't quite see a clear path for me to raise my score.", "limitations": "yes.", "rating": 2, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 1, "originality": 1, "strengths_and_weaknesses": "The primary weakness is the lack of novelty and motivation. KDE is non-parametric and not scalable. In its comfort zone, there are many ways to improve the performance of a standard KDE without invoking diffusion models, and of course the silverman baseline. Theorem 1 is also highly idealized.", "was_revised": true, "final_justification": "My score will remain unchanged. I think a revision would make the paper much stronger.", "rebuttal": "We thank the reviewer cUSv for their review. \n\nWe would like to point out that SD-KDE can be applied effectively without invoking diffusion models, in Figure 5 in the paper, SD-KDE is applied with a score estimate from the Silverman KDE. \n\nW1 (lack of novelty/motivation): As described in our paper, kernel density estimation is an important modeling technique in a variety of fields, such as anomaly detection, clustering, data visualization, and dynamical systems. We provide\n1. A theoretical results for better asymptotic convergence\n2. A simple algorithm to compute the SD-KDE\n3. Practical empirical analysis showing the improved performance of the method\n\nBased on this, we hope that the reviewer will reconsider the low scores given to the paper."}, {"review_id": "PP6ogMbxgI", "reviewer": "Reviewer_4YAs", "summary": "Kernel density estimation is a classical density estimation technique which yields properly normalised nonparametric density estimates. Score based methods match only the score function (gradient of log density), which does not require the normalising constant, and therefore does not yield a density estimate in a classical sense. The authors propose to use information in (an estimate of) the score function to assist in KDE, yielding properly normalised score debiased KDE (SD-KDE). This is achieved through updating the original data samples in one step of the gradient of the score function. A theoretical result shows improved estimation error compared with naive KDE, as well as optimal bandwidth and step size selection, in the case where the exact score function is known. Experiments are demonstrated on a range of low dimensional settings, showing good benefits above classical KDE.", "questions": "I am currently borderline between reject and accept. I err on the positive side (believing in advance that the authors will give a reasonable rebuttal) and give accept for this review. I am unlikely to raise my score due to the main weakness of corollary 1, unless the authors can somehow improve the result or apply it to a concrete setting. Actionable summary of my weaknesses:\n\n- Please discuss whether something more practical can be salvaged from Corollary 1. For example, is it possible to upper bound the integrated RHS? Are there any takeaways for practical score estimation?\n- Please answer my question about whether independent training set is used for score matching versus the KDE step. Include any modifications you might make to an updated manuscript.\n- Please address the minor comments.", "limitations": "Yes.\n\nLimitations are pointed out in the paper, and agree with the major weakness in my review above. I am not concerned about negative societal impact for this broad application-agnostic paper.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths:**\n- Algorithm 1 is very elegant and clean. It is easy to read and implement. The intuitive description provided by the authors about using the score to debias the samples is helpful.\n- I liked that the authors showed two demonstrations of approximate score functions. One involves an initial KDE and then using the score, and the other involves the more standard neural estimator via Hyvarinen score or Fisher divergence. (Unforunately, the neural estimator seems to perform not so well, but this is a separate point). \n\n**Weaknesses:**\n- The presentation of theorem 1 could be improved, in my opinion.\n    - First, there is no need for the first equation, as it is just substituting step 1 into step 2 of algorithm 1, which is already referenced in the theorem text. \n    - Second, I believe \"asymptotically optimal\" is more precisely \"the smallest AMISE\". Please correct me if I'm wrong.\n    - Third, \"Let $\\hat{s}$ be the exact score function of $p$.\" could more precisely be stated as \"Let $\\hat{s}$ be the exact score function of $p$, i.e. $\\hat{s} = s$\". \n- There is a gap between Corollary 1 and a notion of \"error\" obtained by standard scoring algorithms. Unfortunately, I can't really see how corollary 1 is useful in its current form. It would be nicer if we could relate the AMISE to some error in the score estimate (e.g. the error in the Hyvarinen matching). This I see as the biggest weakness of this paper. This is acknowledged in the limitations section, by the authors. Would it be possible to upper bound the first term in the RHS of corollary 1?\n- Figure 16 is qualitative and it is difficult to assess its meaning. I am not convinced that the images on the right are less representative than the images on the left (I don't have any suggestions for improvement, unfortunately).\n- The proof of theorem 1 reveals that some conditions are missing from the statement of theorem 1. Namely, the kernel has to have a convergent Taylor series (satisfying Taylor's theorem)\n\n**Questions:**\n- Does one (or should one) use the same datapoints for both the score estimate and the SD-KDE, or should you use different points? Would it make sense to first estimate the score using an independent pretraining set? Why or why not, and did you try both ways?\n\nMinor:\n- There are three counts of \"the theorem 1\". Recommend changing to \"Theorem 1\", to retain consistency throughout. \n- Line 79 uses the nonstandard notaiton $\\mathcal{N}(0, \\sigma)$, which maybe should be $\\mathcal{N}(0, \\sigma^2)$. \n- I believe \"std\" in the legends in Figure 2 should be $\\sigma$, however this is not mentioned in the text.", "was_revised": true, "final_justification": "I previously noted that I would maintain my score, if the authors gave a good rebuttal. I'm satisfied with their response to my questions and to the points raised by other reviewers.\n\nIt seems that most other reviewers find this an interesting and mostly sound paper (with minor improvements pointed out in the reviews --- different ones each caught by different reviewers), sufficient for NeurIPS. \n\nI am puzzled by Reviewer cUSv's review. Perhaps they have valid points but perhaps not. Given they have not engaged in the discussion so far, I am inclined to make a hard (binary) decision to disregard their review in my final justification.\n\nUPDATE\nI agree that there is a gap between knowing the score function and estimating the score function, and the paper does not address this gap. A much stronger paper would include a result that accounts for inexact score. (Bear in mind, knowing the score function does not mean knowing the normalising constant. For example, we can estimate the posterior of a Bayesian model with the exact score function using this method). However, all things considered, I am still inclined to borderline accept the paper, because it offers quite a new perspective.\n\nAt the end of the day, I will not be upset if the AC decides to reject the paper.", "rebuttal": "We would like to thank reviewer 4YAs for their thoughtful review, which has made our paper better.\n\nW1 (presentation of theorem 1): Thank you for these suggestions, we will update as you suggested to make the presentation more clear and concise. \n\nW2 (corollary 1): The main intention of corollary 1 is to give some intuition on how the bias decompose when we only have an estimate for the score. In the two terms one is proportional to the gradient of p, which will be small near modes, and the other is proportional to p, and will be small in the tails of the distribution. Taken together, we can get practical intuition that the bias may concentrate more on moderate density regions rather than extreme trail or modes. We thank the reviewer for pointing out that this is not central to the paper, and we will hence move it to the appendix for the camera ready.\n\nW3 (MNIST figure): In our revision, we will quantitative evaluations to better support the MNIST results. Specifically, we use Classifier confidence: The average top-1 confidence, \\frac{1}{N} \\sum_{i=1}^{N} \\max_{k} \\, P(y = k \\mid \\text{image}_i),  from a pretrained MNIST classifier on generated samples, which serves as a proxy for sample realism. This is more clearly showing the advantage of our method.\n\nW4 (convergent taylor series condition theorem 1): Thank you for pointing this out, we have updated the script with this condition.\n\nQ1 (same vs different points SD-KDE): In practical settings, often only one dataset of points is provided, hence in our experiments with using the KDE to estimate the Score for SD-KDE, we estimated the score from the same set of datapoints as was used for SD-KDE.\nIn the updated manuscript we will include a detailed analysis of the tradeoff between using independent vs overlapping samples for the estimates. \n\nMinor comments:\nThank you for pointing out the formatting errors, indeed, N(0,σ) should be N(0,σ^2), we will use Theorem throughout, and will update the legend in figure 2 to use σ instead of std for the camera ready."}]

Author rebuttal:

Meta-review: Most reviewers agree the paper provides some compelling theoretical results on the standard problem of density estimation, that are supported by empirical evidence. The approach provides a quite new perspective on KDE and provably show that the knowledge of the score can be used to obtain a more efficient KDE estimator.

Reviewers cUSv shares a concern that the proposed method still suffers from limitations of KDE (scalability in high dimensions) and that it requires the knowledge of the score function. Reviewer f5yx shares some of the concerns, especially whether one can show how an estimate of the score (and not the true score) can provably improve the estimation.

This suggests that the paper opens new perspectives and questions: Can one still get improved estimates, when using an inexact estimate achieving some given error? How should the error scale with data in order to get such improvement? 

We hope that by accepting this initial work, it would encourage more research in that direction.
Decision: Accept (poster)