NeurIPS 2025

SD-KDE: Score-Debiased Kernel Density Estimation
Elliot L. Epstein1∗
Rajat Dwaraknath1∗
Thanawat Sornwanee1∗
John Winnicki1∗
Jerry Weihong Liu1∗
1Stanford University, Stanford, CA 94305, USA
{epsteine, rajatvd, tsornwanee, winnicki, jwl50}@stanford.edu
Abstract
We propose a novel method for density estimation that leverages an estimated score
function to debias kernel density estimation (SD-KDE). In our approach, each data
point is adjusted by taking a single step along the score function with a specific
choice of step size, followed by standard KDE with a modified bandwidth. The
step size and modified bandwidth are chosen to remove the leading order bias in the
KDE, improving the asymptotic convergence rate. Our experiments on synthetic
tasks in 1D, 2D and on MNIST, demonstrate that our proposed SD-KDE method
significantly reduces the mean integrated squared error compared to the standard
Silverman KDE, even with noisy estimates in the score function. These results
underscore the potential of integrating score-based corrections into nonparametric
density estimation.
1
Introduction
Kernel density estimation (KDE) (Rosenblatt, 1956; Parzen, 1962) is a widely used nonparametric
method for estimating an unknown probability density function from a finite set of data points.
The classical KDE effectively smooths the data by convolving with a kernel function, such as the
Gaussian kernel, and then normalizing the result to obtain a density estimate. KDE finds application in
diverse fields such as anomaly detection, clustering (Campello et al., 2013), data visualization (Scott,
2012), nonparametric statistical inference (Guerre et al., 2000; Zhang et al., 2008), and dynamical
systems (Hang et al., 2018).
The classical KDE suffers from a well-known bias-variance trade-off, controlled by the choice of ker-
nel bandwidth (Silverman, 1986). Larger bandwidths lead to smoother estimates with lower variance
but higher bias, while smaller bandwidths yield more variable estimates wi...