Kernel Density Estimation and Non-Parametric Regression
By: Florian Heinrichs
Tags: Statistics, Data Science, Regression, Machine Learning

Kernel Regression

Figure 1: Worldwide Google searches for “chocolate”; x-axis: time, y-axis: searches in percent w.r.t maximum; Image by author

Let’s start with an example. Imagine you are a data scientist working in the chocolate department of a candy factory. (I truly hope that this niche of data science exists.)

You might want to predict the demand of chocolate based on its historic demand and as a first step want to analyze the trend. The demand of chocolate in the years 2004–2020 might look similar to the data in Figure 1.

Clearly, there is some seasonality and the demand increases in winter, but as you are interested in the trend, you decide to get rid of these fluctuations. To do so, you might calculate moving averages with a bandwidth of b months, i.e., for each time instant t you calculate the average of the demand in the time frame from t-b to t+b.

More formally, if we have data observed over time X(1), …, X(n), i.e. a time series, the moving average with bandwidth b can be defined as

It can be seen in Figure 2, that the moving average is somehow a smoother version of the original data and that the degree of smoothness varies, depending on the bandwidth. The larger the bandwidth, the smoother the function.

Figure 2: Moving average with bandwidths 6, 24 and 42; x-axis: time, y-axis: searches in percent w.r.t maximum; Image by author

The choice of the bandwidth is crucial, but it is not clear how to choose it. If the bandwidth is too small, we might not get rid of seasonal fluctuations. If the bandwidth is too large, we might not capture the trend. For instance, if we choose the bandwidth b=0, we have the original data with its seasonality. Contrarily, if b=n, we simply have the average over all observations and cannot see any trend.

In the example, b=6 months is a reasonable choice to “smooth away” the seasonality since we calculate averages over entire years (13 months). However, b=12 or b=18 are equally valid choices. Depending on the choice of b, we give more weight to observations in the same season as our time instant t (b=12) or to the opposite season (b=6 or b=18). A possible solution to mitigate this problem is to give different weights to the observations, thus calculating a weighted mean rather than a simple mean.

Intuitively, observations close to the time instant t are more important and should have larger weights than more distant observations. This allows us to choose a larger bandwidth b while giving more weight to more important observations. For example, we could give weight 1 to the observation at time t and 1-i/12 to observations with distance i, for i=1, …, 11, or generally 1-i/b, for i=1, …, b-1 and bandwidth b (see Figure 3).

Figure 3: Weighted moving average with bandwidths 6, 24 and 42; x-axis: time, y-axis: searches in percent w.r.t maximum; Image by author

This is the basic idea behind kernel estimation: giving different weight to observations with a different distance.

The above choice of weights (1-i/b) is rather arbitrary and other weights can make sense as well. We can generalize the weighted moving average as follows: let b be a bandwidth as before and K a kernel, i.e. a function with some additional properties such as non-negativity [K(x) ≥ 0] symmetry [K(x)=K(-x)] and normalization [the integral over K is 1]. Many kernels have as support the interval [-1, 1], which means they are 0 outside of the interval.

Important examples of kernels are the Epanechnikov kernel K(x) = 3/4 (1-x²) for |x| ≤ 1 and the Gaussian kernel K(x) = 1/sqrt(2π) exp(-x²/2).

Based on the kernel K and the bandwidth b, we define the Nadaraya–Watson estimator as

In Figure 4, we see the Nadaraya-Watson estimator with Gaussian kernel and bandwidth b=12. The choice of kernel and bandwidth remain important, but the estimators with frequently used kernels (such as Epanechnikov, Quartic or Gaussian) are more robust with respect to the choice of the bandwidth than the moving average estimator.

Note that the simple moving average estimator itself is a kernel estimator with the uniform kernel K(x) = 1/2. The weighted moving average estimator is also a kernel estimator, can you figure out the corresponding kernel in this list?

Figure 4: Nadaraya-Watson estimator with Gaussian kernel and bandwidth 12; x-axis: time, y-axis: searches in percent w.r.t maximum; Image by author

Further note: First, the Nadaraya-Watson estimator is often defined based on rescaled time, that is i/n instead of i, and the formula changes accordingly. Second, the estimator can be defined more generally for observations with varying distance.

Kernel Density Estimation

Let’s consider another example. For one reason or another, you might be interested in gas prices in Germany. So you crawl the internet to get the current prices at all 14,000 gas stations. In Figure 5 is a common representation of this data: a histogram. The histogram displays the distribution of the gas price. The bandwidth of the blocks is a crucial parameter and different choices lead to different histograms (similar to different bandwidths of moving average estimators).

Figure 5: Histogram showing frequency of gas prices in Germany (05/12/2020) with 10 and 50 bins respectively; x-axis: price of gas in EUR; y-axis: frequency; Image by author

If we assume that the distribution of the gas price is continuous, we might prefer to estimate and visualize the density function of the underlying distribution. The idea is fairly simple: For any point in the range of the gas prices, we count how often the price is close to this point and give more weight to prices that are close and less weight to those that are more distant.

If the part “different weights, depending on distance” rings a bell, that’s good! Because this was the idea behind kernel regression and if an idea works, we reuse it.

The kernel density estimator of data X(1), …, X(n) is defined very similar to the Nadaraya-Watson estimator. Given a kernel K and a bandwidth h > 0, define

Often, the same kernel functions as in the case of kernel regression are used (e.g. Gaussian, Epanechnikov or Quartic). Kernel density estimates can be interpreted as smooth histograms and give insights about the distribution of the underlying data generating process. The choice of kernel and bandwidth are again crucial (see Figure 6 for different estimators).

Figure 6: KDE of the density of gas prices with different kernels (top: Epanechnikov, bottom: Gaussian) and different bandwidths (left: 0.05, right: 0.1); x-axis: price of gas in EUR; y-axis: frequency; Image by author

Implementation in Python

To showcase kernel regression, we use the count of Google searches for the term chocolate, which can be downloaded at Google Trends. If saved as kde_chocolate.csv, the following Python script calculates the Nadaraya-Watson estimator and plots Figure 4:

Given the prices of gas at different gas stations in Germany with columns ‘station_uuid’ and ‘e5’, saved in kde_gas_data.csv, a plot similar to Figure 6 can be obtained through the following script. The data can be obtained through Tankerkönig’s Azure Repo.

Final Remarks

In both cases (kernel regression and kernel density estimation), we have to select a kernel and a bandwidth. As the frequently used kernels have similar shapes (see Figure 7), the choice of the bandwidth is more crucial. There is a vast amount of literature on bandwidth selection. In applications, two approaches are dominant: rule-of-thumb bandwidth estimators, such as Silverman’s rule of thumb, and cross validation (particularly for kernel regression).

Figure 7: Frequently used kernel functions: Gaussian (top, left), Epanechnikov (top, right), Quartic (bottom, left), Uniform (bottom, right); Image by author

As most kernel functions are smooth (or at least continuously differentiable), the resulting estimators are smooth too. Thus, kernel regression in the presented form only makes sense for sufficiently smooth regression/density functions.

The Nadaraya-Watson estimator is a special case of a broader class of non-parametric estimators, namely local polynomial estimators. In particular, the Nadaraya-Watson estimator is a local constant estimator.

Conclusion

Kernel based estimators are very useful and applicable in many situations. Even the k-nearest neighbors algorithm can be regarded as kernel based estimator with uniform kernel and varying bandwidth. This paradigm opens new possibilities, as we could employ other kernels, thus, giving different weights to different points.

I believe that the concept of kernel based estimation is important for data scientists and hope that you could build some intuitive understanding.

Oh, and by the way, the kernel corresponding to the weighted moving average estimator is the triangular kernel K(x) = (1-|x|) for |x| ≤ 1.