Kernel Regression in Python. How to do Kernel regression by hand in... | by Pawan Nandakishore | Towards Data Science

1 Kernal Regression by Statsmodels
1.1 Generating Fake Data
1.2 Output of Kernal Regression
2 Kernel regression by Hand in Python
2.0.1 Step 1: Calculate the Kernel for a single input x point
2.0.2 Visualizing the Kernels for all the input x points
2.0.3 Step 2: Calculate the weights for each input x value
2.0.4 Step 3: Calculate the y pred value for a single input point
2.0.5 Step 4: Calculate the y pred values for all the input points
2.0.6 Step 5: Visualize the difference between the two methods
3 Conclusion
4 References
This notebook demonstrates how you can perform Kernel Regression manually in python. While Statsmodels provides a library for Kernel Regression, doing Kernel regression by hand can help us better understand how we get to the find result.
First, I will show how Kernel Regression is done using Statsmodels. Next I will show how it is done by hand, then finally overlay both plots to show that the results are the same.
To begin with, lets looks at Kernel regression by Statsmodels
We generate y values by using a lambda function. You can change the lambda function around to see what happens. The x values i.e. the independent variable is controlled by new_x where we have displaced the x value to show that you can have
import numpy as npimport plotly.express as pxfrom statsmodels.nonparametric.kernel_regressionimport KernelReg as kr import plotly.graph_objs as goimport pandas as pd 
np.random.seed(1)# xwidth controls the range of x values.xwidth = 20x = np.arange(0,xwidth,1)# we want to add some noise to the x values so that dont sit at regular intervalsx_residuals = np.random.normal(scale=0.2, size=[x.shape[0]])# new_x is the range of x values we will be using all the way throughnew_x = x + x_residuals# We generate residuals for y values since we want to show some variation in the datanum_points = x.shape[0]residuals = np.random.normal(scale=2.0, size=[num_points])# We will be using fun_y to generate y values all the way throughfun_y = lambda x: -(x*x) + residuals
Let us plot the data. We are going to be using Plotly express through this article for all the plotting.
# Plot the x and y values px.scatter(x=new_x,y=fun_y(new_x), title='Figure 1:  Visualizing the generated data')
Our goal is to fit a curve the above data point using regression. How can we go about it? Using statsmodels it is fairly simple.
The output of kernel regression in Statsmodels non-parametric regression module are two arrays.
1) The predicted y values2) The Marginal Effects
The marginal effects are essentially the first derivative of the predicted value to the independent variable for a univariate regression problem. More on marginal effects can be found here.
fig = px.scatter(x=new_x,y=fun_y(new_x),  title='Figure 2: Statsmodels fit to generated data')fig.add_trace(go.Scatter(x=new_x, y=pred_y, name='Statsmodels fit',  mode='lines'))
To do Kernel regression by hand, we need to understand a few things. First, here are some of the properties of the kernel.
1) The Kernel is symmetric i.e
K(x) = K(-x)
2) Area under the Kernel function is equal to 1 meaning
We are going to use a gaussian kernel to solve this problem. The Gaussian kernel has the form:
Where b is the bandwidth, xi are the points from the dependent variable, and xx is the range of values over which we define the kernel function. In our case xi comes from new_x
kernel_x = np.arange(-xwidth,xwidth, 0.1)bw_manual = 1def gauss_const(h):    """    Returns the normalization constant for a gaussian    """    return 1/(h*np.sqrt(np.pi*2))def gauss_exp(ker_x, xi, h):     """    Returns the gaussian function exponent term    """    num =  - 0.5*np.square((xi- ker_x))    den = h*h    return num/dendef kernel_function(h, ker_x, xi):     """    Returns the gaussian function value. Combines the gauss_const and    gauss_exp to get this result    """    const = gauss_const(h)    gauss_val = const*np.exp(gauss_exp(ker_x,xi,h))    return gauss_val# We are selecting a single point and calculating the Kernel valueinput_x = new_x[0]col1 = gauss_const(bw_manual)col2= gauss_exp(kernel_x, input_x, bw_manual)col3 = kernel_function(bw_manual, kernel_x, input_x)
We want to display the dataframe for a single point xi.
# Dataframe for a single observation point x_i. In the code x_i comes from new_xdata = {'Input_x': [input_x for x in range(col2.shape[0])],        'kernel_x': kernel_x,        'gaussian_const': [col1 for x in range(col2.shape[0])],        'gaussian_exp': col2,        'full_gaussian_value': col3,        'bw':[bw_manual for x in range(col2.shape[0])],        }single_pt_KE = pd.DataFrame(data=data)single_pt_KE
We also want to visualize a single kernel function.
# Plotting a scatter plot of Kernel px.line(x=kernel_x, y=col3, title='Figure 3: Kernel function for a single input value')
We want to visual the kernel K(x)K(x) for each xixi. Below we calculate the kernel function value and store them in a dictionary called kernel_fns which is converted to a dataframe kernels_df. We then use Plotly express to plot each kernel function.
## Plotting gaussian for all input x points kernel_fns = {'kernel_x': kernel_x}for input_x in new_x:     input_string= 'x_value_{}'.format(np.round(input_x,2))     kernel_fns[input_string] = kernel_function(bw_manual, kernel_x, input_x)kernels_df = pd.DataFrame(data=kernel_fns)y_all = kernels_df.drop(columns='kernel_x')px.line(kernels_df, x='kernel_x', y=y_all.columns, title='Gaussian for all input points', range_x=[-5,20])
We will need to calculate the weight for a single input. The weight is calculated using the expression below:
The above equation represents the weights for the ithith element of new_x where xx are all the elements of new_x. The denominator is summed over all the points in new_x. What is interesting to note here is that you are going to be using the kernels for all input points to calculate the weights. The equation above essentially scales weights between 0 and 1.
The equation above has been implemented in the function weights Which gives us the weights for a single input point. . The function takes a single input point and gives us a row of weights. It does this by looping over all the input points while implementing the above equation.
We get the predicted value for the ithith point from :
This equation is implemented in the function single_y_pred . We take a dot product of the row of weights we get from the weights function and the y values from our fake data. The equation above represents that dot product.
def weights(bw_manual, input_x, all_input_values ):     w_row = []    for x_i in all_input_values:         ki = kernel_function(bw_manual, x_i, input_x)        ki_sum = np.sum(kernel_function(bw_manual, all_input_values, input_x))        w_row.append(ki/ki_sum)    return w_rowdef single_y_pred(bw_manual, input_x, new_x):     w = weights(bw_manual, input_x, new_x)    y_single = np.sum(np.dot(fun_y(new_x),w))    return y_singleypred_single = single_y_pred(bw_manual, new_x[0], new_x)
The code below loops over all the input points calculates the predicted values and appends them to Y_pred. Once we have the predicted values, all we need to do now is to visualize them.
Y_pred = []for input_x in new_x:     w = []    Y_single = single_y_pred(bw_manual, input_x, new_x)    Y_pred.append(Y_single)
Now that we have acquired the predicted value by calculating the predicted values manually, we can compare our regression curve to the one we get from statsmodels. We overlap the fits on top of each other and fit that they match perfectly.
data= {'x': new_x, 'y': fun_y(new_x), 'y_manual': np.array(y_all)}fig = px.scatter(x=new_x,y=fun_y(x))fig.add_trace(go.Scatter(x=new_x, y=pred_y, name='Statsmodel KR',  mode='lines'))fig.add_trace(go.Scatter(x=new_x, y=np.array(Y_pred), name='Manual KR',  mode='lines'))
This article shows how we can understand the kernel regression algorithm's inner workings with a simple example that uses generated data. If you learned something from this article do like and share this article.
Thank you for reading!