LOWESS Regression in Python: How to Discover Clear Patterns in Your Data? | by Saul Dobilas | Towards Data Science

Machine Learning is making huge leaps forward, with an increasing number of algorithms enabling us to solve complex real-world problems.
This story is part of a deep dive series explaining the mechanics of Machine Learning algorithms. In addition to giving you an understanding of how ML algorithms work, it also provides you with Python examples to build your own ML models.
What category of algorithms does LOWESS belong to?
How does Locally Weighted Scatterplot Smoothing work?
How can I use LOWESS to identify patterns and predict new data in Python?
Locally Weighted Scatterplot Smoothing sits within the family of regression algorithms under the umbrella of Supervised Learning. This means that you need a set of labeled data with a numerical target variable to train your model.
The below graph is interactive, so make sure to click on different categories to enlarge and reveal more👇.
If you enjoy Data Science and Machine Learning, please subscribe to get an email whenever I publish a new story.
Since LOWESS is a non-parametric fitting technique, you do not need to assume that data follows any specific distribution. However, at the same time, non-parametric fitting means that at the end, you will not have a global equation for you to use to predict the values of new data points. Not to worry, though, as I provide a workaround to this issue in the Python section later in this story.
Given the above advantages and disadvantages, LOWESS is often used to perform the following analysis:
A regression analysis where simple linear regression cannot produce a line of good fit due to data attributes not following a linear relationship. Note, a special case of multiple linear regression known as polynomial regression could also be used in this case.
Data exploration and analysis in the social sciences, such as elections and voting behavior.
Fitting a line to a scatter plot or time plot where noisy data values, sparse data points, or weak interrelationships interfere with your ability to see a line of best fit.
While writing this story, I have assumed that you are already familiar with the ordinary least squares (OLS) regression. Hence, in this section, I only intend to provide an intuitive explanation of how LOWESS splits up the data to perform linear regression on local sections of the data.
If you would like a refresher on OLS regression, feel free to take a look at my linear regression story:
towardsdatascience.com
Said that one important addition to LOWESS over OLS is that it applies weightings as you might have guessed from the algorithm’s name — Locally Weighted Scatterplot Smoothing.
Typically, the algorithm uses a tri-cube weight function (see below), although other functions can also be used.
Weight functionw(x) = (1 - |d|3)3, where d is is the distance of a given data point from the point on the curve being fitted, scaled to lie in the range between 0 and 1
The weight function gives the most weight to the data points nearest to the point of estimation and the least weight to the data points that are furthest away. It is based on the idea that points near each other in the explanatory variable space are more likely to be related to each other in a simple way than points that are further apart.
We will use an example to illustrate how LOWESS works. Let us start by creating a scatterplot where the data points follow a sine wave pattern, but they have some random noise added, making the pattern less obvious.
After that, we perform a LOWESS regression analysis a couple of times using different hyperparameters and add LOWESS curves to the plot:
As you can see, there is a lot of noise in the data, with the relationship between x and y being non-linear (periodic, in fact). Clearly, simple linear regression would not give us a meaningful result here since it can only produce a straight line (unlike LOWESS or polynomial regression).
The first thing that LOWESS does is selects a subset of the data. We can control the size of the subsets by specifying the ‘fraction’ hyperparameter. The two lines that we plotted above have hyperparameter values of 2/3 and 1/5.
Let’s take the orange line as an example, with a default hyperparameter value of 2/3. This means that the LOWESS algorithm takes a subset of two-thirds of the entire data and performs a weighted linear least squares regression over the span of that data while moving by one point at a time and repeating the same process until it has gone through every single point.
Starting from the very left and going right one point at a time, we can see how the orange line remains straight until x=-2.1. This is because the entire data ranges from -2π to +2π (-6.28 to +6.28) with an initial two-thirds window ranging from -6.28 to +2.1. Hence, only when the algorithm goes beyond -2.1 (midpoint between -6.28 and +2.1), it starts encountering new data points within its window, changing the curve's slope.
If we now look at the red line (frac=1/5), we can see how narrowing the window to one-fifth of the data makes the algorithm more sensitive towards local trends. The featured gif image at the beginning of this story illustrates how this red line (frac=1/5) moves across the data performing weighted linear regression. This is the same process as described for the orange line (frac=2/3) except, the window the algorithm sees is narrower.
While in this example, making the window smaller helped us get a better approximation of the sine wave, it is not always desirable to make a ‘fraction’ hyperparameter smaller. This largely depends on the data you are analyzing, as making a window smaller runs a risk of overfitting. You will see this in my next example, where 2/3 works much better than 1/5.
Now that you have a general understating of how the algorithm works, it is time to have some fun and build a model in Python
We will use the following:
House price data from Kaggle
Scikit-learn library to build a simple linear regression model (so we can compare the result to LOWESS)
statsmodels library for LOWESS algorithm
Plotly library for visualizations
scipy library for data interpolation
Pandas and Numpy
Let us start by importing the required libraries.
Next, we download and ingest the data that we will use to build our LOWESS and linear regression models.(source: https://www.kaggle.com/quantbruce/real-estate-price-prediction?select=Real+estate.csv)
We will take ‘X3 distance to the nearest MRT station’ as our input (independent) variable and ‘Y house price of unit area’ as our output (dependent) variable and create a scatterplot to visualize the data.
We can clearly see the relationship between x and y with the house price (per unit area) decreasing as the distance from the nearest MRT station increases.
Since the relationship is somewhat linear, the question is whether LOWESS can give us a better approximation than simple linear regression. Let us find out by fitting the two models.
With fitting done, let us draw simple linear regression and LOWESS lines on the graph.
Visually we can see that LOWESS can fit the data better. This is because the price drop is steeper initially, which then starts to flatten out as the distance from the nearest MRT goes beyond 1000.
Simple linear regression has only one slope parameter meaning that it has the same steepness of the curve throughout. Meanwhile, LOWESS can adjust the curve's steepness at various points, producing a better fit than that of simple linear regression.
Let us now zoom in on the graph to see the difference between the two LOWESS models.
We can clearly see that setting the fraction hyperparameter to 1/5 makes LOWESS a bit too sensitive to the local data fluctuations, giving us a model that is overfitting. This is the opposite of the sine wave example, emphasizing that we need to be careful with hyperparameters and ensure that we choose the best values for the given data.
Unfortunately, the lowess algorithm from statsmodels does not provide us with a predict() method. Hence, we cannot use the model that we have built to give us y values for the new set of x values (i.e., data points that were not in the original data).
Fortunately, we have a solution for that. We can use the interpolation method from the scipy library.
First, we create a function using the interp1d method by passing x and y values from the LOWESS output. Then, we use that newly created function to take new x values and generate y values for them.
Note, in the below code, we used a couple of different options for interpolation. Differences are explained in the code comments. One of them is drawing a line between the two nearest points and calculating a new y value for a given x value, and the other one simply taking the y value from the nearest available point.
At the same time, we have set fill_value to ‘extrapolate,’ which means that if you pass an x value outside of the original range (like 6400 in the example below), you will still get a result based on the extrapolated line. Without it, the algorithm would return ‘NaN.’ However, be careful with extrapolation as you may not always get good results for values far outside the original data range.
Printed output of ynew_linear and ynew_nearest:[45.17484583 38.88067785 33.63954152 30.70005122 28.90428712 27.31620311 26.02059902 11.5419846][45.02258129 38.86385487 33.43419447 31.09566559 28.91596696 27.30837281 26.0121316  11.55394747]
LOWESS is not something that you may want to use in all of your regression models as it follows a non-parametric approach and is quite computationally intensive. However, it is a good way to model a relationship between two variables that do not fit a predefined distribution and have a non-linear relationship.
I hope you enjoyed this story and can use the code and explanations for your own data science projects. If you have any feedback or questions, feel free to reach out.
Cheers! 👏Saul Dobilas
If you have already spent your learning budget for this month, please remember me next time. My personalized link to join Medium is:
solclover.com
Related stories you may like: