LOWESS Regression in Python: How to Discover Clear Patterns in Your Data?
By: Saul Dobilas
Tags: Data Science, Smoothing Algorithm, Regression, Python, Machine Learning

Machine Learning

LOWESS Regression in Python: How to Discover Clear Patterns in Your Data?

LOWESS algorithm finding the trend. Image by author.

Intro

Machine Learning is making huge leaps forward, with an increasing number of algorithms enabling us to solve complex real-world problems.

This story is part of a deep dive series explaining the mechanics of Machine Learning algorithms. In addition to giving you an understanding of how ML algorithms work, it also provides you with Python examples to build your own ML models.

This story covers the following topics:

What category of algorithms does LOWESS belong to?

How does Locally Weighted Scatterplot Smoothing work?

How can I use LOWESS to identify patterns and predict new data in Python?

What category of algorithms does LOWESS belong to?

Locally Weighted Scatterplot Smoothing sits within the family of regression algorithms under the umbrella of Supervised Learning. This means that you need a set of labeled data with a numerical target variable to train your model.

LOWESS’ place within the family of Machine Learning algorithms. Image by author.

Since LOWESS is a non-parametric fitting technique, you do not need to assume that data follows any specific distribution. However, at the same time, non-parametric fitting means that at the end, you will not have a global equation for you to use to predict the values of new data points. Not to worry, though, as I provide a workaround to this issue in the Python section later in this story.

Given the above advantages and disadvantages, LOWESS is often used to perform the following analysis:

A regression analysis where simple linear regression cannot produce a line of good fit due to data attributes not following a linear relationship. Note, a special case of multiple linear regression known as polynomial regression could also be used in this case.

Data exploration and analysis in the social sciences, such as elections and voting behavior.

Fitting a line to a scatter plot or time plot where noisy data values, sparse data points, or weak interrelationships interfere with your ability to see a line of best fit.

How does the Locally Weighted Scatterplot Smoothing algorithm work?

While writing this story, I have assumed that you are already familiar with the ordinary least squares (OLS) regression. Hence, in this section, I only intend to provide an intuitive explanation of how LOWESS splits up the data to perform linear regression on local sections of the data.

If you would like a refresher on OLS regression, feel free to take a look at my linear regression story:

Said that one important addition to LOWESS over OLS is that it applies weightings as you might have guessed from the algorithm’s name — Locally Weighted Scatterplot Smoothing.

Typically, the algorithm uses a tri-cube weight function (see below), although other functions can also be used.

Weight function w(x) = (1 - |d|³)³, where d is is the distance of a given data point from the point on the curve being fitted, scaled to lie in the range between 0 and 1

The weight function gives the most weight to the data points nearest to the point of estimation and the least weight to the data points that are furthest away. It is based on the idea that points near each other in the explanatory variable space are more likely to be related to each other in a simple way than points that are further apart.

Intuitive explanation

We will use an example to illustrate how LOWESS works. Let us start by creating a scatterplot where the data points follow a sine wave pattern, but they have some random noise added, making the pattern less obvious.

After that, we perform a LOWESS regression analysis a couple of times using different hyperparameters and add LOWESS curves to the plot:

LOWESS approximation of the sine wave. Image by author.

As you can see, there is a lot of noise in the data, with the relationship between x and y being non-linear (periodic, in fact). Clearly, simple linear regression would not give us a meaningful result here since it can only produce a straight line (unlike LOWESS or polynomial regression).

The first thing that LOWESS does is selects a subset of the data. We can control the size of the subsets by specifying the ‘fraction’ hyperparameter. The two lines that we plotted above have hyperparameter values of 2/3 and 1/5.

Let’s take the orange line as an example, with a default hyperparameter value of 2/3. This means that the LOWESS algorithm takes a subset of two-thirds of the entire data and performs a weighted linear least squares regression over the span of that data while moving by one point at a time and repeating the same process until it has gone through every single point.

Starting from the very left and going right one point at a time, we can see how the orange line remains straight until x=-2.1. This is because the entire data ranges from -2π to +2π (-6.28 to +6.28) with an initial two-thirds window ranging from -6.28 to +2.1. Hence, only when the algorithm goes beyond -2.1 (midpoint between -6.28 and +2.1), it starts encountering new data points within its window, changing the curve's slope.