MARS: Multivariate Adaptive Regression Splines — How to Improve on Linear Regression? | by Saul Dobilas | Towards Data Science

Machine Learning is making huge leaps forward, with an increasing number of algorithms enabling us to solve complex real-world problems.
This story is part of a deep dive series explaining the mechanics of Machine Learning algorithms. In addition to giving you an understanding of how ML algorithms work, it also provides you with Python examples to build your own ML models.
Before we dive into the specifics of MARS, I assume that you are already familiar with Linear Regression. If you would like a refresher on the topic, feel free to explore my linear regression story:
towardsdatascience.com
What category of algorithms does MARS belong to?
How does the MARS algorithm work, and how does it differ from linear regression?
How can I use MARS to build a prediction model in Python?
Looking at the algorithm's full name — Multivariate Adaptive Regression Splines — you would be correct to guess that MARS belongs to the group of regression algorithms used to predict continuous (numerical) target variables.
Regression itself is part of the supervised Machine Learning category that uses labeled data to model the relationship between data inputs (independent variables) and outputs (dependent variables).
The below graph is interactive, so make sure to click on different categories to enlarge and reveal more👇.
If you enjoy Data Science and Machine Learning, please subscribe to get an email whenever I publish a new story.
You can use multivariate adaptive regression splines to tackle the same problems that you would use linear regression for, given they both belong to the same group of algorithms. A few examples of such problems would be:
Estimating the price of an asset based on its characteristics
Predicting home energy consumption based on time of day and outside temperature
Estimating inflation based on interest rates, money supply, and other macroeconomic indicators
While the list can go on forever, remember, regression algorithms are there to help you when you have a numerical target variable.
The beauty of linear regression is its simplicity, as it assumes a linear relationship between inputs and outputs (except for polynomial regression, a special case of multiple linear regression, used to model non-linear relationships).
However, the interaction between metrics in the real-world is often non-linear, which means that simple linear regression cannot always give us a good approximation of outputs given the inputs. This is where MARS comes to the rescue.
The simplest way to think about MARS is to imagine it as an ensemble of linear functions joined together by one or more hinge functions.
Hinge function: h(x-c) = max(0, x-c) = {x−c, if x>0; and 0, if x≤c}, where c is a constant also known as a knot
The result of combining linear hinge functions can be seen in the example below, where black dots are the observations, and the red line is a prediction given by the MARS model:
It is clear from this example that simple linear regression would fail to give us a meaningful prediction as we would not be able to draw one straight line across the entire set of observations. Simultaneously, polynomial regression would also struggle with this task because of the “sharp angles” seen in the data plot.
However, the MARS algorithm does very well since it can combine a few linear functions using “hinges.”
The equation for the above example:y= -14.3953 + 1.99032 * max(0, 4.33545 - x) + 2.00966 * max(0, x + 9.95293)
The algorithm has two stages: the forward stage and the backward stage.
It generates many candidate basis functions in the forward stage, which are always produced in pairs, i.e., h(x-c) and h(c-x). However, a generated pair of functions is only added to the model if it reduces the overall model's error. Typically, you can control the max number of functions that the model generates with a hyperparameter.
The backward stage, a.k.a. pruning stage, goes through functions one at a time and deletes the ones that add no material performance to the model. This is done by using a generalized cross-validation (GCV) score. Note, GCV score is not actually based on cross-validation and is only an approximation of true cross-validation score, aiming to penalize model complexity.
The result is several linear functions that can be written down in a simple equation like in the example used above.
Since you now have a general understating of how the algorithm works, it is time to have some fun and build a couple of prediction models in Python.
We will use the following:
House price data from Kaggle
Scikit-learn library to build linear regression models (so we can compare its predictions to MARS)
py-earth library to build MARS models
Plotly library for visualizations
Pandas and Numpy
Note that the py-earth package is only compatible with Python 3.6 or below at the time of writing. If you are using Python 3.7 or above, I suggest you create a virtual environment with Python v.3.6 to install py-earth.
Let us start by importing the required libraries.
Next, we download and ingest the data that we will use to build our MARS and linear regression models.(source: https://www.kaggle.com/quantbruce/real-estate-price-prediction?select=Real+estate.csv)
Let us take ‘X3 distance to the nearest MRT station’ as our input (independent) variable and ‘Y house price of unit area’ as our output (dependent, a.k.a. target) variable.
Before we build the models, however, we will create a scatter plot to visualize the data.
Looking at the graph above, we can clearly see the relationship between the two variables. The price of a house unit area decreases as the distance from the nearest MRT station increases.
Now, let’s build multivariate adaptive regression splines and simple linear regression models and compare their predictions.
As you can see, the MARS model added two hinge functions in the forward stage, but then it pruned h(x0–1146.33) from the model in the backward stage. Hence, the final equations for the two models are:
Simple linear regression model:y = 45.85142705777498 - 0.00726205 * xMARS model:y = 31.4145 + 0.0184597 * h(1146.33 - x) - 0.00269698 * x =   = 31.4145 + 0.0184597 * max(1146.33 - x, 0) - 0.00269698 * x
Let us now plot them both on one graph so we can see how they differ.
Note the kink at x=1146.33. This is where the hinge function h(c-x) becomes zero, and the line changes its slope. The graph makes it very intuitive to understand how MARS can better fit the data using hinge functions.
Let us now go up in dimensions and build and compare models using 2 independent variables.
We start by creating a 3D scatterplot with our data. Note, we use the same data as before but add one more independent variable — ‘X2 house age’.
We can see that while somewhat weaker, there is also a relationship between X2 and Y as the price increases when the house age decreases.
Let us now fit multivariate adaptive regression splines and linear regression models.
Since we increased the number of dimensions, we now have two slope parameters in a linear regression model (one for each x). We also have 4 hinge functions that have been added to the MARS model using both independent variables.
Let us plot two graphs to visualize the results, one for multiple linear regression and another for multivariate adaptive regression splines. But before that, we need to generate a mesh with a range of input values and predict output values. This will give us the data for our two graphs.
Now that we have the data ready, let us draw the graphs.
It is easy to see the difference between the two models. Multiple linear regression creates a prediction plane that looks like a flat sheet of paper. Meanwhile, MARS takes that sheet of paper and folds it in a few places using hinge functions, enabling a better fit to the data.
If you wondered what that feature image represented at the beginning of the story, you should now be able to see that it overlays the predictions from linear regression and MARS models to help you see how the prediction outputs differ using each model.
Multivariate adaptive regression splines algorithm is best summarized as an improved version of linear regression that can model non-linear relationships between the variables. While I demonstrated examples using 1 and 2 independent variables, remember that you can add as many variables as you like.
I hope you found this story useful and that you will put what you learned into practice by building and improving your own regression models.
Feel free to reach out if you have any feedback or questions.
Cheers! 👏Saul Dobilas
If you have already spent your learning budget for this month, please remember me next time. My personalized link to join Medium is:
solclover.com
Related stories you may like: