Multiple regression as a machine learning algorithm | by Mahbubul Alam | Towards Data Science

Previously I wrote a couple of pieces on multivariate modeling but they both focused on time series forecasting. If curious, go ahead and check out the posts on vector autoregression and panel data modeling. Writing on multivariate regression (i.e. multiple linear regression) was always on my list but something else was on its way — I started a series on anomaly detection techniques! Once started that series, I could not stop until I wrote 11 consecutive posts.
Today I’m back with multiple regression and here’s the plan — first I’ll define in broad terms what a multiple regression is, and then list some real-world use cases as examples. The second part is going to be a rapid implementation of multiple regression in Python, just to give a broad intuition. In the third part, I’ll dive a bit deeper following the typical machine learning workflow. I’ll end with some additional points to keep in mind while implementing linear regression
Before talking multiple regression let’s talk about simple linear regression.
Simple linear regression is one that has one dependent variable and only one independent variable. If “income” is explained by the “education” of an individual then the regression is expressed in terms of simple linear regression as follows:
Income= b0+ b1*Education+ e
Multiple regression goes one step further, and instead of one, there will be two or more independent variables. If we have an additional variable (let’s say “experience”) in the equation above then it becomes a multiple regression:
Income= b0+ b1*Education+ b2*Experience+ e
So why is multiple regression useful and how are they used in real-world data science?
Regression-based machine learning applications can be grouped into (at least) three different problem domains:
1) Quantifying relationships: We know that through correlation we can find the strength (expressed in terms of a number between 0–1) and direction (positive or negative) of the relationship between two variables. But multiple regression goes a step further and actually quantifies that relationship.
2) Prediction: Machine learning is all about prediction. Since regression has the capacity to quantify relationships, we can turn this capability into solving prediction problems. That means, if we know the level of education and years of experience of an individual, we should be able to predict what the level of salary should be for that individual using a regression equation (as the equation above).
3) Forecasting: The third problem domain for multiple regression is forecasting with time series analysis. Vector autoregression and panel data regression are two powerful forecasting techniques that use the principles of multiple regression.
Now here are some real-world application cases of multiple regression:
prediction of used-car prices based on make, model, year, shift, mpg and color
[make, model, year, shift, mpg] → car prices
prediction of the price for a house in the market based on location, lot size, number of beds, number of baths, neighborhood characteristics etc.
[location, lot size, # beds, # bath, crime rate, school ratings] → house prices
forecasting future revenue based on historical data such as expenditure on ads, marketing, customer characteristics etc.
[past ads expenditure $, marketing $, subscription rate] → revenue at time T
Now it’s time to show how multiple regression works in data scientists’ notebooks. First I will give an intuition with a fire-drill and then will dive a bit deeper. If you want to follow along, you can do so by downloading the automobile dataset from the UCI Machine Learning Repository.
The machine learning objective here is to predict the price of used cars based on their features.
import pandas as pdimport numpy as npfrom sklearn import linear_model
# import datadf = pd.read_csv("../automobile.csv")# select features X = df[["horsepower", "highway-mpg"]]y = df["price"]
# instantiate modelmodel = linear_model.LinearRegression()# fit modelmodel.fit(X, y)
# predicting price based 90 horsepower and 30 mpgmodel.predict([[90, 30]])# predicted price>> 11454.03
You’ve got the intuition with a simplified example of how multiple regression makes prediction of the price of a used car based on two features: horsepower and high-way mpg.
But the world we live in is complex and messy. Each of the steps I’ve shown above will need to be branched out further. For example, in Step 2 we imported data and assigned features to X, y variables, without doing any further analysis. But who doesn’t know that data wrangling alone can take upwards of 80% of all tasks in any machine learning project?
I’m not going to go into every situation you’ll encounter as a data scientist in the real-world but I’ll talk about some fundamental issues which are unavoidable.
I’m going to use a few libraries and modules to do some mandatory tasks.
data wrangling: pandas and numpy
multiple regression model: linear_model from sklearn
splitting training and testing data: train_test_split
model evaluation: r2_score
import pandas as pdimport numpy as npfrom sklearn import linear_modelfrom sklearn.model_selection import train_test_splitfrom sklearn.metrics import r2_score
First I’m reading the dataset in my environment (I’m using Jupyter notebook, by the way) and as a ritual, taking a peek at the first few rows.
# import datadata = pd.read_csv("../automobile.csv")data.head(2)
As you can see, there are many columns that barely fit in the window. So I’m going to get a list of all columns.
# all column namedata.columns>> 'symboling', 'normalized-losses', 'make', 'fuel-type', 'aspiration', 'num-of-doors', 'body-style', 'drive-wheels', 'engine-location', 'wheel-base', 'length', 'width', 'height', 'curb-weight', 'engine-type', 'num-of-cylinders', 'engine-size', 'fuel-system', 'bore', 'stroke', 'compression-ratio', 'horsepower', 'peak-rpm', 'city-mpg', 'highway-mpg', 'price'
(I’m not a car expert so I really don’t know what some of those columns represent, but in real-world data science, this is not a good excuse, some domain knowledge is essential).
I did few other things as part of data exploration and preparation which I’m skipping here (such as checking and converting data types, removing a couple of rows that had symbols like ‘?’ etc.), but suffice to reiterate my previous point that getting the data in the right format can be a real pain in the neck.
We want to predict price, so the dependent variable is already set. Now comes which features to use for prediction. Ideally I’d include all of the features in the initial model, but for this demo I’m choosing 1 categorical feature (i.e. make) and two numeric features (i.e. horsepower and highway-mpg) which I think most people would care about while choosing a car.
Following the ML convention, I’m designating the independent variables as X and the dependent variable as y.
# select data subsetdf = data[["make", "horsepower", "highway-mpg", "price"]]# select data for modelingX = df[["make", "horsepower", "highway-mpg"]]y = df["price"]
I’ve purposefully chosen a categorical variable (i.e. make) to highlight that you need to do some extra work to convert it into a machine-readable format (a.k.a. numbers!). There are several ways to do that such as with Label Encoder or One Hot Encoder — both available in sklearn module. But I’m going with the classical “dummy variable” approach, which converts categorical features into numerical dichotomous variables (0s and 1s).
At this stage, I’m also splitting data into training and testing set for model evaluation.
# create dummy variablesmake_dummy = pd.get_dummies(X["make"], drop_first = True)X = X.drop("make", axis = 1)X = pd.concat([X, make_dummy], axis = 1)# split data into train and testX_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.3, random_state = 0)
It’s always amazing to think that within the whole complex machine learning pipeline the easiest part is (in my opinion of course!) actually specifying the model. It’s just three easy steps of instantiate — fit — predict as in most ML algorithms. Of course, you have to parameterize the model and iterate it several times until you get the one that satisfies your criteria, but still, this step of the model building process gives me the least headache.
# instantiate model = LinearRegression()# fitmodel.fit(X_train, y_train)# predicty_pred = model.predict(X_test)
Now comes the moment of truth — how well does the model perform? There are many ways to evaluate model performance but in classical statistics, the performance of linear regression models is evaluated with R2 — which gives a value between 0 and 1, and the higher the R2 the better the model.
# model evaluationscore = r2_score(y_test, y_pred)print(score)# score>> 0.8135478081839133
In our demonstration we are getting a R2 value of 0.81, meaning 81% of the variation in the dependent variable (i.e. used car price) can be explained by the three independent variables (i.e. make, horsepower and highway-mpg). Of course, this metric can be improved by including more variables and trying with different combinations of them and by tuning model parameters — the topic of a separate discussion.
I felt like I could write forever on multiple regression, there are so many areas to cover but I have to stop somewhere. Here are few additional things to keep in mind while building a linear regression model for real-world application development:
while selecting features check for correlation between dependent and each independent variable separately. If they are not correlated, remove the feature from the model;
check for multicollinearity, the relationship between independent variables. Remove correlated features to avoid model overfitting;
there are a few ways to choose variables for the model. Forward selection and backward elimination are two of them. As the names suggest, in this process you add or remove one variable at a time and check mode performance;
I used R2 for model performance evaluation, but some people choose other metrics such as AIC, BIC, p-value etc.
To recap:
Multiple regression is a machine learning algorithm to predict a dependent variable with two or more predictors.
Multiple regression has numerous real-world applications in three problem domains: examining relationships between variables, making numerical predictions and time series forecasting.
If you have everything ready for model building, its implementation can be as simple as 4 easy steps as demonstrated in the “Fire Drill”. However, real hard work is wrangling with data and finding the right model through an iterative process.
I hope this was a useful post. If you have comments feel free to write them down below. You can follow me on Medium, Twitter or LinkedIn.